Choosing mechanical systems

[prophet]

Quote from WDGann:
1st system:
Low Sharpe... wouldn't trade it. These kinds of system is more vulnerable to the changing markets that there are higher risk of the system deteriorating (I've statistically tested it so I'm talking from experience).

True, though I was wondering if that system could be traded discretionarily. It’s performance trends very well locally, though it could potentially stagnate and go nowhere.

Quote from WDGann:
Finally, the data is too short to analyze that the system is good.

That’s all the data I have available. I have some different data going back further, but I can’t obtain that data in real time because I was using QuoteTracker to log ticks, and QT was doing some filtering I can’t replicate.

Here is a system with QT data going back further:

 

[trend456]

Prophet,

No offense.....

The data is really short. You need to get more data if you can. Whether you use forward opt. or any technique to create a system, you need to have more data. The amount of data u are using is way to short.

If it's good with more data trade them all by allocation.

Good Luck.

Trend

 

[nononsense]

Quote from trend456:

Prophet,

No offense.....

The data is really short. You need to get more data if you can. Whether you use forward opt. or any technique to create a system, you need to have more data. The amount of data u are using is way to short.

If it's good with more data trade them all by allocation.

Good Luck.

Trend

trend456,

You are quite right. This is why I referred to Prophet's systems as "poor". Prophet had his doubts, otherwise he would not have posted.

Testing these proposed systems thoroughly - 400 to 500 tradingdays - will in all likelihood bring out some quite unforeseen "events". If his systems still stand up, Prophet does not need our help. In fact for an analytically minded person like Prophet, the whole business turns around "the risk of ruin". A lot has been published about this also intuitive notion. I found it most difficult to quantify for a real market tradingsystem and I am still trying.

In my experience, you are still in the beginning of the tunnel if you base your expectations on the very short test runs now presented by Prophet. I learned this the hard way!

 

[ArchAngel]

Quote from prophet:
Networks seem much better at forecasting semi-profitable system performance than trying to forecast market prices directly.

There are a number of useful applications of nets, but they're definitely not a panacea. Keep in mind that nets are also real good at memorizing optimal PAST behavior (thereby making their trained results look impressive) but then being totally useless going forward. You need lots of care in selecting the training set and then a sizable forward testing set against which to validate the results.

 

[prophet]

Quote from nononsense:

Testing these proposed systems thoroughly - 400 to 500 tradingdays - will in all likelihood bring out some quite unforeseen "events". If his systems still stand up, Prophet does not need our help. In fact for an analytically minded person like Prophet, the whole business turns around "the risk of ruin". A lot has been published about this also intuitive notion. I found it most difficult to quantify for a real market tradingsystem and I am still trying.

In my experience, you are still in the beginning of the tunnel if you base your expectations on the very short test runs now presented by Prophet. I learned this the hard way!

Sorry for the delay in responding to your messages.

1) I have tested my systems on longer data, starting on 3/13/03. I didn’t present these results because (1) this data has different characteristics to the data I’m using now, so it isn’t a fair test and (2) I can’t obtain this data in real time, so I stopped testing with it. See the attached gif for that performance. This system is identical to the third system report I posted. In my mind, this gives me extra confidence.

2) These systems trade constant size and have trading rate limits built into the network. This greatly diminishes the risk of blowing up. I’ve tested these systems trading up to 15 trades/day, and although the return/risk or Sharpe ratio is much lower, there are no major risk events since 3/13/03. Although I would like to have enough data to simulate back to 9/11, I’m sure most mechanical systems can not handle events like 9/11.

3) It is hard enough to design a fully mechanical, profitable system simulated on 40 or 100 days (all the data I have). It would be much harder to develop a such a system, trading 1 to 2 times/day for the last 400 or 500 trading days… 2 years! Markets have undoubtedly changed a lot in the last 2 years. I think it’s much more profitable to design a statistically robust system, adapted to more recent market conditions, say for the last 150 trading days. Of course this all depends on how robust the system is, how often it trades, and the method of risk management. Is the system’s edge rather narrow in scope, more likely to fail, or is it broad and adaptive?

 

[prophet]

Quote from ArchAngel:

There are a number of useful applications of nets, but they're definitely not a panacea. Keep in mind that nets are also real good at memorizing optimal PAST behavior (thereby making their trained results look impressive) but then being totally useless going forward. You need lots of care in selecting the training set and then a sizable forward testing set against which to validate the results.

These networks use past training examples and walk-forward validation/testing sets. The simulations I posted are all walk forward.

I have become very acquainted with the tendency of networks to over- fit the training examples, or memorize patterns in past data that falsely appear to be predictive and don’t generalize. It helps to keep the degrees of freedom to a minimum and use training example filtering methods.

 

[bdixon619]

Quote from peterfigliozzi:

Some very nice ideas I must say I know not much about.



But what you described above is called a bad system.

Also, by "system" I see we might be talking about very different animals. The "system" I am talking about cannot adapt to jack shit. It is more like an ambush, it sits around and waits for a certain setup, using very simple rules. After entry, if the price action is not immediately favorable, fitting the "winners profile", it exits. Otherwise it makes money. The only drawback is that it is mostly out of the market. Some might say that's not a drawback.

Not that it means much, but this is a smart way to trade.

 

[nononsense]

Quote from prophet:



Sorry for the delay in responding to your messages.

1) I have tested my systems on longer data, starting on 3/13/03. I didn’t present these results because (1) this data has different characteristics to the data I’m using now, so it isn’t a fair test and (2) I can’t obtain this data in real time, so I stopped testing with it. See the attached gif for that performance. This system is identical to the third system report I posted. In my mind, this gives me extra confidence.

2) These systems trade constant size and have trading rate limits built into the network. This greatly diminishes the risk of blowing up. I’ve tested these systems trading up to 15 trades/day, and although the return/risk or Sharpe ratio is much lower, there are no major risk events since 3/13/03. Although I would like to have enough data to simulate back to 9/11, I’m sure most mechanical systems can not handle events like 9/11.

3) It is hard enough to design a fully mechanical, profitable system simulated on 40 or 100 days (all the data I have). It would be much harder to develop a such a system, trading 1 to 2 times/day for the last 400 or 500 trading days… 2 years! Markets have undoubtedly changed a lot in the last 2 years. I think it’s much more profitable to design a statistically robust system, adapted to more recent market conditions, say for the last 150 trading days. Of course this all depends on how robust the system is, how often it trades, and the method of risk management. Is the system’s edge rather narrow in scope, more likely to fail, or is it broad and adaptive?

Prophet,

Building a mechanical system is of course a very personal experience. I think we would both agree on this.

I fully share your desire for robust systems. However why do you want a "robust"system to be adaptive? I have looked at many possibilities and centered my search on a robust system, like you did. For me it should be (1) non-adaptive and (2) able to trade several markets (albeit somewhat similar, I do only index futures).

My main criteria are standard deviation of profit over average profit and a speed of growth measure also involving drawdown. I found that I did best in this by looking over at least 2 years of data. I worked hard to get a complete tick database for most markets I am interested in and found that if things keep within bounds for a period of 2 years or so, there will be no surprises in any other period. However, I could not get this kind of result if I attempted to make systems adaptive. Maybe one day I will again experiment with this but not right now,

On your observation about catastrophic events (9/11), this fear is the truly limiting factor in the growth mechanism. I went through a lot of risk of ruin analysis but I am worrying a lot about this. Catastrophies fall outside of my models. Spikes with subsequent exchange decisions are another threat. Perhaps the best solution for limiting your exposure is to look for systems with very short overall "in market" time.

I also spent a lot of time with neural networks. Not anymore. I believe a lot in mathematics but also believe that personal observation is sometimes a better technique to discover profitable features than neural networks. You need lots of patience though! I plan to use some neural networks to manage actual trading operations though. This is only refinement.

Prophet, glad to have heard about your work. Lots of luck to you.

 

[prophet]

Quote from nononsense:

I fully share your desire for robust systems. However why do you want a "robust"system to be adaptive? I have looked at many possibilities and centered my search on a robust system, like you did. For me it should be (1) non-adaptive and (2) able to trade several markets (albeit somewhat similar, I do only index futures).

I work with a trader who trades robust systems that are mechanically non-adaptive consisting of a few fixed rules even though these rules require periodic adjustment and external/discretionary risk management. He claims his systems are non-adaptive. Yet when you think about it, his intelligence and discretion constitutes a major adaptive element. Many of the steps he takes to adjust the rules can be automated. So why not automate the adjustments and simulate/trade it walking forward? That way you know the adaptive elements are working properly.

The most important adaptive element in my opinion is the decision to trade or not trade a system at each moment. Almost any system ever applied to trading or forecasting markets will have situations where it will do well, and situations where it does poorly. So rather than trying to come up with the perfect set of non-adaptive rules, that are consistently profitable, why not take several mediocre systems and try to characterize their win/loss profitability patterns over time, forecasting their profitability?

I can possibly make the case that my systems are adaptive and robust because they employ several (8 to 64) papertrading subsystems with a network forecasting the future performance of each subsystem, identifying subsystems most likely to succeed. The network makes it adaptive. The diversification through multiple subsystems makes it robust.

Quote from nononsense:
I also spent a lot of time with neural networks. Not anymore. I believe a lot in mathematics but also believe that personal observation is sometimes a better technique to discover profitable features than neural networks. You need lots of patience though! I plan to use some neural networks to manage actual trading operations though. This is only refinement.

Most neural net learning algorithms seem to converge much too slowly (or not at all) due to the noisy nature of price data and the network being too expressive for the amount of CPU available to train it. I use linear networks that although less expressive, achieve fast and full convergence. These include correlation/covariance, linear regression, constrained least squares, analysis of the singular value decomposition, and/or a-priori non-linear input stages. They’re much simpler and robust to noise than neural networks, yet so often overlooked.

Is there anyone else trading adaptive mechanical systems like this?

 

[trend456]

Hey prophet,

You still need more data samples. It might not be necessary but it would really help you.

I'm not going to spend time explaining why? Or maybe I just don't know why? LOL

Anyways.... just my 2 cents.....

Good Luck.

trend