Trading Systems Robustness & Market Structure
- Thread starter CPTrader
- Start date
You could average the returns from the walk forward period and compare that average to the average returns from the in sample period. Student's T Test for a smaller sample or a Chi Squared test might help here. You'd test the hypothesis that one average is the same as the other and get a confidence level that your out of sample testing is working better than expected or decaying.
With respect to testing on only recent years, I'd argue that a system is a business. Even something as simple as a McDonald's restaurant has changed a great deal in 25 years, so I wouldn't expect a fast food idea that worked in the 1980s to work today.
I like the idea of taking out the X largest wins of a system too, just to eliminate the exogenous shocks that skew some of the models and see if they make money day to day when things are quiet.
Just some ideas...
Tim
Curtis does a thorough explanation of this paradox in his book Way of the Turtle, chapters 11-13. You may read an extract at: http://www.tradingblox.com/tradingblox/optimization_paradox.htm
Everything doesn't have to and can't be automated.
The questions that need to be asked is what / which part of market has a robust tendency. Or which part of the system is good to optimize and which parts are dangerous to be optimized.
The questions that need to be asked is what / which part of market has a robust tendency. Or which part of the system is good to optimize and which parts are dangerous to be optimized.
I'm actually struggling with this very problem-- if a system works now, but it didn't in the past ('97 and before) is it robust?
Along these lines I think it's important, as already mentioned to define the margins for working vs. broken. Do you think using a normal distribution is acceptable? Has anybody actually been in this situation?
Along these lines I think it's important, as already mentioned to define the margins for working vs. broken. Do you think using a normal distribution is acceptable? Has anybody actually been in this situation?
CPTrader,
You said:
"It is commonly agreed that the measures of a robust trading system are:
1. Common sense and valid underlying logic.
2. Identical parameters, rules for all markets
3. Consistent profitable performance across multiple markets in multiple market complexes
4. Simple rules"
I strongly disagree with 2) and 4) above, both from my own experience and from logic.
Why should the parameters be identical? Markets differ widely in their nature and operation. In most other things we do there is variation: shoes come in different sizes because people aren't identical and their feet are different. We drive differently depending on road and traffic conditions.
What makes the markets different than just about everything else?
I've found it much more effective to create a separate trading system for each market I trade as opposed to trying to force all markets to fit in to the same sized shoe.
I suspect the inventor of these rules was somebody selling a âhow to trade courseâ and they wanted to limit their efforts so it was a good idea to simplify everything to sell more units.
Jerry
You said:
"It is commonly agreed that the measures of a robust trading system are:
1. Common sense and valid underlying logic.
2. Identical parameters, rules for all markets
3. Consistent profitable performance across multiple markets in multiple market complexes
4. Simple rules"
I strongly disagree with 2) and 4) above, both from my own experience and from logic.
Why should the parameters be identical? Markets differ widely in their nature and operation. In most other things we do there is variation: shoes come in different sizes because people aren't identical and their feet are different. We drive differently depending on road and traffic conditions.
What makes the markets different than just about everything else?
I've found it much more effective to create a separate trading system for each market I trade as opposed to trying to force all markets to fit in to the same sized shoe.
I suspect the inventor of these rules was somebody selling a âhow to trade courseâ and they wanted to limit their efforts so it was a good idea to simplify everything to sell more units.
Jerry
The paradox is only in his mind. There is no paradox. Optimized systems are fitted for vest past performance. Past performance is no guarantee of futures results I heard. Simple.
Paradox is an apparent contradiction that is resolved when one of the false premises is removed and replaced by a true one. In this case, we have a true conclusion, no paradox.
Bill
Hi CPTrader,
You have a good outline, I'll offer some of my experiences with the above points:
1. Common sense and valid underlying logic.
-- Sure. KISS is the pertinent idea here, market moves are generally either catalyst driven (price/volume or news impulses) or range bound. Any systems that correctly identifies a basic market behavior can be successful.
2. Identical parameters, rules for all markets
-- No way. The SPY acts much differently than APPL as does the ES compared to Gold. Each product has a flow and pace that needs specialized attention and in many cases a completely different risk assessment. I will say that products can be classified into groups where identical parameters/rules will work: for example, an SP strategy should also work on the Russel, Nasdaq and Dow. By nature, these markets generally behave in similar fashion. Another group would be tech stocks with Avg. volume greater than 2mil shares. Testing a system in the SPY and then testing it in AAPL proves nothing... great if it works on both, but, if it was designed for SPY and doesn't work on AAPL that is no reason to conclude the system is not robust.
3. Consistent profitable performance across multiple markets in multiple market complexes
-- See my response to point 2.
4. Simple rules
-- Not necessarily. There are a wide range of behaviors that can be exploited each requiring in some cases a number of good rules. From my experiences rules need to be "tuned", that is you can start with a basic premise like "fade gaps", but, you need to add a lot of support to this premise via stats to identify when a gap is most likely to fill. The rules here can get complicated fast and they should IMO...
In terms of market structure, I would say that volatility and liquidity are most important. A system is robust if it performs well in high and low volatility environments. If you want to scale the system then liquidity is very important obviously. The other points are irrelevant IMO because all is reflected in price. Good volatility = good profit potential and good liquidity = easy entry/exit and scalability.