Backtesting's Flaw

K

KS2007

I think the main reason that otherwise profitable strategies fail once automated, is because of backtesting. In my experience, the 'tweaking' for a very specific set of data doesn't allow for future performance. Although the backtesting results will look fantastic, it is not a true representation of what the system will do in the future. If you can create a system that requires no (or VERY few) 'tweaks,' in the backtesting phase, you may be onto something! Also, another good test is to run it on other markets. If you design the system on the S&P but it demonstrates comparable performance in other markets, it shows universality, and suggests that the system may have value for future trading.

...your thoughts?
 
KS - I think people put too much into backtesting and then are shocked that it doesn't work in real time. Fully automating a trading methodology is not easy to say the least.
 
Quote from KS2007:

I think the main reason that otherwise profitable strategies fail once automated, is because of backtesting. In my experience, the 'tweaking' for a very specific set of data doesn't allow for future performance. Although the backtesting results will look fantastic, it is not a true representation of what the system will do in the future. If you can create a system that requires no (or VERY few) 'tweaks,' in the backtesting phase, you may be onto something! Also, another good test is to run it on other markets. If you design the system on the S&P but it demonstrates comparable performance in other markets, it shows universality, and suggests that the system may have value for future trading.

...your thoughts?
More...

Yes.

People are tempted to curve fit past data to minimise system drawdowns. Yes, this is inconsistent design! Either you have a system that is generally applicable but will sometimes go into drawdown or you have a system that worked beautifully on the test data but is not applicable generally.

Bliss.
 
3D modeling of results usually can help you stay away from the over optimized results section. If a small change in the parameters makes a big change in the results, your 3D map will show a very steep peaked hill but if small changes of any of the parameters does not affect the results much you will have a nice rounded hilltop. Pick the hilltop part.
 
@KS2007,

that's not only a problem of automated trading, but generally of system trading.

In addition or better alternative to test your system in other markets, I recommend to test the system also with synthetic data ("data scrambling").

With synthetic data here I think not about total randomly generated data, but about data which is generated using the original historical data of the special instrument you will trade (e.g. DJ, SP500, DAX etc.), change the time series of the data set thru random (monte carlo simulation based) components and optionally customize the data generation with your future estimations (e.f. higher, lower volitality in future...).

So you get alternative test data for new backtests which can avoid curve fitting and over optimization.

If such a system then also shows acceptable results in an additional system stress test (e.g. again monte carlo simulation based), so perhaps you have found a system with a higher rate of survive at the markets...:-)

bye,
zentrader
 
You must log in or register to reply here.
Back
Top