Optimizing and forward testing

J

jcl

It is known that you have to use out-of-sample data for testing, otherwise the test will generate far too optimistic, meaningless results. However, as I learned the hard way, it also matters how you select the out-of-sample data.

Previously I split the historic price data in 3-week intervals, and used 2 weeks for optimizing the strategy, and every third week for testing. This way I got very good results for some strategies, such as Sharpe ratios up to 5 and annual returns of 700%.

When testing the same strategies in a different way - the whole first 70% for optimizing, the 30% rest for testing - the Sharpe ratios went down to 2. When testing them in real trading, the results are comparable to this 70/30 split method, not to the previous 3-weeks method.

I'm sharing this info here in case someone runs into the same problem. But I'm not sure why the 3-weeks data split gives much more unrealistic results than the 70/30 method. Maybe someone has an idea?
 
Yes, this make sense. Not only the price data, but also the closeness of the test market to the optimizing market has certainly an effect on the result. I just was surprised how large that effect is, although I used price data from 4 years for optimizing.

Your PDF is interesting. I found a different method meanwhile that generates enough out-of-sample data - the walk-forward test. How do you scramble the data? I have experienced with random bar offsets, but that does not change the overall market situation, only the individual bars.
 
hi,

the principal method is to save the relative distances of the OHLC bars (O-H-L-C) and the relative distance of the last Close to the current Open (refC - O). The second step is resampling. In my software I've also a bunch of different methods, including a block mechanism, which can resample not only single bars but blocks of bars. Also this has the risk to divide historical related bar groups - but on the other side it's also not sure that history repeats in future... :-)

bye,
Volker
 
Did this happen with only one price series or with many?

If the latter you may be on to somerhing. Why don't you do the actual trading the same way and see what happens?
 
Volker, thanks for the info, I'll make some tests with MC.

- It happens with all price series that the strategy uses: 6 currencies, 5 stock indices, oil, gold, and silver CFDs. Life trading seems to confirm it. However I only life trade since 3 months, which gives not enough data. Anyway 240% is good enough for me at the moment, although 700% would be nicer.
 
You must log in or register to reply here.
Back
Top