Out of sample testing

D

Dhalsim

When is it okay to use out of sample data again?

If I have a basic non-optimised system with just entries tested and it works on the in sample and then I test on out of sample which it passes. If I add a secondary entry rule or filter which improves performance further, can I be justified in using the same out of sample data? This is usually a situation when I think of something that can improve strategies much later after completing the original system.

If we then add stops (which is going to have a big effect on performance) can we then use the out of sample data again?

Same for targets, can we use out of sample data again for this?
 
There's no hard and fast rule, but in general curve fitting is more likely the more you "look" at each data set.

Good practice is to use 3 sets, the training set (original data), a test set (out-of-sample set) and then a cross-validation set (second out-of-sample set). You can do walk-forward in lieu of a third set, if doing so is practical given the trading frequency of the system.

Otherwise all the rules of thumb for sound backtesting apply: use as few parameters as possible, choose parameters which are robust across a range of values, test across a range of symbols and market conditions, verify that adjusting a given parameter's value one way or the other has a roughly similar effect on performance in all data sets.
 
Avoid backtesting. Try to think of a model no one can backtest and then trade it forward to see if good. Only way to make money. Backtesting is for losers.
 
Specterx said:
There's no hard and fast rule, but in general curve fitting is more likely the more you "look" at each data set.

Good practice is to use 3 sets, the training set (original data), a test set (out-of-sample set) and then a cross-validation set (second out-of-sample set). You can do walk-forward in lieu of a third set, if doing so is practical given the trading frequency of the system.

Otherwise all the rules of thumb for sound backtesting apply: use as few parameters as possible, choose parameters which are robust across a range of values, test across a range of symbols and market conditions, verify that adjusting a given parameter's value one way or the other has a roughly similar effect on performance in all data sets.
More...

I use random out sample periods e.g testing from 1997 to 2018, this gives 30% random quarters as out of sample.

Also, is it not wise to have as much test sample data as possible? This gives the systems more data to train and should improve predictive power.

Do you ever have a market inefficiency you want to exploit and then test the basic entry principle behind this potential inefficiency. If this passes the in sample across multiple correlated markets then immediately test the out of sample? This way you don't waste weeks or months trying to build a system around a principle rule which had no edge, to begin with. Is this okay to do or should one finish all the entry criteria (can take weeks) then test the finished entry model on the out of sample?
 
What I often do is create/optimize on a current dataset of about the last 10 years. If everything looks ready, I look at the 10 years before that and if that period's results are roughly similar. I'm mostly looking for a similar risk profile (avg. return vs dd), not that the returns and drawdowns are exactly the same.
 
You must log in or register to reply here.
Back
Top