How much out of sample testing is enough?

There is no need for a relationship between the size of in-sample (used to develop the strategy) and out-of-sample testing. Although Pardo does recommends an out-of-sample size of 10-20% of the optimization data. This seems arbitrary to me.

My approach is to specify the size of the out-of-sample data set independently of the size of the training set.

1. Define the minimally statistically significant sample size one requires, take you pick: 25, 30, 35, 50, or 100+. The larger the better, but at least 25.

2. Count the number of if statements in the strategy that result in trading decisions. You want the size of the out-of-sample test to generate a number of trades >= MinSampleSize*NbrIfStatements.

3. Better yet, you want the number of trades to be >= SampleSize for each path through your strategy’s decision tree. You will have to instrument you strategy during the in-sample optimization to get the fraction of trades that follow any given path. The path with the least fraction of trades then determines the size of the out-of-sample data set: NbrOfTrades > MinSampleSize/SmallestFraction. The number of bars required for the out-of-sample test = NbrOfTrades*AveNbrBarsPerTrade, where AveNbrBarsPerTrade is also collected by the in-sample instrumentation.

#3 is harder to implement but is superior to #2 because one gets a statistically significant number of trades to test on each path though your decision tree.
 
The question is not how much out of sample testing is required but how valid the premise. For example if we had a intermarket bond system which used silver as an intermarket with only 2 parameters , that would require less out of sample than a pattern discovered from data mining, in that case ,unless the pattern makes sense it might not be safe to trade even with 20 years of out of sample data.

The key is you need to understand the premise so you also understand when the system has stopped working because the premise is no longer valid .
 
Using daily data there is not much room for taking large samples.

The two best methods I have used include walk forward sampling.

Divide the data into blocks of daily bars.

Train on block A, sample with block B.

Train on block A + B sample with block C

Train on block A + B + C sample with block D

Another approach is to Train on block A and sample on block D, Train on block B and train on block J. Repeat again and again pairing 'train blocks' with 'sample blocks'. This approach may work for single market price action, but doing this with more complex intermarket models is difficult.

Both of these approaches are necessary because the total number of observations (daily bars) are so small.

When your sample results match your train results you can take some confidence in your results.

If your approach is just using price action alone and a single market you can use daily data older than 5 years. Anything intermarket that is older than 5 years may not give you any consistent results as many intermarket relationships have changed.

Try and get 30 or 60 minute data, just by improving your data sizes the results are going to probably improve.

Good luck,
:)
 
TradersStudio ships with a free walk forward macro , which works for any systems which has parameters without changing the code for the system. The macro creates a series of reports using Excel OLE and it is all automated. The only thing required is Excel is on the machine. It also works on a basket of markets so you can do easy walk forward testing across a portfolio based system. The macro and supporting Excel code is open so you can change it and add new features if you would like.

You are correct this is a very powerful methodology and that is why I write a three part series on walk forward testing earlier this year in Futures Magazine
 
Quote from hypostomus:

No amount of backtesting with the most fabulous data is going to help you if you don't understand what the fuck is going on NOW.
More...

Agreed, but 30 out-of-sample trades per degree of freedom is a good start.

--laz
 
I asked Alan Farley a similar question about backtesting when I first started trading. His answer was "Life is too short to backtest. I just trade". He also told me "You just gotta get in there and lose some money. Just start small"

I dont totally agree with him on this but what he said did help me focus on trading my ideas instead of doing rigorous amounts of testing on them. IMO most people who spend too much time testing probably dont have the stomach or financial stability to trade.

You really wont have many ideas to backtest unless you've actually traded. Some of the best system/mechanical traders were discretionary traders at one point in there life and this is the point where they learned their approach.
 
Hello:

Alan's comments are not really helpful to traders. I guess I would hope he wasn't serious. Depending on the market you trade, it is important to have an edge. That edge can be either a historical tendency, or a true statistical edge based on some element of human behavior. Either way, you should realize that in most markets, there are a variety of participants some of whom may have an advantage over you, if you don't have a good basis for making a trade.

In addition, I have found that traders tend to hesitate to pull the trigger when they don't have confidence in their signals. Testing when done correctly, gives you that confidence.

While few things in life are cut and dried, I will continue to test BEFORE I trade.

Good luck
Steve
 
You must log in or register to reply here.
Back
Top