I am designing an intraday system. I was wondering if backtesting a strategy for a period of 2 months using a 5 minute bar period should be enough data for the system to be tested on. What would your suggestion on the length of the period be?
Backtesting time period
- Thread starter NachiketJoshi
- Start date
Yes, I'm trading stocks. How did you come to the number of 6 months? Should it be a good idea for the period to be based on market cycles? I mean my thinking is, to test the system works well for any particular market cycle.
The idea that backtests should have a minimum number of trades is rooted in the concepts of sampling, margins of error, confidence levels and statistical inference. By careful study of a relatively small amount of data (the sample), the backtester hopes to draw conclusions about a very much larger set of data (the population) without having to study the whole population.
The main attraction of testing over a shortened period of historical data is that it avoids the cost of acquiring all available historical data, and to a lesser extent that it requires shorter processing times.
In a backtest, the random variable being sampled is usually the trade PnL, so that the ânumber of measurementsâ is equivalent to the number of trades. So, you need to have enough trades (rather than enough bars), and that is what determines the minimum amount of time required for your backtest.
However, for a sample to be useful it must not be biased, and this is often as much of a problem for the backtester as whether or not the backtest has enough trades. If the strategy being tested works best in certain market conditions, and if the historical data sample chosen for testing exhibits precisely those conditions, then the backtest will not provide on its own a reliable indicator of how that strategy will perform over all conditions. This is one of the strongest reasons in favor of not using shortened data sets at all for testing, but instead using as much relevant data for in-sample and out-of-sample tests as you can get your hands on!
Another condition a sample must satisfy is that it must be taken from the correct population. So, if say your strategy trades the ES only in the overnight Globex session, do not backtest using regular trading hoursâ data. Or if your strategy trades the ES, be cautious testing it on S&P 500 or SPY data. Etc.
Sorry, I didnât give you an answer for how many trades a backtest should have at a minimum!
[I am reading the following data from statistical tables calculated for Normal Distributions, which should be fine for large populations where the Central Limit Theorem can be assumed to apply.]
Say you estimate a strategy may have traded 10,000 times (the population) âsince time beganâ:
How many trades do you need in your backtest to estimate the average trade PnL with a margin of error less than 5% with a confidence level of 95%?
 370 trades.
Or with a margin of error less than 1% with a confidence level of 99%?
 6,239 trades.
Or if you estimate your strategy may have traded 100 times âsince time beganâ:
5% margin of error, 95% confidence level?
 80 trades.
1% margin of error, 99% confidence level?
 99 trades.
[I am reading the following data from statistical tables calculated for Normal Distributions, which should be fine for large populations where the Central Limit Theorem can be assumed to apply.]
Say you estimate a strategy may have traded 10,000 times (the population) âsince time beganâ:
How many trades do you need in your backtest to estimate the average trade PnL with a margin of error less than 5% with a confidence level of 95%?
 370 trades.
Or with a margin of error less than 1% with a confidence level of 99%?
 6,239 trades.
Or if you estimate your strategy may have traded 100 times âsince time beganâ:
5% margin of error, 95% confidence level?
 80 trades.
1% margin of error, 99% confidence level?
 99 trades.
Thank you! Appreciate your inputs.
If there is serial correlation of trades then this cannot apply because the CLT assumes i.i.d. samples. I know there are ways to relax the requirement but I am not sure how your calculations will change neither I have seen tables for that.
Then "since time began" is very abstarct. Do you mean the Big Bang?
I suspect we cannot know the population size or even worse the theoretical popualtion size is close to infinity. Therefore the larger the sample the better it is.Quote from ronblack:
Then "since time began" is very abstarct. Do you mean the Big Bang?
Population is taken to be the total number of trades in your backtest. Although having large population set increases the accuracy of results, it's not essential(Law of large numbers).
Imo you should use as many bars possible as needed to include all possible normal market conditions but as @abattia showed with examples that depends on distribution population and required confidence. But if you are trading E-mini intraday you should pay attention to this randomization study because your chances of making any money are very low unless you have an exceptional edge. If you are trading forex this is the relevant study. Both are must reads.
2 months of data seems very very less. I had used 1 minute historical data on 10 years of data to back test my strategy so as to ensure it would have passed through all possible scenarios and conditions (Bullish / Bearish / Cyclic / Noisy...).