Backtesting time period
- Thread starter NachiketJoshi
- Start date
Murray Ruggiero
Sponsor
Yes, it's really important that you have at least one bear and one bull market in your data set. I would say 10 years is required. I use 10-15 years of data even on my intra-day strategies.
Can you explain how did you arrive at the number 3000?
One of the biggest threats to successful optimization/backtesting is the possibility that the results are just a curve-fit (or "over-fit", if you use that instead …)
i.e. they are just the strategy settings that would have worked best by chance over the historical data used, but contain no “intelligence” about price action or market structure that will enable the strategy to perform well if future price action is different … which of course it will be…
... like a lottery winner wins by chance but has no edge in terms of winning again in the future…
One way to have a shot at beating the curve fitting genie is to make use of “In Sample” and “Out of Sample” data. Optimization is undertaken using the “In Sample” data, and then the optimized settings are used to simulate performance in the “Out of Sample” data. Then statistics is used to compare the two populations (i.e. Population 1 - “In Sample” trades and their associated performance statistics, and Population 2 - “Out of Sample” trades and their statistics). With statistics, significence levels are established to test the null hypotheses that - say - the “Out of Sample” average trade, “Out of Sample” PF, etc. is the same as the “In Sample” average trade, etc.
That is, confidence levels are established to test the hypotheses that Out of Sample performance is the same as In Sample performance. If it is, then this strengthens the possibility that the settings identified “In Sample” reflected [i.e. “intelligently”] some aspect of market structure or behavioural finance that allowed the strategy to perform the same way “Out of Sample”, too. If not, then the probability increases that the “In Sample” findings were no more than a meaningless curve-fit. In which case we abandon them and move on to search for alternative settings.
So the question of “how many trades are needed for a backtest?” can also be answered by approaching it from the above angle: … how many trades are required to produce two big enough trade populations (“In Sample” and “Out of Sample”) that statistics can be used to attack the curve fitting genie at whatever significance level (e.g. 90%, 99%, 99.9%, etc.) is required.
i.e. they are just the strategy settings that would have worked best by chance over the historical data used, but contain no “intelligence” about price action or market structure that will enable the strategy to perform well if future price action is different … which of course it will be…
... like a lottery winner wins by chance but has no edge in terms of winning again in the future…
One way to have a shot at beating the curve fitting genie is to make use of “In Sample” and “Out of Sample” data. Optimization is undertaken using the “In Sample” data, and then the optimized settings are used to simulate performance in the “Out of Sample” data. Then statistics is used to compare the two populations (i.e. Population 1 - “In Sample” trades and their associated performance statistics, and Population 2 - “Out of Sample” trades and their statistics). With statistics, significence levels are established to test the null hypotheses that - say - the “Out of Sample” average trade, “Out of Sample” PF, etc. is the same as the “In Sample” average trade, etc.
That is, confidence levels are established to test the hypotheses that Out of Sample performance is the same as In Sample performance. If it is, then this strengthens the possibility that the settings identified “In Sample” reflected [i.e. “intelligently”] some aspect of market structure or behavioural finance that allowed the strategy to perform the same way “Out of Sample”, too. If not, then the probability increases that the “In Sample” findings were no more than a meaningless curve-fit. In which case we abandon them and move on to search for alternative settings.
So the question of “how many trades are needed for a backtest?” can also be answered by approaching it from the above angle: … how many trades are required to produce two big enough trade populations (“In Sample” and “Out of Sample”) that statistics can be used to attack the curve fitting genie at whatever significance level (e.g. 90%, 99%, 99.9%, etc.) is required.
Last edited:
Am I right that there is some confusion here regarding the use of the terms sample and population? I think a sample is what you get from backtests. Population is what defines the actual distribution of your system returns. I think someone else before also mixed up those.
http://www.dissertation-statistics.com/population-sample.html
Again, I think the way to put it is : "how many trades are required to produce two big enough trade samples"
The answer IMO is many thousands because the actual population of trades is very large.
I just saw this but I am not sure I fully understand it. Can this bound be used to determine the number of trades based on win rate?
Last edited:
Yes, you're right I have mixed up sample and population. Thanks!
Thanks!
It probably could if you equate p to win rate ...
Referring to wikipedia entry for Chernoff bound ... "the Chernoff bound requires that the variates be independent". So you have to assume that trades are independent of each other, and ignore serial correlation. So it is just a rough approximation to what is really happening, but may still be useful ...
Last edited:
Thinking further about this, I think it effectively gives a way to estimate how many trades you need in each of your In Sample and Out Of Sample "samples" to be "(100 - epsilon) % certain" that say the two average trade measurements can be compared. ...
Some of you are focusing way too much on the statistics, without really seeing the forest from the trees.
If you have only a small subset of data, there is no reason to believe the same conditions will necessarily persist on a new small subset of data. Market properties change a lot, and therefore a lot of market conditions should be required to give you more confidence, not a lot of trades in a small subset of market conditions.
If you have only a small subset of data, there is no reason to believe the same conditions will necessarily persist on a new small subset of data. Market properties change a lot, and therefore a lot of market conditions should be required to give you more confidence, not a lot of trades in a small subset of market conditions.
