how much data is needed for backtesting?

[FulltimeTrainee]

I have developped a "profitable" system upon ym's last 3 months of data (from 9th Mars to First Juin). The percent profitable is 40.87% and the profit factor is 2.1. While the performance report looks all beautiful, I know that this isn't reliable yet since the amount of data is quite limited. My questions to those experienced system traders are the following:

What is the minimum amout of data need to backtest?

Since the market is changing, would the old data (like those older than a year) biase the developpement of a system ? In another word, Should I just use the most recent data to develop my system in order to approach the most recent tendency of the market ?

If so, I will have to update the parameters of my system often, like every week or every month, right?

Thank you!

 

[JimmyJam]

here's one guess.

at least two different market cycles.

we're in a bull cycle

you need a bear cycle

and then it would be great to have a "range bound" cycle as well

so, oh, at least 5 yrs ...

otherwise you're going to wake up one days and your "system" will have stopped working:eek:

best,

jimmy

 

[Markus]

There are a few factors that you need to consider when determining the period for backtesting your trading system:

Trade frequency
How many trades per day does your trading system generate? It's not important how long you backtest a trading system; it's important that you receive enough trades to make statistically valid assumptions*: If your trading system generates three trades per day, i.e. 600 trades per year, then a year of testing gives you enough data to make reliable assumptions*. But if your trading system generates only three trades per month, i.e. 36 trades per year, then you should backtest a couple of years to receive reliable data.

Underlying contract
You must consider the characteristics of the underlying contract. The chart below shows the average daily volume of the e-mini S&P:

It doesn't make sense to backtest a trading system for the e-mini S&P before 1997, because the contract simply didn't exist! In my opnion it doesn't make sense to backtest an e-mini trading system before 2002 because at that time the market was completely different; less liquidity and different market participants. I believe that a reliable testing period for the e-mini S&P are the years 2002 - 2004.

Margin of Error
The more trades you use in your backtesting (without curve-fitting), the higher the probabilities that your trading system will succeed in the future. Look at the following table:

Number of Trades 50 100 200 300 500
<>Margin of Error 14% 10% 7% 6% 4%

The more trades you have in your backtesting, the smaller the margin of error, and the higher the probability of producing profits in the future.

My 2 cents.... Hope it helps

Markus

 

[FulltimeTrainee]

Performance report

 

[FulltimeTrainee]

p1

 

[FulltimeTrainee]

p2

 

[FulltimeTrainee]

There's another thing I worry about is the optimisation. The performance of the system is 5 times better with the optimisation. There are 4 time zones for trading in this system, and each use the same indicators for entry and exit, but the parameters are different. Those parameters were found with optimisation. I'm affraid that the optimisation is overly done which boost up the misleadingly good results.

 

[radolym]

Is this an intraday system or EOD? Also, can you test on basket of stocks in Tradestation or these results based on 1 stock?

 

[nononsense]

Quote from FulltimeTrainee:

There's another thing I worry about is the optimisation. The performance of the system is 5 times better with the optimisation. There are 4 time zones for trading in this system, and each use the same indicators for entry and exit, but the parameters are different. Those parameters were found with optimisation. I'm affraid that the optimisation is overly done which boost up the misleadingly good results.

You're almost there. Start working now on predicting the transitions from one "zone" to another.
If everything goes well, this might require you only 5 to 10 years.

PS: don't get sidetracked by "Trend" guru's & addicts.

 

[99atlantic]

Quote from FulltimeTrainee:

There's another thing I worry about is the optimisation. The performance of the system is 5 times better with the optimisation. There are 4 time zones for trading in this system, and each use the same indicators for entry and exit, but the parameters are different. Those parameters were found with optimisation. I'm affraid that the optimisation is overly done which boost up the misleadingly good results.

as long as the system is profitable in its base level, and surrounding optimization paramteres are profitable, you're probably okay. E.g., if 22 is optimal and 20,21, 23, etc.. are also profitable, you should be okay; but it 22 is optimal and 20 or 24 or arodun there tanks and looses you money......