Which back test is more dependable?

[kiev]

I do back test on Strategy A with Day bar data in past 10 years, and do back test on Strategy B with Minutes bar data in past 2 years, Can you tell me, comparatively, which test is more dependable?

 

[NetTecture]

Depends on your strategy lengths. EOD based strategies will be nice with the first. Try backtesting an intraday strat with the first and you realize it is stupid to even try.

 

[dom993]

IMO ... 2 years of historical data isn't enough to be representative of the various modes a market can be in.

You could have 5000 trades in a 2 years (intraday) backtest, and still have a large exposure to the next regime change.

If your intraday strategy has good results on 2 years, you should consider purchasing additional historical data to extend its test coverage.


Re. the EOD strategy, 10 years is only 2500 bars ... in such a small space, it doesn't take a lot to reach overfit-land.

 

[Wide Tailz]

Quote from dom993:IMO ... 2 years of historical data isn't enough to be representative of the various modes a market can be in.

This has also been my experience with hourly bar strats - they needed about 5 years of data to be robust in the forward test.

 

[kiev]

Quote from dom993:

Re. the EOD strategy, 10 years is only 2500 bars ... in such a small space, it doesn't take a lot to reach overfit-land.

Can you tell me how many bars you think enough to avoid overfit?

If you use 20 years data, the company which can exist 20 years have very different properties from others.
Do you think the strategies working well on these over 20 years old companies can be applied to others?

 

[dom993]

Quote from kiev:

Can you tell me how many bars you think enough to avoid overfit?

If you use 20 years data, the company which can exist 20 years have very different properties from others.
Do you think the strategies working well on these over 20 years old companies can be applied to others?

I have a system which generates ~ 10,000 trades in 6 years on 1 market, and I am still suspecting overfit ...

... if your EOD system trades stocks, then you should be able to reduce the risk of overfit by backtesting over a reasonably large set of stocks. Going to 20 years only doubles the size of the backtest, using 20 stocks on 10 years is already 10 times larger than 20 years on 1 instrument.

Robert Pardo in his book (Evaluation & Optimization of Trading Strategies) gives some metrics to assess if a system is overfit or not.

 

[kiev]

thanks for your information.

 

[QuantWizard]

Quote from kiev:

thanks for your information.

If you're worried about over fitting you should also keep the number of parameters to a minimum...

 

[lindq]

Quote from kiev:

I do back test on Strategy A with Day bar data in past 10 years, and do back test on Strategy B with Minutes bar data in past 2 years, Can you tell me, comparatively, which test is more dependable?

It depends on the instrument(s) you are trading. If you are backtesting a portolio of stocks, EOD data can often be suspect in terms of daily highs/lows. What you see on a daily bar may have been logged by a data provider, but could not have been actually traded. Going to intraday data can help, but itself is no guarantee that a trade could have been executed.

That's the limitation of backtesting. Its a guide, but no guarantee of profits.

 

[toc]

Backtest is no match for real time observation of a system. Also remember that markets tend to change constantly and a system has to adapt to the changes or go out of business.