How much backtesting?

[harrytrader]

Yes I agree. This is essential because all statistic inference is only true within that implicit premisce of independancy. If one trade depends on another there is a risk of overestimating. If you take one trade and then another 1 minute later this is obvious. But it can be also the same thing with two trades that has been taken the same week since because of market symetry property market cycle tend to repeat more or less although in counterclock path and if your system is also symetrical these two trades will be in fact dependant.

So as a practical protocol, one can say that statistically it is at least needed 35 items (100 is better of course) in a sample that must be sampled randomly among a population of hundreds of trading days if one want to be sure that the sample is not biased by a too much small size of the population. If one hasn't such a huge number of trading days then rather take a survey approach by splitting days into several market contexts and sub-sampling within each context so as to constitute the whole sample. Of course I simplify but that's the basic idea. If you want to refine with monte-carlo and things like that it's up to you but sometimes it's not more worth than a gadget approach although it is at least useful to have a better visual feeling.

Quote from axeman:

For a short term scalping system, 100 may be enough,
but not if they all occur within a single week

I would trade several hundred, but takes samples from
several years and different seasons, etc.

peace

axeman

 

[Lawrence Chan]

You will need at least some significant amount of data for various known market cycle to be sure.

1. bull period
2. bear period
3. topping period
4. bottoming period
5. and very important for scalping system - congestion period

For each type of market environment you need at least 1 to 2 months worth of data.

One very interesting time period are the months we've just experienced because it is a WAR period but there were news fed out in real-time that shocked the market every hour.

Since this particular period is not a normal condition, you may want to go back to some other historical data for your testing.

 

[jack hershey]

Do random trades and consecutive trades in two separate back tests.

If and when your tests for statistical sig between the two fail, then you have enough of a sample. The minimum sample size is the sum of the two runs.

Differing algorithms require different sample sizes.

 

[spreadem]

Yes yes yes.... but my point is, systems come in all kinds of flavors, and a system with a 25 in a row losing streak should not automatically be disqualified. Money management can sometimes solve issues like this.

Yes, but your backtesting also gives an indication of the tendencies of the trading system. If that trading systems has tendencies to have long periods of drawdowns, it's valuable to the trader to determine if he/she would be comfortable psychologically with that kind of trading systems.

Therefore backtesting should not only tell the trader if the trading system works or not, but it should also tell the trader the tendencies of the trading system.

 

[axeman]

Excellent points.

I like to take a lot of time to discover the "nature" my trading
systems. To get a good "feel/understanding" of what the best,
worst and normal cases are.

peace

axeman

Quote from spreadem:

Yes, but your backtesting also gives an indication of the tendencies of the trading system. If that trading systems has tendencies to have long periods of drawdowns, it's valuable to the trader to determine if he/she would be comfortable psychologically with that kind of trading systems.

Therefore backtesting should not only tell the trader if the trading system works or not, but it should also tell the trader the tendencies of the trading system.