How far back is enough for a backtest?

M

markc

I know this question has been addressed in the past, but I have not seen this particular situation discussed before.

I have spent some time designing and backtesting some simple strategies in TS2000i based on daily S&P bars. They mostly exit on the close and enter on the open. Regardless of what system I use - they all seem to do poorly prior to 1997/1998 and do quite well from then up till now.

I understand the market went electronic around this time with the introduction of the emini. I am unsure whether this caused the subsequent increase in volatility or if it was related to the market nearing the top and then the sell off over the past couple of years or so.

So, I 'could' just look at the data I have from 1997/8 till now and base my results on that because the 'market changed', or I could take 10 years worth and use that - because we could go back to a low-volatiliy environment from which it appears we came. The problem is my systems don't make money prior to 1997 so I really have no clue how much confidence I can place in the backtest - and the probability of my systems performing in the future.

I'm a bit stuck and wonder if other people have seen a similar situation in their own models. If you have, what makes you confident of future performance?

Attached is an equity curve trading a single contract from 1990 - 2003.

Mark
 

Attachments

in addition to that, I would like to ask why it is that in testing two systems (for stocks, not futures), system A performs moderately well in a backtest, system B performs wonderfully, yet in forward testing system A performs very well and system B is horrible, even after accounting for slippage and commissions. Knowing that back-tests are not predicative of future performance (which would argue against backtesting wouldn't it?) what does this disparity of performance imply of either system? And what steps are further needed to explore? Thanks.
 
in the creation of my trading system, i divided my data into three parts and tested my ideas on each section. for me the most important was to have at least 150 trades in each section. if a section of back data did not have 150 trades then i would increase the historical data until i created at least 150 trades. then i would compare the three sections. but the most important thing to always be aware of is the question "is it going to work in the future".
 
i. During the earlier period your system simply did not make money (but neither did it lose you money) it stayed flat - this I would take as a positive point, if the market does return to the conditions where your system performs "neutral" at least you aren't going to give it your money back

ii. Look at the market conditions where your system performed neutral - then use a filter.

iii. You have a few downs during your march to glory, look into what the market was doing during this period, again can you filter out these market conditions

iv. Your systems makes some then loses some (but less), can your personality cope with this or will you lose faith in your system
 
Quote from progers82:

in the creation of my trading system, i divided my data into three parts and tested my ideas on each section. for me the most important was to have at least 150 trades in each section. if a section of back data did not have 150 trades then i would increase the historical data until i created at least 150 trades. then i would compare the three sections. but the most important thing to always be aware of is the question "is it going to work in the future".
More...

Thanks for your thoughts. The model doesn't have that sort of trade frequency - but I like the idea of comparing different periods.
 
IF it doesn't have that frequency of trades, then imo backtesting can't be relied on and is subject to optimization/curve fitting.
 
Quote from thewoodcutter:

i. During the earlier period your system simply did not make money (but neither did it lose you money) it stayed flat - this I would take as a positive point, if the market does return to the conditions where your system performs "neutral" at least you aren't going to give it your money back

ii. Look at the market conditions where your system performed neutral - then use a filter.

iii. You have a few downs during your march to glory, look into what the market was doing during this period, again can you filter out these market conditions

iv. Your systems makes some then loses some (but less), can your personality cope with this or will you lose faith in your system
More...

Yes, I noticed that is was flat too back in the early 90's. So, it would certainly be better to tread water than go backwards. :)

I will follow up your suggestion on looking at the market periods where it was flat - I still think it is due to volatility changes. The drawdowns experienced on the 'march to glory :) will also be examined.

It is a low hit-rate approach, with tight stops and the w/l ratio skewed in my favour. I know I can handle this type of trading - as long as I am confident the wins will come.

Thanks for the feedback.

Can't get back here until tomorrow - thanks for the replies so far, some useful tips.

Mark
 
Quote from progers82:

IF it doesn't have that frequency of trades, then imo backtesting can't be relied on and is subject to optimization/curve fitting.
More...

Aahh, sorry, my mistake - 505 trades in 13 years. The period of that weird equity curve I posted.

I haven't got many variables to mess with and there are no indicators etc. Basically just fading certain gaps and trying to catch a possible volatility breakout. Enter on the open and exit on the close. Two types of entry, and two exits. (Stopped out, or at the close)

I'm only a novice at system building. But some of the results are encouraging me to try and turn ideas into workable models.
 
You must log in or register to reply here.
Back
Top