Just how reliable is backtesting?

d08 said:
Well, curve-fitting implies to me that the strategy doesn't work on any out of sample data.
More...
The 'hate' against curve-fitting assumes that curve-fitting on training data implies no such curve fitting on out-of-sample data. The 'hate' is also likely just being regurgitated without giving it much thought.

But, in fact, the best system will fit both training and out of sample data...that is--curve fit.
 
Related to the topic of backtesting, but a subject so rarely touched upon...is it possible to trade a full day and still be productive in your backtesting during the evenings (and weekends)? Quite frankly, the intense focus and levels of concentration required during a full day's trading leave me mentally drained at the end of the day's play (even after a relatively lucrative day in the markets).

When I read stories written by some ET posters about how their systems were built, little mention is made of how they balanced the backtesting work with other commitments, such as earning a living from existing trading systems or other means, trading preparation and planning, ongoing evaluation, trading education, some form of physical exercise for a healthy mind, family commitments, and dare I mention it...a "balanced" life with some form of social aspect. :wtf:

The impression I get from many extremely successful traders who built something really substantial and scaleable, is that all of these other commitments were put on hold for several months or years at a time, and quite possibly no trading at all took place until their system was complete, or at least until the bulk of the system design and backtesting was complete.

Is this impression of mine in line with the reality? That is, you can trade full time, or you can do research and backtest full time, but undertaking both roles at the same time with a high degree of success is unrealistic. I am interested to hear about others experiences dealing with this conundrum.
 
I am usually up 18 hours a day as I trade in the Americas, Asia, and Europe. I certainly do spend more time back testing then I do trading. I do not like to trade blind so if I do not have time to backtest and trade and I do not trade.

I do not do much trading in the in-between hours, so I do exercise in those hours. Most of my trading is done near close.
 
benwm said:
Related to the topic of backtesting, but a subject so rarely touched upon...is it possible to trade a full day and still be productive in your backtesting during the evenings (and weekends)? Quite frankly, the intense focus and levels of concentration required during a full day's trading leave me mentally drained at the end of the day's play (even after a relatively lucrative day in the markets).

When I read stories written by some ET posters about how their systems were built, little mention is made of how they balanced the backtesting work with other commitments, such as earning a living from existing trading systems or other means, trading preparation and planning, ongoing evaluation, trading education, some form of physical exercise for a healthy mind, family commitments, and dare I mention it...a "balanced" life with some form of social aspect. :wtf:

The impression I get from many extremely successful traders who built something really substantial and scaleable, is that all of these other commitments were put on hold for several months or years at a time, and quite possibly no trading at all took place until their system was complete, or at least until the bulk of the system design and backtesting was complete.

Is this impression of mine in line with the reality? That is, you can trade full time, or you can do research and backtest full time, but undertaking both roles at the same time with a high degree of success is unrealistic. I am interested to hear about others experiences dealing with this conundrum.
More...

There talking EA's here, so you write the code, hit the Test button, wait 5mins out hey presto how did it do, failed, change a few things keep going to the curve works with the history.

Personally, as a normal daytrader, back testing isn't really that useful, it's about reacting live where 1 second can make the difference between $200 profit and market jumping 20pts and wiping it out ( happened Thursday, finger going to Close didn't get there in time LOL )

Self employeed so don't work full time hours well rarely, I fit trading around it.

Stress levels are getting better as I trade more recently, but 2 - 3 hours and I'm wiped.
#
 
userque said:
But, in fact, the best system will fit both training and out of sample data...that is--curve fit.
More...

This means that you will NOT "Re-curve fit daily, if trading daily". Your system should be self-learning or self-adapting.
The fact that it will change every day proofs that it is worthless. A good system works always in all market circumstances. The only thing that can influence the results is how big the trends are. If very trending, profits will be bigger and win rate will be higher, and in non trending markets profits will be smaller and win rate will be lower.
But the system on itself will always work and will not have to be curve fitted every day.
 
i am nobody said:
This means that you will NOT "Re-curve fit daily, if trading daily". Your system should be self-learning or self-adapting.
The fact that it will change every day proofs that it is worthless. A good system works always in all market circumstances. The only thing that can influence the results is how big the trends are. If very trending, profits will be bigger and win rate will be higher, and in non trending markets profits will be smaller and win rate will be lower.
But the system on itself will always work and will not have to be curve fitted every day.
More...

LOL
 
Turveyd said:
There talking EA's here, so you write the code, hit the Test button, wait 5mins out hey presto how did it do, failed, change a few things keep going to the curve works with the history.

Personally, as a normal daytrader, back testing isn't really that useful, it's about reacting live where 1 second can make the difference between $200 profit and market jumping 20pts and wiping it out ( happened Thursday, finger going to Close didn't get there in time LOL )

Self employeed so don't work full time hours well rarely, I fit trading around it.

Stress levels are getting better as I trade more recently, but 2 - 3 hours and I'm wiped.
#
More...

You're describing the very basic approach to backtesting that's just parameter optimization. There are many aspects like obtaining non OHLCV data, processing it, combining it with OHLCV in certain circumstances. Then there's analyzing liquidity, slippage, how to dist/acc orders to minimize impact etc. Analyzing how multiple systems behave in the same portfolio is also important in respect to drawdowns and correlation. There's much more to it than just "EA" parameter modifications.
 
userque said:
Exactly! Re-curve fit daily, if trading daily.
More...

You do that? I would consider that system useless. The system should be adaptive and the only need for substantial modification would happen not more than 3-4 times per year.
Not much changes in 1 day when it comes to the structure of the markets, therefore daily optimization would be unnecessary.
 
d08 said:
You do that? I would consider that system useless. The system should be adaptive and the only need for substantial modification would happen not more than 3-4 times per year.
Not much changes in 1 day when it comes to the structure of the markets, therefore daily optimization would be unnecessary.
More...
Yes, I do that. I disagree with your premise.
 
You must log in or register to reply here.
Back
Top