Just how reliable is backtesting?

nxt7 said:
So my backtesting model which is supposed to give atleast a 4.6 win/loss ratio from a sample size that covered 68% of the backtested price data, has been giving me nothing but losses these past few weeks. Good thing it was only minor losses otherwise I would have almost certainly blown up. Has made me question just how much faith one should put in backtesting... Like how do I even find out my model is now completely crap based on recent price data skewing the win/loss ratio way down or that I have to wait months before a successful trade? Obviously I can't
More...
Backtesting is futile. You are now learning that lesson. Forward testing is all that is useful in the plan development process.
 
dratsum said:
Backtesting is forward testing for those doing it right.
More...

Yep, if you've only got 1Min OHLC data, then only build and back test systems which work on the Close of the Bar that way ( open of a new bar maybe ? ), the back test will be identical to a live test.

Simples!
 
Turveyd said:
Yep, if you've only got 1Min OHLC data, then only build and back test systems which work on the Close of the Bar that way ( open of a new bar maybe ? ), the back test will be identical to a live test.

Simples!
More...

You're completely ignoring the topic of curve-fitting of parameters. You only described how to do a very basic back-test properly.
 
d08 said:
You're completely ignoring the topic of curve-fitting of parameters. You only described how to do a very basic back-test properly.
More...

Ahhh but all you can do is Curve fit, run 2 data sets, 1 6months and before, 1 last 6months, test on the older 1, then confirm still viable on newer 1.

Just got to hope your curve fit, still has some profit making time left, before you have to re curve fit.
 
d08 said:
Well, curve-fitting implies to me that the strategy doesn't work on any out of sample data.
More...

Exactly!
If you start to curve fit you will always run behind the facts. In best case you will find what was the best setup in past. But you need a setup for the future.
 
What you folks are describing with the "curve fitting" is exactly why you have a separate training set and test set. It is not statistically correct to validate your learned parameters (the aforementioned "curve") against the same set you learned them from.
 
Turveyd said:
Ahhh but all you can do is Curve fit, run 2 data sets, 1 6months and before, 1 last 6months, test on the older 1, then confirm still viable on newer 1.

Just got to hope your curve fit, still has some profit making time left, before you have to re curve fit.
More...
Exactly! Re-curve fit daily, if trading daily.
 
You must log in or register to reply here.
Back
Top