Just how reliable is backtesting?

[lindq]

So my backtesting model which is supposed to give atleast a 4.6 win/loss ratio from a sample size that covered 68% of the backtested price data, has been giving me nothing but losses these past few weeks. Good thing it was only minor losses otherwise I would have almost certainly blown up. Has made me question just how much faith one should put in backtesting... Like how do I even find out my model is now completely crap based on recent price data skewing the win/loss ratio way down or that I have to wait months before a successful trade? Obviously I can't

You don't know if your system is in a drawdown, or broken.

The fact that you don't indicates a lack of confidence in your system.

You simply have not lived with it long enough to make a judgement. That ONLY comes from experience.

You may have something valuable. Or may not. Only time will tell. No shortcut for that.

 

[Xela]

Then what do you use

Personally, I trade from price action, without indicators. But that has nothing to do with the discussion, really ... I'm not suggesting that "nobody should use indicators" just because I prefer not to.

 

[Simples]

So my backtesting model which is supposed to give atleast a 4.6 win/loss ratio from a sample size that covered 68% of the backtested price data, has been giving me nothing but losses these past few weeks. Good thing it was only minor losses otherwise I would have almost certainly blown up. Has made me question just how much faith one should put in backtesting... Like how do I even find out my model is now completely crap based on recent price data skewing the win/loss ratio way down or that I have to wait months before a successful trade? Obviously I can't

Try to list your assumptions, then what might possibly break them.

How long is your backtest? What is the reasoning behind your trade plan? If it's just an MA crossover system, why do you expect consistent profits (edge) from using an average of prices as triggerline? If this instrument had stellar historical results, why wouldn't it return to the mean of the other instruments now? Where in the business cycle is the market now, and are you playing the right direction or waiting out possible adversity? Do you have enough trust in your system, robustness and financial stamina to keep at it, or may that mean you go bust? If you forget everything you think you know and just look at the long-term chart, what might be happening right now do you think and why would your system outsmart the market?

This is a hard simple game. Are you prepared to double down on time and avoiding spending too much on dead ends? What's keeping you from seeing what happens down the road (several months from now)?

 

[qxr1011]

So my backtesting model which is supposed to give atleast a 4.6 win/loss ratio from a sample size that covered 68% of the backtested price data, has been giving me nothing but losses these past few weeks. Good thing it was only minor losses otherwise I would have almost certainly blown up. Has made me question just how much faith one should put in backtesting... Like how do I even find out my model is now completely crap based on recent price data skewing the win/loss ratio way down or that I have to wait months before a successful trade? Obviously I can't

The main problem with back-testing is not just how well its results, based on the past data, will indicate the future performance of the method.

The main problem of the back-testing is that it does not test the trader himself !

So when in the future the day will come and the shit will hit the fan, the method may still be working, but the trader (who was never tested) will start pissing in the pants and abandon the method....

 

[userque]

Backtesting is extremely reliable if done properly. The backtesting should trade exactly as you would trade forwards.

There's much more to it, but two main points are proper use of out-of-sample data and avoiding leaks from 'future' data.

 

[Stockmarketmap]

A good, informative interview on "backtesting" systems.
Is it skill or is it luck ...

 

[zdreg]

How did you create your model?

I think a lot of people do backtesting wrong. In machine learning there's a concept of a training set and a test set -- you'd *never* take the results of the model on the training set as any indication of the model's strength. Instead you have to see how well it generalizes to new data. You may have committed a similar error when you created your model.

sounds like you learned from the author of the master algorithm
http://www.washington.edu/news/2015...domingos-author-of-the-master-algorithm/:cool:

 

[syswizard]

A good, informative interview on "backtesting" systems.
Is it skill or is it luck ...

Good interview, but his book is not really selling that well.
However, this guy's book is:
Wow, 247 reviews !! 5 star average rating !!!
This guy must be on to something....

 

[Surgo]

sounds like you learned from the author of the master algorithm
http://www.washington.edu/news/2015...domingos-author-of-the-master-algorithm/:cool:

Err... "Cool in a crisis? UW Emergency Management seeks volunteer responders"? Not sure what that has to do with anything. I'm just talking about basic statistics concepts that you'd learn in the first week of any college-level machine learning class.

 

[zdreg]

Err... "Cool in a crisis? UW Emergency Management seeks volunteer responders"? Not sure what that has to do with anything. I'm just talking about basic statistics concepts that you'd learn in the first week of any college-level machine learning class.

re: interesting interview
correction: http://www.washington.edu/news/2015/09/17/a-q-a-with-pedro-domingos-author-of-the-master-algorithm/