Just how reliable is backtesting?

[nxt7]

So my backtesting model which is supposed to give atleast a 4.6 win/loss ratio from a sample size that covered 68% of the backtested price data, has been giving me nothing but losses these past few weeks. Good thing it was only minor losses otherwise I would have almost certainly blown up. Has made me question just how much faith one should put in backtesting... Like how do I even find out my model is now completely crap based on recent price data skewing the win/loss ratio way down or that I have to wait months before a successful trade? Obviously I can't

 

[Surgo]

How did you create your model?

I think a lot of people do backtesting wrong. In machine learning there's a concept of a training set and a test set -- you'd *never* take the results of the model on the training set as any indication of the model's strength. Instead you have to see how well it generalizes to new data. You may have committed a similar error when you created your model.

 

[d08]

Try to limit the amount of parameters/filters, the more trades you filter out the more unlikely it is to work with forward data.

 

[nxt7]

How did you create your model?

I think a lot of people do backtesting wrong. In machine learning there's a concept of a training set and a test set -- you'd *never* take the results of the model on the training set as any indication of the model's strength. Instead you have to see how well it generalizes to new data. You may have committed a similar error when you created your model.

lol machine learning is way beyond my league.

 

[nxt7]

Try to limit the amount of parameters/filters, the more trades you filter out the more unlikely it is to work with forward data.

Literally just using 1 parameter and thats a moving average. I don't know how much simpler I can get

 

[K-Pia]

lol machine learning is way beyond my league.

But not the point.
Do not overfit on training set.
And test forward on untrained data.

 

[K-Pia]

Literally just using 1 parameter and thats a moving average. I don't know how much simpler I can get

How much backward have you backtested ?
A simple MA system might be good for daily data.
And some ticker are more appropriate than other for that.

 

[nxt7]

How much backward have you backtested ?
A simple MA system might be good for daily data.
And some ticker are more appropriate than other for that.

The entire historical data of the instrument

 

[K-Pia]

The entire historical data of the instrument

Ok. Maybe there are time the system performed similarly.
For exemple in time of low volatility ? Don't know.
It's not only about the size of the data set.
But about the conditions, underlyings.

 

[d08]

MA has a period and the type of MA used was also probably not selected at random. Personally I'd never base trade decisions solely on a moving average but that's just me.