Walk forward results

[peppermint_tea]

Hi,

I did a walk forward backtest, 4 years INS, 1 year OOS, starting in 2007. I optimised one parameter to find the best value for each INS and used that ‘best’ parameter for the next OOS.

Then once I got to the end of the walk forward, I simply added up each ‘best’ parameter value for all the OOS’s and took a simple average of it, then tested that average on the whole OOS length (2011-2023)

Over the same OOS date length (2011-2023), my results were a lot better when I used the average than if I used the best parameter for each OOS and stitched them together.

New to walk forward testing so just want to be sure i'm on the right track.
Have i done my testing ok/correctly?

Thanks

 

[MarkBrown]

mainly what matters is the variance of the optimal settings or the settings you used.

 

[peppermint_tea]

mainly what matters is the variance of the optimal settings or the settings you used.

thanks, I simply used a parameter ranging from 50% to 130% and optimised it in steps of 10, so only 8 iterations. I don’t like to optimise much. Simpler is better imo

 

[MarkBrown]

thanks, I simply used a parameter ranging from 50% to 130% and optimised it in steps of 10, so only 8 iterations. I don’t like to optimise much. Simpler is better imo

i guess what i was asking is there a setting that endured all the testing - and if so what was the deviation from it's optimal?

 

[peppermint_tea]

i guess what i was asking is there a setting that endured all the testing - and if so what was the deviation from it's optimal?

Not sure how to answer your question to be totally honest, apologies for my lack of understanding, trying to learn as i go.
Here are the parameters from the INS that were the 'best' for the OOS :

OOS 1 110%
OOS 2 110%
OOS 3 110%
OOS 4 90%
OOS 5 90%
OOS 6 100%
OOS 7 90%
OOS 8 120%
OOS 9 80%
OOS 10 130%
OOS 11 120%
OOS 12 120%
OOS 13 120%

Average : 107%

 

[MarkBrown]

The standard deviation of each reading from the mean is approximately 14.72% points.

Best if that number less than 5%.... better if below 3 and good at 2.

 

[Statistical Trader]

As soon as you took the average and then tested that on 2011-2023, you are no longer doing OOS testing because all the so-called "OOS" periods were used to determine the parameter value you decided to use. So, those results are just in-sample testing and so it's not surprising that they out-performed your walk forward testing. The walk forward is a more honest assessment of the results you might get in the future.

There's nothing wrong with your parameter changing over time IF you have reason to believe that different market periods will behave differently for your strategy and IF you have reason to believe your strategy is able to adapt to such changes via changes to the value of the parameter and also IF you have reason to believe four years of training is a good length to tune your parameter for the following year. That's an awful lot of "if"s and so from that perspective, it's preferable to have a stable optimized parameter value with stable results in OOS data, as @MarkBrown was saying.

What Mark is probably saying is that you should see if there is a robust parameter value that did well throughout rather than trying to get the optimal parameter value each time. In other words, if p1 consistently makes a good profit across all OOS periods, that would probably be preferable to changing the parameter value every year unless doing so results in a substantial improvement in your outcome (...and you have reasons to believe you are able to optimize as the market changes...)

There's a lot more that could be said, but I would need to know much more about the situation.

If I were you, I might run an analysis of your 8 parameter values x your 17 years (136 runs) just to see if there are some parameter values that consistently did well across time, etc. I know that's backtesting, but I wouldn't be doing this to optimize at this stage, just to get a sense of the "landscape" you're working with.

We could also talk about *what* you are optimizing. Return? Something adjusted for risk? Etc. That's also important.

 

[peppermint_tea]

As soon as you took the average and then tested that on 2011-2023, you are no longer doing OOS testing because all the so-called "OOS" periods were used to determine the parameter value you decided to use. So, those results are just in-sample testing and so it's not surprising that they out-performed your walk forward testing. The walk forward is a more honest assessment of the results you might get in the future.

That was exactly what i thought may be the case, i just wasn't sure.

What Mark is probably saying is that you should see if there is a robust parameter value that did well throughout rather than trying to get the optimal parameter value each time. In other words, if p1 consistently makes a good profit across all OOS periods, that would probably be preferable to changing the parameter value every year unless doing so results in a substantial improvement in your outcome (...and you have reasons to believe you are able to optimize as the market changes...)

Thats a good point, i'll look into that.

This is the first time i've ever tried to get the optimal parameter each time like this using walk forward. What i normally do is take my data, split it up into 3, test INS, then test on the two OOS parts. I just thought i'd explore walk forward to see what it may offer.

Normally I do some changes to parameters when i'm INS just to try some different ideas out, eg stop distances or rule parameters, but i never 'run' a bunch of optimizations and just pick the best. I keep to round numbers, simple rules, and few rules, but most importantly a sound idea from the start.

We could also talk about *what* you are optimizing. Return? Something adjusted for risk? Etc. That's also important.

I wasn't optimizing for any individual thing, like net profit for example. What i did was, for each iteration I recorded the net profit, profit factor, max dd, etc, then i would just choose which parameter looked the best/most robust according to those results. Thats the one I then chose for the next OOS.

Thanks for your replies both, its really appreciated to have an input from people who are well versed in this type of thing a lot more than i am.

 

[peppermint_tea]

The standard deviation of each reading from the mean is approximately 14.72% points.

Best if that number less than 5%.... better if below 3 and good at 2.

Really interesting this is. I get that now (I think!). What you're saying is you don't want much variance from the mean because it kind of says your 'optimal' parameter is too erratic each time. If it is within 5% of the mean you know you have some consistency/the values aren't too far removed from the mean.

 

[MarkBrown]

Really interesting this is. I get that now (I think!). What you're saying is you don't want much variance from the mean because it kind of says your 'optimal' parameter is too erratic each time. If it is within 5% of the mean you know you have some consistency/the values aren't too far removed from the mean.

right like "little red riding hood" find the partridge that is not to cold and not to hot but just right.. lol