"Rigorous" walk-forward backtesting?

[mizhael]

I've heard that the most rigorous backtesting mechanism is the "walk-forward" optimization where you optimize your parameters dynamically.

This maximally emulates the way strategy developers backtest and apply our trading strategies: at any point of time, one optimizes the parameters based on the data from the past history, and then apply the "optimized" model onto a certain future period; after a while, one re-calibrates by refitting the model to updated data.

After the "walk-foward optimization", there are only two parameters that are global and floating - the lookback period in the "walk-forward optimization" (i.e. on each optimization, what past data does on optimize his model on), and the forward step size (i.e. how often does one re-calibrate)...

Any thoughts on how to choose these two global floating grand parameters?

If I run another bigger optimization on top of these "walk-forward optimizations", and I found that:

1. The overall Sharpe Ratio is 2.09 if I re-calibrate the model every 12 months, and use the past 6 months data in each optimization.

2. The overall Sharpe Ratio is 1.98 if I re-calibrate the model every 1 day, and use the past 6 months data in each optimization.

And based on above results, I choose to re-calibrate the model every 12 months, instead of every 1 day, I feel I am just data-mining... what do you think?

Any better approaches?

Thanks a lot!

 

[Wide Tailz]

Either way it looks like you're coming out ahead. The big risk is that the assumptions behind the original system become invalid some day and the system blows up your account, not whether your sharpe ratio goes down 10%.

I've seen 80% / 20% recommended many times, but I'm too new at this to have any war stories.

 

[jcl]

Quote from mizhael:

I've heard that the most rigorous backtesting mechanism is the "walk-forward" optimization where you optimize your parameters dynamically.

This maximally emulates the way strategy developers backtest and apply our trading strategies: at any point of time, one optimizes the parameters based on the data from the past history, and then apply the "optimized" model onto a certain future period; after a while, one re-calibrates by refitting the model to updated data.

After the "walk-foward optimization", there are only two parameters that are global and floating - the lookback period in the "walk-forward optimization" (i.e. on each optimization, what past data does on optimize his model on), and the forward step size (i.e. how often does one re-calibrate)...

Any thoughts on how to choose these two global floating grand parameters?

If I run another bigger optimization on top of these "walk-forward optimizations", and I found that:

1. The overall Sharpe Ratio is 2.09 if I re-calibrate the model every 12 months, and use the past 6 months data in each optimization.

2. The overall Sharpe Ratio is 1.98 if I re-calibrate the model every 1 day, and use the past 6 months data in each optimization.

And based on above results, I choose to re-calibrate the model every 12 months, instead of every 1 day, I feel I am just data-mining... what do you think?

Any better approaches?

Thanks a lot!

The forward step size is a compromise between test duration and test accuracy. As a rule of thumb, a step size should cover at least 20 trades.

The lookback period is important for performance and depends on your strategy. Some strategies require a long and others a short lookback period. The reason is that the markets changes over time. You don't want to have too many different market situations in the lookback - it's just the opposite of a fixed parameter optimization. The lookback period should cover several 100 trades for a good parameter quality.

We found that a lookback/test ratio of 85:15 works best for most strategies.

Your strategy seems to be quite robust when it has about the same results with a 2:1 as well as a 150:1 ratio.

WFO requires parameter-free strategies, so you also need a mechanism that regularly recalculates the parameter while trading. When your trade platform does not support this, you'd need restart the strategy, download newest price data, and run a WFO cycle again after every step. This would be more difficult with a one day step size, than with a one month step size.