Walk-Forward Testing and Optimization

[intradaybill]

Quote from jcl:

I think not, as I'm meanwhile so familiar with this forum that I can already identify the trolls. At least one.

It is good that you can at least identify yourself after failing to understand that WFO is optimization. I think you should try somewhere else. There is no ground for you here.

 

[Rationalize]

Quote from jcl:

... Instead, you either have a parameter free system that automatically adapts to the market with a frequency equivalent to the WFO OOS period - that's what Zorro does. Or you have to manually download new price data and run a new parameter optimization every couple of weeks. Both methods keep the strategy at its effectivity maximum.

Are you contending that there is an ideal OOS length?

Because, with a short OOS, you look great in test, re-adjust a lot when live, curve fit & fail.

With a long OOS, you look poor in test, and fail.

Sorry to be blunt here, but where's the edge?

 

[jcl]

There is indeed no ideal OOS length.

In real trading, the strategy will obviously work the better, the more often the parameters are adapted. Of course you won't adapt them every week - that would only cause minimal parameter changes. But you can see in the WFO how often you must adapt them for keeping the strategy effective. The frequency of parameter adaption has nothing to do with overfitting. If the optimization process causes overfitting, the strategy will already fail in the WFO, and you won't trade it anyway.

In the WFO process, the OOS must have sufficient length to generate enough trades for a statistically relevant test. We found that this is sometimes critical, so Zorro uses oversampling for generating more trades. But this depends on the strategy.

Tradestation has a cluster analysis function for determining the optimal OOS length, but this is normally not necessary. A IS/OOS ratio in the 85:15 range works almost always.

 

[ssrrkk]

Quote from jcl:

There is indeed no ideal OOS length.

In real trading, the strategy will obviously work the better, the more often the parameters are adapted. Of course you won't adapt them every week - that would only cause minimal parameter changes. But you can see in the WFO how often you must adapt them for keeping the strategy effective. The frequency of parameter adaption has nothing to do with overfitting. If the optimization process causes overfitting, the strategy will already fail in the WFO, and you won't trade it anyway.

In the WFO process, the OOS must have sufficient length to generate enough trades for a statistically relevant test. We found that this is sometimes critical, so Zorro uses oversampling for generating more trades. But this depends on the strategy.

Tradestation has a cluster analysis function for determining the optimal OOS length, but this is normally not necessary. A IS/OOS ratio in the 85:15 range works almost always.

I am still thinking that you have mistaken local fits as a solution to overfitting. There is no free lunch ever, only trade-offs. A local fit is still an overfit in that it still has issues with generalizability. The solution to overfitting I think cannot be to overfit locally over and over again.

But here's one thing that you might be getting at that I might leave open as a possibility. You might be saying something like this: there is a finite-length memory or autocorrelation in the price trajectories of markets. This autocorrelation length in your estimation is about 10 weeks for certain instruments. Therefore, you can only hope to fit locally and use that fit to predict for the next 10 weeks. Beyond those 10 weeks, the signal disappears under the noise. Therefore, there is no hope to make a global fit beyond 10 weeks of training. In fact the academic literature is filled with these kinds of observations about autocorrelations.

This is similar to weather forecasting. You can be most assured about forecasting the next hour and perhaps up to the next day. Your prediction accuracy will go down for 3 day forecasts. Predicting beyond 10 days is pretty much hopeless. This is because weather models are always necessarily local fits to the current weather patterns and the immediate past. The same thing can be said about predicting the trajectory of hurricanes. You see a cone shaped forecast showing where it may go, and as time passes, the possibilities of where it might end up becomes larger and larger (the uncertainty becomes larger) until the prediction becomes useless (it can be anywhere on this side of the earth, etc.).

 

[jcl]

Quote from ssrrkk:

You might be saying something like this: there is a finite-length memory or autocorrelation in the price trajectories of markets. This autocorrelation length in your estimation is about 10 weeks for certain instruments. Therefore, you can only hope to fit locally and use that fit to predict for the next 10 weeks. Beyond those 10 weeks, the signal disappears under the noise. Therefore, there is no hope to make a global fit beyond 10 weeks of training. In fact the academic literature is filled with these kinds of observations about autocorrelations.

No, the 10 weeks are not an autocorrelation period. The autocorrelation horizon is much shorter. The 10 weeks are rather a stationarity period specific for the strategy. The strategy assumes a stationary market in relation to its adaptable parameters. You must also not forget that when you have 10 weeks OOS, you have about 2 years IS. That is not a local fit.

WFO is also no solution for overfitting. An overfitted strategy stays overfitted no matter how often you run a WFO with it. WFO is only a test that detects overfitting.

Assume that you've developed and optimized a strategy and the backtest tells you 100% annual profit. Now you run a WFO. The result: 10% annual loss. You now go: "Ugh, WFO killed my strategy." But you can be glad for that, otherwise you had the 10% loss in your account. Not WFO, but overfitting killed your strategy.

Now assume that WFO didn't give you 10% loss, but 80% profit. This is a worse result than the backtest, but still good enough. So you start trading it with regular parameter adaption. It works well 6 months, then you get a nasty drawdown.

Without WFO, you now had to think. Either the drawdown is a statistical fluctuation, then you can sit it out. Or the market has changed, then you have to stop the strategy immediately. What should you do?

When your strategy however was WFO tested over a long enough simulation period, you can continue it in both cases. If the reason was a market change, the strategy will likely adapt to it, because its regular parameter adaption was WFO confirmed. This way WFO gives you some more confidence about life trading your strategy.

 

[ssrrkk]

Quote from jcl:

No, the 10 weeks are not an autocorrelation period. The autocorrelation horizon is much shorter. The 10 weeks are rather a stationarity period specific for the strategy. The strategy assumes a stationary market in relation to its adaptable parameters.

WFO is also no solution for overfitting. An overfitted strategy stays overfitted no matter how often you run a WFO with it. WFO is only a test that detects overfitting.

Assume that you've developed and optimized a strategy and the backtest tells you 100% annual profit. Now you run a WFO. The result: 10% annual loss. You now go: "Ugh, WFO killed my strategy." But you can be glad for that, otherwise you had the 10% loss in your account. Not WFO, but overfitting killed your strategy.

Now assume that WFO didn't give you 10% loss, but 80% profit. This is a worse result than the backtest, but still good enough. So you start trading it with regular parameter adaption. It works well 6 months, then you get a nasty drawdown.

Without WFO, you now had to think. Either the drawdown is a statistical fluctuation, then you can sit it out. Or the market has changed, then you have to stop the strategy immediately. What should you do?

When your strategy however was WFO tested over a long enough simulation period, you can continue it in both cases. If the reason was a market change, the strategy will likely adapt to it, because its regular parameter adaption was WFO confirmed. This way WFO gives you some more confidence about life trading your strategy.

It seems to me that you refuse to see the connection between autocorrelation, memory / persistence length, stationarity of model parameters, and model generalizability...

 

[jcl]

Quote from ssrrkk:

It seems to me that you refuse to see the connection between autocorrelation, memory / persistence length, stationarity of model parameters, and model generalizability...

I think I know this connection quite well, at least I'm paid for writing algorithms and software based on it, together with many other professionals that are in test software development. But if you think that all trade platform developers are wrong in this point, please enlighten me. I promise to inform the industry immediately.

 

[ssrrkk]

Quote from jcl:

I think I know this connection quite well, at least I'm paid for writing algorithms and software based on it, together with many other professionals that are in test software development. But if you think that all trade platform developers are wrong in this point, please enlighten me. I promise to inform the industry immediately.

Well if you know this, then I don't know how you can flat out deny the relationship between the ACF, PACF and model stationarity.

Anyway, I am not arguing for or against WFO or any of that. I am just debating the validity of using successive local fits to overcome loss of generality of global fit models, that's all. And I don't claim to have any answers either; I am just bringing up questions like any scientifically minded person would.

 

[shopster]

In trading, the cycles of the game go in and out of phase.

all depends on how the " big dogs piss in the tall grass " on a daily basis.

this applies to forex , futures et al.

read hurst and ehlers.

once you learn to read the market signage offered up in live time you get to win.

In forex, the banks reset the game and have a 5 day plan.

they clear the board , blow out the stops and load up the retail pendings for the breakout guys at both ends of the channel box.

then they move the price action out for the next 3 levels of rise or fall.

while you sit there licking your wounds on your latest 25 tick stopped out trade.

" stuffing a digitally coded sack of rules that have been curve fitted up against a chunk of historical price data " and then running it in a live market .......is a death spiral that will suck your equity dry.

get your head wrapped around this........
market news has absolutely nothing to do with the direction of the game.

there is a 5 day plan that starts to execute on sunday nite open and runs for the entire week.

support and resistance rules this game.

they test and retest the price to the " left side of the hard right edge..."

if you cannot see these setups and the cycles going in and out of phase, you are doomed to fail.

digital coded does not cut it.
you can walk it forward and tinker with it to infinity.

back to the lion's den to snack on the next breakout trader.
another loser strategy.

cheers,

s

 

[Joe]

Heads up guys, Rita & JCL are the same person. Also, JCL/Rita you can not advertise on this site, we've been doing this for over 10 years, just repost and do not link to your site. Last warning.