Walk-Forward Testing and Optimization

trader3cnd · Mar 7, 2012

Does it make any sense? Has anyone used it with success or is it just another buzzword of the software industry?

IMO an optimized system - is an optimized system, whether that is done stepwise or not. More importantly, I could think of some cases where walk forward optimization would produce worse results than optimizing over the whole sample.

jcl · Mar 7, 2012

The answer of your questions is yes. Walk forward optimization makes sense, is necessary for successful strategy development, and produces worse test results.

When you develop and optimize a strategy, you don't want good test results. You want realistic results that reproduce real trading. Backtests can't do this. This can only be achieved with walk forward tests.

intradaybill · Mar 8, 2012

Quote from jcl:

The answer of your questions is yes. Walk forward optimization makes sense, is necessary for successful strategy development, and produces worse test results.

Produces worse test results? I'm not sure what you want to say.

Regardless, nobody is going to take your word for it. Do you have some examples or you're talking under the influence?

jcl · Mar 8, 2012

Quote from intradaybill:

Produces worse test results? I'm not sure what you want to say.

Regardless, nobody is going to take your word for it. Do you have some examples or you're talking under the influence?

I'm always talking under the influence of the mind

.

I have plenty examples, as I've meanwhile used WFO for about 25 strategies since last October. Here's one example, with a short WFO tutorial:

Link Deleted Spam

There are two reasons why a walk forward test often gives you worse test results than a backtest or a parallel out-of-sample test. First, it eliminates most overfitting effects, and second, it tests in a slightly different market situation than you had when optimizing. Both effects make your tests more realistic, which usually means a worse test result.

goodgoing · Mar 8, 2012

Quote from jcl:

Both effects make your tests more realistic, which usually means a worse test result.

If "realistic" means worse, would that make "reality" mean disaster? Because in reality there are unknown effects that make things even worse.

Why do you think optimizing during a choppy market period and then trading during a trending market is any good? It is a bad idea. WFO is a bad idea in general. A parameter optimization session should include a wide variety of market conditions. Like most things in trading, some guy once mentioned the method in his book and some other guys continue talking about it. Most of those who use it in actual trading fail. This is the truth.

There is an interesting ongoing discussion in this forum:

http://groups.google.com/group/adaptrade-builder/t/3f6e7dcbf68c5ea2?hl=en

Some people who actually work with systems claim OOS testing is useless.

jcl · Mar 8, 2012

Quote from goodgoing:

If "realistic" means worse, would that make "reality" mean disaster? Because in reality there are unknown effects that make things even worse.

Why do you think optimizing during a choppy market period and then trading during a trending market is any good? It is a bad idea. WFO is a bad idea in general. A parameter optimization session should include a wide variety of market conditions. Like most things in trading, some guy once mentioned the method in his book and some other guys continue talking about it. Most of those who use it in actual trading fail. This is the truth.

There is an interesting ongoing discussion in this forum:

http://groups.google.com/group/adaptrade-builder/t/3f6e7dcbf68c5ea2?hl=en

Some people who actually work with systems claim OOS testing is useless.

Ok, there are some valid arguments but also some misconceptions.

The effects that amount to a difference between testing and trading are well known and based on relatively simple math. WFO is one method to deal with those effects. There are several other methods, but they are more complicated or less effective.

Some years ago, algo developers believed that strategy parameters should be optimized for a long time span and cover as many different market conditions as possible. Then the strategy was considered stable. As it turned out, it was not. Strategies optimized this way tended to become unprofitable in real trading after a remarkably short time.

No matter how many market conditions you want to cover with your parameters, you'll always encounter a completely new market situation in real trading after a while.

You can do a little experiment: Take one of your profitable strategies, and optimize and test it in the traditional way with a simulation period of 4 years, 6 years, and 8 years. You will almost always find that the 4 years test is the best, 6 years is worse, and 8 years is the worst - no matter in which year you start the simulation. This should answer the question about the usefulness of WFO.

intradaybill · Mar 8, 2012

Quote from jcl:

You can do a little experiment: Take one of your profitable strategies, and optimize and test it in the traditional way with a simulation period of 4 years, 6 years, and 8 years. You will almost always find that the 4 years test is the best, 6 years is worse, and 8 years is the worst - no matter in which year you start the simulation. This should answer the question about the usefulness of WFO.

Obviously, that happens because the less the data the better the fit. In other words, walk forward is a series of best fits. If the OOS segments are small in relation to IS, you get a very fitted startegy.

Try to do a walk forward with IS equal to OOS and come back. Are you a newbie? You sound like someone who has started less than 6 months ago and thinks he knows everything.

After 15 years you'll realize you know nothing. You will read your messages here and feel some kind of an emptyness and embarashment.

Rita · Mar 8, 2012

Quote from jcl:

Here's one example, with a short WFO tutorial:

http://zorro-trader.com/en/tutorial_fisher.htm

There are two reasons why a walk forward test often gives you worse test results than a backtest or a parallel out-of-sample test. First, it eliminates most overfitting effects, and second, it tests in a slightly different market situation than you had when optimizing. Both effects make your tests more realistic, which usually means a worse test result.

In the example that you posted, you're getting a better and not a worse test result with WFO. Is this an exception?

Second question, which software do your recommend for WFO? Is there free software that can do that?

Wide Tailz · Mar 8, 2012

For one who keeps bringing up God in response to Mike's hounding, you seemed to have forgotten one Eternal Truth:

"Judge not, lest ye be judged"

intradaybill · Mar 8, 2012

Quote from Wide Tailz:

For one who keeps bringing up God in response to Mike's hounding, you seemed to have forgotten one Eternal Truth:

"Judge not, lest ye be judged"

Yes, remember that because your are the one who judges all the time. Also top playing other's advocate. You sound like a high school dropout.