Walk-Forward Testing and Optimization
- Thread starter trader3cnd
- Start date
I'm more known as Bob & Alice...
This is of course very valid. I just have a different opinion about locally and globally fit models, and would even go so far to dispute that a globally fit model really exists. A WFO cycle normally uses about 2 years for the IS period, and a fixed parameter strategy might use 4 years. So I fail to see why 2 years constitute a local fit and 4 years don't. But this can be debated.
Of course global and local has only relative meaning: price or energy or voltage or any coordinate system only has relative meaning in the world. The point I am debating is whether there is a real signal that can be modeled adequately and this pertains to the issue of stationarity. If the markets are not stationary, but there is an autocorrelation, then perhaps that means you can model it approximately assuming stationarity only within a small timeframe. But that model will only work near that point where you trained. It is the same as perturbation theory or truncated Taylor series expansions about a point. So this argues for your case, i.e., that it might make sense to use a successive local fits (like a piece-wise linear approximation to a curve).
If you optimize a system just before a choppy market turns into a trend, then you miss the trend. Trading system performance is path dependent. Only global optimization can take care of that to some extend. Local optimization can impose a large penalty because of frequent switching points. The fact of the matter is that any type of optimization is bad in trading system development and whoever presents WFO as a solution has a commercial reason for doing that and also probably purposely deceiving people.
I have to say this is a bit of an overstatement and I disagree. For every problem there is an "optimal amount of optimization". The point at which optimization starts to hurt you is when you add more parameters (or complexity) than there are effective degrees of freedom in the data. Alternatively this can be measured by the amount of information that the added model complexity explains. There are various measures for it, such as the partial F, the BIC and the AIC.
All traded strategies are optimized. As soon as you manually select some algorithms or parameters, you perform optimization, sometimes to a high degree. For instance, the turtle strategy was highly optimized. The only difference between computer and manual optimization is that with a computer, you can control and measure the amount of optimization.
Do you think that the following startegy is optimized?
If close of today > high of yesterday then buy at the close
I agree but since it is very difficult to measure DoF and added complexity of non-stationary systems, optimization must be avoided at all costs. This is my point. Mathematically you are correct (as always) but in practice it hardly works.
Edit: Plus I hope you understand that "optimal amount of optimization" leads to an infinite regression and thus it cannot be resolved to a sound deductive procedure.
I assume that this is not a traded strategy, otherwise I had to fear for your financial future.
But if it were, you had probably run backtests and selected this strategy from a pool of other variants that compare the open, low, close, or average of yesterday, or the day before yesterday, and so on. Selecting the best strategy is already an optimization process. When you select the best asset for that strategy, it's another optimization. This causes data mining bias and makes it likely that the strategy fails in life trading.
You can use WFO even for this primitive strategy. In this case WFO would repeat the strategy and asset selection process in every cycle. If then always the above strategy would come out as a result, and would perform well in the following OOS test, then you could with some confidence trade it. I hope this illustrates the method.
Human behavior in groups can only be calculated so far. This may be the source of your 20 excursion into what doesn't work.
I sense that you are looking for certainty, however you must realize the deal you strike with the market when you provide your services....... you take on risk in exchange for potential reward.
It's a business. You never know if you're going to make it until you do. But people who make it have certain thingz in common......
