Have you actually tested this out for yourself? Or are just putting forward a hypothesis?
Is Walk-Forward (out of sample) testing simply an illusion?
- Thread starter pursuit
- Start date
Have you actually tested this out for yourself? Or are just putting forward a hypothesis?
Not an illusion at all if done properly with close to zero degrees of freedom.
First, before doing any WFA, a trader needs to understand why, and how their model(s) gives them any kind of competitive advantage in the marketplace. Also make sure they're executable in real-time before moving into the analysis phase. Assuming doesn't cut it. Real money needs to be put on the line.
Second, unless i'm marketing to investors, I don't give a shit about Sharpe, Sortino, Treynor and whatever metrics so called quants use, and I don't need R, MatLab, etc... to do effective WFA. A custom built Excel sheet works just fine. As a lone wolf, I can only keep track of a limited amount of variations/models.
First, before doing any WFA, a trader needs to understand why, and how their model(s) gives them any kind of competitive advantage in the marketplace. Also make sure they're executable in real-time before moving into the analysis phase. Assuming doesn't cut it. Real money needs to be put on the line.
Second, unless i'm marketing to investors, I don't give a shit about Sharpe, Sortino, Treynor and whatever metrics so called quants use, and I don't need R, MatLab, etc... to do effective WFA. A custom built Excel sheet works just fine. As a lone wolf, I can only keep track of a limited amount of variations/models.
What you've described is not walk forward analysis/optimization.
True WFA is robust. I know I'll have to explain/simplify...so I'll just get to it:
1. Optimize over days 1-50.
2. See how it worked on OOS (Out of Sample day 51. It must be out of sample because day 51 didn't exist during the analysis/optimization of days 1-50).
3. Day 51 closes.
4. Optimize over days 2-51.
5. See how it worked on OOS day 52.
6. Repeat.
What you did say, and I agree with, is that optimizing by holding data out for *validation* is, many times, not much different that optimizing over *all* the data.
Let me simplify further:
WFA *tests* on *true* OOS data. In the above example, our hero optimizes *after* the close of the dow/nasdaq/etc. markets (4 pm ET)...but before the next market open.
Then, the next day--after the next market close, our hero sees how well his forecast did.
Repeat.
It doesn't get more robust than proper WFA.
There was a very nice study done by someone at DB (I think) that shows how your "percieved" Sharpe grows over a number of optimization passes. It looked pretty scary, IMHO.
My approach is "hypothesis -> study -> first pass strategy -> live trading in small size -> improvement based on real results". In most cases, the causes for failure are aspects that can not be accounted for in paper trading (fills, borrow, information delays).
Also, I am almost never ready to deploy a strategy unless I have a solid fundamental hypothesis regarding the source of alpha. There are no free lunches, only cheap lunches or stolen lunches. If it's the latter, I'd like to know what I am paying and if it's the former, I'd like to know who I am stealing it from.
As you correctly my point is about traditional optimization/backtesting not the "WFA". As you describe WFA is basically "rolling window" optimization.
Yes...also known as "sliding window" optimization.
Well... then it's SOLVED. All we need is to use "sliding window optimization" and our system will always work live and will work forever. /s
You're welcome /s
How have your systems tested with WFA fared live? Was performance in line with testing? If so, for how long?
Note: I can't be manipulated into revealing information I wouldn't normally reveal. I'm a pretty good Texas Hold'em player.
This is your thread. I haven't started a thread revealing my data. If you want me to compile data for you, I can quote you a price.