Your Thoughts On Walk-forward Testing To Assess Robustness

[shatteredx]

Fair question. But it’s a technique some people use both for robustness testing and also staying on top of the system so it adapts with current markets. Idea is in real world trading you can keep the core system the same but have a short term optimisation regularly to keep the parameters as close to current market conditions.

Right, it's just a drastic shift in the amount of training data, from 4 years to 1 month.

I don't think it's reasonable to train a strategy on just 1 month of data, even as a test of robustness.

I could be wrong!

 

[tommo]

How can you adopt a strategy to accommodate random? Especially when the random can cause move of such varying intensity?

I understand that to mean you assume all market movements are random?

 

[traider]

Things have changed. It's a bit more difficult to manage trump tweets these days.

And I'm not even joking, it's a big problem for automated strategies these days.

It should affect strategies that are delta netural less. Are your automated strategies all breakouts of some sort

 

[10_bagger]

It’s as useless as backtest

Through the years, i have spent thousands of hours back testing. Every strategy that i trade go through the same process.

1. Theory.
2. Back test and collect as many samples as you can or until you feel that the sample size is statistically relevant.
3. Forward test and trade a small amount relative to normal trading size.
4. If its proven profitable increase trading size based on your overall profits.

So i'm sorry, i have to strongly disagree with this statement. I have a system that i'm trading now that is extremely systematic where there's hardly any thinking involved. If the numbers add up you hold/buy and if it doesn't add up you exit/short. I would be willing to go as far as saying the money that i make is through back testing and not live trading, because in live trading all i'm doing is pushing buttons.

 

[traderslair]

It should affect strategies that are delta netural less. Are your automated strategies all breakouts of some sort

My experience is more in the market making sector so these trump tweets are a bit of a disaster for a some strategies running in fpga code

 

[stepandfetchit]

running in fpga code

Curious with your statement "running in fpga code", can you elaborate?

 

[traderslair]

Curious with your statement "running in fpga code", can you elaborate?

It's a pain to put new code up there. Things that have been working for decades start making losses because of trump tweets. We don't run code on servers, we run it on FPGA

 

[Steve Alexander]

Hi,

Just wondered what your thoughts were on walk forward optimization as a way to test the robustness of a system?

I have done conventional walk-forwards, so for example build a systems based on 4 years of data (sample of around 350 trades) and then test that system on the most recent two years of data. I have systems that work on this basis which is a good sign you have robustness in your strategy. However if I try a walk forward optimization (so for example optimize every 30 days then restest 10 days out of sample) over the same period it destroys the backtest.

Part of me thinks this means the system is not robust, the last 6 weeks or so of optimized settings should have some predictive value on the next 2 weeks.

However, the same system is profitable on a basic 4 year in-sample versus 2 year out of sample test with no parameter changes. Which is a positive sign.

I find it very hard to get any walk forward optimization backtests to produce profitable systems. I don't want to cheat myself by trading non robust systems but at the same time don't want to set an unrealistic bar whereby I throw out good systems.

What are your thoughts?

I think it's a waste of time.

 

[guowei58]

Hi,

Just wondered what your thoughts were on walk forward optimization as a way to test the robustness of a system?

I have done conventional walk-forwards, so for example build a systems based on 4 years of data (sample of around 350 trades) and then test that system on the most recent two years of data. I have systems that work on this basis which is a good sign you have robustness in your strategy. However if I try a walk forward optimization (so for example optimize every 30 days then restest 10 days out of sample) over the same period it destroys the backtest.

Part of me thinks this means the system is not robust, the last 6 weeks or so of optimized settings should have some predictive value on the next 2 weeks.

However, the same system is profitable on a basic 4 year in-sample versus 2 year out of sample test with no parameter changes. Which is a positive sign.

I find it very hard to get any walk forward optimization backtests to produce profitable systems. I don't want to cheat myself by trading non robust systems but at the same time don't want to set an unrealistic bar whereby I throw out good systems.

What are your thoughts?

There are too few trades for the backtest and optimization. Typically you need 2k-3k of data points for a backtest. if you don't have this many trades, then trade more instruments using the same rules, or break down your trades into daily P&L to increase sample size.

the optimization needs a lot more trades too.