Optimization and Walk-Forward

[birzos]

Am absolutely convinced people who say backtesting have never actually built a strategy and run it live. Signals change interbar, meaning with backtesting you only see the close, if it's a 60m bar you do not see the signals during that period. You can also bet that your strategies profitability will go down, not up, when live.

If your strategy is bar close then that's different but you still have slippage and spread issues. You also have to take in to account if you have one trade open another should not kick in, which means that it's all about when you switch it on, so need to build in rules to reset for example closing trades for the weekend, for the month. The theory is great, but the last 5% is what really makes the money and that is where all the problems occur which backtesting cannot take in to account.

 

[Simples]

Am absolutely convinced people who say backtesting have never actually built a strategy and run it live. Signals change interbar, meaning with backtesting you only see the close, if it's a 60m bar you do not see the signals during that period. You can also bet that your strategies profitability will go down, not up, when live.

If your strategy is bar close then that's different but you still have slippage and spread issues. You also have to take in to account if you have one trade open another should not kick in, which means that it's all about when you switch it on, so need to build in rules to reset for example closing trades for the weekend, for the month. The theory is great, but the last 5% is what really makes the money and that is where all the problems occur which backtesting cannot take in to account.

If you actually try backtesting, you'll see 4 price levels during each bar: Open High Low Close (OHLC) as well as Volume. In theory, there's no chance of price going beyond these levels for that bar.

No need to just test for Close. What may happen in practice though, is that the exchange/data feed doesn't provide the real ranges during the day in order to cover up their own screw-ups, but that should be rather rare depending on the market.

If you're interested in what happens in the time between Open and Close, you're simply analysing on too high a timeframe for that.

Having a trade open is in practice no problem as you no longer have funds to open new positions.

 

[KMeriwetherD]

Am absolutely convinced people who say backtesting have never actually built a strategy and run it live. Signals change interbar, meaning with backtesting you only see the close, if it's a 60m bar you do not see the signals during that period. You can also bet that your strategies profitability will go down, not up, when live.

If your strategy is bar close then that's different but you still have slippage and spread issues. You also have to take in to account if you have one trade open another should not kick in, which means that it's all about when you switch it on, so need to build in rules to reset for example closing trades for the weekend, for the month. The theory is great, but the last 5% is what really makes the money and that is where all the problems occur which backtesting cannot take in to account.

You can write back testing code to account for all kinds of things. My partner and I would factor in closing trades on Friday, closing trades before contract roll, or even block out periods where you would only follow the system during certain hours. Yes, there will be uncertainty inside the candles. To counter this we would trade the system the same way we tested it. For example, for testing we would trade the current candle based the indicator signal for the previous candle. This eliminates hindsight bias since in back testing you would not know how the signal would play out over the course of the current candle but you would know what happened in the previous candle. Then in real time we would trade it the same way.... using the signal for the previous candle on real time price. (These were 1 hour candles)
That being said, slippage and commissions are a BIGGIE. This is especially true for intraday systems that take 3 or more trades a day.

 

[PetaDollar]

The search for the right backtesting-walk forward procedure is just the Search for the Holy Grail cast in a different way (the usual way being the trading rules). What should you do? Learn to gamble. Learn to capture a specific market behavior and apply your system when and where there's a decent chance it will work.

 

[supracharger]

Andrew, is this part of the walk-forward testing processes? Thanks, Andrew.

In-Sample and Out-of-Sample are two terms that one needs to understand to at least be able to find profitable systems that actually work live. However, the terms are not solely connected to walk-forward Optimization. Additionally, Walk-Forward Optimization is supposed to minimize Curve-Fitting, but I was told by a Good System Trader that Walk-Forward Opt. causes more Curve-Fitting to happen than just a normal Opt. because it uses less data to Opt., even though it walks forward with that small data set.

 

[Sergio77]

Am absolutely convinced people who say backtesting have never actually built a strategy and run it live. Signals change interbar, meaning with backtesting you only see the close,

I am convinced you comment about backtesting without knowing the subject.

You can backtest with tick data if you like but even with daily you can properly account for all OHLCVB and any fundamental factors you like.