Confirmation bias or valid procedure?

[Danielbanker]

Limited backtest-data is one problem I have. That’s right. My H1-data feed only goes until 2014. But that’s not what this is here about.

I'm curious if a backtested strategy is valid if it works only in some specific markets and in some timeframes.

Trying to understand why the results are like they are is a good hint. I think there is a pattern in which markets the strategy works and in which not.

Im already in a low-risk forward Test since beginning of November. I have 75% successful trades and a profit-risk-ratio of 1.45 at this moment. But I only did 8 trades.

Backtest results are good, nearly my strategy was tested in various backtests by other traders. And it fits perfect to my trading possibilities. I give it a try.

Get successful or die trying

Thanks for all the answers.

 

[globalarbtrader]

Hey everybody,

I have a (in my view) good strategy. Some combination of CCI, oversold and mean reversion. My backtest results give back some good results. >60% profitable hit rate, > 1.5 profit factor. Long story short: I‘m pleased in the first step.

I have a question to all the backtest experts. My strategy works in some markets very well (e.g. GBPNZD or EURJPY or NDX) in some absolutely worse (e.g. EURUSD, GOLD, ES).

>Is it a valid procedure to pick that instruments with a given strategy, that works in the backtests?

And in my backtests the strategy works perfectly in the EuroStoxx50 Future H1, but not in M30. In YM perfect in M30, but not that good in M15. In german DAX perfect in M5, good in M15 and M30 but bad in H1. In WTI good in M30 and H1 but bad in M15.

>Is it a valid procedure to pick exactly that time period in which the strategy works best?

Or is picking the best working instruments and the best working time periods in this instruments some kind of confirmation bias?

Am I on the wrong way?

Don't do this.

A) You are using information you didn't have at the start of the backtest. Not sure if confirmation bias is the correct term, but it's definitely bad (I call this implicit fitting).

B) It's unlikely the differential in performance is statistically significant or meaningful.

Use all your instruments and allocate risk in a backward looking fashion over the backtest using a robust methodology


GAT

 

[tommcginnis]

Some combination of CCI, oversold and mean reversion.

Oops. Forgot to mention: this are *all* "mean reversion" concepts -- there is no independent reinforcement from using the three. Which puts a very big hole in things..... Sorry about my miss.

 

[guru]

Limited backtest-data is one problem I have. That’s right. My H1-data feed only goes until 2014. But that’s not what this is here about.

I'm curious if a backtested strategy is valid if it works only in some specific markets and in some timeframes.

Trying to understand why the results are like they are is a good hint. I think there is a pattern in which markets the strategy works and in which not.

Im already in a low-risk forward Test since beginning of November. I have 75% successful trades and a profit-risk-ratio of 1.45 at this moment. But I only did 8 trades.

Backtest results are good, nearly my strategy was tested in various backtests by other traders. And it fits perfect to my trading possibilities. I give it a try.

Get successful or die trying

Thanks for all the answers.

IMHO both answers (yes & no) are valid, because some strategies may work on a group of instruments and others may not. It would be great to use a strategy that works on many instruments but this may lower your chances or invalidate the strategy and then you’d have to either find a new strategy or rely on one that’s fit to specific instrument. Each approach will have drawbacks and benefits.
One of algo traders and book writers, Ernie Chan, stated that using separate strategies for different instruments worked best for him.
His blog may also be useful to you, for example where he discusses an H1 example specific to AUDCAD, but not all Forex pairs, and where at the same time he describes how to deal with and optimize strategies based on smaller data sets:
http://epchan.blogspot.com/2017/11/optimizing-trading-strategies-without.html?m=1

 

[Danielbanker]

IMHO both answers (yes & no) are valid, because some strategies may work on a group of instruments and others may not. It would be great to use a strategy that works on many instruments but this may lower your chances or invalidate the strategy and then you’d have to either find a new strategy or rely on one that’s fit to specific instrument. Each approach will have drawbacks and benefits.
One of algo traders and book writers, Ernie Chan, stated that using separate strategies for different instruments worked best for him.
His blog may also be useful to you, for example where he discusses an H1 example specific to AUDCAD, but not all Forex pairs, and where at the same time he describes how to deal with and optimize strategies based on smaller data sets:
http://epchan.blogspot.com/2017/11/optimizing-trading-strategies-without.html?m=1

Thank you! I appreciate!

 

[Danielbanker]

Oops. Forgot to mention: this are *all* "mean reversion" concepts -- there is no independent reinforcement from using the three. Which puts a very big hole in things..... Sorry about my miss.

Yep. All in all it’s a simple mean reversion strategy. No rocket science.

 

[guru]

Limited backtest-data is one problem I have. That’s right. My H1-data feed only goes until 2014. But that’s not what this is here about.

I'm curious if a backtested strategy is valid if it works only in some specific markets and in some timeframes.

Trying to understand why the results are like they are is a good hint. I think there is a pattern in which markets the strategy works and in which not.

Im already in a low-risk forward Test since beginning of November. I have 75% successful trades and a profit-risk-ratio of 1.45 at this moment. But I only did 8 trades.

Backtest results are good, nearly my strategy was tested in various backtests by other traders. And it fits perfect to my trading possibilities. I give it a try.

Get successful or die trying

Thanks for all the answers.

Actually I just remembered when/where specifically Ernie Chan talked about using single strategy vs different strategies for different instruments (his primary focus is Forex but here he uses futures as examples):

 

[nooby_mcnoob]

IMHO both answers (yes & no) are valid, because some strategies may work on a group of instruments and others may not. It would be great to use a strategy that works on many instruments but this may lower your chances or invalidate the strategy and then you’d have to either find a new strategy or rely on one that’s fit to specific instrument. Each approach will have drawbacks and benefits.
One of algo traders and book writers, Ernie Chan, stated that using separate strategies for different instruments worked best for him.
His blog may also be useful to you, for example where he discusses an H1 example specific to AUDCAD, but not all Forex pairs, and where at the same time he describes how to deal with and optimize strategies based on smaller data sets:
http://epchan.blogspot.com/2017/11/optimizing-trading-strategies-without.html?m=1

I disagree. Treat the algorithm as a blackbox. Here is the long blackbox: EMA(15) > EMA(30)

How many instruments will "pass" this in backtests? This is why I personally would want to evaluate why it is working on a certain set of instruments because then you would need another filter to apply to filter those out from the universe.

 

[guru]

I disagree. Treat the algorithm as a blackbox. Here is the long blackbox: EMA(15) > EMA(30)

How many instruments will "pass" this in backtests? This is why I personally would want to evaluate why it is working on a certain set of instruments because then you would need another filter to apply to filter those out from the universe.

A black box shouldn’t matter. For example a strategy (whether black box or not) may detect corporate buy-back pattern that applies to one company but not another.
But stocks are different and difficult, so I do trade vast number of stocks using few strategies. Though at the same time I have dozens of strategies per each stock/company, and may still add some of them to strategy pool.

While with futures and forex there may be different trading patterns, for example depending on the weather, airlines buying crude, others hedging against gas, specific banks/countries buying currencies, etc.
I’d expect those patterns to change over time, but nevertheless I get different results using different strategies.
And I hate to refer to just one guy, but Ernie stated that results speak for themselves, even though he and everyone else assumes that using single strategy across multiple instruments sounds more logical.
If your results show something different then this confirms that different strategies may work differently, so no one may even agree on what works best...

 

[panzerman]

...If you can trade profitably...

That is a mighty big IF. Most people never get to the point of trading profitably. Instead of spending years bouncing from one trading idea to another and losing lots of money looking for profitability, why not apply a bit of science to the trading hypothesis first? Conduct proper backtesting methodology first and see if results in positive expectancy before risking your bankroll.

Why not do this? Because proper backtesting to avoid biases is hard and most people don't know how to do it.