Confirmation bias or valid procedure?

[Danielbanker]

Hey everybody,

I have a (in my view) good strategy. Some combination of CCI, oversold and mean reversion. My backtest results give back some good results. >60% profitable hit rate, > 1.5 profit factor. Long story short: I‘m pleased in the first step.

I have a question to all the backtest experts. My strategy works in some markets very well (e.g. GBPNZD or EURJPY or NDX) in some absolutely worse (e.g. EURUSD, GOLD, ES).

>Is it a valid procedure to pick that instruments with a given strategy, that works in the backtests?

And in my backtests the strategy works perfectly in the EuroStoxx50 Future H1, but not in M30. In YM perfect in M30, but not that good in M15. In german DAX perfect in M5, good in M15 and M30 but bad in H1. In WTI good in M30 and H1 but bad in M15.

>Is it a valid procedure to pick exactly that time period in which the strategy works best?

Or is picking the best working instruments and the best working time periods in this instruments some kind of confirmation bias?

Am I on the wrong way?

 

[nooby_mcnoob]

My humble opinion is that you would be overfitting by being selective and thus doing yourself a disservice. You need to understand why this happens first.

 

[gaussian]

Hey everybody,

I have a (in my view) good strategy. Some combination of CCI, oversold and mean reversion. My backtest results give back some good results. >60% profitable hit rate, > 1.5 profit factor. Long story short: I‘m pleased in the first step.

I have a question to all the backtest experts. My strategy works in some markets very well (e.g. GBPNZD or EURJPY or NDX) in some absolutely worse (e.g. EURUSD, GOLD, ES).

>Is it a valid procedure to pick that instruments with a given strategy, that works in the backtests?

And in my backtests the strategy works perfectly in the EuroStoxx50 Future H1, but not in M30. In YM perfect in M30, but not that good in M15. In german DAX perfect in M5, good in M15 and M30 but bad in H1. In WTI good in M30 and H1 but bad in M15.

>Is it a valid procedure to pick exactly that time period in which the strategy works best?

Or is picking the best working instruments and the best working time periods in this instruments some kind of confirmation bias?

Am I on the wrong way?

If your backtesting procedure does not include 2007-2009 you are just riding a rocket ship up and probably overfitting.

 

[tommcginnis]

"Confirmation Bias" -- an attribution of truth or veracity to a hypothesis when chosen outcomes match it -- is not appropriate (IMO) to the use of financial forecasts. Why? Because your first purpose is not *science* -- the finding of TRUTH -- but profit.

You need to understand why this happens first.

And so, while MY HEART agrees 100% with Noobs, my brain observes quietly, "It doesn't matter. If you can trade profitably, and have stops in place should the {Black/Grey Box} magic stop, then your responsibility is not to find TRUTH but to exploit profitable exchange."

Thus, if you can REMIND yourself, "I don't know why this works -- and I do not want to trade it blindly for the next 5-10 years" it would remain your responsibility to milk those profitable outcomes from the market for every trade you can manage. Period.

Take a simple trend-following algo, for example. We have the classic questions:
• what is a trend? {I define it. I decide.}
• when does it begin? {See above.}
• when does it end? {See above.}
• if I don't know when it ends or begins, isn't the operative rule, "Cut your losses short, let your winners run" ??? {You bet your bippy it is.}

Nowhere in there did you hear anything about "No! You must define it for eternity! Absolute terms! Inviolable standards!!" Even the term "over-fitting" gives little creed to the fact that {something} was fit in the first place. Hello!

 

[nooby_mcnoob]

"Confirmation Bias" -- an attribution of truth or veracity to a hypothesis when chosen outcomes match -- is not appropriate (IMO) to the use of financial forecasts. Why? Because your first purpose is not *science* -- the finding of TRUTH -- but profit.

And so, while MY HEART agrees 100% with Noobs, my brain observes quietly, "It doesn't matter. If you can trade profitably, and have stops in place should the {Black/Grey Box} magic stop, then your responsibility is not to find TRUTH but to exploit profitable exchange.

Thus, if you can REMIND yourself, "I don't know why this works -- and I do not want to trade it blindly for the next 5-10 years" it would remain your responsibility to milk those profitable outcomes from the market for every trade you can manage. Period.

Fair enough.

 

[pipeguy]

Hey everybody,

I have a (in my view) good strategy. Some combination of CCI, oversold and mean reversion. My backtest results give back some good results. >60% profitable hit rate, > 1.5 profit factor. Long story short: I‘m pleased in the first step.

I have a question to all the backtest experts. My strategy works in some markets very well (e.g. GBPNZD or EURJPY or NDX) in some absolutely worse (e.g. EURUSD, GOLD, ES).

>Is it a valid procedure to pick that instruments with a given strategy, that works in the backtests?

And in my backtests the strategy works perfectly in the EuroStoxx50 Future H1, but not in M30. In YM perfect in M30, but not that good in M15. In german DAX perfect in M5, good in M15 and M30 but bad in H1. In WTI good in M30 and H1 but bad in M15.

>Is it a valid procedure to pick exactly that time period in which the strategy works best?

Or is picking the best working instruments and the best working time periods in this instruments some kind of confirmation bias?

Am I on the wrong way?

So you suspect this to be confirmation bias because at first you believe that your EA works perfectly, then test it on markets see positive results and argue then these positive results goes as a confirmation to your assumption. Am I right?

If yes, then there is a room for doubts only if you assume that a good trading system should be able to work on any markets. Proving that you can pick holes in your strategy then.

 

[fan27]

If your backtesting procedure does not include 2007-2009 you are just riding a rocket ship up and probably overfitting.

Agreed. @Danielbanker, what is the backtest date range?

 

[minmike]

I don't undersatnd the H1 M30 designations, but I believe they are time designations?

Some strategies work better at some times than others. Usually for identifiable reasons. Perfectly reasonable to only trade most profitable times.

Seems to work better in less liquid markets than more liquid. Probably means you are assuming you can execute on stale quotes.

I'm guessing you are believing you can trade on stale quotes. Last trade outside current bid ask. Is you average trade bigger than the bid/ask of the particular contract?

 

[Overnight]

I don't undersatnd the H1 M30 designations, but I believe they are time designations?...

H1 = one-hour bar charts. M30 = 30-minute bar charts.

 

[smallfil]

Hey everybody,

I have a (in my view) good strategy. Some combination of CCI, oversold and mean reversion. My backtest results give back some good results. >60% profitable hit rate, > 1.5 profit factor. Long story short: I‘m pleased in the first step.

I have a question to all the backtest experts. My strategy works in some markets very well (e.g. GBPNZD or EURJPY or NDX) in some absolutely worse (e.g. EURUSD, GOLD, ES).

>Is it a valid procedure to pick that instruments with a given strategy, that works in the backtests?

And in my backtests the strategy works perfectly in the EuroStoxx50 Future H1, but not in M30. In YM perfect in M30, but not that good in M15. In german DAX perfect in M5, good in M15 and M30 but bad in H1. In WTI good in M30 and H1 but bad in M15.

>Is it a valid procedure to pick exactly that time period in which the strategy works best?

Or is picking the best working instruments and the best working time periods in this instruments some kind of confirmation bias?

Am I on the wrong way?

A method that is valid will work whatever you trade. The Turtle Trading System was used to trade forex, stocks, futures, etc. All showed profitable results. Not to mean you will not have losing trades, just that overall, after all your trades are added up, it shows a profit most years. I had designed a trading system that was hitting 80% winners and substantial gains overall. Of course, my backtest was limited because I was doing it manually which was very tedious. Still, I figured it might be worth testing so, placed actual trades to test it. It was a huge failure. Most of the trades were losers and worst, slippage was even worst! I did not count on the slippage having that huge effect but, it did! Your high win percentage of 60% probably, will not hold up. I highly, suspect it will not hold up for long. Your backtest already show that! Your edge is what matters, not your winning percent. Winning percent means nothing. The average size of your winning trades compared to the average size of the losing trades matter more!