Data mining bias and outlier trade outcomes

[logic_man]

I was reading a review of Aronson's book on evidence-based TA and the reviewer stated that one sign that data mining bias was not present in an optimized backtest would be that there were few outliers in the backtest results.

I take this to mean that your backtest should show that your results are primarily driven by a few really great trades and that the rest of your trades should sort of average out to something near 0.

For those who've looked into this in depth, does this seem like a reasonable takeaway? If so, anyone have any sort of heuristic on what this would look like quantitatively? What if I have a backtest that shows that 10% of the trades generate 50% of the profits? Or 20% generate 80% of the profits? My hunch is that there isn't a hard and fast line here, though.

Overall, the point seems valid because the alternative would seem to be that your backtest showed that nearly every trade was a home run, which seems more unlikely to be a characteristic of a strategy that would work out of sample.

 

[sheepsucker]

Could also be outliars to the negative.
Steady profits and a few big losers.

 

[trade4ever2day]

Aronson sounds too dogmatic to me without any theoretical justification for his dogmas. Trading is all about actual performance. No serious trader cares about "optimized backtested data-mining results". If an optimized backtested system with data-mining bias works and makes money it is good enough for me.

 

[logic_man]

Quote from alexandermerwe:

Aronson sounds too dogmatic to me without any theoretical justification for his dogmas. Trading is all about actual performance. No serious trader cares about "optimized backtested data-mining results". If an optimized backtested system with data-mining bias works and makes money it is good enough for me.

I agree that there is little that can be called valid "theory" in interpreting trading results, partially because there's nothing quite like trading to serve as a model for interpretive methodologies and because the market doesn't follow the laws of nature or physics.

That said, if it happens to be true that many successful traders have methods which fit the profile of "a few big wins, a lot of little wins and losses and a few (but fewer than the big wins) large losses", that enables a trader to judge if the profile of a new method is aligned with that.

Kind of like in baseball scouting, you know that not every guy who's 6ft2 and 220lbs can crush the ball, but the first guys you look at are the guys who fit that profile, not the guys who are 5ft8 and weigh 170, even though you occasionally get a guy with that build who can hit home runs.

 

[logic_man]

Quote from sheepsucker:

Could also be outliars to the negative.
Steady profits and a few big losers.

Right. After I posted that question, I realized that something needed to be said about losers. My guess is that the number of "big" losers, in most or all successful systems, would be smaller than the number of "big" winners and that should be reflected in the back-testing.

 

[trade4ever2day]

Quote from logic_man:

Right. After I posted that question, I realized that something needed to be said about losers. My guess is that the number of "big" losers, in most or all successful systems, would be smaller than the number of "big" winners and that should be reflected in the back-testing.

Sounds like that you (and maybe Aronson) claim that a large payoff ratio (avg. win/avg. loss) is required to have significance. But that is maybe equivalent to saying that only trend-following can be significant. This is empirically false as HFT with uniform win/loss distribution is the most profitable method. Yes, many authors say things that defy empirical facts.

 

[dom993]

Quote from logic_man:

I was reading a review of Aronson's book on evidence-based TA and the reviewer stated that one sign that data mining bias was not present in an optimized backtest would be that there were few outliers in the backtest results.

I take this to mean that your backtest should show that your results are primarily driven by a few really great trades and that the rest of your trades should sort of average out to something near 0.

For those who've looked into this in depth, does this seem like a reasonable takeaway? If so, anyone have any sort of heuristic on what this would look like quantitatively? What if I have a backtest that shows that 10% of the trades generate 50% of the profits? Or 20% generate 80% of the profits? My hunch is that there isn't a hard and fast line here, though.

Overall, the point seems valid because the alternative would seem to be that your backtest showed that nearly every trade was a home run, which seems more unlikely to be a characteristic of a strategy that would work out of sample.

The data-mining bias is relevant to systematic search for an "edge", that is to say systematically trying 1,000's of signals to select the best performing in backtesting.

All the book revolves around techniques to ensure statistical significance of a backtest, specifically by refuting the null hypothesis that the selected system has in fact no predictive power (no edge).

I read the book not too long ago, I do not remember anything discussing the trade distribution profile in itself as indicative of statistical significance or not.

I would argue that the fewer the big wins outliers, the better the system actually is ... because the end-result is less dependent on a few large winners, which might or not be present in the future, and more importantly might be missed in the future for an endless list of reasons.

 

[logic_man]

Quote from alexandermerwe:

Sounds like that you (and maybe Aronson) claim that a large payoff ratio (avg. win/avg. loss) is required to have significance. But that is maybe equivalent to saying that only trend-following can be significant. This is empirically false as HFT with uniform win/loss distribution is the most profitable method. Yes, many authors say things that defy empirical facts.

Fair point. I was not being inclusive in my comment on the type of system profile which would be indicative of a highly successful trader, since a very high winning percentage combined with wins and losses of approximately equal size would also be highly profitable.

 

[logic_man]

Quote from dom993:



I read the book not too long ago, I do not remember anything discussing the trade distribution profile in itself as indicative of statistical significance or not.

I would argue that the fewer the big wins outliers, the better the system actually is ... because the end-result is less dependent on a few large winners, which might or not be present in the future, and more importantly might be missed in the future for an endless list of reasons.

Hmm, maybe the reviewer didn't quite get the point, which is always possible.

Your argument about outliers is interesting to me because one of the things I consistently see (doesn't mean it's correct, obviously) is that if you are doing more of a trend-following strategy, you want to see outliers because that is your strategy catching the long tails in the market's price distribution. Yes, the market sometimes goes into "hibernation" with regards to these long tails (I would say that my ES system is experiencing this at the moment, actually and, as a result, my trade frequency is way down), but they eventually always come back, so while you cannot predict the timing or size of these outliers, you can build a system to take advantage of their inevitable appearance.

 

[dom993]

OK, don't confuse your trend following strategy's runners by design with outliers. What you can do, though, is look at the distribution of your winners, and call the top 5% outliers.

With that done, remove them from your trading results, replace them by average wins, and assess the impact ... that will tell you how much your results are "dependant" on these outliers.