Data mining bias and outlier trade outcomes

[abattia]

Quote from logic_man:
I was reading a review of Aronson's book on evidence-based TA and the reviewer stated that one sign that data mining bias was not present in an optimized backtest would be that there were few outliers in the backtest results...

I would understand this as follows:
For the good, “non curve-fit” system, good performance results from uncovering some underlying truth relating to market microstructure.
The performance of a curve-fit system on the other hand only appears good because of a serendipitous synchronization between entries/exits and price action.
The key point here about a curve-fit system is that the outcome is random, and just happens to look like good performance. Any such random outcome will be the result of multiple trades. What permutation of multiple random trades (that appear to reflect “good performance”) is more likely? (a) A series of multiple random trades where all are good trades? Or (b) a series of multiple random trades where most are distributed equally about small wins and losses but a few “outliers” substantially lift the overall result?
IMO (b) is more likely than (a) to be the profile of a curve-fit system.

 

[trade4ever2day]

Quote from abattia:

IMO (b) is more likely than (a) to be the profile of a curve-fit system.

IMO neither is an indication of a random system. Both can be random and both can have significance.

The key here abattia is that nobody knows exactly the link between "curve-fit" and "random systems" when it comes to the markets. Only OOS testing can provide a partial answer and actual trading provides the ultimate check.

 

[logic_man]

Quote from abattia:

I would understand this as follows:
For the good, “non curve-fit” system, good performance results from uncovering some underlying truth relating to market microstructure.
The performance of a curve-fit system on the other hand only appears good because of a serendipitous synchronization between entries/exits and price action.
The key point here about a curve-fit system is that the outcome is random, and just happens to look like good performance. Any such random outcome will be the result of multiple trades. What permutation of multiple random trades (that appear to reflect “good performance”) is more likely? (a) A series of multiple random trades where all are good trades? Or (b) a series of multiple random trades where most are distributed equally about small wins and losses but a few “outliers” substantially lift the overall result?
IMO (b) is more likely than (a) to be the profile of a curve-fit system.

In the case of (a), how would you define "good" trades? If I were defining it, I would define it not just as winning trades of various sizes, but as losing trades in which you "lost well". What that would tell me is that the system had an objective way of "cutting your losers" in a way that minimized your opportunity cost of the market then reversing in your favor. I say that because I take it for granted (as you probably do as well) that no system is 100% accurate.

But, I can still see (b) as the profile of a trend following system which generates its profits in accordance with the 80/20 rule, i.e. 80% of profits from 20% of trades. The question would then be how much deviation from that 80/20 rule is allowable. Could 5% of trades generating 75% of profits still be a non-curve fit system?

It appears there is some level of unavoidable subjectivity in determining the conclusions of any trade outcome analysis.

 

[logic_man]

Quote from alexandermerwe:

IMO neither is an indication of a random system. Both can be random and both can have significance.

The key here abattia is that nobody knows exactly the link between "curve-fit" and "random systems" when it comes to the markets. Only OOS testing can provide a partial answer and actual trading provides the ultimate check.

I agree that the problem of what can you actually know about the markets is central to any kind of attempt to assess your risk of failure once you go live or once you continue to trade a system which is in a drawdown mode.

Still, it seems to be the prudent thing to do to explore all avenues of measurement of these risks prior to deploying capital. Someone told me once that airline flight plans begin to be modified as soon as the plane's wheels leave the runway and the crew has to react to actual, not hypothetical, conditions, but that still doesn't stop the airlines for generating a plan for the entire flight before anyone even gets on the plane.

 

[Trading Blox]

For my two cents, I agree that there is a subjective nature to assessing a trading system. But the robustness of a system is also measurable, and a rigorous application of robustness tests to a trading system can go a long way toward identifying overly curve-fit trading systems....Trading Blox is founded on the idea that traders can collect, measure, analyze and learn from back-tested results to avoid the pitfalls of curve-fitting...