Quote from LeeD:
Harris does backtest. However, apporach he advocates is to avoid manually picking patterns for a trading system and instead automate search for best patterns.
Because traditional trading system qualities such as percentage of winners, profit factor, Sharpe ratio and many others don't say anything about how likely it is that a trading system actually works as opposed to be purely reesult of overfitting.
Using Harris's approach it is all too easy to create a trading system which relies on a large number of patterns. The system may have great performance statistics in the backtest. However, each pattern may appear in the whole data only a couple of times, which is absolutely insufficient to determine if the pattern works. Such a system has no potential to perform in real world but on paper it will look great.
Finding a system that has good backtest results is effectively a datamining exercise. So, distingushing between good results and bad results is far from trivial. Read more here.
Ah, thank you for that concise cogent summary. I will offer a wholly personal view on that approach. I have found in my own practice that the statistical validity of a tested hypothesis works better in live testing if you do the following:
form the hypothesis based on apparent repeated observation of price action, like "price tends to break out of the clean side of a consolidation rather than the raggedy side" (not a valid hypothesis, BTW)
make sure that the hypothesis has some valid explanation in market principles, like "the big boys are holding the line until they have discouraged the weak hands"
run an initial test based on eyeballed estimates of likely loss and profit stops, and proceed if this has positive expectancy
optimize the stop loss with no profit target, and proceed to profit optimization if the best stop loss is weakly profitable or weakly unprofitable
test far enough back in time to get supposedly statistically significant sampling, but not so far back that volatility variations confuse the issue
do rolling tests of the same time duration as time proceeds and estimate the stability of the hypothesis as volatility changes.
I believe in doing this going forward, not backward in time, as a paradigm shift may have occurred in the past, and you want to be au courant.
P.S.: Thanks for that link. And I forgot to add that one requirement in hypothesis formation is that it helps sometimes if the hypothesis is ridiculously absurd.
