I've spent a lot of time thinking about using various automated methods to discover trading patterns/strategies and I'm coming to the conclusion that it's mathematically impossible.
When you apply some automated discovery mechanism you either have no preconceived notion or only a very general notion for what you are looking for. You then use some search technique to look at huge number of possible relationships and indicator/price permutations. If you are using a dumb random search approach you can easily search millions of possibilities. If you use a smart directed search algorithm (ie genetic algorithm), you can search the equivalent of billions or tens of billions of permutations.
When you look at so many possibilities, you are guaranteed to find quite a few methods that work incredibly well simply by chance. Even worse, some of these methods will pass any statistical test you throw at it (sharpe ratio, t-test, edge test, correlation test, etc.) since all these test compare the observed results vs what you'd expect randomly. But when you look at a 1,000,000 random samples, you will obviously have some that perform better than 99.999% of random, thus passing any possible test you throw at it.
So even if there are meaningful patterns that your search discovers, they will be intermingled with numerous patterns that work simply by chance. And there is absolutely no way to actually separate them out.
I'd be curious to hear if anyone sees a flaw in this reasoning.
-bulat
When you apply some automated discovery mechanism you either have no preconceived notion or only a very general notion for what you are looking for. You then use some search technique to look at huge number of possible relationships and indicator/price permutations. If you are using a dumb random search approach you can easily search millions of possibilities. If you use a smart directed search algorithm (ie genetic algorithm), you can search the equivalent of billions or tens of billions of permutations.
When you look at so many possibilities, you are guaranteed to find quite a few methods that work incredibly well simply by chance. Even worse, some of these methods will pass any statistical test you throw at it (sharpe ratio, t-test, edge test, correlation test, etc.) since all these test compare the observed results vs what you'd expect randomly. But when you look at a 1,000,000 random samples, you will obviously have some that perform better than 99.999% of random, thus passing any possible test you throw at it.
So even if there are meaningful patterns that your search discovers, they will be intermingled with numerous patterns that work simply by chance. And there is absolutely no way to actually separate them out.
I'd be curious to hear if anyone sees a flaw in this reasoning.
-bulat