The key point here is not to propose patterns based on empirical observations but rather on a proposed mechanism of the underlying market participants. If you simply tried to find patterns with no physical mechanism in mind, humans will be slower and an inferior to a machine, I agree. But what the machine cannot do is propose an underlying mechanism aka theory. This is where humans can beat machines.
Pattern Based Strategy Design
- Thread starter frostengine
- Start date
The key point here is not to propose patterns based on empirical observations but rather on a proposed mechanism of the underlying market participants. If you simply tried to find patterns with no physical mechanism in mind, humans will be slower and an inferior to a machine, I agree. But what the machine cannot do is propose an underlying mechanism aka theory. This is where humans can beat machines.
In that case i partially agree, but you can also argue that the mechanism of the underlying market participants would be discovered indirectly with a few specific "patterns". Although the expression (string of input values and math) wouldn't be as clear as when formulated by a human, it could very well describe what the theory wanted to in the first place without the researcher actually being aware of what he is looking at. Also, you can include any physical mechanism in your backtesting and/or formulate it as external values if you code the backtesting engine yourself, like a proper researcher should.
Exactly. I said that under the assumption that frosty and srk are retailers, without access to a data centre with 100s of CPUs at their disposal.
In the end, it's a business like any other. You need to either make a serious investment or be extrelemy innovative to do much with little resources. Maybe frostengine and some others need to be reminded of that, which isn't a bad thing. What's bad is when retailers give up too easily and never assume that the hypotheses they're testing might not in fact be broken, but need an increasing amount of investment (eg a lot of CPUs, better connections, better data, etc) to be figured out completely.
Braincell you seem to think CPU power is the answer to the problem.
After seeing what a simple optimizer can do to curvefit a simple strategy so it doesnt work out of sample I am convinced CPU power is not the answer.
After seeing what a simple optimizer can do to curvefit a simple strategy so it doesnt work out of sample I am convinced CPU power is not the answer.
Optimizers are notorious for doing that. Pattern search or any kind of data mining is different, and there is a lot to discuss on how to avoid a curvefit while doing that. It also depends on the method used for the search. Neural networks have a higher probability of curvefit, GA of existing strategies slightly lower if handled correctly, and then there is GP and more modern techniques. If you only tried one approach and created a conviction for yourself, that's your choice. There is a lot of text written by others, not me, to provide proof that search algos work not just for trading but for voice recognition, chemistry analysis, etc. If they didn't we wouldn't have so many books and papers written on the subject, and also hedge funds which use the technology with success. The devil is in the details.
Your assumption is right regard to my personal trading endeavors. However, in my day job, I am a computational scientist, and I have built large parallel systems (infiniband networked, multi-smp "fat node" systems with several hundred cpus), and I have designed and implemented parallel algorithms to leverage the hardware (via MPI as well as direct ibverbs implementations). We are also looking at mixed parallel multi-CPU-GPU systems. I have thrown large scale computations to very difficult problems, but it is still hard to beat human ingenuity / creativity because humans literally have the ability to break the rules.
Its more of a "custom" hybrid approach. Closest representation I guess would be a classification tree.
When I first started designing this algorithm, I also experienced that it was better at filtering trades from existing strategies. Eventually I had a "light bulb" moment and realized what was stopping it from being proficient at generating trade signals. That idea formed the basis of what became this "hybrid" pattern searching system.
I agree, having more computational power may be nice, but is far from required to extract meaningful patterns out of market data.
I see your point, not sure if I can agree yet though. Need to do some more research.
I just hate the details
