Since you are no longer using it do mind posting your set up? I would love to take a llok to see what you were doing. I am sure you can still find your set up at this link. http://www.trade-ideas.com/RecentWindows.html
Thanks,
D
Is data mining for trading patterns impossible?
- Thread starter bulat
- Start date
Since you are no longer using it do mind posting your set up? I would love to take a llok to see what you were doing. I am sure you can still find your set up at this link. http://www.trade-ideas.com/RecentWindows.html
Thanks,
D
Here's an interesting academic paper which also talks about the near impossibility of automatically identifying valid trading patterns, even when you know for sure that they exist in the solution space that you are searching (which you actually dont know).
http://www.intranet.management.mcgill.ca/homepage/profs/sarkissian/papers/pspur.pdf
http://www.intranet.management.mcgill.ca/homepage/profs/sarkissian/papers/pspur.pdf
The paper described in non-layman's term what I said in my previous message 
So I will just summarize it here,
The more explosive patterns (i.e. fits the tested data better) are chosen by most market participants as these patterns are more obvious to find (i.e. simple data mining) thus will not last (i.e. not working in the future).
The key to correctly identify recurring patterns is consistencies in the patterns' behaviour in various type of market conditions. Consistency can be measured by means of stable percentage in their edge.
As I mentioned before, one important area of data mining in financial data that is not really touched by both academics and actual market participants is the behaviour of smaller time frames (e.g. 15-sec, 5-sec, etc.) due to various reasons like not enough computing power, hard disk space, etc., so this area will be an area that gives an edge to the smaller players in the market.
Notice that many successful independent daytraders are tape readers - the lowest time frame of all data, even though they use many other forms of technical analysis to assist them in their final decisions.
So I will just summarize it here,
The more explosive patterns (i.e. fits the tested data better) are chosen by most market participants as these patterns are more obvious to find (i.e. simple data mining) thus will not last (i.e. not working in the future).
The key to correctly identify recurring patterns is consistencies in the patterns' behaviour in various type of market conditions. Consistency can be measured by means of stable percentage in their edge.
As I mentioned before, one important area of data mining in financial data that is not really touched by both academics and actual market participants is the behaviour of smaller time frames (e.g. 15-sec, 5-sec, etc.) due to various reasons like not enough computing power, hard disk space, etc., so this area will be an area that gives an edge to the smaller players in the market.
Notice that many successful independent daytraders are tape readers - the lowest time frame of all data, even though they use many other forms of technical analysis to assist them in their final decisions.
hard to do? yes.
impossible? no.
one should not automatically accept results that are derived from data mining processes, but one should not automatically reject them neither. you need a very analytical approach to understand what is valid or not.
as for taleb, he is right on some things, but he's also wrong by being unwilling to be more open-minded.
his theories are worthwhile, but only to an extent, because he's the extremist. his vision is limited to black and white and nothing in between. people who worship him like god obviously have not had much success with any pattern recognition, and people who disregard him completely obviously have not experienced enough of disorderly events to understand that some of taleb's warnings have validity.
Hello:
I am going off to have Easter dinner, and thought I would look in on this post.
The reason that patterns do not have a significant edge, is that the underlying price series does not exhibit stationarity. Its interesting that with all the extensive background that you folks claim to have, you don't (none of you) mention this very simple statistical "fact of life".
If you would learn to characterize the data series properly so that you know when price exhibits stationarity (or is likely to exhibit stationarity) you could then look for patterns with confidence that they are the result of non-random behavior.
Basic examples of stationarity occur at or arround earnings reports, economic reports, end of month (also known as "window dressing") bond and note auctions, etc. This is quite basic stuff.
As for how to do this (whether "by eye" or with a software approach) it is best to use both. A few posters have already published methods that work. Use search.
Best to all,
Lefty
I am going off to have Easter dinner, and thought I would look in on this post.
The reason that patterns do not have a significant edge, is that the underlying price series does not exhibit stationarity. Its interesting that with all the extensive background that you folks claim to have, you don't (none of you) mention this very simple statistical "fact of life".
If you would learn to characterize the data series properly so that you know when price exhibits stationarity (or is likely to exhibit stationarity) you could then look for patterns with confidence that they are the result of non-random behavior.
Basic examples of stationarity occur at or arround earnings reports, economic reports, end of month (also known as "window dressing") bond and note auctions, etc. This is quite basic stuff.
As for how to do this (whether "by eye" or with a software approach) it is best to use both. A few posters have already published methods that work. Use search.
Best to all,
Lefty
that's really beyond the scope of this thread because i don't know what your data was tested on and how it was tested. the only thing i can say is that one should try to cover and review *all* bases thoroughly when studying a system. investigate not just the core but every event. yes, it can be an excrutiatingly tedious process, and this is where human is stronger than machine. interrogate why things happen or why they don't. machines can crunch numbers but they don't have that intuitive judgement, so they can't perform that effectively. (maybe they can, but i haven't encountered any system like that yet.) but humans can.