Data Mining for Trading Short-term Price Patterns

T

theShark20

As an aspiring systematic trader, my assumption is that there must exist some "price patterns", or regularities which are profitable when traded over the long term, the so-called "Statistical Edge".

With no prior trading experience (I am a Mechanical Engineer), I was skeptical about the existence of these patterns and had no idea at all how to discover them. I therefore used data mining to try to discover these price patterns. After years of trial and error, I found a couple of patterns that seem to actually work and I have been profitable for the last 2 years. So I start to convince myself that trading might be (slightly) better than the casino.

However I still have a hard time to understand my existing price patterns (why do they work? Why do they suddenly stop working? ) and I find very difficult and time consuming to discover new profitable patterns. I think there must be "a better way"

I read an interesting paper "Data Mining for Financial Applications" from Boris Kovalerchuk and Evenii Vityaev which states :
"The current efficient market theory/hypothesis discourages attempt to discover long-term stable trading rules/regularities with significant profit. This theory is based on the idea that if such regularities exist they would be discovered and used by the majority of the market players. This would make rules less profitable and eventfully useless or even damaging. [...] Data mining does not try to accept or reject the efficient market theory. Data mining creates tools which can be useful for discovering subtle short-term conditional patterns and trends in wide range of financial data. This means that retraining should be a permanent part of data mining in finance and any claim that a silver bullet trading has been found should be treated similarly to claims that a perpetuum mobile has been discovered."

This time-varying market efficiency and the short life time of price patterns make sense to me and match my experience.

  • What is your experience regarding the life time of price patterns?
  • What data mining techniques are appropriate to discover subtle short-term conditional patterns and trends ?
  • Is there a way to identify when a price serie is predictable/random without specifying a trading rule (e.g. Hurst exponent) ?
 
I have done some years of (manual) pattern search
on daily stock index data and found patterns with a couple years of lifetime till
10 or 15 years. However, the latter ones are rare
in terms of the frequency they occur and the number of patterns that
can be found.

In my opinion there is no general test wether an instrument is in "random walk mode" or not. Only thing you can do is to check if a specific pattern occurs at the moment or not. Another one might occur
when the first one is temporarily not occuring.

Hurst exponent: As I understand this an alternative to testing for autocorrelation. Although I don't know what is exactly calculated I assume it will not detect any assymetric or nonlinear patterns.

Which datamining techniques did you use?
The most important part in datamining is after having identified patterns, to statistically check if they are a result of random walk curve fitting or if they describe patterns that really exist.
 
if you are interested in data mining for price patterns there is a lot of stuff in this blog: http://www.priceactionlab.com/Blog/

Hurst exponent value depends on the lookback period like all other indicators. Very volatile indicator, virtually of no practical use for short term traders. Same as autocorrelation. A better alternative to life time is robustness. BTW price patterns is the only way to go.
 
Thanks for your answers. I fully agree with you that statistical significance is key in data mining for finance although it is completely ignored by most retail data mining/trading software in the market !

As of today, I do not use data mining and search the patterns "manually". I come up with new pattern ideas by looking at charts, reading papers or forums. Then I test the validity of these ideas on historical data using backtesting.
However, this way of looking for new patterns has some limitations :

1) It requires to form an hypothesis on the patterns or to specify trading rules
2) It requires frequent retraining because the patterns are not stable over time and evolve (intermittent patterns, short lived patterns, pattern drift, decrease of profitability of the patterns etc.)
3) It requires to create a full trading system (entry, exit, money management) to assess the profitability of the pattern

My current development framework is no longer good if you assume that patterns are short lived or intermittent and that frequent retraining of the patterns is required.
For example the equity curve or equity curve metrics (Net Profit, Max Drawdown) is not a good indication of the performance of the system unless the system activation/deactivation criteria and retraining is taken into account.
 
You're actually in a pretty good place. Like you, I have an engineering background, which means we come to trading unencumbered with silly assumptions like EMH, which somebody with an economics background gets burdened with. Just start looking at price movements with fresh eyes and ask yourself what makes sense and what doesn't. Am I making assumptions that can't ultimately be justified? Am I stuck in a rut as to how I approach this problem (pattern recognition, etc.) or am I thinking outside the box?

Remember: if economists had all the answers, they'd all be rich.

A wise man once said "No other professionals are more deserving of the question 'If you're so smart, why aren't you rich?' than economists." All those people do all day is study money: where it comes from, where it goes, what it does between coming and going. With such life-long professional focus on money, how is it that so few of them wind up in the top income-tax bracket?

Think better than the economists. You're an engineer; this is not a great challenge. :)
 
vinc said:
this is stupid american clishe and it's beyond me why a smart engineer would ever bring it up..
this is guy is undoubtedly not as smart as you and probably that's why he is poor..
http://en.wikipedia.org/wiki/Grigori_Perelman
More...
It is often a stupid cliche because not everybody is doing what they do just for money. But some people obviously are in it for the money, e.g., traders, and most economists. And of course some stupid people do get rich (pro athletes, other 'celebrities', heirs and heiresses, lotto winners).

The question is aimed primarily at so-called experts who talk about how to make money. Most trading 'gurus' get their money from book sales and seminars, not from actually trading themselves.
 
theShark20 said:
Thanks for your answers. I fully agree with you that statistical significance is key in data mining for finance although it is completely ignored by most retail data mining/trading software in the market !

As of today, I do not use data mining and search the patterns "manually". I come up with new pattern ideas by looking at charts, reading papers or forums. Then I test the validity of these ideas on historical data using backtesting.
However, this way of looking for new patterns has some limitations :

1) It requires to form an hypothesis on the patterns or to specify trading rules
2) It requires frequent retraining because the patterns are not stable over time and evolve (intermittent patterns, short lived patterns, pattern drift, decrease of profitability of the patterns etc.)
3) It requires to create a full trading system (entry, exit, money management) to assess the profitability of the pattern

My current development framework is no longer good if you assume that patterns are short lived or intermittent and that frequent retraining of the patterns is required.
For example the equity curve or equity curve metrics (Net Profit, Max Drawdown) is not a good indication of the performance of the system unless the system activation/deactivation criteria and retraining is taken into account.
More...


I believe that the hypothesis based approach is the way to go, specially if you are mining the data, since you need to have an idea to test and a clear goal for labelling your data.
What ML techniques are you using?
I've played around with Neural nets (a few flavours of sigmoid/arctan/Hopfield, softmax/binary output, probabilistic trigger), random forests, kmeans... I've got some interesting results but hit a bottleneck on my infrastructure (mostly on validation of results & run time) that I've been fixing for the last 8 months...

And I've heard some pretty good things about SVMs
 
theShark20 said:
As an aspiring systematic trader, my assumption is that there must exist some "price patterns", or regularities which are profitable when traded over the long term, the so-called "Statistical Edge".

  • What is your experience regarding the life time of price patterns?
  • What data mining techniques are appropriate to discover subtle short-term conditional patterns and trends ?
  • Is there a way to identify when a price serie is predictable/random without specifying a trading rule (e.g. Hurst exponent) ?
More...

Ever wonder why Subway is often the number one franchise to buy into every year if one wanted a better than average chance of success dealing with a restaurant? History of it working, nothing fancy, nothing too technical, store baked bread is biggest reason, it is fresh, by the time to pile all the unhealthy food into it, what you remember the most is the bread. Trading is the same way. I spent years wasting my time on trying to find the things the Big Boys weren't doing that worked and then I could corner the market, yeah right.

There aren't any new patterns, big brokerage have techs working 24 hours a day and where does all the big money go, pretty easy once you learn to trade like they do. How DUMB DUMB DUMB was I thinking of trying to beat them. What you want to do is steal the money from the other DUMB DUMB DUMB uneducated traders. The ones who think they finding something that only they discovered, LOL. It is like those who data mine commodity spreads back fifteen years looking at magical dates of when to put on and take off, there are no magical dates, but what that is crops go into the ground during seasons, based on rain and sun they grow, lack of either they die, if another country grows too much, that is not factored into those special dates.

Am sure you been profitable cause of rigid money management rules, you have a trading plan which most who enter into trading lack. All Price Patterns that were written in most charting books occur for reasons, and usually the reasons have to do with where to screw the smaller traders who are unskilled and under capitalized. That's It!!!

Having books going back to early 1900s I can view what Wheat did, guess what, not much has changed in over 100 years of charting, they didn't have computers then except paper and pencil, chart patterns are chart patterns BUT when too many are doing them, someone is going to pull the plug and throw heavy volume to hit the stops. How many times have you witnessed price breaking trendline, going up for a day, just enough for under capitalized to move up there stops only to see price come down to hit the stops, then guess what? Moves on back up in same day, LOL.

After 3 1/2 decades of trading, for longer term trading, where one enters means less and less to me as long as money management rules are very strong. Using one minute and less timeframes for scalping is a different story as that type of price is more random, entries must be addressed firmly to keep losses as low as possible, Price management, Time management of winners are studied, very little gained from losses with the exception of finding patterns that would have stopped us from taking these trades altogether. A method that produces 80% winners are not worth me taking as my risk is often more than what my gains will be and I average down on every trade. Look at it this way, what if you could find a pattern that breakevens plus one tick 65% of the time, and your money management rules allow you to have less than 10% losses, would you come out ahead? Not all winners/losers would be full targets, not all breakeven trades plus one tick have price going against position.

One element I had to learn was adapting and knowing chart patterns and why they happen, THEN I can adapt whatever I was doing to make it work again. It s funny, people buy systems cause mainly they can't build a system cause lack of experience, system stalls cause conditions change as many systems are created optimally, so they have to change but buyers go back to not knowing what to adapt cause lack of experience, then they keep buying systems, hopeless.
 
You must log in or register to reply here.
Back
Top