Analyzing similar patterns in the past

[ph1l]

Thank you, this is great explanation of the idea! May I ask you some questions?
1. Have you used this in the real trade or this is just research now?
2. Did you try to add some other factors like corporate or macroeconomics news? For example, exclude from 5000 similar chart patterns those which didn't have any important news?
3. What software do you use for this analysis?
4. Did you try something else (DTW for example) instead of Euclidian metric? Euclidian metric requires the same windows length. But the closely-matching chart patterns can have small difference in the time window.
5. Did you analyze the shape of the chart patterns after the closely-matching prior chart patterns? Let's say for example take in account not only median and std of price change but Euclidian metric std.
6. How did you get 873,185 historical charts? By sliding window with one day offset?
7. Is your risk-adjusted return a difference between return and mean return of all 415ETF? Or something like SP500?

1. Have you used this in the real trade or this is just research now?
A few weeks ago, I started trading this (long trades only) with small amounts of real money for passively-managed, unleveraged, non-inverse ETFs (including ETNs, UITs, Grantor Trusts, and Commodity Pools). Most of the potential assets have asset class Equity according to etfdb.com, but there are a few with asset classes Alternatives, Commodity, Real Estate, and Volatility that are considered for trades. And although analyzed, I won't trade in anything that sends me a K-1 tax form (hard for U.S. tax filing).

I have traded this strategy before with some real money comparing chart representations of S&P 500 constituents with the S&P 500. This worked somewhat, but I wanted to try broader classes of assets to be less dependent on a single market.

2. Did you try to add some other factors like corporate or macroeconomics news? For example, exclude from 5000 similar chart patterns those which didn't have any important news?
This strategy only uses representations of price charts. Another strategy I'm using (genetic programming) does have other kinds of data but not news or fundamentals.

I have tested the genetic programming strategy with fundamental stock data from Morningstar in combination with price data to predict future price direction. But I gave up on using that because it's not clear when the data is available relative to the period for the data (e.g., Morningstar might add different companies' annual report data at different times). And the finest fundamental data granularity is trailing twelve month data which Morningstar calculates by extrapolating annual and quaterly data. So the fundamental data wasn't consistent enough for me to trust using it.

3. What software do you use for this analysis?
I developed the software for this k-nearest neighbor strategy and the genetic programming strategy. Some details are in
https://www.elitetrader.com/et/thre...your-edge-for-2019.329802/page-8#post-4809209
and
https://www.elitetrader.com/et/thre...your-edge-for-2019.329802/page-8#post-4809655

4. Did you try something else (DTW for example) instead of Euclidian metric? Euclidian metric requires the same windows length. But the closely-matching chart patterns can have small difference in the time window.
I tried unweighted Euclidean-type distances, and found the weighted versions with newer data more heavily weighted seemed to work better. I haven't tried any other kinds of distances measures. Assuming DTW stands for dynamic time warping, this looks like something worth investigating.

5. Did you analyze the shape of the chart patterns after the closely-matching prior chart patterns? Let's say for example take in account not only median and std of price change but Euclidian metric std.
No, I just calculated performance measures for the simulated future part of the prior chart patterns. I guess I could consider calculating distance metrics on this simulated future part of the prior chart patterns and use this to help decide to take a trade or not.

6. How did you get 873,185 historical charts? By sliding window with one day offset?
Yes. Each historical chart has 21 days of future data, 1 day of for the simulated trade entry, and 252 days for the history. And this would go back for up to 2,772 days (11 years) -- less when there is not enough historical data.

7. Is your risk-adjusted return a difference between return and mean return of all 415ETF? Or something like SP500?
The risk-adjusted return used for the simulated future part of the historical charts is the mean daily loss-equivalent-to-gain return divided by the sample standard deviation of the daily loss-equivalent-to-gain daily.

The daily loss-equivalent-to-gain return is ((todayClose - yesterdayClose) / min(todayClose, yesterdayClose)). So if the price goes from 100 to 120, the loss-equivalent-to-gain return is (120 - 100) / 100 == 0.2. If the price goes from 120 to 100, the loss-equivalent-to-gain return is (100 - 120) / 100 == -0.2. So if the price goes from 100 to 120 and back to 100, the mean of the loss-equivalent-to-gain returns would be (0.2 + -0.2) / 2 == 0.

 

[aimaster]

1. Have you used this in the real trade or this is just research now?
A few weeks ago, I started trading this (long trades only) with small amounts of real money for passively-managed, unleveraged, non-inverse ETFs (including ETNs, UITs, Grantor Trusts, and Commodity Pools). Most of the potential assets have asset class Equity according to etfdb.com, but there are a few with asset classes Alternatives, Commodity, Real Estate, and Volatility that are considered for trades. And although analyzed, I won't trade in anything that sends me a K-1 tax form (hard for U.S. tax filing).

I have traded this strategy before with some real money comparing chart representations of S&P 500 constituents with the S&P 500. This worked somewhat, but I wanted to try broader classes of assets to be less dependent on a single market.

2. Did you try to add some other factors like corporate or macroeconomics news? For example, exclude from 5000 similar chart patterns those which didn't have any important news?
This strategy only uses representations of price charts. Another strategy I'm using (genetic programming) does have other kinds of data but not news or fundamentals.

I have tested the genetic programming strategy with fundamental stock data from Morningstar in combination with price data to predict future price direction. But I gave up on using that because it's not clear when the data is available relative to the period for the data (e.g., Morningstar might add different companies' annual report data at different times). And the finest fundamental data granularity is trailing twelve month data which Morningstar calculates by extrapolating annual and quaterly data. So the fundamental data wasn't consistent enough for me to trust using it.

3. What software do you use for this analysis?
I developed the software for this k-nearest neighbor strategy and the genetic programming strategy. Some details are in
https://www.elitetrader.com/et/thre...your-edge-for-2019.329802/page-8#post-4809209
and
https://www.elitetrader.com/et/thre...your-edge-for-2019.329802/page-8#post-4809655

4. Did you try something else (DTW for example) instead of Euclidian metric? Euclidian metric requires the same windows length. But the closely-matching chart patterns can have small difference in the time window.
I tried unweighted Euclidean-type distances, and found the weighted versions with newer data more heavily weighted seemed to work better. I haven't tried any other kinds of distances measures. Assuming DTW stands for dynamic time warping, this looks like something worth investigating.

5. Did you analyze the shape of the chart patterns after the closely-matching prior chart patterns? Let's say for example take in account not only median and std of price change but Euclidian metric std.
No, I just calculated performance measures for the simulated future part of the prior chart patterns. I guess I could consider calculating distance metrics on this simulated future part of the prior chart patterns and use this to help decide to take a trade or not.

6. How did you get 873,185 historical charts? By sliding window with one day offset?
Yes. Each historical chart has 21 days of future data, 1 day of for the simulated trade entry, and 252 days for the history. And this would go back for up to 2,772 days (11 years) -- less when there is not enough historical data.

7. Is your risk-adjusted return a difference between return and mean return of all 415ETF? Or something like SP500?
The risk-adjusted return used for the simulated future part of the historical charts is the mean daily loss-equivalent-to-gain return divided by the sample standard deviation of the daily loss-equivalent-to-gain daily.

The daily loss-equivalent-to-gain return is ((todayClose - yesterdayClose) / min(todayClose, yesterdayClose)). So if the price goes from 100 to 120, the loss-equivalent-to-gain return is (120 - 100) / 100 == 0.2. If the price goes from 120 to 100, the loss-equivalent-to-gain return is (100 - 120) / 100 == -0.2. So if the price goes from 100 to 120 and back to 100, the mean of the loss-equivalent-to-gain returns would be (0.2 + -0.2) / 2 == 0.

I have tested the genetic programming strategy with fundamental stock data from Morningstarin combination with price data to predict future price direction. But I gave up on using that because it's not clear when the data is available relative to the period for the data.

There is an active competition now on kaggle.com about market prediction using stock markets news and price data. 10-years data was provided for the competition. According to many discussions on kaggle.com - it's difficult to get any significant edge from news. Moreover adding news analysis to strategies based on only price data usually worsen the strategy result.

Assuming DTW stands for dynamic time warping

Yes, DTW stands for dynamic time warping. The benefit is it can compare time series with different length. But you needs to tune some more parameters like penalties for different time series length.

Thank you again for your the detailed response.

 

[userque]

I have tested the genetic programming strategy with fundamental stock data from Morningstarin combination with price data to predict future price direction. But I gave up on using that because it's not clear when the data is available relative to the period for the data.

There is an active competition now on kaggle.com about market prediction using stock markets news and price data. 10-years data was provided for the competition. According to many discussions on kaggle.com - it's difficult to get any significant edge from news. Moreover adding news analysis to strategies based on only price data usually worsen the strategy result.

Assuming DTW stands for dynamic time warping

Yes, DTW stands for dynamic time warping. The benefit is it can compare time series with different length. But you needs to tune some more parameters like penalties for different time series length.

Thank you again for your the detailed response.

Thanks for your informative posts.

I understand the use of DTW for speech recognition, etc.; because the phrase "THE CAT IN THE HAT" is the same phrase, regardless of cadence.

However, I would not assume that a stock pattern in one time frame will give a similar and generally successful prediction in another time frame. (And in the context of k-Nearest Neighbor, different time frames would likely have different nearest neighbors. IOW, DTW would effectively be 'over-riding' kNN and resulting in something equivalent to [greatly?] increasing the curse of dimensionality in the kNN algorithm.)

More simply: DTW will result in over-fitting, no?

Any studies showing the viability of using DTW with regard to financial market forecasting?

 

[ph1l]

Thanks for your informative posts.

I understand the use of DTW for speech recognition, etc.; because the phrase "THE CAT IN THE HAT" is the same phrase, regardless of cadence.

However, I would not assume that a stock pattern in one time frame will give a similar and generally successful prediction in another time frame. (And in the context of k-Nearest Neighbor, different time frames would likely have different nearest neighbors. IOW, DTW would effectively be 'over-riding' kNN and resulting in something equivalent to [greatly?] increasing the curse of dimensionality in the kNN algorithm.)

More simply: DTW will result in over-fitting, no?

Any studies showing the viability of using DTW with regard to financial market forecasting?

Any studies showing the viability of using DTW with regard to financial market forecasting?
Here are some (plus see the attached pdf files):
http://system.tradingninja.com/how-to-use-dtw-dynamic-time-warping-in-trading/
http://jonathankinlay.com/2016/08/dynamic-time-warping/
https://systematicinvestor.wordpress.com/2012/01/20/time-series-matching-with-dynamic-time-warping/

I'm testing my k-nearest neighbor strategy with dynamic time warping instead of Euclidean-type distance. Dynamic time warping looks like it might be helpful so far.

 

[murray t turtle]

Hi
These are the top most similar patterns in the past for some symbols with their further continuation. Do you find such type of analysis useful for manual trading?

View attachment 198619 View attachment 198620 View attachment 198621

%%
Yes.
Once you have the main trend, based on 200 day moving average + discretion[In other words 5 minutes above or below 200 dma means not much @ all.]

And oil may/maybe not highly correlated with S&P 500/SPY;
but since oil has no earnings , may not mean much @ all.And oil pays no dividends+ you may not have even meant to put them next to each other , like some put QQQ next to SPY. But even those stock ETFs can diverge easy, from each other.

AS far as ''news'', like one market maker said- you will most likely will get the news before i do , i have other things to watch .Good question

 

[ph1l]

Any studies showing the viability of using DTW with regard to financial market forecasting?
Here are some (plus see the attached pdf files):
http://system.tradingninja.com/how-to-use-dtw-dynamic-time-warping-in-trading/
http://jonathankinlay.com/2016/08/dynamic-time-warping/
https://systematicinvestor.wordpress.com/2012/01/20/time-series-matching-with-dynamic-time-warping/

I'm testing my k-nearest neighbor strategy with dynamic time warping instead of Euclidean-type distance. Dynamic time warping looks like it might be helpful so far.

I found dynamic time warping code from
http://sites.labic.icmc.usp.br/prunedDTW/
and ported it to my k-nearest neighbor strategy.

For testing, I ran this on data after the close on March 8, 2019 from 415 ETFs. The data had 875,888 representations of daily price charts of length 252 trading days plus corresponding future simulated entries at the next the next day's close with simulated exits 21 trading days later. The price charts were scaled and weighted so more recent times have higher weights. The tests used data for up to 2,772 charts per ETF (approximately 11 years of charts for a single ETF -- fewer charts when less history was available).

The dynamic time warping tests use a warping window which is a percent of the chart duration (252 trading days) rounded down to the nearest integer. The warping window is the maximum number of days before and/or after a particular day the algorithm can check for comparing the data on another chart. A warping window percent of zero is equivalent to using Euclidean distance without warping.

For each of the 415 ETFs, each test with a different dynamic warping window percent found the closest 5,000 charts by distance compared to the chart ending March 8, 2019 and then looked at the future simulated entries:

The values in the spreadsheet are the means for each statistic over the 415 ETFs. The distance values use a different unit of measure than previous images I posted, so these distance values aren't directly comparable to the previous images.

The results show as the warping window percents increased, the mean distances of the 5,000 closest historical charts decreased, and the mean risk-adjusted future returns increased. Dynamic time warping appears to find better matches than Euclidean distance, and these better matches have better future returns.

 

[murray t turtle]

Thanks for your informative posts.

I understand the use of DTW for speech recognition, etc.; because the phrase "THE CAT IN THE HAT" is the same phrase, regardless of cadence.

However, I would not assume that a stock pattern in one time frame will give a similar and generally successful prediction in another time frame.

Any studies showing ......?

%%
You are right;
market makers can do fine, small time frames.BUT commissions , slippage, stuff like bad earnings or good earnings can easy trainWreck small time frames/ forecasting ========================================================================================.

 

[aimaster]

In continuation with the theme of similar pattern searching and analyzing…

I have launched a free service which allows to:
- specify sample chart pattern;
- search for financial symbols where the pattern are available at the moment;
- search for the most similar patterns on the history price data and get trade stats for them.

How it works

Let assume you have found a pattern on a chart which you think gives you some advantages in your trading. So you want: a) check this pattern on the history b) check if this pattern is available somewhere for trade now.

Let’s consider, for example, a range breakout after the previous up movement on EURUSD:

Take start and end the pattern dates - 2017/12/18 00:00 и 2018/01/12 23:00 and go to the service.

Specify pattern to search:

pattern definition method = Specify start/end dates of the pattern
financial symbol = EURUSD:
timeframe = 1 day:
pattern period is from 2017/12/18 00:00 till 2018/01/12 23:00 (specify dates of our pattern):

Select symbols where to search for. By default all available symbols are selected – let it be as is:

Press Search, get the result:

— Chart of the pattern which we are searching for:

— Financial symbols where the pattern are available now for trade. Unfortunately there are no such patterns now:

— Top 50 the most similar patterns in the history:

— General stats for found patterns:

— Separately stats for buy only and sell only trading:

There are several search settings. You can, for example, specify how many similar patterns to search in the history charts or specify period for trading stats calculation:

There are a few financial symbols available for search, but you can upload your quotes data.

If you are interested in trying this service — you are welcome:
https://ai-finmarkets.com/fin/srv/price_patterns/service_search_price_patterns/