Out of sample testing of patterns

[intradaybill]

Quote from Hugin:

A sound selection process can improve your results significantly, but defining this process seems to be as hard as it is finding an edge. Hugin

This is what it boils down to, I agree.

Now, one more question that bothers me. The developer of APS, Michael Harris, published the output of the program in 2002 in Traders' magazine, in the case of some QQQQ patterns. Then, in 2008, or after 6 years, he showed in an article (link below) that the patterns were still overall profitable. I think 6 years in an efficient market would be plenty of time to erase any edge, unless a monkey could have a higher edge. The overall profit factor is 1.53 in the out_of_sample from 2002 to 2008 and this is much higher than what would have been obtained purely by chance. His profit and stop were equal to 7% but the volatility in the QQQQ market was significant, especially in the beginning and ending period I calculate the 14-day ATR to be around 6%. Although the market had an uptrend it basically ended up where it started (buy_and_hold gain almost 0).

Article link

Any thoughts about the issues of time and how fast should any edge disappear?

 

[Hugin]

Quote from intradaybill:

This is what it boils down to, I agree.

Now, one more question that bothers me. The developer of APS, Michael Harris, published the output of the program in 2002 in Traders' magazine, in the case of some QQQQ patterns. Then, in 2008, or after 6 years, he showed in an article (link below) that the patterns were still overall profitable. I think 6 years in an efficient market would be plenty of time to erase any edge, unless a monkey could have a higher edge. The overall profit factor is 1.53 in the out_of_sample from 2002 to 2008 and this is much higher than what would have been obtained purely by chance. His profit and stop were equal to 7% but the volatility in the QQQQ market was significant, especially in the beginning and ending period I calculate the 14-day ATR to be around 6%. Although the market had an uptrend it basically ended up where it started (buy_and_hold gain almost 0).

Article link

Any thoughts about the issues of time and how fast should any edge disappear?

No good answers on this one I’m afraid.

This is obviously just speculation but maybe it could have to do that different strategies run by the big boys feed on one another so that the edge will come and go depending who is currently stronger. Which means it is hard to target a certain edge without bringing in people with other objectives that will bring it back (I’ve read some articles on behavorial finance along these lines where they model this tug of war using only two types of investors – trend followers and value investors – the resulting market becomes very similar to real ones).

Another thing is that it might have to do with volume. We could never run our stuff with the volumes a fund would require. So maybe you can ignore the big sharks and it just comes down to having the largest teeth in your little pool? Or behave like a pilot fish feeding on the left overs?

Our experience is that things are very intertwined. For example we have found that the larger the window we use for training data the more stable the resulting strategy will become in walk-forward/live trading and the longer the edge will hold. But walk-forward results will be weaker than for a strategy trained with less data. On the other hand strategies trained with little data are more uncertain going forward and edges disappears more abruptly (probably signalling a larger degree of fitting to training data).

Our longest running strategy has been running since 2006 and even if we have updated it roughly once a year the original one still has a small edge. This was trained with data from 1998-2004 so it would still make money after 6 years.

 

[pwrtrdr]

Maybe I don’t get it, but I would be more interested in finding what is "making" the patterns.... the behavior or correlation behind it, understand that. Wait for that "change" and use that as my indication to sell or buy. In other words, I believe these patterns do exist. However they won’t be there too long because the underlying reason or causation will change. Identify the change and trade that..

A random pattern ( 1800 bars) is not something you can build a business around.... understanding 50 patterns ( or 50 equities) and the causation of the pattern is in IMO

Quote from ronblack:

I need your opinion on this one. I used 10,000 bars of 60-minute QQQQ data for the in_sample and 1,800 bars for out_of_sample. I had APS Automatic Pattern Search find patterns in the in_sample with pf> 1 and 4% target and stop.

APS generated 84 patterns. I then had the program test them out_of_sample and it turned out that 40 of them remained profitable with pf >1. Out of those 40, 15 patterns had success rate of 100% (no losers).

My questions are:

A. Should I consider using only the patterns with 100% success rate in out_of_sample?

B. Should I assume that those patterns with 100% success rate will at some point soon revert towards mean value and instead use other patterns.

C. Should I consider using only the patterns with the highest number of out_of_sample trades? or those with the highest number of trades in the in_sample and with pf > 3 for example?

D. Any combination of the above or even none of the above?

Thanks

 

[Code7]

Quote from intradaybill:

This is what it boils down to, I agree.

Now, one more question that bothers me. The developer of APS, Michael Harris, published the output of the program in 2002 in Traders' magazine, in the case of some QQQQ patterns. Then, in 2008, or after 6 years, he showed in an article (link below) that the patterns were still overall profitable. I think 6 years in an efficient market would be plenty of time to erase any edge, unless a monkey could have a higher edge. The overall profit factor is 1.53 in the out_of_sample from 2002 to 2008 and this is much higher than what would have been obtained purely by chance. His profit and stop were equal to 7% but the volatility in the QQQQ market was significant, especially in the beginning and ending period I calculate the 14-day ATR to be around 6%. Although the market had an uptrend it basically ended up where it started (buy_and_hold gain almost 0).

Article link

Any thoughts about the issues of time and how fast should any edge disappear?

How can you tell readers that buy-and-hold return of 62 % is almost zero?

Those eight long-only patterns were tested from 05/07/2002 to 08/22/2008. QQQQ went up from 29.27 to 47.49 in that period. If you randomly selected entry dates in that period and used the same exits, you would get the same winning percentage.

I calculated that some time ago.
http://www.elitetrader.com/vb/showthread.php?s=&postid=2690096#post2690096
http://www.elitetrader.com/vb/showthread.php?s=&postid=2691679#post2691679

 

[intradaybill]

Quote from Code7:

How can you tell readers that buy-and-hold return of 62 % is almost zero?

Those eight long-only patterns were tested from 05/07/2002 to 08/22/2008. QQQQ went up from 29.27 to 47.49 in that period. If you randomly selected entry dates in that period and used the same exits, you would get the same winning percentage.

I took the 12/31/2008 closing of 29.74.

Here is the Michael Harris results with APS v8.0. First in_sample results is from 19900711 to 20020507. The pattern numbers are different in this version.

The next results if from a forward test from 20020507 to 20081231. The average profit factor was 1.3125.

The next result is from 20090101 to 20091231. 26 winning trades and only 1 losing trade. Average win rate is 96.29%. You will say that the market went straight up during that period but hey you paper trader, eveything is clear on hindsight to you. The system though took the trades. Did you take random entries in that period? You should be rich by now along with all the other random traders.

 

[intradaybill]

Out_of_sample 1

 

[intradaybill]

Out_of_sample 2

 

[Code7]

There are lots of small flaws in what you did. You can't average individual profit factors of 8 different patterns into 1.3125 because some patterns have more trades than others, apart from slightly different amounts of avg_win and avg_loss. Only two of your in-sample patterns have identical stats compared to Harris' in-sample patterns. So I really can't tell if you used the same patterns or not.

But the big question is, did Harris' original 8 patterns perform better than randomly selected trades? Because if not, the results have to be attributed to the long-only decision and the exits of 7 % stop and target. The price patterns were just useless accessory then. That actually was my whole point. I already showed that Harris' patterns were no better than random in the 6-year period he published results for.

Now go ahead and keep talking out of your ass by calling me a paper trader.

 

[intradaybill]

Quote from Code7:

Now go ahead and keep talking out of your ass by calling me a paper trader.

I need a system to trade. I cannot do random entries, your benchmark of evaluation. You are applying the wrong concept from statistics to trading. The issue with trading is whether you have a system that enforces discipline and has a winning bias. A random entry generator is the exact opposite at minus infinity. IF a system is random then why taking the trade now and not 30 minutes or 1 day latter?

Actually it appears that you do not even know how to paper trade correctly. You are another programmer who lurks around trading forums waiting to bash traders like me using words such as "random entries". Are you an idiot or what? This is a trading forum.

 

[goodgoing]

Quote from Code7:

There are lots of small flaws in what you did. You can't average individual profit factors of 8 different patterns into 1.3125 because some patterns have more trades than others, apart from slightly different amounts of avg_win and avg_loss. .

The trade-weighted PF is around 1.29 based on my calculations from the figure. It can be much higher if you use the various patterns with some clever money management.

Quote from Code7:

But the big question is, did Harris' original 8 patterns perform better than randomly selected trades?

Which random set of all possible random sets? You can find a random set that outperforms any system and one that underperforms it. This is the reason people use a benchmark of some kind, like free risk return, the return of some index or another benchmark portfolio. I understand you are probably a programmer and you have never taken a course in finance but think before you spew crap.

Quote from Code7:

The price patterns were just useless accessory then. That actually was my whole point. I already showed that Harris' patterns were no better than random in the 6-year period he published results for.

I do not have the time to study the Harris patterns but are you suggesting that the most valuable accessory of a trader is a pair of dice?

http://en.wikipedia.org/wiki/D20_System

I think most programmer chose their profession after having wasted their youth playing video games. Many of the decisions made in video games are based on random number generators.

Roll your dice then...geek...