More on APS and out_of_sample testing

goodgoing · May 9, 2010

I start a new thread to hopefully discuss some results I get from APS and ideas for better testing. The procedure I follow is:

1. I use two historical data files, the in-sample and the out-of-sample.
2. I run an APS search in the in-sample
3. I test the patterns in the out-of-sample using the APS test function
4. I get the APS code for Amibroker and program the system there
5. I use the Amibroker backtest report to check out-of-sample results

Test parameters: Profit target = 1%, Stop loss = 2%. Minimum pf = 1.5

Data: QQQQ daily
In-sample: 19900711 to 20020507 (As in the study by Harris)
Out-of-sample: 20020507 to 20080822

- Results of the APS search: 1 long and 2 short patterns

http://www.4shared.com/photo/TrT007n5/QQQQ_20020507_T1S2.html

- Results of APS out-of-sample testing:20020507 to 20080822

http://www.4shared.com/photo/ewAG_24R/QQQQ_20080822_T1S2.html

The profit factor is about 1.38

- This is interesting next: results for just all of 2008- see chart

http://www.4shared.com/photo/HwD_1Nq5/QQQQ_2008_T1S2.html

This is the Amibroker report:

http://www.4shared.com/photo/43WON86f/AB_QQQQ_2008_T1S2.html

This was a 100% hit rate for all three patterns during 2008.

Question: what criteria would you apply for deciding whether to keep this system after 2008?

This is an important question I do not know how to answer properly. I have another question I think is important becuase I am having some doubts about the significance of out-of-sample testing. Is it maybe better to use the full history available to search for patterns, use some (?)criteria for determining their significance and then hope they will perform well in real trading?

Kevin Schmit · May 9, 2010

Quote from goodgoing:

Data: QQQQ daily
In-sample: 19900711 to 20020507 (As in the study by Harris)

Q's did not trade until 1999. SPY's started in 1993. AFAIK, no American ETF's were trading in 1990.

goodgoing · May 10, 2010

Quote from Kevin Schmit:

Q's did not trade until 1999. SPY's started in 1993. AFAIK, no American ETF's were trading in 1990.

Thanks Kevin, you are technically correct. The data for QQQ prior to inception of the fund were provided to vendors by the company based on extrapolations but you are correct, they do not present actual trades.

I have repeated the study using SPY data since inception. I used an in-sample from 1993 to the end of 2008 and the out-of-sample 20090101 - 20100507.

Data: SPY daily
APS search parameters: Profit target = 1%, Stop loss = 2%. Minimum pf = 1.5 Minimum trades = 50 Minimum success rate for long/short = 75%.

- Results of the APS search: 10 long and 1 short patterns

http://www.4shared.com/photo/ZQ9pf-dD/SPY_APS_IN_SAMPLE.html

- Results of APS out-of-sample testing:

http://www.4shared.com/photo/76BHOv4v/SPY_APS_OUT_OF_SAMPLE.html

This is the Amibroker report:

http://www.4shared.com/photo/_98Z59mA/SPY_2009_2010_T2S4.html

The success rate out-of-sample is 66.67% and the profit factor pf = 1.34

This is the Amibroker report when all bars are equally weighted for long trades only (no commission effect taken into account):

http://www.4shared.com/photo/X1Ytw_uX/SPY_ALLBARS_2009_2010_T2S4.html

The success rate out-of-sample is 69.32 % and the profit factor pf = 1.12, much less than the APS pattern pf.

Question 1: would you consider trading this system?

Question 2: is it maybe better to skip out-of-sample testing and use the entiry price history in the APS search?

jimbojim · May 11, 2010

Quote from goodgoing:

Question 1: would you consider trading this system?

Question 2: is it maybe better to skip out-of-sample testing and use the entiry price history in the APS search?

Q1: Maybe

Q2: I think it is better because you save time. Out_of_sample testing is just stupid and useless. If your system selection is based on systems that perform well in out_of_sample plus in the in_sample then this is equuivalent to a system that performs well in the full sample.

intradaybill · May 11, 2010

Quote from jimbojim:

Q1: Maybe

Q2: I think it is better because you save time. Out_of_sample testing is just stupid and useless. If your system selection is based on systems that perform well in out_of_sample plus in the in_sample then this is equuivalent to a system that performs well in the full sample.

Out_of_sample testing is necessary IMO, especially when optimization of system parameters is involved. One could debate whether OHLC patterns are subject to optimization but many see selection as a form of optimization.

There is no doubt in my mind that the best out_of_sample testing is one that involves actual trades. It takes money to make money.

goodgoing · May 14, 2010

This is the last study I will post using APS again for SPY daily data pattern search with profit objective = 2% and stop-loss = 3%. After a number of studies I have done including those I posted here, I have concluded that APS has high potential in discovering systems that involve some sort of short-term market timing.

I used again an in-sample from 1993 to the end of 2008 and the out-of-sample 20090101 - 20100507.

Data: SPY daily
APS search parameters: Profit target = 2%, Stop loss = 3%. Minimum pf = 1.5 Minimum trades = 29 Minimum success rate for long/short = 75%.

- Results of the APS search: 10 long and 3 short patterns

http://www.4shared.com/photo/DdL8uQ7G/SPY_APS_IN_SAMPLE.html

- Results of APS out-of-sample testing:

http://www.4shared.com/photo/p8DTD0hd/SPY_APS_OUT_OF_SAMPLE.html

Out of the 13 patterns, 5 had a 100% win rate and 1 pattern had a 0% win rate in the out_of_sample . Overall, 8 long patterns and 2 short patterns were profitable and 1 long pattern had 0 trades.

This is the Amibroker report for the out_of_sample:

http://www.4shared.com/photo/JSuFcwuC/SPY_2009_2010_T2S3.html

The success rate out-of-sample is 75% and the profit factor pf = 2.17

This is the Amibroker report when all bars are equally weighted for long trades only (no commission effect taken into account):

http://www.4shared.com/photo/dAp0JGPe/SPY_ALLBARS_2009_2010_T2S3.html

The success rate in the out-of-sample is 64.6 % in this case and the profit factor pf = 1.20, much less than the APS pattern pf.

The conclusion is that APS patterns, at least for the examples I posted here and for several others I have studied, exhibit high survivorship rates in out_of_sample testing. Keep in mind that this offers no guarantee that the performance will be maintained under actual trading conditions but as far as the software goes these studies answered several questions I had about its capabilities. Great tool. Use it with caution like all tools, they can be a help or inflict damage if not used properly.

ronblack · May 16, 2010

goodgoing, I like your examples. I like the fact that you are using lower avg win to avg loss ratio with higher win rate. The reason for this is - surprisingly - not understood by many traders I meet in trading conferences. Everyone is shooting for high avg win to avg loss while forgetting the volatility. AFAIK, this is a major cause for trader failure.

Regarding APS, if you know what you are doing this program will pay for itself in a few trades and afterwards make you money. But it is not for everyone. I do not recommend it to people who do not have knowledge of probability and statistics. I met once a very smart trader from Europe in a conference and when we opened our laptops we both started APS, that was really funny. He explained to me how he used AND in addition to OR to combine patterns in a strategy. Just keep that in mind. When you use AND you demand is for extra confirmation. It does not have to be concurrent in time, you can allow of few bars for that like in

//N is number of bars, K number of patterns

flag() = 0;

for j = 0 to N-1 do
for i =1 to K do
If patternk then flag(i) = 1;

sumflag = 0;
for i = 1 to k do begin
sumflag = flag(i)+sumflag;

If sumflag > 2 then buy (sell)

Good work, all the best.

intradaybill · May 19, 2010

Quote from ronblack:

goodgoing, I like your examples. I like the fact that you are using lower avg win to avg loss ratio with higher win rate. The reason for this is - surprisingly - not understood by many traders I meet in trading conferences. Everyone is shooting for high avg win to avg loss while forgetting the volatility. AFAIK, this is a major cause for trader failure.

Regarding APS, if you know what you are doing this program will pay for itself in a few trades and afterwards make you money. But it is not for everyone. I do not recommend it to people who do not have knowledge of probability and statistics. I met once a very smart trader from Europe in a conference and when we opened our laptops we both started APS, that was really funny. He explained to me how he used AND in addition to OR to combine patterns in a strategy. Just keep that in mind. When you use AND you demand is for extra confirmation. It does not have to be concurrent in time, you can allow of few bars for that like in

//N is number of bars, K number of patterns

flag() = 0;

for j = 0 to N-1 do
for i =1 to K do
If patternk then flag(i) = 1;

sumflag = 0;
for i = 1 to k do begin
sumflag = flag(i)+sumflag;

If sumflag > 2 then buy (sell)

Good work, all the best.

Excellent post. Just correct the first if-statement to:

for j = 0 to N-1 do
for i =1 to K do
If pattern(i) then flag(i) = 1;

jimbojim · May 21, 2010

Quote from intradaybill:

There is no doubt in my mind that the best out_of_sample testing is one that involves actual trades. It takes money to make money.

I agree that the ultimate test is the real market but nobody is willing to spend a fortune testing systems, the objective is to make fortunes, not to spend it.

ronblack · May 22, 2010

Quote from intradaybill:

Excellent post. Just correct the first if-statement to:

for j = 0 to N-1 do
for i =1 to K do
If pattern(i) then flag(i) = 1;

Thanks for the correction of the pseudocode.

I have implemented this approach to results I get from APS and in certain cases I get significant improvement in performance. I appears that some patterns are more suitable as entry signals and others as confirmation signals. This is part of the idea behind this approach.