Aronson's Bogus Trading Tests

[kut2k2]

The anti-TA crowd makes much of the fact that in his book Evidence-Based Technical Analysis, David Aronson ran backtests on an awesome 6400 "trading rules" and found them all wanting.

As is often the case, things aren't nearly so cut and dried.

First of all, there weren't 6400 different trading rules. There were 6400 different parameter sets applied to a measly FOUR (4) trading rules. That is a horse of an entirely different color. The proof is here:

http://www.evidencebasedta.com/significancetestresultsof6402rules.txt

But even more damning is this statement from the Amazon page:

"... using twenty-five years of historical data to test 6,400 binary buy/sell rules on the S&P 500."

Wait ... did that say binary? Yes, it did.

Binary is code for stop-and-reverse (SAR). SAR is based on the assumption that a market is always trending, alternately up or down. As if chop isn't real and plentiful.

No reasonable trader uses SAR. No evidence supports its main assumption.

Is there any worse set of trading rules than SAR? Arguably not.

So David Aronson found that his 6400 SAR-based tests were all failures.

What a shocker.

And in other breaking news, water is still wet.

 

[jem]

interesting

 

[dtrader98]

"... using twenty-five years of historical data to test 6,400 binary buy/sell rules on the S&P 500."

Wait ... did that say binary? Yes, it did.

Binary is code for stop-and-reverse (SAR). SAR is based on the assumption that a market is always trending, alternately up or down.
What a shocker.

I don't know how you arrived at that comment. How is binary a code for stop-and-reverse SAR?

My understanding is that binary buy/sell rules simply means a set of indicator and/or signal logic based rule outputs that are comprised of two (binary) states: buy and sell. Those signals could be applied in trending or choppy markets.

He used something like 40 indicators and series, along with parameter permutations to arrive at the 6400 rules used in statistical comparison to random signals. I haven't read the book in a few years, but don't necessarily agree with all the conclusions. For one, the set of rules he tested were by no means comprehensive.

 

[volpunter]

agree, the comment by OP is outright bizarre. Sure the book (which I also read and flipped through years ago) does not do a fair job at looking at different asset classes, holding periods, strategy approaches. But the binary comment by OP is a bit odd and entirely unrelated.

I don't know how you arrived at that comment. How is binary a code for stop-and-reverse SAR?

My understanding is that binary buy/sell rules simply means a set of indicator and/or signal logic based rule outputs that are comprised of two (binary) states: buy and sell. Those signals could be applied in trending or choppy markets.

He used something like 40 indicators and series, along with parameter permutations to arrive at the 6400 rules used in statistical comparison to random signals. I haven't read the book in a few years, but don't necessarily agree with all the conclusions. For one, the set of rules he tested were by no means comprehensive.

 

[kut2k2]

I don't know how you arrived at that comment. How is binary a code for stop-and-reverse SAR?

My understanding is that binary buy/sell rules simply means a set of indicator and/or signal logic based rule outputs that are comprised of two (binary) states: buy and sell. Those signals could be applied in trending or choppy markets.

He used something like 40 indicators and series, along with parameter permutations to arrive at the 6400 rules used in statistical comparison to random signals. I haven't read the book in a few years, but don't necessarily agree with all the conclusions. For one, the set of rules he tested were by no means comprehensive.

Look at the official link :

http://www.evidencebasedta.com/significancetestresultsof6402rules.txt

There are no 40 indicators and series.

There are only 4 trading rules tested on a single series (SP500).

The first rule is designated TT-1-3 to TT-40-205.

The second rule is designated TI-1-3 to TI-40-205.

The third rule is designated D-1-3-10-15 to D-12-40-20-60.

The fourth rule is designated E-1-1-10-15 to E-12-40-20-60.


As far as binary, that means two and only TWO signals : Buy and Sell (short). No exit signal. Always in the market. IOW, stop-and-reverse.

You agreed as much here:

" My understanding is that binary buy/sell rules simply means a set of indicator and/or signal logic based rule outputs that are comprised of two (binary) states: buy and sell. Those signals could be applied in trending or choppy markets."

By contrast, my system is trinary: Buy, Sell (short), and Exit.

 

[volpunter]

sounds more like you bought the book and are pissed off that it does not provide the value you hoped for. Chalk it up as sunk cost.

Look at the link :

http://www.evidencebasedta.com/significancetestresultsof6402rules.txt

There are no 40 indicators and series.

There are only 4 trading rules tested on a single series (SP500).

The first rule is designated TT-1-3 to TT-40-205.

The second rule is designated TI-1-3 to TI-40-205.

The third rule is designated D-1-3-10-15 to D-12-40-20-60.

The fourth rule is designated E-1-1-10-15 to E-12-40-20-60.


As far as binary, that means two and only TWO signals : Buy and Sell (short-sell). No exit signal. Always in the market. IOW, stop-and-reverse.

You agreed as much here:

" My understanding is that binary buy/sell rules simply means a set of indicator and/or signal logic based rule outputs that are comprised of two (binary) states: buy and sell. Those signals could be applied in trending or choppy markets."

By contrast, my system is trinary: Buy, Sell (short), and Exit.

 

[wrbtrader]

The anti-TA crowd makes much of the fact that in his book...

I think if we stop making these TA threads to attract the anti-TA crowd, there will be less problems at this forum and there will be less reasons for an ET member to transform into a troll.

Instead, just do a trade journal or discuss an actual TA method being used (e.g. The ACD Method thread), it will be easier to get moderators and Baron to remove them.

 

[kut2k2]

I think if we stop making these TA threads to attract the anti-TA crowd, there will be less problems at this forum and there will be less reasons for an ET member to transform into a troll.

Instead, just do a trade journal or discuss an actual TA method being used (e.g. The ACD Method thread), it will be easier to get moderators and Baron to remove them.

I made the thread to post evidence that Aronson's conclusions don't apply to real traders, because he never tested a system that a real trader would use. Smacking anti-TAers is just a side benefit. If they can't deal with the truth, how is that my problem?

 

[dtrader98]

Look at the official link :

http://www.evidencebasedta.com/significancetestresultsof6402rules.txt

There are no 40 indicators and series.

There are only 4 trading rules tested on a single series (SP500).

The first rule is designated TT-1-3 to TT-40-205.

The second rule is designated TI-1-3 to TI-40-205.

The third rule is designated D-1-3-10-15 to D-12-40-20-60.

The fourth rule is designated E-1-1-10-15 to E-12-40-20-60.


As far as binary, that means two and only TWO signals : Buy and Sell (short). No exit signal. Always in the market. IOW, stop-and-reverse.

You agreed as much here:

" My understanding is that binary buy/sell rules simply means a set of indicator and/or signal logic based rule outputs that are comprised of two (binary) states: buy and sell. Those signals could be applied in trending or choppy markets."

By contrast, my system is trinary: Buy, Sell (short), and Exit.

Right. Ok. I guess you'd have to have the book to follow the methodology closer to what I'm seeing. But flipping back through the end, there are three categories representing major themes of TA 1) trends 2) extremes and transitions 3) divergence. I can understand and agree if you think those categories are limited, but, I also find his approach a lot more useful than the typical TA proponent that comes on the board and shows three or four anecdotal charts as if they are gospel.

The 40 underlying signals are some raw time series and some transformations of raw time series that those three basic methods are applied around (with parameter permutations like lookback period).

Again, I'm going by memory, but I agree he is always in, which is definitely a shortcoming.
I'd much prefer to see tests with three states covered; long, short, flat. It would be a lot harder to set up the testing comparisons-- but, hey, that just gives creative people some room to think outside of the box... What's nice about the book, at least, is that it gives us some ideas about how statisticians approach testing around trading signals and systems; along with shortcomings.

 

[kut2k2]

Right. Ok. I guess you'd have to have the book to follow the methodology closer to what I'm seeing. But flipping back through the end, there are three categories representing major themes of TA 1) trends 2) extremes and transitions 3) divergence. I can understand and agree if you think those categories are limited, but, I also find his approach a lot more useful than the typical TA proponent that comes on the board and shows three or four anecdotal charts as if they are gospel.

The 40 underlying signals are some raw time series and some transformations of raw time series that those three basic methods are applied around (with parameter permutations like lookback period).

Again, I'm going by memory, but I agree he is always in, which is definitely a shortcoming.
I'd much prefer to see tests with three states covered; long, short, flat. It would be a lot harder to set up the testing comparisons-- but, hey, that just gives creative people some room to think outside of the box... What's nice about the book, at least, is that it gives us some ideas about how statisticians approach testing around trading signals and systems; along with shortcomings.

If it's hard to test realistic trading systems (long, short, flat), then that doesn't speak well of the usefulness of the Aronson testing methodology. The always-in systems he did test just have no relation to real-world trading, which makes me wonder what was his point?

Funny how none of this came out years ago. It's taken this long to get to the basic shortcomings of the book.