Reasons for failur in automatic trading systems

BertH · May 12, 2006

Quote from acrary:

Vast majority of systems fail because they're not based on a edge in the market. Most are based on fitting a method to the past data. Even with a walk forward test they're just wasting their time. Take the trades from backtesting....put them through a Monte Carlo process...save the 95% drawdown level...watch how often it's hit in out of sample tests. Pretty conclusive evidence the method was based on past market conditions and not a edge. Just about all commercially available systems are built on market conditions and just about all fall apart over time.

Acrary, I am not over here often, but I have read some of your intriguing takes. Can you elaborate a bit on an "edge" vs market conditions? I think I understand that market conditions can render some strategies relatively useless, but how is an edge different in that it can deal with varied market behavior?

Thank you.

Lawrence Chan · May 12, 2006

A simple way to explain market condition vs edge is to use an example.

In a bear market (like the crash from 2000-2002), you can have pretty random or obscure rules to short, as long as you do not use stop or having very wide stop, those trades your system take will eventually turn into profitable ones. Such systems always fail in sideway market or bull market. It was the market condition that makes those systems profitable.

An edge with a particular market, however, is more stable and usually works well in various market conditions. The issue is, an edge could disappear or stop working too when there is something fundamentally changed the market you trade. e.g. change of trading time, change of minimum tick size, etc.

BertH · May 12, 2006

I understand your example regarding benefiting from market conditions, but I guess I need to further research the term "edge." I am working on something myself, and I wouldn't want it to be too dependent on temporary conditions.

Thanks for your efforts.

acrary · May 12, 2006

Quote from BertH:

Acrary, I am not over here often, but I have read some of your intriguing takes. Can you elaborate a bit on an "edge" vs market conditions? I think I understand that market conditions can render some strategies relatively useless, but how is an edge different in that it can deal with varied market behavior?

Thank you.

Here's a example. Years ago when the Stock Traders Almanac first came out one of the tendencies described was the last two days and first 5 days of the month being market movers. I verified the condition existed and persisted from month to month in the SP futures. I could have stopped there and used a envelope strategy to try to catch any trend starting around the end of a month. This would be how a person with only market data might analyze the market. I went on to try and figure out why the market moved during those days. I knew mutual funds were big and thought maybe it was caused by inflows into the funds. There were and are sources of mutual fund flows available so I got some data and did some tests. I found that fund flows in and out had a lag due to reserves in the funds, but the tendency was for large inflows to be put to work within 5 days after receipt and outflows caused selloffs in the funds around 3 days after the funds were disbursed. I setup the software to do a weekly trade based on buying when inflows were strong and sell when inflows were weak. If the inflows remained on the side I was already in then no new trade was initiated. From 2001 - 2002 there were no losing trades and the average winner was 28k per-contract in the SP. That's a difference between mapping to market data and pulling the profits out of the market based on a edge.
I went on to build a whole series of models based on this edge things like ... if the market early in the week went against my edge then I'd initiate new positions to take advantage of the opportunity knowing they'd be active within a day or two.
Anyway, there's a example for you. There's lots of these edges around...just don't expect to find them in the o, h,l, c data.

TSGannGalt · May 12, 2006

Quote from BertH:

Acrary, I am not over here often, but I have read some of your intriguing takes. Can you elaborate a bit on an "edge" vs market conditions? I think I understand that market conditions can render some strategies relatively useless, but how is an edge different in that it can deal with varied market behavior?

Thank you.

Other examples of edges from the past:

SOES Bandit, 99 lot fills, Bond vs SP trades, SP/ES arb, Floor Traders (pre ES, there's still some but a lot has been exhausted), CRT...

Umm... in other semantics... low commission, fast computer, low latency, lots of capital, programming... etc. etc.

there's tons of them...

BertH · May 12, 2006

Quote from acrary:

Here's a example. Years ago when the Stock Traders Almanac first came out one of the tendencies described was the last two days and first 5 days of the month being market movers. I verified the condition existed and persisted from month to month in the SP futures. I could have stopped there and used a envelope strategy to try to catch any trend starting around the end of a month. This would be how a person with only market data might analyze the market. I went on to try and figure out why the market moved during those days. I knew mutual funds were big and thought maybe it was caused by inflows into the funds. There were and are sources of mutual fund flows available so I got some data and did some tests. I found that fund flows in and out had a lag due to reserves in the funds, but the tendency was for large inflows to be put to work within 5 days after receipt and outflows caused selloffs in the funds around 3 days after the funds were disbursed. I setup the software to do a weekly trade based on buying when inflows were strong and sell when inflows were weak. If the inflows remained on the side I was already in then no new trade was initiated. From 2001 - 2002 there were no losing trades and the average winner was 28k per-contract in the SP. That's a difference between mapping to market data and pulling the profits out of the market based on a edge.
I went on to build a whole series of models based on this edge things like ... if the market early in the week went against my edge then I'd initiate new positions to take advantage of the opportunity knowing they'd be active within a day or two.
Anyway, there's a example for you. There's lots of these edges around...just don't expect to find them in the o, h,l, c data.

Thatâs a great example, acrary. I appreciate your providing that stimulus for my mind to strive that direction.

If, for exampleâs sake, I note that a stock trending strongly upward and then coming back to the 10 day MA over a few days on decreasing volume is a good long play, then I need to seek a reason behind it versus just seeing (or thinking I see) the phenomenon to be pretty successful over the past two years?

BertH · May 12, 2006

cohvi--'pologize for getting away from your thread.

acrary, thanks again for your reply. I am sure with my enthusiasm and quest for learning, I could drive ya nuts. Would try the PM thing again but realized you'd be overwhelmed in a short while with such requests. So, will use the search engine as much as possible.

jerryz · May 13, 2006

say you are in a bear market, but the data shows that if you buy at the open every day and sell 30 minutes later, you make money on average.

is this an edge?
or is this a condition of the first 30 minutes of the market?

you then try to
1) find concrete reasons why this strategy works, and
2) do a monte carlo test of other long only, 30 minute holding period trades that can begin at any time of the day. (is this the correct way?)

you now have 4 possible results:

Code:

                      Monte-Carlo (significant)      Monte-Carlo (insignificant)
Real reason exits
No reason exits

which of these results is an edge?

TSGannGalt · May 13, 2006

Code:
Quote from jerryz:

say you are in a bear market, but the data shows that if you buy at the open every day and sell 30 minutes later, you make money on average.

is this an edge?
or is this a condition of the first 30 minutes of the market?

you then try to
1) find concrete reasons why this strategy works, and
2) do a monte carlo test of other long only, 30 minute holding period trades that can begin at any time of the day. (is this the correct way?)

you now have 4 possible results:

Code:

Monte-Carlo (significant) Monte-Carlo (insignificant) Real reason exits No reason exits

which of these results is an edge?

??? Ummmm.... arrhhh...

No Reason Exits ... w/ insignificant Monte-Carlo

*doh*

jerryz · May 13, 2006

typo, i meant to type

Reason exists
No real reason exists

Code:
Quote from jerryz:

say you are in a bear market, but the data shows that if you buy at the open every day and sell 30 minutes later, you make money on average.

is this an edge?
or is this a condition of the first 30 minutes of the market?

you then try to
1) find concrete reasons why this strategy works, and
2) do a monte carlo test of other long only, 30 minute holding period trades that can begin at any time of the day. (is this the correct way?)

you now have 4 possible results:

Code:

Monte-Carlo (significant) Monte-Carlo (insignificant) Real reason exits No reason exits

which of these results is an edge?