When a system spectacularly fails

jcl · May 8, 2012

The equity curve below is from a WFO test of a simple trend following system from a trading book:

This is an interesting curve. From 2004 to 2010, the system generates each year about the same profit with a quite good Sharpe ratio. When you trade such a system for 6 years, you certainly grow confident that you have a steady source of income.

However in fall 2010, the same system suddenly starts to generate huge losses, and does not recover, even though its parameters are regularly adapted to the market by WFO optimization. The GBP/USD market must have changed by the end of 2010 in a way that even parameter adaption can not save the system. Nothing in the 2004-2010 performance curve and no advanced test, such as WFA or Monte Carlo analysis, would foretell such a dramatic change.

No matter how well you know a system, it seems to be always possible that it suddenly starts generating huge losses without apparent reason.

braincell · May 8, 2012

Quote from jcl:

No matter how well you know a system, it seems to be always possible that it suddenly starts generating huge losses without apparent reason.

Yep, that's right.

What else can anyone say.

I always recommend running multiple strategies at once, perhaps a slightly lower chance they all fail at once.

trhudson · May 8, 2012

A few things that are obvious, although in hindsight (these could be measured in real-time):

The market has dropped significantly and likely to be trading in a much tighter range on an intra-day / day-to-day basis. When trading a trend-following system, this leaves much less movement for the trader to profit from.

A trend-following system will not work (has a much lower chance of working) when the market is range bound. You could implement a filter to determine the strength of a trend in order to capture only trending markets. (The ADX would be a good start on daily charts).

When developing intra-day systems, we always put limits on how tight a market is allowed to get. Our research shows that when a market tightens up on a daily basis, the trend-following systems are less likely to be profitable. At that time, one could find something else to trade, take a longer-term trade, or wait until a trend and a reasonable trading range returns.

ssrrkk · May 8, 2012

Quote from jcl:

The equity curve below is from a WFO test of a simple trend following system from a trading book:

This is an interesting curve. From 2004 to 2010, the system generates each year about the same profit with a quite good Sharpe ratio. When you trade such a system for 6 years, you certainly grow confident that you have a steady source of income.

However in fall 2010, the same system suddenly starts to generate huge losses, and does not recover, even though its parameters are regularly adapted to the market by WFO optimization. The GBP/USD market must have changed by the end of 2010 in a way that even parameter adaption can not save the system. Nothing in the 2004-2010 performance curve and no advanced test, such as WFA or Monte Carlo analysis, would foretell such a dramatic change.

No matter how well you know a system, it seems to be always possible that it suddenly starts generating huge losses without apparent reason.

Simple answer: non-stationarity. The training data does not contain all information. It is also a form of overfitting. Your model thinks that all cases have been covered in the past data which is not true -- another symptom of optimizing when there is little to no long-term signal.

logic_man · May 8, 2012

Quote from ssrrkk:

Simple answer: non-stationarity. The training data does not contain all information. It is also a form of overfitting. Your model thinks that all cases have been covered in the past data which is not true -- another symptom of optimizing when there is little to no long-term signal.

Is the lack of long-term signal a function of the parameters chosen? I know that without knowing them, it's difficult to say, but is this generally true?

braincell · May 8, 2012

Quote from logic_man:

Is the lack of long-term signal a function of the parameters chosen? I know that without knowing them, it's difficult to say, but is this generally true?

The signal is the market logic or market "truths" that have some underlying or fundamental regularities to them. How can anything be a function of an attempt to describe that (with parameters) signal? You're completely getting something in reverse, or are confused, and I'm not even sure what.

Parameters can sometimes match the signal, but generally it can never be known to what extent - because nobody can claim what a true fundamental market signal really is.

logic_man · May 8, 2012

Quote from braincell:

The signal is the market logic or market "truths" that have some underlying or fundamental regularities to them. How can anything be a function of an attempt to describe that (with parameters) signal? You're completely getting something in reverse, or are confused, and I'm not even sure what.

Parameters can sometimes match the signal, but generally it can never be known to what extent - because nobody can claim what a true fundamental market signal really is.

I think it's a terminology thing and that the word "function" wasn't appropriate there because it has another meaning which doesn't align with what I was asking. What I meant is whether or not that poster thought that the cessation of the strategy's ability to capture profits was a reflection of the underlying fact that the parameters in the model only temporarily happened to enable profitable trades due to some temporary luck and that the parameters weren't measuring anything "fundamental" at all.

failed_trad3r · May 8, 2012

Tested it on other pairs? I have noticed when you simply take a subset of 10 different markets for one system, if one market starts failing other markets pick up the slack, if the system is robust.

braincell · May 8, 2012

Quote from logic_man:

I think it's a terminology thing and that the word "function" wasn't appropriate there because it has another meaning which doesn't align with what I was asking. What I meant is whether or not that poster thought that the cessation of the strategy's ability to capture profits was a reflection of the underlying fact that the parameters in the model only temporarily happened to enable profitable trades due to some temporary luck and that the parameters weren't measuring anything "fundamental" at all.

I see. Ok in that case: what came first - the chicken or the egg? ; )

jcl · May 9, 2012

Quote from failed_trad3r:

Tested it on other pairs? I have noticed when you simply take a subset of 10 different markets for one system, if one market starts failing other markets pick up the slack, if the system is robust.

The results are similar with most other pairs. That's because currencies are not really uncorrelated. Interestingly, the price curves of many currency pairs have a low correlation, but the equity curves of systems with the same currencies have often a much higher correlation.