How Have Properly Backtested Systems Failed In Actual Trading?

M

magicwords

I'm a stock trader and with the use of a mechanical trading system, I believe I've had some moderate success with respect to my return and drawdown preferences.

Still, I remain mindful that my system is bound to fail eventually. With that, how have your properly backtested systems failed in actual trading?

I believe that a lot of you seem to know how to properly backtest systems. So I'd like to know when these good-in-paper systems failed in actual trading.

Was it because you arbitrarily skipped trades? Did you change some components of the system? Or perhaps the system just, well, degraded?
 
I have backtested a fair number of systems, and my two cents on how real-time systems don't match up to backtested ones is in price target exits. If you use target limit exits, a backtested system result may show a price print at the target exit, indicating a profitable close. In reality, though, if the target limit exit price is at the extreme of a price bar, that trade may not actually close due to not filling.

With fast trading instruments such as futures, just because a price prints at the top (or bottom) of a price bar, signaling a profitable limit exit, that does not guarantee a successful fill. Instead, it may mean falling back through breakeven and hitting a stop exit later... which is why I always use breakeven stops when within a few ticks of my targets.

So, do not assume a profitable limit exit just because the target price prints on the chart. There is no way to decipher, from an historical chart, if every trader waiting in line at that limit price got a fill. As a rule of thumb for such backtesting, I only count a target limit exit as profitable if I see that price PLUS a tick, better assuring me that, in the real world, a fill would be likely at the limit price.

Example, if the target limit exit for a trade on the ER2 is at 795.50, and the price ONLY prints at 795.50, there is no assurance such an exit would have actually been filled. This is dangerous to system traders as it skews historical results into hypothetical. The last thing we as traders need is to keep assumptions, rounding, etc. in our system results. A price print one tick better than the target, to 795.60, assures me that most everyone who wanted to get filled at 795.50, did get filled, including me.

There are other deficiencies in backtesting, but this is the issue I know best, so I hope it helps to answer your question.
 
Yes, don't assume a fill unless price went through your target. Don't assume a backtest that is not tick by tick is accurate either. Assumptions made about what happens inside a bar can be way off from reality....
 
Quote from magicwords:

I'm a stock trader and with the use of a mechanical trading system, I believe I've had some moderate success with respect to my return and drawdown preferences.

Still, I remain mindful that my system is bound to fail eventually. With that, how have your properly backtested systems failed in actual trading?

I believe that a lot of you seem to know how to properly backtest systems. So I'd like to know when these good-in-paper systems failed in actual trading.

Was it because you arbitrarily skipped trades? Did you change some components of the system? Or perhaps the system just, well, degraded?
More...

Don't put too much faith in backtesting. You should view it as a possible performance not actual historical performance. It is like looking through clouded glass - you get an idea of what you are looking at, but until you actually trade the system in real time, in different conditions you will not understand performance. Just like a photo doesn't tell you everything about the experience.

I have backtested many systems that fail in real time. Sh it happens. Just cos she looks good in the picture, doesn't mean she will look good in the flesh.

I am definitely an advocate for mechanical systems and backtesting, just put the results in context.

pneuma
 
The biggest error in system trading is in the use of back-testing by the author. Back-testing can be done the correct way or the incorrect way.

Most out of inexperience choose the latter.
 
Quote from magicwords:


I believe that a lot of you seem to know how to properly backtest systems. So I'd like to know when these good-in-paper systems failed in actual trading.
More...

Systems that trade equities for any period beyond scalping are highly dependent on the market environment.

So a close look at system performance in relation to the broader market will usually reveal what to look for in the future.

A 'failure' in a system that's been successful shouldn't come as a surprise. If it does, well, you probably shouldn't be trading the system in the first place.
 
The REAL PROBLEM IMHO is that the chart you see and the patterns we observe are illusions. I am currently trading a non-linear supersystem for the nasdaq index.. I analyze the cash index, trade the futures.. I have proven IT WORKS!

One problem I see for most system designers is that the tools they use are inadequate to describe the TRUE MATH inderlying the "chart" image. Under the surface the mathematics for 5 similar chart patterns can be "wildly different"! This is where one can trade all 5 times and get a different result each time!!

The tools you use need to project for you the correct mathematical "image" necessary to define the "pattern" correctly. It is then that your performance will improve dramatically.

Another problem is that some of these patterns are more static, others dynamic/fleeting. Your profit taking needs to be flexible enough to take money off the table yet allow a % of the position to go with the trend.. the more dynamic the pattern, the faster the profit should be locked down!

Finally, these patterns have multiple levels associated with them -like an onion -it is incredibly difficult, but possible, to peel back each layer such that a raw model can be "cleaned" and traded more effectively.

Mh

PS. Dark Pools are proving to be a potentially vexing problem for any system designer -some of the tools that were VERY effective in the past (say 2001) are less effective now. It can be tricky but those patterns, if defined carefully (mathematically correct), are consistent! :cool:
 
Quote from pneuma:

I have backtested many systems that fail in real time.
More...
Have you guys found similarities between these good backtests that failed in actual trading? Maybe by looking at their similarities and spelling them out we will have an idea on whether a good backtest is likely to fail in actual trading.

For example, just for the sake of argument, maybe one similarity is in the systems' having too many parameters (say 20). So when we stumble across a system that backtests well but has 30 parameters, we already have something telling us beforehand that the system may not perform as well in actual trading.
 
Quote from TSGannGalt:

All you guys are retarded.

Markets are always changing.
More...
(i love those small little inbetween comments of the
senior guys. dropping at some place, make noise, walk
along ... probably unscubscribing as soon as the post
was finished ... has some aesthetic value ...)
 
You must log in or register to reply here.
Back
Top