Limitations in backtesting

oddiduro · Dec 5, 2003

I have run one proven money maker, and several experimental tests side by side with a backtester. Signal in real time are often different than signals in the backtest. Once such system is a net loser in the backtester, while profitable in real time.

The best backtester in the world can only grossly duplicate real time market conditions, I have found.

Has anyone found any anamolies in backtesting they would like to share?

Regards
Oddi

hypostomus · Dec 5, 2003

...I find very often that E-Signal's historical one-minute charts are off by one tick from the realtime data at critical price points, such as the open or failures. This can lead my systems to have two optimized entry points. I simply trade whichever signal pops first. Not knowing the details of the system in question, I can only offer (out of sheer ignorance, no offense intended) that if historical tick variations are enough to hose the results, the system is marginal.

The systems I trade are insensitive to that, because a signal which fails to appear in backtesting will be compensated for by a signal which failed to appear in real time. I simply go to IB and demand that they give me the missing real-time trade.

maxpi · Dec 5, 2003

One thing I did that worked in backtesting and failed in reality was to place stops so perfectly that I altered the market. As the price swept down towards my stop it made one more tick downward to take out my stop!!

I heard from one guy that would test his systems realtime with two different data providers and used the one that worked best for him. It's just crazy.

hypostomus · Dec 5, 2003

...I think what we are saying is that good entries and exits in today's low volatilities may hang on a tick. Re your stop problem, the opposite often happens, too: price will kiss the optimized stop and rebound. That's why I often reinforce my position one tick away from the optimized stop. What the hell. As Livermore was quoted as saying, "Always trade when there is the least possibility of a loss."

oddiduro · Dec 5, 2003

Quote from hypostomus:

...I find very often that E-Signal's historical one-minute charts are off by one tick from the realtime data at critical price points, such as the open or failures. This can lead my systems to have two optimized entry points. I simply trade whichever signal pops first. Not knowing the details of the system in question, I can only offer (out of sheer ignorance, no offense intended) that if historical tick variations are enough to hose the results, the system is marginal.

The systems I trade are insensitive to that, because a signal which fails to appear in backtesting will be compensated for by a signal which failed to appear in real time. I simply go to IB and demand that they give me the missing real-time trade.

What I am speaking of specifically is that there are systems that backtest poorly and do well in realtime. Any system can be made to work with money management, some work better in real time than with backtesting.

There may be people out there throwing away good systems because of a marginal backtest.

I now advocate testing all systems in real time to falsify the loss as well as the gain.

Of course, this cost some money. But the results may be well worth it.

Regards
Oddi

hypostomus · Dec 5, 2003

...I am having trouble understanding how that could be the case. Without getting proprietary, can you give an example of how there could be a real-time element which is not encompassed by the backtesting, but which makes a trade? I invariably find that the reverse is true, that the real-time trading is worse than the backtesting, even accounting for experienced slippage in market orders.

OddTrader · Dec 5, 2003

Quote from oddiduro:

I have run one proven money maker, and several experimental tests side by side with a backtester. Signal in real time are often different than signals in the backtest. Once such system is a net loser in the backtester, while profitable in real time.

2 cents:

Would it be possible that the actual orders placed while real-time trading could Not be all-time and everytime filled (without failure) immediately upon every signal generated by the system(s) (therefore skipping some losing trades, possibly in a systematic way)?!

OddTrader · Dec 5, 2003

Quote from oddiduro:

I now advocate testing all systems in real time to falsify the loss as well as the gain.

Of course, this cost some money. But the results may be well worth it.

Obviously, you must be joking (in order to collect feedback

?)!

oddiduro · Dec 6, 2003

Quote from hypostomus:

...I am having trouble understanding how that could be the case. Without getting proprietary, can you give an example of how there could be a real-time element which is not encompassed by the backtesting, but which makes a trade? I invariably find that the reverse is true, that the real-time trading is worse than the backtesting, even accounting for experienced slippage in market orders.

Easily.

Any system that requires entry on the immediate appearance of a signal, rather than at the end of a bar. Obviously, the longer the time frame, the worse the effect. All oversold/ overbought crossing of an oscillator will not be acted on in a backtest until the end of the bar.

The are some signals that occur in real time that are not seen in delayed feeds.

hypostomus · Dec 6, 2003

...thanks for explaining. I think I understand what you mean. I do all my backtesting so that the entry/exit decisions are made at the end of the current bar, and trade with market orders at the open of the next bar. I once modified one of my profitable codes for mid-bar decisions and entries/exits using limit orders, and got better results, assuming that the limit orders would be filled.

However, if I am understanding you correctly, the discrepancy between the backtested performance of a system and its real-time performance with mid-bar entries could be resolved by coding and testing mid-bar entries and exits, trying as best you can to write algorithms for where your intuition/experience tell you to trade.

Also, can you elaborate on signals that appear in real time that don't occur in the data feed? If you mean those due to one tick variations due to data base time differences, then I understand.

Apologies if I come across as dense and argumentative (it's hard-wired in). I always enjoy exchanging messages with you because I learn more from you than you learn from me!