Backtesting vs. Realtime results

J

jonnyman

I've been reading these forums for over a year now and I've recently begun development of an automated trading system. It's going well, but I was just curious to those who have systems running:

What was the accuracy of the backtesting results vs the realtime numbers?

Was there any reliable standard that existed? i.e. if your backtesting makes $x then real time it made 10% of $x

Or were they completely off.

Just curious to see the difference, it'd be helpful if some people experienced in this could share their thoughts.

I'm not looking for specific numbers or strategies, just generalizations (unless you want to be specific).

Thanks - I've had fun perusing these forums for a long time.
 
My personal experience is the following : there is no answer to your question, sorry. It really depends. I've seen systems with a higher return in live trading than in backtested trading and the contrary too.

I would take 75% as a low. Under this, your system might be in trouble. But as I say, there is no rule.
 
I think science has it about right.

At least with the stuff I trade, you can consider that you have two systems - a backtest system, and the real system.

Yeah they may have the same rules, but data errors and problems, bad/no fills and poor/no liquidity right at your buy/sell levels, goofy spreads, busted trades, funky pre/post market action, ISP problems, broker problems, exchange problems, halted stocks, code logic and programming errors, computer problems, survivorship bias (in stock data), screwups and mis-guided system overrides, etc, etc, make the real world challenging!
 
Heh, yeah guys that's exactly why I'm asking. I'm not looking for THE answer, just what some automated traders found when they moved from backtesting to live trading.

If 5 or 6 people posted what the difference was profit wise I think it would be interesting.

I'm hoping just to get an idea of the amount of profits lost, so I could see a minimum profit needed in backtesting before even ATTEMPTING to put it live.

I'm hoping that the system will generate 10%-15% of the profits the backtest claims to make.
 
JOHNNY................ personally i will answer it two ways...........for position trading systems i have had very good reliable results ..........but for intraday scalping systems i have found reproducing backtested results totally unreliable.
 
i think it is essential to look at the specific features of a system plus the process of system finding to judge the component of fitting.
we took a year of data and drew several hundred random trades. the chance to see a modified sharpe ratio of above 1 is 3%. thus, if you search for systems by "merely" making run after run you would by pure chance "detect" systems. no need to say that their expectancy is 0 - tradingCost.

(fair to say that the comparison between random entries and an algorithm which produces trades are two different stories. i would think that it is more difficult to find a random algorithm which produces a sharpe of above 1 for several hudnred trades.)

some people try to capture the problem by analysing the statistics of the market as such. "edge" is the common word for that.

IMHO the discussion on this lacks the most important aspect: the way of testing. two testers having two different paradigms will contradict. peaches and apples. that kind of thing.

out of sample testing helps. walk forward optimisation helps. higher number of trades helps. robustness in different market regimes helps. similar performance in different markets helps. papertrading helps. being critical to one's ego helps :).
 
Depends what the system is trading and how you've tested it. Liquidity and fills are a problem, as is survivorship bias in backtesting. Without more information about your system (ie. what you're trading, time frame and avg win) it's difficult to offer much insight as to the extent to which they will be different.
 
Thanks for the input.

What I'm really asking though is what YOUR experience has been.
I don't expect you to analyze what MY system would do. I wouldn't presume to expect that anyone could tell me what would happen in my specific circumstances.

This is more a discussion than me asking for help.

Anyone who has automated systems that trade NYSE and NASDAQ care to share experiences?

Thanks for the posts so far!
 
I cant seem to find any discussion on Sharpe Ratios in the forums, so I'll add my question here-

I have a low Sharpe Ratio system of .4, which returns 30% annually without leverage.

Im told that good systems have ratios of 1, 2, even 4.

Is there something wrong with a system with a low ratio?

I dont know exactly how it generates this value, as my software outputs the results.

Thanks
 
You must log in or register to reply here.
Back
Top