Backtesting vs Simulation vs Actual Trading

This system is *almost* fully automated. Almost meaning:

The software does the whole analysis, then propose some trades long or short and how much money to put into each trade. And present a dialog box with 2 buttons (Send, Cancel) and some textbox where I can change how much money to put into the trade, stop/limit... Usually I don't change anything and all I have to do is press "Send" and that's it, the software fires off the trade to IB.

That is what I am assuming in simulated results.

However, in real life I skim over the news for each trade proposed to see if there are any major news. If there are no major news I place my trades, if there are some major news I might or might not follow the recomendation.

Of course a lot of other factors get into play (loosing/winning streaks, bug in my software, ...) and can influence me. And I might or might not follow the system to the letter. In general, I follow it to the letter 95% of the times.

But still because of the slippage, busted trades (happened twice), partial/no fill... The results differ greately from the simulation results.
 
Hi All,
I know something about 'Backtesting vs Simulation vs Actual Trading'.

Nobody will dispute the soundness of keeping (sufficiently or superiorly) profitable trading strategies a secret.

Now, developing such an admissible strategy depends mostly on Backtesting & Simulation methodologies. Nay, Backtesting & Simulation methodologies often grow and originate together with a trading strategy under development.

Don't expect to find in any place:
(1) Actual Trading strategies (that work);
(2) Backtesting methods useful for (1);
(3) Simulation methods useful for the above.

If you are not (yet) convinced about these truths, you will still keep on wasting a lot of time and probably spending a lot of money at 'mathematical-savvy' or 'computer-savvy' sounding snake oil vendors.

nononsense
 
nononsense,

I agree totally with your statements, but I don't think it really fits in this thread.

This is about:
If somebody develops a strategy, backtest it, then simulate it... What should he expect once he starts trading it with real money.

This thread is not about:
Should somebody buy a backtested , simulated strategy?

Anyhow, I agree that if somebody thinks a strategy is good, it shouldn't be sold, but traded instead. But for other people that want to post on this thread, please keep it to the following subject:

If somebody develops its own strategy, backtest it, then simulate it... What should he expect once he starts trading it with real money.
 
You have listed at least four reasons why real trading may differ from backtesting/simulated trading:

1. “ If there are no major news I place my trades, if there are some major news I might or might not follow the recomendation.”

2. “Of course a lot of other factors get into play (loosing/winning streaks, bug in my software, ...) and can influence me. And I might or might not follow the system to the letter. In general, I follow it to the letter 95% of the times.”

3. “. . . slippage . . .”

4. “. . . busted trades (happened twice), partial/no fill . . .”

If you are actually trading the system in a discretionary manner it is not possible to backtest or simulate those factors (including fear and greed) that goes into your decision making process; additionally you may be introducing lag into the process resulting in additional slippage. If you have not used realistic slippage your backtesting will look far better than what is actually possible. Finally, using limit orders can also create unrealistic expectations: there may have been only one trade at your limit price during the backtest, and in the real world the odds are that it’s not going to be your trade.

Bottom line: in live trading the system differs from the backtesting because you have introduced a human into the process (with emotions and lag) and you have not fully accounted for slippage and limit orders.
 
Quote from StocksSniper:

nononsense,

I agree totally with your statements, but I don't think it really fits in this thread.

This is about:
If somebody develops a strategy, backtest it, then simulate it... What should he expect once he starts trading it with real money.

This thread is not about:
Should somebody buy a backtested , simulated strategy?

...
More...
Hi StocksSniper,

Simply trying to squeeze something of value into your thread.
You claim to "totally agree with my statements", but the above makes me seriously doubt as to whether you understand my point.
 
Quote from WolfVector:

You have listed at least four reasons why real trading may differ from backtesting/simulated trading:

1. “ If there are no major news I place my trades, if there are some major news I might or might not follow the recomendation.”

2. “Of course a lot of other factors get into play (loosing/winning streaks, bug in my software, ...) and can influence me. And I might or might not follow the system to the letter. In general, I follow it to the letter 95% of the times.”

3. “. . . slippage . . .”

4. “. . . busted trades (happened twice), partial/no fill . . .”

Bottom line: in live trading the system differs from the backtesting because you have introduced a human into the process (with emotions and lag) and you have not fully accounted for slippage and limit orders.
More...

There is also a 6th point..
Backtesting and simulation are consider to be close systems compare to live trading where it's an open system. In backtesting/close system, your trade entry is just overlaying on the market without considering your trade entry....where as in live trading once you enter in a trade, you are introducing 'chaos' into the system and utlimately screws up the market movement. So you are sorta like "the butterfly" who flaps its wings in Hong Kong which then cause a series of natural events to later created a hurricane Katrina in the USA. (Simple chaos theory example). So without the butterfly, there might not be hurrican Katrina... aka...with your entry, the market might not do what it was suppose to do..and vice versa.

So no matter if you made the live trading execution to perfection without slips and everything, it will still be less than the backtesting results. This is especially true if backtesting less than 15 minutes timeframe.
 
One of the biggest problems with backtesting is that simulation orders can get filled at prices that do not exist in the time and sales listing but fall within the daily range even though they are not valid tick values. Many of the backtesting programs show them as fills because the value happens to be in the daily range.

Another issue to address is limit orders. For example, a buy order that falls on the low of the day or a sell order that falls on the high of the day. In reality, your chances of a fill in either situation would be nil in most cases it is so rare to actually get the high or low fill (unless you are on the other side of the trade).

Do any backtesting platforms address these issues.
 
bwc,

Very well said.

Did anybody try to backtest a system introducing some "chaos" variables?

I think it will be a very good subject to study. And compare Backtesting, Simulation, Backtesting with some sort of chaos variables, and then Real trading.

I would work on it if I knew how to mathematize/formalize some "chaos" variable, but I have no idea how to do it and if it is possible at all.

Any suggestions are appreciated.

Thanks.
 
agtrader777,

The first issue doesn't happen if you use limit orders, because then you will probably be filled at a better price than the simulated.

But then you fall into the second issue.

A way I found to mitigate these problems is to use limit orders in simulation, but then assume that I won't get filled unless the price goes x% above/below my limit.

This wouldn't eliminate completely the no-fill, partial-fill issue, but might help giving a more accurate result.

The bigger the x%, the more reliable the backtesting will be.
 
You must log in or register to reply here.
Back
Top