Why is my backtest overly optimistic?

[guru]

Also, think about what happens when one person places a limit order on open (LOO) while you place MOO order. Possibly without your order the opening price would be different. Basically you may be worsening the opening price by buying without caring for that price, while without you the opening price could be different. (though that could happen in live trading, not paper)

 

[nijshar28]

I’ve had similar issues years ago when I first started backtesting similar strategies. Don’t remember details, as I’ve had to solve a lot of problems over time, but I suspect that you’re using historical data that is based on the Open prices that aren’t really relating to MOO orders for those illiquid stocks. Due to low volume those stocks could open 1+ minutes after market opens, and be traded at limit prices/orders, while you get screwed by using market orders. The official opening prices can be based on the first trade, after market opens.
Basically your assumptions may be wrong.

Secondly, it could be due to overfitting, which plagues every trading strategy. That always happens when you tweak your strategy until you get positive results. Though indeed may not happen if you simply try to emulate MOO orders, in which case you’d face other issues like above.

You might be onto something. Generally, I see better results on the backtest when I use MOO as opposed to MOC. MOC results are also highly unrealistic though. So I feel something else must be going on.

Just out of curiosity what do you trade now? Futures?

 

[guru]

You might be onto something. Generally, I see better results on the backtest when I use MOO as opposed to MOC. MOC results are also highly unrealistic though. So I feel something else must be going on.

Just out of curiosity what do you trade now? Futures?

I stream paper penny stock trades on Twitter, and was trading some of them live until recently, but now I’m focusing on options as they are more scalable. Never traded futures.

 

[nijshar28]

When you BUY, you should using the ASK price and when you SELL you should be using the BID and do not assume volume > than the bid/ask size, correct? (your reply on slippage seems off). Since you state you are using MOO and MOC the price you assume should attempt to reflect that. Note: You stated these are illiquid products! --Your comments seem to imply you are not addressing this properly.

Not sure I understand.

In my backtest I use either the Open, or the Close trade price I have for that day. I don't have Bid / Ask historical data.

In my backtest, I assume that my order (which is <= 1% of prior days volume) will be filled at those prices.

In my forwardtest, I just send market orders (also limited to 1% of prior days volume) around the opening/closing and get simulated fills at the top of the book Bid / Ask prices around the opening/closing times.

The paper trading simulator is handled by the broker, so I do not have full control of what they do. But I benchmarked the fill prices against the opening/closing trade prices for that day and generally they are not too far off. From what I understand, the broker does not model market impact, but if the volume is insufficient my orders do not get filled / filled fully.

 

[taowave]

How many BP's (per trade) in slippage "destroys" the system?

Hey guys. I have been backtesting and paper-trading my strategy for a few months now. I can't help to notice that my backtest looks more optimistic than my paper trading results.

Even before I started paper trading, I suspected that something is off about my backtest as it exhibits double digit Sharpe ratios, virtually no downside risk, and returns that are frankly unrealistic.

I tried running some tests, like introducing programming assertions that look for lookahead bias and interchanging blocks of code between my backtesting and forward testing software. Even though I flushed out some minor issues this way, unfortunately, I still cannot pinpoint the main problem.

Has someone encountered this before? I feel that any advice would be helpful at this point. Thank you.

 

[nijshar28]

How many BP's (per trade) in slippage "destroys" the system?

I would have to get back to you on that. But if I remember correctly adding .002 was significant (not sure about destroys through). Is there a rule of thumb here?

 

[Metamega]

Wouldn’t the simple process be to take your paper trading/forward test results and backtest over the period of the forward test and compare trades/executions?

 

[nijshar28]

Wouldn’t the simple process be to take your paper trading/forward test results and backtest over the period of the forward test and compare trades/executions?

Yes! That's what I have been attempting for the past couple of weeks. It was somewhat informative, and I uncovered a few minor issues this way but I didn't quite get to an 'Aha!' moment.

If I substitute the execution prices and shortlists from my forward test to my backtest, I get something that is very close to my forward test (I believe the remaining difference is due to the interest I pay on borrowed stocks, which I don't simulate in my backtest yet).

The problem is that I don't have a lot of data collected from my forward test (about 3-4 weeks).

I am not sure if it is my strategy that is underperforming in the past few weeks due to unique market conditions, or there's some elaborate bias going on and everything I get from my backtest is in fact garbage. And my actual strategy doesn't work, which is kind of what I see in my forward test.

I know my writing is confusing. Sorry about that. But then again it has been a confusing journey

 

[SunTrader]

Open and closing periods are the most volatile, excluding intra day news event generated moves.

Assumptions are made using opening/closing prices which mostly don't hold true.

The only real test to trust is real trading.

 

[taowave]

How does 20 Bps relate to avg Bid/Offer spread??

I would have to get back to you on that. But if I remember correctly adding .002 was significant (not sure about destroys through). Is there a rule of thumb here?