Why is my backtest overly optimistic?

[nijshar28]

Hey guys. I have been backtesting and paper-trading my strategy for a few months now. I can't help to notice that my backtest looks more optimistic than my paper trading results.

Even before I started paper trading, I suspected that something is off about my backtest as it exhibits double digit Sharpe ratios, virtually no downside risk, and returns that are frankly unrealistic.

I tried running some tests, like introducing programming assertions that look for lookahead bias and interchanging blocks of code between my backtesting and forward testing software. Even though I flushed out some minor issues this way, unfortunately, I still cannot pinpoint the main problem.

Has someone encountered this before? I feel that any advice would be helpful at this point. Thank you.

 

[stepandfetchit]

Hey guys. I have been backtesting and paper-trading my strategy for a few months now. I can't help to notice that my backtest looks more optimistic than my paper trading results.

Even before I started paper trading, I suspected that something is off about my backtest as it exhibits double digit Sharpe ratios, virtually no downside risk, and returns that are frankly unrealistic.

I tried running some tests, like introducing programming assertions that look for lookahead bias and interchanging blocks of code between my backtesting and forward testing software. Even though I flushed out some minor issues this way, unfortunately, I still cannot pinpoint the main problem.

Has someone encountered this before? I feel that any advice would be helpful at this point. Thank you.

If you provide some details, it is possible flaws can be identified, however without specific information, we can only speculate. 1st speculation: What product are you trading and are you using a reasonable slippage?

 

[Snuskpelle]

Most common bugs (as per my experience):

• Lookahead bias (any hints about the future will generally be snapped up by machine learning algorithms to make "predictions". This can happen in very subtle ways.
• Failure to model slippage. A huge range of algorithms will appear profitable without slippage, e.g. it can manifest as "trading" the bid-ask spread with a very simple mean reversion algo.
• Failure to model fills (impact and latency). Market orders of size can't be filled without increasing slippage. Limit orders are much worse, and will be filled at other market participants' discretion (adverse selection). I've stopped bothering with limit orders for this reason.
• Failure to model commissions. Depending on your platform this can also cause issues (e.g. likely on IB). Thankfully, this is an easier problem to solve than the others since it can be negotiated, so you shouldn't discard an algo just because IB would charge you too much.

 

[Tradex]

Most common bugs (as per my experience):

• Lookahead bias (any hints about the future will generally be snapped up by machine learning algorithms to make "predictions". This can happen in very subtle ways.
• Failure to model slippage. A huge range of algorithms will appear profitable without slippage, e.g. it can manifest as "trading" the bid-ask spread with a very simple mean reversion algo.
• Failure to model fills (impact and latency). Market orders of size can't be filled without increasing slippage. Limit orders are much worse, and will be filled at other market participants' discretion (adverse selection). I've stopped bothering with limit orders for this reason.
• Failure to model commissions. Depending on your platform this can also cause issues (e.g. likely on IB). Thankfully, this is an easier problem to solve than the others since it can be negotiated, so you shouldn't discard an algo just because IB would charge you too much.

Excellent post.

 

[Tradex]

Hey guys. I have been backtesting and paper-trading my strategy for a few months now.

Did you code it with C++, Python or other similar computer languages?

I suggest you compare the backtest results with a "manual" backtest of each completed trade, this will allow you to find the error quickly.

 

[nijshar28]

Thank you guys. I appreciate your input.

Here's some additional info about my system:

I trade stocks, generally small / micro, illiquid names, hundreds of them. Long / short.

I am using market orders and either the open, or closing prices to simulate trades. In my paper trading, I use simulated MOO, or MOC orders.

In order to limit slippage, I limit my trades to 1-2% of prior day's volume. I think that since I am trading on the open, or the close, the slippage should be less relevant. Though I understand that I can still impact the price, or not get filled if the stock is very thinly traded that day (which happens occasionally in my paper-trading simulator btw).

The paper simulator uses the top of the book to simulate trades, so market depth should not factor in the discrepancy I observe between my backtest and paper trading simulator.

Sometimes the stocks I want to short are non-shortable, but generally it is a small fraction.

My system is coded in Python.

 

[taowave]

When you say backtest,are there optomized parameters that held up during WFA?

What's your holding period..?
I am guessing hours at most

How long was

 

[nijshar28]

When you say backtest,are there optomized parameters that held up during WFA?

What's your holding period..?
I am guessing hours at most

How long was

Yes. I had to look up what WFA stands for but that is exactly how I backtest.

I trade daily. So my holding period is at least 1 day.

 

[guru]

Thank you guys. I appreciate your input.

Here's some additional info about my system:

I trade stocks, generally small / micro, illiquid names, hundreds of them. Long / short.

I am using market orders and either the open, or closing prices to simulate trades. In my paper trading, I use simulated MOO, or MOC orders.

In order to limit slippage, I limit my trades to 1-2% of prior day's volume. I think that since I am trading on the open, or the close, the slippage should be less relevant. Though I understand that I can still impact the price, or not get filled if the stock is very thinly traded that day (which happens occasionally in my paper-trading simulator btw).

The paper simulator uses the top of the book to simulate trades, so market depth should not factor in the discrepancy I observe between my backtest and paper trading simulator.

Sometimes the stocks I want to short are non-shortable, but generally it is a small fraction.

My system is coded in Python.

I’ve had similar issues years ago when I first started backtesting similar strategies. Don’t remember details, as I’ve had to solve a lot of problems over time, but I suspect that you’re using historical data that is based on the Open prices that aren’t really relating to MOO orders for those illiquid stocks. Due to low volume those stocks could open 1+ minutes after market opens, and be traded at limit prices/orders, while you get screwed by using market orders. The official opening prices can be based on the first trade, after market opens.
Basically your assumptions may be wrong.

Secondly, it could be due to overfitting, which plagues every trading strategy. That always happens when you tweak your strategy until you get positive results. Though indeed may not happen if you simply try to emulate MOO orders, in which case you’d face other issues like above.

 

[stepandfetchit]

When you BUY, you should using the ASK price and when you SELL you should be using the BID and do not assume volume > than the bid/ask size, correct? (your reply on slippage seems off). Since you state you are using MOO and MOC the price you assume should attempt to reflect that. Note: You stated these are illiquid products! --Your comments seem to imply you are not addressing this properly.