Backtesting: My trading strategy works too well

[Victory5]

Too*


Seems like a weird problem to have but here is a brief summary of where I am at.

I have been trading for about 3 years now. I recently got tired of losing on emotional, baseless trading (blew out multiple accounts) and decided to put pen to paper and organize a few of my trading systems I had kicking around in my head.

I am currently finishing up a swing trading system that leverages a very basic technical pattern with a few option plays. I went out a got historical option data for as far back as I could and started to manually run through the numbers. The signal I use hits about once every 5 weeks and there have been 75 entries since 1/09. Most importantly, I did not cherry pick my data when backtesting. I came in with a set idea and although I changed around a few specifics with my option strikes for example, I did not try to make random rules to isolate the best results and enhance my returns.

Now, I have data back to 2009 (I am using VXX, the ETN was issued first in January of '09) so its only 6 years, but over that time period if you put 3k into the system, you would have somewhere between 100-300k today based on how aggressive you were reinvesting profits. Furthermore, there appears to be little correlation between market performance and my strategies performance. For example, 2011 was my 2nd year when backtesting, but this was obviously the worst year for the market since 09. At the same time, 2013 was the markets best year, but also my worst.

I have gone through and double checked my math. I have actual option data (not guessing vol, etc). I have accounted for commission using Interactive Brokers pricing, which I plan to use, as well.

Obviously slippage may come into play, but what else am I doing wrong? I have faith in my strategies, but it seems like all of this is too good to be true? I know backtesting does not guarantee future profits/results so does anyone else have any experience with things seeing 'too good to be true'?

 

[i am nobody]

Too*


Seems like a weird problem to have but here is a brief summary of where I am at.

I have been trading for about 3 years now. I recently got tired of losing on emotional, baseless trading (blew out multiple accounts) and decided to put pen to paper and organize a few of my trading systems I had kicking around in my head.

I am currently finishing up a swing trading system that leverages a very basic technical pattern with a few option plays. I went out a got historical option data for as far back as I could and started to manually run through the numbers. The signal I use hits about once every 5 weeks and there have been 75 entries since 1/09. Most importantly, I did not cherry pick my data when backtesting. I came in with a set idea and although I changed around a few specifics with my option strikes for example, I did not try to make random rules to isolate the best results and enhance my returns.

Now, I have data back to 2009 (I am using VXX, the ETN was issued first in January of '09) so its only 6 years, but over that time period if you put 3k into the system, you would have somewhere between 100-300k today based on how aggressive you were reinvesting profits. Furthermore, there appears to be little correlation between market performance and my strategies performance. For example, 2011 was my 2nd year when backtesting, but this was obviously the worst year for the market since 09. At the same time, 2013 was the markets best year, but also my worst.

I have gone through and double checked my math. I have actual option data (not guessing vol, etc). I have accounted for commission using Interactive Brokers pricing, which I plan to use, as well.

Obviously slippage may come into play, but what else am I doing wrong? I have faith in my strategies, but it seems like all of this is too good to be true? I know backtesting does not guarantee future profits/results so does anyone else have any experience with things seeing 'too good to be true'?

Be careful with back testing. There can be different problems. You might be the cause, but also the software you use can be the cause. I have already found errors in the way a very well know software package did back test. Backtest were very good, almost no losing trades, and in reality a lot of losing trades. The software made errors in the interpretation of the rules.
The more complex your system, the bigger the risc that you made an error in programming the entry and/or exit rules.

Maybe do a check by backtesting manually some trades and compare the results with the automated backtesting.

 

[Victory5]

I backtested all of the trades manually in excel.

 

[AAAintheBeltway]

Too good to be true usually is exactly that. It's impossible to offer suggestions without knowing the logic and backtesing methodology, but in general, curve-fitting is the culprit. You somehow incorporated parameters to capture random events that will not occur in the future. The other possibility is faulty logic, eg using todays' close to trigger a buy on close. Obviously, slippage and limitations on something like shorting coudl be factors as well.

There is a formula, which unfortunately I have forgotten, that estimates the probablility of curve-fitting in a system. I do recall that the most important factor was degrees of freedom in the system, ie how many parameters you have. More than a couple and you can be on thin ice.

Obviously, the only real test is to apply it on a walk forward basis, ie test on data not used in the system development.. Or trade real money with it and see what happens.

 

[sprstpd]

Easiest way to figure out if there is an issue with your backtesting is to start trading it for real with small amounts of money and see if the numbers line up. If they don't, you'll know in a hurry.

 

[d08]

What about the spread? Did you buy on last or ask? That's the killer when it comes to most options strategies.

 

[Victory5]

What about the spread? Did you buy on last or ask? That's the killer when it comes to most options strategies.

Well VXX options are pretty liquid and there is always good volume on the strikes I am planning on buying (part of the adjustment I made was due to the lack of volume on some of the deep ITM puts/calls having next to no volume).

I decided on going with the last rather than the ask.

Im buying at the close, the ask price given from the dataset I have, like any ask you could find online is the last quote of the day, which includes after hours. So if the underlying moves, even a bit after hours, the lack of volume and wide spreads will not actually represent the option price at close.

Even if volume is low, the higher volume around the close makes it more likely that the last price was sometime around 4, even though its not perfect, it seemed to be the best way to go about things.

I do admit there is likely some slippage with either course you follow and I tried to account for that by lowballing things if I was given the chance too, but I would agree that the point you raised is probably the most significant issue facing me atm.

 

[Victory5]

Too good to be true usually is exactly that. It's impossible to offer suggestions without knowing the logic and backtesing methodology, but in general, curve-fitting is the culprit. You somehow incorporated parameters to capture random events that will not occur in the future. The other possibility is faulty logic, eg using todays' close to trigger a buy on close. Obviously, slippage and limitations on something like shorting coudl be factors as well.

There is a formula, which unfortunately I have forgotten, that estimates the probablility of curve-fitting in a system. I do recall that the most important factor was degrees of freedom in the system, ie how many parameters you have. More than a couple and you can be on thin ice.

Obviously, the only real test is to apply it on a walk forward basis, ie test on data not used in the system development.. Or trade real money with it and see what happens.

Thanks for the reply, agree 100% with your points. If you do find that formula let me know, it would obviously be helpful to see if I did fall victim to the processes you are describing.

 

[aqtrader]

Be careful with back testing. There can be different problems. You might be the cause, but also the software you use can be the cause. I have already found errors in the way a very well know software package did back test. Backtest were very good, almost no losing trades, and in reality a lot of losing trades. The software made errors in the interpretation of the rules.
The more complex your system, the bigger the risc that you made an error in programming the entry and/or exit rules.

Maybe do a check by backtesting manually some trades and compare the results with the automated backtesting.

Actually, there are many traps in the back-testing. historic data, time frames, time ranges, data quality, subjective selection of markets, stocks, and errors in using historical data, software problem, etc, these are all problems that causes often too-good-to-be-true results.

 

[Victory5]

In regards to curve fitting, I chose my entry and exit points based on an underlying philosophy and strategy before I took a look at the charts and before I started to backtest. I did make adjustments to some of the option specifics as I mentioned above, but I did not change the underlying strategy at all.