Back Testing Survivor Bias
- Thread starter joshdy186
- Start date
Interesting question - when I saw the title I assumed you might mean survivorship bias among *strategies* that you were backtesting.
As for survivorship bias in the data you're using, I would have thought that the use of out of sample data at the end of any optimisation or selection procedure would be the natural solution.
Feel free to expand on your thinking in more detail
As for survivorship bias in the data you're using, I would have thought that the use of out of sample data at the end of any optimisation or selection procedure would be the natural solution.
Feel free to expand on your thinking in more detail
How does "survivor bias" apply to a universe of stocks being studied?
In my experience (about 7 years of testing and trading) survivorship bias in short term trading systems is not that big of a deal.
In the stocks I track and trade, most will go through quite a few cycles of up and down trends, before they finally succumb and get delisted.
Yes, the final dive to below my low buy point (about $6) may cause a largish loss, but most of the time I have already banked many profitable trades on that stock. So the net is overall profits.
At least that is the way it has worked for me. Others may differ.
Good luck to all.
In the stocks I track and trade, most will go through quite a few cycles of up and down trends, before they finally succumb and get delisted.
Yes, the final dive to below my low buy point (about $6) may cause a largish loss, but most of the time I have already banked many profitable trades on that stock. So the net is overall profits.
At least that is the way it has worked for me. Others may differ.
Good luck to all.
Agreed,
survivor bias is more of a problem when you are dealing with trading the IBD100 or the components of QQQQ.
Unless you've tracked those memberships over time, you'll run into a bias big time because you are backtesting on only the current list.
Testing against a wide range of stocks will most likely reduce survivor bias to a reasonable level.
-Mike
survivor bias is more of a problem when you are dealing with trading the IBD100 or the components of QQQQ.
Unless you've tracked those memberships over time, you'll run into a bias big time because you are backtesting on only the current list.
Testing against a wide range of stocks will most likely reduce survivor bias to a reasonable level.
-Mike
I think it's useful to distinguish two problems related to your question:
1. Getting historical data that includes delisted securities
2. Developing a backtesting strategy that reveals to what extent your back testing results are suffering from survivorship bias.
For the first part, you might want to look into data offered by Tradestation. They claim "you now have access to history on all stocks, including those stocks that no longer trade, giving you the ability to test strategies on a larger number of stocks".
1. Getting historical data that includes delisted securities
2. Developing a backtesting strategy that reveals to what extent your back testing results are suffering from survivorship bias.
For the first part, you might want to look into data offered by Tradestation. They claim "you now have access to history on all stocks, including those stocks that no longer trade, giving you the ability to test strategies on a larger number of stocks".
Doesn't seem like a very good solution - buyouts, mergers, or failure to comply with various exchange rules (e.g., minimum liquidity) result in a lot of delistings while the stock trades over $10 or $20. And then, of course, such a rule doesn't account for the cases where a stock goes from $20 to $4 and is delisted in one day on some very bad news (the "surprise delisting bias" would still be present.)
Also, back during the dot-com blowup, there were plenty of stocks that were kicked to the pink sheets while trading over $10. Simply sticking to higher priced stocks is certainly no guarantee that your stock won't get delisted as you're trading it!
I suspect selecting only stocks with higher share prices might even actually bias towards selecting stocks that were delisted in backtesting - consider that many troubled companies, in a desperate attempt to avoid getting booted, implement a reverse stock split, adjusting their share price back up before the end comes.
Another thing to consider is the fact that the average and the median share price have changed over the years. Back in the 1980s, fewer companies (proportionally) listed on the NYSE traded below $10, yet many (most) of them went on to be delisted for one reason or another. If you were talking about OTC stocks, on the other hand, a much larger proportion were trading below $10 a share back then than today. So any fixed threshold is likely to get less accurate the further back you test.
And of course, to implement any check based on the share price, you'd need a comprehensive database of historical stock splits to get the correct adjusted price. That probably won't be any easier to compile than a complete database of the actual share prices with the delisted stocks included, frankly.