Zacks Backtesting Yields Over %50

Zack's Research Wizard suffers from many data pitfalls. The worst of all being survior bias. In other words stocks that go bankrupt or get bought out no longer exist in there database so this skews backtests as being more favorable in many cases than what would have be acheived in real time. Also they only have slpit adjusted prices not prices as they were in real time. So any screen with a price component is useless. This was the case a year ago and as far as I know still is. I also found several data errors when I was using them and they were never able to give me an explanation. Good marketing, terrible data. There are much better historical databases out here for a lot less.
 
Bignatty is correct. Research wizard connects to production databases while TSE connects to backtesting databases. the backtesting databases are split adjusted and have been adjusted for survirorship. they used to offer an internet based version of tse that you could get on trial for a few weeks. you may want to try that.
 
I talked to people in ZACKS and I even forwarded the postings from other forums stating that there is survivorship bias in their system. They said there used to be but not anymore.

So I took a look at my database randomly and I was not able to find any survivorship bias. Here is what I did;

Survivorship bias means the tendency for failed companies to be excluded from performance studies due to the fact that they no longer exist. Well when I checked my historical data I found many many tickers that do not exist anymore. They existed many years ago but no longer. This proves that the failed, purchased, bankrupt, merged companies are not taken away from the historical database, they are still there.

Secondly,I have screens that yields more than 50% with 1 stock every week, buying Monday selling Friday and winning ratio is over 80%. So far no problems.

I must be honest, in the beginning which is June 2007, I found many data missing or wrong or N/A, so after constant communication with ZACKS team, they corrected all issues. It took me many months to reslove all these issues but they were helpful. At this point when I run any historical data, I am not catching any errors or N/A s anymore.

Also someone said "Also they only have slpit adjusted prices not prices as they were in real time". I do not understand why this is a problem if I am buying and selling in the same week.

Please comment.
 
Not having split adjusted prices is a problem in backtesting because if for example you have price as a screen factor i.e. price>5 then the stocks picked by the backtester will not be the same as they would have been in real time because the prices have been split adjusted.

Also to check the survior bias see if ENRON is still in the database.
 
I agree with a previous poster who said you need to track the screens realtime. That's the only way you know if the lists that Zacks spits out are aligned with the backtesting. One specific problem I found with Zacks data was with earnings surprises. I had a long strategy that looked at the 4wk performance of companies after they had positive earnings surprises, and I had unbelievable results. Only after I started tracking it did I realize that the Zacks backtester was including stocks that hadn't had their earnings come out yet. For example, if the backtester was creating the list for the first week of February, it was incorporating positive earnings surprises that came out for the entire month of February (I think the specific field was "earnings surprises in the last month" or something like that). So, of course if the backtester picks a company that it already knows will have a positive earnings surprise then the returns will be huge.

As a result, I couldn't trust any strategy that relied on data that had unexpected timing. That including any earnings data or analyst upgrades/downgrades.
 
This is something like the reason I dumped Zacks in the first place. I remember their vaunted "earnings surprises" didn't work out as they projected. And "earnings surprises" were supposed to be their competitive advantage. So much for Zachs. That and the fact they had to use salespeople.

But if it works out let us know. Maybe they've worked out all the kinks. :D

Quote from HotTip:


As a result, I couldn't trust any strategy that relied on data that had unexpected timing. That including any earnings data or analyst upgrades/downgrades.
More...
 
I use the screener knowing full well that it has survivorship bias and I've always suspected that it had other data problems too so I'm not surprised by the finding that it includes earnings surprises that were not yet reported. That's pretty damning. That's "look ahead bias".

Does anybody use earnings estimate revisions? The brokers who supply that data come and go and some have decided to tighten up dissemination of that data.

Tragically, Reuters discontinued its premium screening product. It was outstanding in its detail and functionality. They refunded me and apologized that customer uptake was too low (I suspect it was cannabalizing their institutional product).

I agree with other posters here that you have to walk forward these strategies in real time. Invariably it doesn't work as well as backtested results.

Finally, several quant hedge funds were doing long-short strategies that are related to the type of screens that Zacks Screener does. OF COURSE these guys are using teams of quants, computing power and institutional grade data. But despite that, their edge got shellacked in August and again in January. Look at AQR capital and others. Cliff Asness founded AQR and holds a Phd from the University of Chicago. He was a quant at Goldman too. I'm noticing other hedgies that are like AQR are geting slammed too. So FYI
 
You must log in or register to reply here.
Back
Top