https://fivethirtyeight.com/features/worst-pollster-in-world-strikes-again/
The Worst Pollster in the World Strikes Again
By Nate Silver
Dirty little secret: I sometimes write material in the late evening that will be posted the following morning. This is one of those instances. The early word, however, is that a Zogby poll to be released today will show Barack Obama’s presidential approval numbers at around 50-50.
As of this writing, the Pollster.com averagehas Barack Obama’s numbers at 59.3 percent approve and 33.8 percent disapprove; the Real Clear Politics averageis slightly more favorable to Obama, at 61.2 percent approve and 30.5 percent disapprove.
So a Zogby poll that put Obama’s numbers at roughly 50-50 would be a significant outlier. Outliers are nothing new, however, when it comes to Zogby polls. They are, in fact, the rule and not the exception.
Let me qualify this a bit: Zogby International conducts two types of polls. One type are conventional telephone polls. Zogby’s telephone polls, while prone to somewhat wild fluctuations and subject to their share of erratic results (such as predicting a 13-point win for Barack Obama in the California primary; Obama lost by 9 points), are actually not terrible, and did fairly well on November 4th.
Zogby, however, also conducts Internet-based polls. These polls are conducted among users who volunteer to participate in them, first by signing up at the Zogby website (you can do so yourself here) and then by responding to an e-mail solicitation. These Internet polls, to the extent they rely on voluntary participation, violate the most basic precept of survey research, which is that of the random sample. And as you might infer, they obtain absolutely terrible results.
Because I’m writing this post ahead of time, I’m not 100% certain that the particular poll in question is an Internet poll. Zogby, which is probably aware of the poor reputation of its Internet polls, has begun to go to some length to conceal their origin, usually reserving notification that the poll was conducted online for the fine print. However, I’m about 99.8% certain that this is an Internet poll, as allof the polling that Zogby has recently conducted on Presidential approval have been of the Internet variety. And that polling, by the way, has produced some very strange results: in an Internet pollconducted from January 22-26, for example, in the immediate aftermath of Barack Obama’s inauguration, Zogby had Obama’s job approval at 52/29, while the average of polls from five other agencies (Gallup, Hotline, Rasmussen, FOX and Democracy Corps) conducted at the same time put the numbers at avergae of 62/19.
Let’s take a look at the track record of Zogby’s Internet polling. Zogby conducted his last series of Internet polls for last year’s Presidential election in mid-October. He missed 3 of 11 states, and was off on the final margin by an average of 5.4 points.
These are not good results — the average miss on election day among all presidential polls in these states was around 2,5 points; Zogby Interactive’s average error was twice that. And Zogby is probably fortunate that he confined his polling to these 11 states only. When he had conducted a broader series of Internet polling back in June, it had produced some truly head-scratching results:
Note that Zogby had Barack Obama winning states such as Arkansas, Arizona and South Carolina, results which no other pollster saw at any point during the election cycle. He also had Obama polling within 5 points of John McCain in Oklahoma, which Obama would go on to lose by 31 points, and Tennessee, which he’d lose by 15.
Did the Zogby Internet polls just have a bad year? No. If anything, their performance was much improved from 2006, when Zogby had surveyed a wider number of contests. Below are the Internet polls that Zogby put out on October 27, 2006 in advance of that year’s senatorial elections. His average miss was 8.7 points, including six misses of 10 points or more, and one miss of almost 30 points:
Zogby’s gubernatorial polling that year was similarly error-prone. His Internet polls missed the margin in 19 gubernatorial contests by an average of 7.7 points, and he called 5 of the 19 election wrong, including a couple of states (like Arkansas, Colorado and Wisconsin) where the outcome was never in much doubt.
All told, between 48 contests that he’s surveyed over the past two election cycles, Zogby’s Internet polls have been off by an average of 7.6 points. This is an extreme outlier with respect to absolutely anyone else in the polling community.
These Internet polls, simply put, are not scientific and should not be published by any legitimate news organization, at least not without an asterisk the size of an Alex Rodriguez steroidal syringe. But I’ll bet you that Matt Drudge already has the siren cued up by now.
