Reality Check: Public Opinion Polling is Doing Just Fine

Reality Check: Public Opinion Polling is Doing Just Fine

By David Dutwin – January 27 at 11:30am

Polling is under siege. Recent failures in election polling in England, Israel, and a few high-profile states in the U.S. midterm election have been squarely attributed to an increased unwillingness of many Americans to take surveys (what the industry terms “declining response rates”). Some in Congress have called for significant reductions in funding for the U.S. Census and making data collection for the bureau voluntary, despite the critical role such data play in hundreds of government policy decisions — not to mention businesses who use such data for a range of strategic decision making.

Yet, with apologies to Mark Twain, the simple truth is that the death of probability based, scientifically rigorous and highly precise polling has been greatly exaggerated. Public opinion polling is doing just fine. Polls serve a critical role in democracy and, contrary to polling’s noisy critics, continue to provide highly accurate estimates of public sentiment.

Recent research of telephone polling by the Pew Research Center, ABC-Washington Post polls and CBS-New York Times polls finds little growth in inaccuracies in polls over the past 20 years, despite response rates having dropped from the 30 percent range to the single digits (1). Indeed, reported estimates show absolutely no increase in bias, as illustrated in this chart. Random surveys of Americans, dialed via cell phones, attain remarkably accurate cross-sections of Americans by most key demographics like age, race and level of education. It is easy to claim that millennials don’t use their phones for anything but texting, but when the phone rings, they answer. (Arguably, having not lived through the past decade of telemarketing landline glut, they may even approach telephone surveys with less reticence, not more.)

reality check 1

Other evidence of polling’s resistance to error comes from polling aggregator Nate Silver, who examined how far off predictions have been for almost every election held in the U.S. since 2000. Notably, Silver did not identify any decline in the accuracy in presidential, gubernatorial, House or Senatorial polls.

reality check 2

And it makes sense. Scientifically, it does not matter whether 1 in 5 or 4 in 5 Americans hang up on pollsters, as long as those that hang up are largely similar to those that participate. While there are differences between those who participate and those that do not, nearly 20 years of research finds there to be far more similarities between the two groups than differences. Democracy would likely be better-served if more people chose to participate in polls, but at the same time, lack of participation is not currently leading to a significant impact in overall accuracy.

Accuracy is not Newsworthy

When pollsters “miss,” it is inevitability big news. This has been true since 1948, when some pollsters predicted a Thomas Dewey win over Harry Truman. It is not exactly newsworthy to also talk about the fact that pollsters then went on a 64-year run of correct predictions, when finally in 2012 one or two firms predicted a close Mitt Romney win. Pollsters do not bat .1000. But their accuracy is noteworthy, if not newsworthy.

Polling is now a field with many approaches and techniques, and some are better than others. Low-cost, low-quality alternatives to traditional telephone polling will inevitably miss the mark. But even high-quality polls will sometimes be wrong, particularly in an age of deep partisan divide, where many elections are characterized by one- or two-point victories, not 10-point landslides.

It has been said that political polling is uniquely challenged in that the population being surveyed does not yet exist at the time of the survey: It is surveying a future population of who actually shows up at the polls on Election Day. Pollsters have to rely upon statistical modelling to fill the gap, and because who shows up has been more variable of late, there is much more pressure to get the models right. It must be noted that these models have worked tremendously well for decades, and recent advances offer the promise of maintaining past performance, at the very least.

Polling has faced challenges in every decade of the modern age, and the industry has met and successfully overcome each of them. Polling is not infallible, and no one knows what the future will bring. There will surely be future misses, but polls continue to be the most trustworthy and reliable opportunity for people to have a voice.

If you get called, choose the right to be heard over your right to refuse. Do not undermine the role of polling in democracy with unfounded claims of its demise.

(1) David Dutwin and Trent Buskirk, “Trends in Survey Error in the Age of Declining Response Rates,” unpublished manuscript currently under peer review.

David Dutwin is Executive Vice President and Chief Methodologist for SSRS, a survey research firm located outside of Philadelphia, PA. He is also a Senior Fellow at the Program for Opinion Research and Election Studies at the University of Pennsylvania and serves on the Executive Council of the American Association for Public Opinion Research. Opinions expressed here are solely his own.