After watching the polling and results from Pennsylvania’s 18th Congressional District special election, I’m worried that our collective conversation about polls hasn’t evolved since the 2016 election.
On the eve of Tuesday’s special election, Monmouth University, a very well-respected pollster, showed Democrat Conor Lamb with a six-percentage-point lead over Republican Rick Saccone in its prominent turnout model.
So when Lamb ended up with a less than 1,000-vote lead over Saccone, people have personally asked me, “How did Monmouth get it so wrong?”
My answer: They didn’t.
The poll was perfectly fine and reflected what ultimately has played out: That Lamb was a favorite to win but that Saccone still had a reasonable shot at victory.
The conversation mirrors one I’ve had before, following President Donald Trump’s victory over Hillary Clinton. Most polls showed Clinton with a slight lead and most of the media coverage suggested that Clinton was overwhelmingly favored to win.
The correct interpretation of those polls was that, yes, Clinton was favored, but she wasn’t an overwhelming favorite, and based on the historical accuracy of polling averages, Trump was just a normal polling error away from winning.
So where do we keep going wrong and how can we fix it to better inform the conversation around polls?
Here are four areas where I think we can do better:
1. Averages over single polls
This seems simple, but it’s worth repeating after the coverage I saw in PA-18: we know from the past that the polling average is usually a better predictor than any one poll.
In PA-18, Monmouth had Lamb up by six, but the final average of polls released during the final three weeks of the campaign had Lamb ahead by only 2.5 percentage points, less than half what Monmouth showed.
Prior to the Pennsylvania result, the average difference between the final three-week polling average and the final result in 15 prior House special elections since 2004 with at least two surveys was 3.3 points. The average difference between each individual poll and the final result in those races was 4.6 percentage points. (In Pennsylvania 18, the average error for each individual poll was 3.9 points compared to an error of 2.3 points when we average all the polls together.)
Had the media been more explicit about the average instead of the individual poll, it would have clearer to readers and viewers that Lamb’s lead was not only much smaller than Monmouth’s main model showed but far from safe given the historical accuracy of polling.
2. Understanding sampling margin of error
The sampling margin of error, as it is usually reported, applies to one candidate’s percentage of the vote, not the margin between them. The sampling margin of error of that estimate is usually (though not always) approximately double the sampling margin of error for one candidate.
When a pollster reports that the sampling margin of error on a poll is about +/- 3.7 percentage points (as CNN non-approved RABA Research showed in their PA-18 survey), that means the sampling margin of error for the margin between the candidates is about +/- 7.4 percentage points. In other words, the 4-point Lamb advantage RABA had was well within the margin of error, not outside of it, as the very good blog PoliticsPA reported.
3. Understanding true margin of error
There’s more potential error than just sampling error when it comes to electoral polling. One potential additional pitfall for instance is that we don’t know who is going to turn out to vote, which is why Monmouth offered multiple turnout models. This means that even if you have a poll that perfectly reflects the population you polled that population may not be the one that actually votes.
Even if the Monmouth poll had been the only poll we had, I would have thought that either candidate had a real chance of winning. History tells us that the final polls have a significantly larger “true” margin of error than the sampling error alone indicates. In special elections, the true margin of error for any individual poll in special elections since 2004 has been about +/- 13 percentage points with 95% confidence! This is far wider than the average error, which tends to downplay the chance of an outlier. In PA-18, the 6-point Lamb lead Monmouth found could have reasonably led to a result of Lamb losing by 7 or winning by 19 percentage points.
Reporters should note that the sampling margin of error really doesn’t reflect the true potential for the polls to miss the result. That was a big problem in 2016, when the final polls actually showed a race that could easily tilt toward Trump given how predictive the polls have been historically. I think the inability for the media to note the error beyond the sampling error is still a big problem.
4. A small lead is a small lead
A small lead is still a small lead even if a number of polls show it. Conceivably, the chance of an error declines if pollsters using different turnout models all show one candidate ahead by a small margin (i.e. a candidate’s lead is not dependent on getting the right turnout). But it in no way eliminates the potential for error.
One of the reasons I think the media overhyped Clinton’s lead in 2016 is that most of the polls had her ahead, even if the lead was below 5 percentage points in most swing states and nationally.
I got the sense that was repeating in Pennsylvania 18 when the two final polls of the campaign had Lamb ahead by the most he had been in any public polls. That is, people saw multiple polls with Lamb ahead and thought he was going to win.
Averaging polls does help to shrink the true margin of error, but it’s still quite large. The true margin of error is about +/- 10 percentage points for special elections. It’s smaller than that in presidential polls (especially national polls), though it was still far wider than Clinton’s lead in the final weeks of the campaign.
The bottom line for me is that polls are a great tool for understanding the public mood. They are not fool-proof, however, and the media (myself included) need to fully grasp and communicate that they are far from perfect.