POLLS GOT IT WRONG (AGAIN) BUT DONâT LOSE FAITH IN QUANTITATIVE RESEARCH
Jim Mann from the numbers lab looks at the results from the US elections
Like many, I woke unusually early on Wednesday and reached nervously for my mobile phone. It was US election night and I was eager to see if, from my perspective, crisis had been averted or the world really had gone mad. Before I had a chance to tap my favourite news app I noticed a message from my brother: âAnother resounding victory for the polls bruv!â Detecting sarcasm (Iâm smart like that) I knew this could only mean one thing. Sure enough, Trump was well on course to a victory that nobody, least of all the pollsters, was anticipating. For the third time in eighteen months (following the UK general election and EU referendum) the pollsters had got it wrong!
In the period since May 2015, Iâve had countless debates with polling sceptics like my brother. His, fiercely articulated, view is that polling is not simply inaccurate, it also has the potential to sabotage itself. Heâs not alone in this belief. Behavioural economics shows that people generally wish to follow the herd. Therefore, a poll showing that the majority think in a particular way is likely to influence, albeit subtly, what they themselves believe. Furthermore, there are those that cite the possibility that polls could impact rates of voter turnout. After all, why bother to turn out to vote if the polls have created a strong belief that your favoured candidate is either assured of victory or has no chance of winning?
Polling, when first popularised by George Gallup in the 1930s, was hailed for the positive contribution it made to the democratic process. Gallup himself was, understandably, steadfast in this belief. Elmo Roper, another pioneer of the public opinion poll, described it rather hyperbolically as âthe greatest contribution to democracy since the introduction of the secret ballotâ. But there have always been critics, and the anti-polling arguments inevitably gain traction when the pollsters get it wrong. Failure is not a modern phenomenon either. Immediately prior to the 1948 election George Gallup predicted that Dewey would beat Truman in the election and stated, unwisely as it turns out, âWe have never claimed infallibility, but next Tuesday the whole world will be able to see down to the last percentage point how good we areâ. Dewey lost. The anti-polling lobby had a field day.
So criticisms of polling arenât new and, letâs be honest, they would remain niche concerns if the polls were accurately predicting results. But theyâre not and on the back of a series of high profile failures itâs increasingly common to deride polling as a âdevalued pseudo-science conducted by charlatansâ. Yep, my brother again. I hate to give him the last word so, in order to provide a flavour of wider opinion, Iâll quote the Guardianâs post-election editorial instead. âThe opinion polls and the vaunted probability calculus rarely trended in his (Trumpâs) direction; both are discredited today.â
The purpose of this blogpost is not to defend political polling; I have my own concerns in that direction and itâs undeniable that the work of pollsters is becoming harder, due to a combination of methodological issues and a more fluid, less predictable, political landscape. However, for the sake of fairness Iâd like to mention two things, neither of which is intended to exonerate the practice.
First, most polls reflect public sentiment within a nationally representative sample. In the main, but not exclusively, the polls conducted immediately prior to the election found that, by a relatively small margin, more Americans intended to vote for Clinton than Trump. In this they were correct. At the time of writing, the figures show that 59,814,018 Americans voted for Clinton whilst 200,000 fewer (59,611,678) voted for Trump. However, due to the distribution of votes and the vagaries of the US political system, this translated into 279 Electoral College votes for Trump and 228 for Clinton.
Second, most polls conducted by reputable polling organisations produced results that placed the result well within the margin of error. âWhatâs that?â I hear you ask. Well, tucked away at the end of most reports based on a public opinion poll will be a small note about margin of error. This margin will differ depending on the number of people interviewed for the poll but, for a standard sample size of 1,000, the margin of error is +/- 3.5%. This essentially means that if the poll results show that Clinton is projected to win 47% of votes, the reality is likely to be somewhere between 50.5% and 43.5%. Within this context, the result of the election was well within the margin of error of most polls. It wasnât so much the polls that got it wrong, it was the reporting of the polls that failed to sufficiently stress that the result really was too close to call. But people donât like uncertainty so these boring, statistical caveats tend to get overlooked.
OK, but I said this blog wasnât designed to defend polling. So what is it about? Well, I donât feel the need to defend polling because Iâm not a pollster. However, I am a market researcher working with quantitative surveys and, what concerns me, is the fear that growing scepticism around polling will negatively impact trust in all forms of numbers-based research into public attitudes. Maybe Iâm just a worrier and people are perfectly able to distinguish between different forms of survey based research. However, my own experience suggests that isnât always the case.
In May 2015 I was working at the Guardian. The Guardian has invested significantly in data journalism over recent years and coverage and analysis of polls was given a high degree of prominence in the run up to the UK general election. At the Editorial conference, held the day after the election, the mood was subdued. When the conversation turned to the failure of the polls some journalists questioned the prominence given to polling numbers, especially as those numbers didnât chime with their instincts and the evidence of their own, on the ground, experiences. The upshot was a policy decision, only recently reversed, that editorial coverage of polling should be suspended. The coverage of polls in the run-up to the US election was reported under the banner âSceptical pollingâ, which gives a pretty good indication of the mood around the organisation. Â
As Head of Consumer Insight at The Guardian, a key element of my role was to advocate for use of consumer research and promote evidence-based strategic decision-making. My internal clients were ranged on a spectrum that ran from research enthusiasts to rejecters. This latter group, a minority it should be said, believed there was little to gain from engaging with research. The great polling disaster of 2015 provided a tailor-made reason to disengage. After all, research had been shown, in the most public way imaginable, to be unreliable and wrong! Hadnât it?
Iâm sure the Guardian is like most organisations in having research stakeholders ranging from enthusiasts to sceptics. To the latter group I would make this plea; donât conflate political polling and other forms of quantitative market research and do not deny yourself and your business an incredibly powerful, consistently proven aid to decision making simply because political polling has been shown to not be a perfectly accurate crystal ball. As mentioned, polling isnât quite as inaccurate as some would have you believe. Furthermore, the stakes are simply much higher for polling: A couple of percentage points either way (generally within the margin of error, remember) is the difference between two diametrically-opposed outcomes and the profound repercussions associated with that. In contrast, if a representative survey of consumers in a particular sector suggests that awareness of your brand currently stands at 34% whilst that of a competitor is 64%, does it really make a huge difference to the decisions your company will take if the reality is a couple of percentage points either side? Â
Of course, some decisions do require a higher degree of accuracy. In these instances, market researchers have two huge advantages over pollsters. We can increase the number of people interviewed in the study, thus reducing the margin of error and increasing confidence levels. We can also utilise robust sampling techniques such as random probability sampling. Generally speaking, neither of these options is available to pollsters because they are simply too time consuming. Pollsters are required to provide an almost instantaneous reading of public sentiment, before new events have a chance to change it, and anything that slows that process is, by necessity, discarded. If pollsters were given the freedom to use these tools, itâs likely they would provide far more accurate predictions. How do we know? Well, following the 2015 general election most polling companies conducted re-contact surveys with pre-election poll respondents to try and understand what went wrong. What they discovered was that, even when conducting post-event research, they were unable to accurately replicate the result. The inquiry conducted by the Polling Council of Great Britain concluded that the reason was their use of (attitudinally) unrepresentative samples drawn from panels and that a random probability sampling approach (that gives every member of a target population an equal chance of participating in the study) would counteract the problem. Tellingly, the survey that best replicated the election result was the British Social Attitudes (BSA) survey conducted by NatCen Social Research. Need I say that BSA is based on a large sample (3,000) and utilises random probability sampling?
Iâve rambled on too long and exceeded my word count limit by a distance so Iâll finish by saying this: The great jazz musician, Duke Ellington (or possibly Richard Strauss, itâs disputed) is quoted as saying âthere are only two types of music: good and badâ. Market research is much the same. When done properly it is an incredibly powerful diagnostic and forecasting tool that can provide a highly accurate picture of consumer sentiment as it currently exists. Pollsters, through no fault of their own, are sometimes unable to do it.
Researchers, however, can and do. We hope you can see from this that the numbers lab fall in to the former (good) group so, if you want our thoughts on anything, please just get in touch