Impact case study database

a. Polling errors and methodologies in the UK

This strand of the research has had two related impacts: i) Impact on the conduct and reporting of opinion polling in the UK through changes to the rules of the BPC and to the methodology and procedures of UK polling organisations, and ii) Impact on public policy, in the form of recommendations by the House of Lords Select Committee on Political Polling and Digital Media for regulation of the polling industry.

i) The changes in the rules of the BPC were set out in a statement on 31 March 2016 which states, “the Council has agreed to implement immediately rule changes that will (i) require greater transparency about how polls have been weighted, (ii) specify what changes, if any, have been made since a company’s previous published poll in how the data have been weighted or otherwise adjusted, and (iii) place an obligation on members to supply to any inquiry or committee that has been established by the BPC the micro data set for any poll in which that inquiry or committee has an interest”. The BPC subsequently introduced a new requirement on its members to publish a statement of the level of uncertainty of poll estimates: “the new BPC rule has been introduced in response to recommendation 11 of the inquiry into the 2015 British General Election Polls that was chaired by Prof. Patrick Sturgis” (1 May 2018). [ 5.1]

In addition to these changes to the rules of the BPC, the inquiry also influenced the methodologies of individual polling organisations. In a report published on 19 September 2017 [ 5.2] the BPC summarises the changes that member organisations made to their procedures in response to the inquiry report. These include changes to sampling and weighting procedures, the adoption of model-based turnout adjustment procedures, and new methods of treating respondents who do not report a vote intention. Nick Moon, Secretary of the BPC, said: “The vast majority of the recommendations were both accepted and acted upon and that demonstrates both the seriousness that the polling industry put on it and also of the kind of quality they thought it had” [ 5.3].

As publicly acknowledged on their blog posts [ 5.4] the Sturgis Inquiry’s findings shaped the methodological strategies of pollsters YouGov and Survation, the latter subsequently being the most accurate polling company in 2017 with a polling error of less than 1%.

Interviews with pollsters following the 2019 general election linked the strong performance of the industry to improvements in sampling and weighting introduced in the wake of the Sturgis inquiry [ 5.5]. For example, Adam Drummond of Opinium, the most accurate pollster in the 2019 election, has said “The overall thinking behind the BPC [Sturgis] Report has enormously informed our broader approach.” Anthony Wells of YouGov observed “I’ve always thought it was a very useful report, and that core finding that it was the sampling which was the problem was correct and what people needed to address. The companies that did get it right were the companies who had gone back and addressed that problem.”

Additionally, the research has been reported on extensively in the national and international media [ 5.6]. A Guardian editorial described the Polling Inquiry Report as “the nearest thing to a definitive explanation”. The insights of the inquiry continue to inform the practice and understanding of polling in the UK today. Research by Jennings has more widely informed coverage of polling methodologies, and their vulnerabilities. This has contributed to media and public understanding of polling in the 2016 EU referendum and the 2017 and 2019 UK general elections – via a range of commentaries, interviews and references to research in media reports [ 5.6].

ii) The research impact on policymaking has come about primarily as a result of its influence on the conclusions and recommendations of the House of Lords Select Committee on Political Polling and Digital Media, to which Jennings gave oral and written evidence, in addition to an informal closed briefing. The resulting report published in April 2018 [ 5.7] made recommendations on the regulation of the polling industry including, notably, that a ban on the publication of opinion polls in the days leading up to an election should not be introduced. The written evidence given by Jennings on the measurement of polling accuracy was cited extensively in the report (para. 39), while it also drew on his evidence on the comparative performance of the UK industry internationally and that there was no global crisis in polling (also see Section iii). Overall, his evidence was cited nine times in the report.

b. Polling estimation

Jennings’ research with the ‘Polling Observatory’ has been used by Ofcom in its regulation of election-related broadcast material in the UK since 2015. Its estimates have been included in digests of evidence produced by Ofcom (March 2017, March 2018, February 2019, April 2019 and November 2019) and its reviews of the Ofcom list of major political parties (March 2015 and March 2016). These informed Ofcom’s assessment of current party support, as indicated by opinion polls. Adam Baxter, Principal in Standards and Audience Protection for Ofcom notes [ 5.8]:

“In carrying out its duties in the regulation of election-related broadcast material, Ofcom must rely on accurate summaries of evidence of current support, as shown in opinion polls in the four nations of the UK. Ofcom has therefore used the Polling Observatory’s aggregated summaries of the various polling companies’ Great Britain-wide opinion polls as a proxy for gauging levels of current support in England. The Polling Observatory figures have been an invaluable resource in enabling Ofcom to reach decision in the area of elections. Further, as an aid to broadcasters, Ofcom has included the Polling Observatory figures in the digests of evidence of past electoral support (i.e. election results) and evidence of current support (in the form of opinion polls) that we published ahead of the various elections that took place on: 4 May 2017; 8 June 2017; 3 May 2018; 2 May 2019; and 12 December 2019. We also included the Polling Observatory figures in the digests of evidence of past electoral support (i.e. election results) and evidence of current support (in the form of opinion polls) that we published ahead of the elections that were due to take place on 7 May 2020 and which were subsequently postponed…”

This research has thus informed the regulation of election broadcasting and official guidance provided to broadcasters. The Polling Observatory estimates were for a period published on the websites of The Daily Telegraph and The New Statesman, and additionally were featured in reporting in The Guardian, The Independent, The Times and other national media outlets [ 5.6].

c. Polling accuracy

Jennings’ research on cross-national and over-time trends in polling accuracy has informed understanding of the performance of election polling worldwide in both the media and the polling industry. Jennings’ study of election polling accuracy in 45 countries [ 3.6] was also cited in the House of Lords report (2018, para. 42) [ 5.7], and was discussed by the leading US survey organisation Pew Research Center (2018) in a feature on ‘why well-designed polls can be trusted’ and in its field guide to polling for the 2020 presidential election [ 5.9]. The analysis and data were used extensively by the industry Inquiry into the 2019 Australian federal election polls, enabling benchmarking of the polling error in Australia [ 5.10]. The research has also been used by the President of the World Association for Public Opinion Research (WAPOR), Professor Claire Durand, in assessment of performance of the industry [ 5.11].

The study findings were promoted by the leading US commentators on polling/elections [ 5.13], including Nate Silver, who wrote in a piece for ABC News’ FiveThirtyEight:

“Despite often inaccurate and innumerate criticism over how polling fared in events like Brexit, a recent, comprehensive study of polling accuracy by Professor Will Jennings of the University of Southampton and Professor Christopher Wlezien of the University of Texas at Austin found polling accuracy has been fairly consistent over the past several decades in a variety of democratic countries in Europe, Asia and the Americas. The media narrative that polling accuracy has taken a nosedive is mostly bullshit, in other words. Polls were never as good as the media assumed they were before 2016 — and they aren’t nearly as bad as the media seems to assume they are now. In reality, not that much has changed.”

The study was highlighted further by Harry Enten, Senior Writer and Analyst for CNN Politics, on the leading podcast Pod Save America (which averages more than 1.5m listeners an episode) and has been the subject of commentaries from survey professionals including Scott Keeter, Senior Survey Advisor for the Pew Research Center, and Scott Clement, Polling Director for the Washington Post [ 5.6].

The polling industry has noted the important counter to ‘prevailing narratives’ regarding polling accuracy; for example, Ben Page, Chief Executive of Ipsos MORI says the analysis “is a healthy corrective to suggestions that polls are getting less and less accurate” while Anthony Wells, YouGov describes the findings as “incredibly useful to us in the industry and … to the cause of good science and good governance” [ 5.5]. The article [ 3.6] has an Altmetric score of 1,317 (19 December 2020), placing it in the top 4,200 articles out of 16.4 million tracked by Altmetric in terms of attention via social media attention, news coverage, and other modes of engagement. Altmetric noted 99 news stories in 85 international news outlets, and a total of 1,229 tweets from 1,007 users with an upper bound of 13.0 million followers.

The data collected for this study has also been used in coverage of elections across Europe by media organisations such as The Economist (French presidential elections 2017, Brazilian presidential elections 2018, UK general election 2019) and El Pais (Spanish federal elections 2016) [5.6]. As such it has enabled evidence-based analysis of elections. In June 2017, Idrees Kahloon, Data journalist for The Economist noted [ 5.12]:

“Professor Jennings's work has been invaluable to our coverage of European elections in the past year. By our count, at least 10 articles would not have been possible without his exhaustive collection of international polls—and many more stories have cited his data in discussing historic polling error. Our French election model , which was, to our knowledge, the first of its kind put out by a major news organisation, produced probability estimates using Professor Jennings's data. Our coverage of the recent British election was also improved by these data: we showed how pollsters generally oversold Labour support, and how much the dramatic drop in Conservative vote-share compared with past elections. A long three-page story on polling we ran just last week was also helped greatly by discussions with Professor Jennings. And we look forward to doing similar work with the upcoming German elections. We're certainly proud of our coverage of polling, which we try to explain to our readers accurately and with statistical rigour. But we also know that this quality of coverage only comes with the helpful expertise that the professor has so kindly lent to us.”

G. Elliott Morris, also Data journalist for The Economist, describes the Jennings and Wlezien dataset as “indispensable” and “a staple of our commentary on elections in almost every country” through underpinning his work in forecasting national elections.

Jane Frost, Chief Executive of the Market Research Society (MRS), the UK’s leading professional body for market research, highlights the broad importance of this programme of research to the £4.8 billion sector [ 5.13]:

“The MRS strives to promote the belief that “Evidence Matters”. This is particularly the case in relation to the performance of polls and their potential impact in turbulent times. Only truly robust independent studies have any chance of moving a polarised debate forward. The Sturgis Inquiry and Jennings study of polling accuracy worldwide were widely accepted for their results and their independence and have been instrumental in moving the debate on polling onto more constructive shared ground. The Inquiry to which they contributed is widely cited by MRS in, inter alia, evidence to the House of Lords investigation into polling.”