Impact case study database

Political polling plays an important role in shaping party strategies, media presentation of election campaigns, and public perceptions of the political landscape. The information it provides therefore has the potential to influence democratic processes. Because election polls are highly visible, their quality and accuracy can also have a major effect on the reputation of, and public confidence in, surveys and public opinion research more widely. The work outlined above has delivered direct impacts in several areas relevant to this.

Changing the rules and regulations of political polling in the UK: members of the British Polling Council (BPC) include all the major polling companies who carry out regular election polls in the UK. The work of the Inquiry panel and the resulting report [1] led directly to changes in BPC rules about the ways in which political polling is conducted and reported. In June 2016, the Council confirmed its agreement to immediately implement rule changes corresponding to the reporting and transparency Recommendations 7-9 of the report [1] (but postponed a response to Recommendation 10, on pre-registration of polls) [A]. In May 2018, it also responded to Recommendation 11 by introducing a new requirement for its members to publish a statement about the level of uncertainty in poll estimates of parties’ vote shares [B]. Responding to the methodological Recommendations (1-5), the BPC further advised that it was “ for individual member companies to decide how best to take these forward”. Some of the ways in which member companies did this are discussed below (see “Impacts on industry practice”).

The work has also had impacts on the regulation of polling in the UK via its use in public policy debate on the subject. Impacts here derive primarily from its influence on the conclusions and recommendations of the House of Lords (HoL) Select Committee on Political Polling and Digital Media, appointed on 29 June 2017 to consider the effects of political polling and digital media on politics. The Committee’s final report was published on 17 April 2018 [D]; the government’s response to it was received on 15 June 2018 [E]. Sturgis served as Specialist Advisor to the Committee and five other members of the Inquiry panel - including Kuha - gave evidence or provided briefings to it. The Committee’s report [D] refers extensively to [1], which helped set the parameters for its own inquiry. Thus, for example, the Committee explains that, since it had been “ comprehensively covered” in [1], “ We have not...attempted to replicate this work by delving in detail into the methodological causes of polling errors” [D*, para 13, p. 11 ] . The Chair of the Committee, Lord Lipsey, reported in a 2019 interview with the University of Southampton that: “ the BPC inquiry was one of the key cornerstones of our work constructing the Lords inquiry and in informing our view of the present polling outlook” [G, p.30 ].

The Select Committee’s recommendations on the regulation of the polling industry included a substantially expanded oversight and advisory role for the BPC. It did not, however, propose government regulation of political polling or banning polls close to elections, both of which it had considered. The spirit of these recommendations was echoed in the government response: “ Polling standards should remain self-regulated by the industry [… who…] already have a strong incentive to update and improve their techniques, especially if flaws are uncovered” [E, p. 1 ].

Impacts on industry practice: a second major area of impact - resulting largely from these effects on the rules and recommendations governing polling in the UK - has been changes in methodology and procedures within the polling industry . These have had direct effects on decisions and operations within, and outcomes for, individual polling companies. According to an associate director at Opinium, the principal value of the BPC Inquiry after the 2015 election was that it: “… was able to point quite definitively to a system-wide problem, larger than just an issue which might have only been affecting our Consumer Panel or other things which would have been unique to one company” [G, p. 37 ]. This more rigorous, system-wide understanding underpinned by [1] helped public opinion research companies to identify and address problems in their methods. In a BPC report published in 2017 [C], nine BPC-member companies summarised changes to their polling procedures in response to [1]. These particularly included changes to sampling, weighting, and turnout adjustment procedures.

The influence of the panel’s work on industry practice is also apparent in evidence provided to the HoL Select Committee. Joint evidence submitted by eight of the companies explains:

“The Joint Inquiry led by Professor Patrick Sturgis into political polling…after the 2015 General Election identified that the main cause of polling error was unrepresentative samples. As such, one of the most pressing issues for the industry to tackle has been to improve the quality of their sample, an undertaking pursued across the membership of the British Polling Council.” [F, PPD0014, p. 154 ]

Similarly, YouGov made explicit reference to its own work to address the key findings of the Inquiry, including a substantial investment in efforts to improve recruitment to its panel of respondents [F, PPD0016, p. 549 ]. Its Director of Political and Social Research confirmed that “ the [changes] that came out directly from the report findings were…the introduction of quota and weighting by political interest and the introduction of education weights” [G, p. 12 ].

The polling Inquiry has also had impact beyond the UK. A comparable inquiry was recently conducted along similar lines in Australia to examine the performance of the polls before the federal election there in 2019 [H]. According to the Chair of that inquiry panel: “ Not only was the Sturgis et al. review heavily cited, it was also tremendously influential in helping to frame the Australian inquiry and guide some our analysis” [I].

Benefits to the polling industry

Improving polling accuracy: the polling industry’s efforts appeared to bear fruit in the polls for the General Election in December 2019, for which the companies further refined their methodologies within the framework outlined in the Inquiry report [J]. The polls correctly predicted the general result of the election, and the BPC described their predicted vote shares as “ more accurate … than in any contest since 2005” [K].

Commercial benefits: the changes in industry practice have had important knock-on effects on the success of individual companies - and, thereby, on the industry’s contribution to the UK economy. Market research and public opinion research are big business in the UK. According to MRS, the UK’s research market is “ second only to the United States”, employing up to 73,000 people and generating GBP4.8 billion in annual gross value added. Election polling represents only a small part of this business, but one which is important because of its unique visibility and accountability. This makes it “ a ‘shop front’ for polling organisations - an activity aimed at increasing their public profiles and advertising their accuracy” [D, Appendix 5, p. 99 ]. The reputation of that “shop front” was badly damaged in 2015. As well as improving the reliability of polling methods, evidence submitted to the Select Committee acknowledged the impacts of [1] on transparency in the industry. Providing evidence for the BPC, its President Professor Sir John Curtice credits [1] with influencing the fact that: “ …if any company changes in any way the way in which it has collected or estimated its voting intention data during an election or referendum campaign, it has to make that public” [J, QQ 139–147, p. 89 ]. These changes have already begun to improve the polling industry’s reputation among the media and other stakeholders. According to Opinium, [1] itself was “ extremely valuable in that it showed that we were being transparent and were committed to resolving the various issues”. Public engagement work by the Inquiry panel was also “ extremely helpful” in “rehabilitating a bit of the industry's image after 2015” [G, pp. 37-38 ] . By providing actionable (and now actioned) recommendations to improve polling, the Inquiry has contributed to restoring confidence in this highly visible facet of the UK’s “business of evidence” industry.

Benefits to users of polling data: beyond the commercial benefits for pollsters and their industry, changes in polling practices have also delivered important benefits to those who use polling data. Key users and consumers of political polls include press and online media covering elections, political parties running in those elections, and voters themselves.

It is generally agreed that pre-election polling helps shape media coverage and therefore the “narrative” of elections; in turn, this may be seen to affect public perceptions of an election and decision-making by political parties. This view was echoed in evidence given to the HoL Select Committee by the Director of Editorial Policy and Standards at the BBC: “ Our concern about the 2015 and 2017 general elections and the Scottish and EU referendums was the capacity of the polls to influence the journalistic narrative of those election campaigns” [D, para. 82, p. 27 ]. Certainly, it is well known that political parties take an interest in the results of more prominent newspaper polls, as well as using private polling to inform decision-making, and the HoL Select Committee heard evidence that this can influence the strategic approach and decision-making of those parties [D, para. 83-91, pp. 27-29 ].

The report [1] has improved understanding among journalists about the reporting of polls. A representative of the BPC explained that it provided “ a very useful tool for the media to help understand exactly what had gone wrong [in 2015]…It made it easier for them to in turn explain to their readers [and] viewers what had happened” [G, p. 26 ]. The report also underpinned the development of new media recommendations about the use of polling data, now published by both BPC and MRS [L]. According to the Director of Deltapoll:

“…following the Inquiry the British Polling Council hardened its recommendation to journalists on how they should use and treat polling. I think lots of people now don’t look at individual polls to the extent that they did, but view them as part of a series and a trend and look at the general patterns rather than individuals, which obviously implies outliers probably get less recognition than they used to.” [G, p. 35 ]

Finally, political polls are one of the most prominent forms of information available to the voting public. One cannot make any very strong claims about whether and how polling may affect voting, as the empirical evidence on this question is limited. Whatever such effects there may be, however, it is clearly preferable that decision-making is based on accurate information. As the HoL Select Committee concluded:

“…voting intention polls play a hugely significant role in shaping the narrative around political events such as elections and referendums. Given the impact that they can have on political discourse, they will inevitably influence public behaviour and opinions, even if only indirectly. It is therefore vital that work continues in order to try to improve polling accuracy and that this is done as transparently as possible.” [D, para. 92, p. 29 ].

The work of the polling Inquiry has contributed to the achievement of this goal. By helping to improve the quality of polling, it has improved the accuracy of the information available to those who use voting data, helping them to develop better-informed campaign strategies and voting choices, and to present and receive fairer and more accurate election coverage.