Impact case study database
Improving public confidence in political polling
1. Summary of the impact
Political polling plays a crucial role in shaping the narrative of election campaigns, influencing party strategies, media reporting, and the vote choices of individual citizens. Public confidence in the accuracy of polling was damaged by their collective failure to predict the outcome of the 2015 UK General Election, leading to the establishment of an inquiry led by Professor Patrick Sturgis. Research by Professor Sturgis has specifically led to:
Changes in the rules of the British Polling Council (BPC) and the Market Research Society (MRS) and in the methodological procedures used by UK polling firms as a result of the BPC-commissioned independent inquiry into polling, known as the Sturgis Inquiry. Sturgis has since served as a member of the panel of an equivalent inquiry in Australia.
UK policy influence through the House of Lords Select Committee on Political Polling and Digital Media for which Sturgis served as Specialist Advisor and fed into key recommendations.
Contributed to enhanced media and public understanding of political polling through high-profile media coverage of the Sturgis Inquiry and the Lords Select Committee.
2. Underpinning research
Prior to Chairing the BPC/MRS Inquiry, Sturgis published research which established him as a leading international expert in the field of survey methodology and which provided the basis of the research conducted on the 2015 pre-election polls. The underpinning interdisciplinary research at UoS sought to understand which methodologies are most effective for improving survey accuracy. Sturgis led a significant body of research between 2008 and 2016 that focused on survey errors that arise due to the differential propensity of individuals to take part in surveys [ 3.1, 3.2] and on errors of measurement in survey questionnaires [ 3.3, 3.4].
Research in 2009 [ 3.4] involving a survey experiment which manipulated the order that respondents administered vote choice and economic evaluation concluded that survey respondents tacitly align evaluations of the national economy with previously stated attitudes and behaviour. An investigation into the direction and magnitude of the relationship between interviewer skill in obtaining contact and cooperation, and correlated interview error [ 3.2] identified the variability of survey outcome based on the characteristics of the interviewer rather than the interviewee. Other examples of analysis of survey methodologies were explored in a review of whether to insert a ‘middle response’ such as neutral into surveys and whether this is assessed accurately [ 3.3] and an exploration of the number of calls needed to get a representative sample in face to face surveys [ 3.1].
Sturgis’ research in his role as Professor of Research Methodology in the Department of Social Statistics and Demography and Director of the ESRC’s National Centre for Research Methods (NCRM) led to his appointment as chair of the polling inquiry (commissioned by the BPC) in 2015 – later known as the ‘Sturgis Inquiry’ – following the significant errors in the polling industry’s predictions of the 2015 General Election result. A Guardian editorial on 14 May 2015 stated that ‘the campaign had more data than any before it, and yet the predictions were some of the worst. Serious flaws in the polls were exposed; fixing these is a democratic priority’ and that ‘the review launched by the BPC is crucial in all this. It is imperative that is does not become an inside industry job. It is encouraging that Professor Patrick Sturgis, a research methods expert, will be in the chair’’.
The key underpinning research [ 3.5 and 3.6] was undertaken between June 2015 and December 2016 with long-time collaborator Professor Will Jennings and comprised an in-depth analysis of the errors in the 2015 General Election opinion polls. The main conclusion of the research that informed the Sturgis Inquiry findings was that the errors were caused by unrepresentative sampling; the way that polling organisations recruited sample members resulted in systematic over-representation of Labour voters and under-representation of Conservative voters. The weighting and adjustment procedures applied to the raw data did little to mitigate this error.
The research was able to rule out a range of other potential causes, including: turnout weighting; postal voting; overseas voting; and late swing. The 2016 Sturgis Inquiry report makes a number of recommendations for changes to the rules of the BPC, for the conduct of opinion polls by opinion polling companies, and for how the results of opinion polls should be reported in the media. The findings set out in the research and the report’s conclusions were subject to a very high level of media interest and acted upon by the polling industry.
3. References to the research
3.1 Sturgis, P. Williams, J. Brunton-Smith, I. and Moore, J. (2016) Fieldwork effort, response rate, and the distribution of survey outcomes: a multi-level meta-analysis, Public Opinion Quarterly, 81(2), 523-542. https://doi.org/10.1093/poq/nfw055
3.2 Brunton-Smith, I. Sturgis, P. and Williams, J. (2012) Is success in obtaining contact and cooperation correlated with the magnitude of interviewer variance? Public Opinion Quarterly, 76 (2): 265-286. https://doi.org/10.1093/poq/nfr067
3.3 Sturgis, P. Roberts C. and Smith, P. (2014) Middle Alternatives Revisited: how the neither/nor response serves as a face-saving way of saying ‘I don’t know’, Sociological Methods and Research, 43 (1), 15-38. https://doi.org/10.1177/0049124112452527
3.4 Sturgis, P. Choo, M. and Smith, P. (2009) Response Order, Party Choice, and Evaluations of the National Economy: A Survey Experiment. Survey Research Methods, 3(1), 7-12. https://doi.org/10.18148/srm/2009.v3i1.129
3.5 Sturgis, P. Kuha, J. Jennings, W. Baker, N. Callegaro, M. Fisher, S. Green, J. Lauderdale, B. and Smith, P. (2017) An assessment of the causes of the errors in the 2015 UK General Election Polls. Journal of the Royal Statistical Society: Series A (Statistics in Society). https://doi.org/10.1111/rssa.12329
3.6 Sturgis, P. Baker, N. Callegaro, M. Fisher, S., Green, J. Jennings, W., Kuha, J., Lauderdale, B. and Smith, P. (2016) Report of the Inquiry into the 2015 British general election opinion polls, London: Market Research Society and British Polling Council. https://eprints.soton.ac.uk/390588
4. Details of the impact
This research has had three related impacts. The first impact is on the conduct and reporting of opinion polling in the UK through changes to the rules of the British Polling Council (BPC) and the Market Research Society (MRS) and to the methodologies and procedures of UK polling organisations. The second relates to public policy, in the form of recommendations by the House of Lords Select Committee on Political Polling and Digital Media for regulation of the polling industry. The third impact is through public engagement mainly through high levels of media coverage contributing to progress in the rebuilding of public confidence in polling after the blow to the polling industry’s credibility of the 2015 General Election prediction.
(i) Changes in Opinion Polling Industry Practice in the UK
The Sturgis Inquiry led to changes in the rules of the BPC. These were set out in a BPC statement on 31 March 2016 which states [ 5.1], “ the Council has agreed to implement immediately rule changes that will (i) require greater transparency about how polls have been weighted, (ii) specify what changes, if any, have been made since a company’s previous published poll in how the data have been weighted or otherwise adjusted, and (iii) place an obligation on members to supply to any inquiry or committee that has been established by the BPC the micro data set for any poll in which that inquiry or committee has an interest.” In a subsequent press release dated 1 May 2018 [ 5.2], the BPC introduced a new requirement on its members to publish a statement of the level of uncertainty of poll estimates, “ the new BPC rule has been introduced in response to recommendation 11 of the inquiry into the 2015 British General Election Polls that was chaired by Prof. Patrick Sturgis”.
In addition to these changes to the rules of the BPC, the Inquiry also influenced the methodologies of individual polling organisations. In a report published on 19 September 2017 [ 5.3] the BPC summarises the changes that member organisations made to their procedures in response to the Inquiry report. These include changes to sampling and weighting procedures, the adoption of model-based turnout adjustment procedures, and new methods of treating respondents who do not report a vote intention.
Individual pollsters have also publicly acknowledged the impact of the Sturgis Inquiry’s findings on their subsequent methodological strategies. For example, in a blog post on 6 May 2018 [ 5.4], YouGov acknowledged the Sturgis Inquiry as an influence on its approach to increasing sample representativeness that “ both our internal investigation and the British Polling Council’s inquiry into what went wrong found that the main problem with the 2015 polls was sampling’ and ‘we have addressed the cause: we have focused on recruiting people who pay less attention to politics onto our panel. We have started sampling and weighting our polls by whether people voted at the last election, their level of education and how much attention they pay to politics. On the basis of the results of the Scottish, Welsh and London elections, this new approach appears to be working well.”
In a blog post on 14 June 2017 [ 5.5], Survation similarly acknowledged the influence of the Sturgis inquiry on its methodological approach which made it the most accurate pollster in 2017 (with a polling error of less than 1%): “ Patrick Sturgis chaired an enquiry into the polling debacle of 2015. He recommended that pollsters improve their results by focusing on finding a representative sample. This election has once again validated Survation’s forward looking approach to our sampling, and we will continue to focus on improving our methodology in line with his conclusions; we will not try to fix historical errors by changing our methods in untested ways.’
The Sturgis Inquiry continues to inform and be referenced by the polling industry today. Ahead of the 2019 General Election, the BPC summarised the principal changes in the conduct and reporting of polls, making reference to the Sturgis Inquiry Report in a statement on 29 November 2019 for understanding of the methodology of the polls [ 5.6]. The day after the 2019 General Election, the BPC reported “ after being widely criticised for their performance in the 2015 and 2017 general elections, the polls have proven more accurate in the 2019 election than in any contest since 2005.” [ 5.7]
Interviews with pollsters following the 2019 general election linked the strong performance of the industry to improvements in sampling and weighting introduced in the wake of the Sturgis report. Adam Drummond of Opinium, the most accurate pollster in the 2019 election, has said “The overall thinking behind the BPC [Sturgis] Report has enormously informed our broader approach.” Anthony Wells of YouGov observed “I’ve always thought it was a very useful report, and that core finding that it was the sampling which was the problem was correct and what people needed to address. The companies that did get it right were the companies who had gone back and addressed that problem.” [ 5.8]
The reach of the Sturgis Inquiry on the polling industry is now extending internationally; Sturgis was appointed in May 2019 as a member of the panel for the Association of Market and Social Research Organisations (AMSRO) Polling Inquiry in Australia, which issued an interim discussion paper on disclosure standards in May 2020 and published its final report in November 2020. AMSRO conducted a review of political polling methods in Australia to determine why all the election polls incorrectly called the outcome in the Federal election in May 2019 and how methods can be improved in future. This has echoes of the reasons the Sturgis Inquiry was initiated and is similarly being led by the industry employing independent experts from Australia and overseas, one of which is Sturgis, the only representative from the UK. Chair of the Australian Polling Inquiry Darren Pennay says: “Not only was the Sturgis et al. review heavily cited [in our report] it was also tremendously influential in helping to frame the Australian inquiry and guide some our analysis.” [ 5.9]
(ii) Public policy
The research impact on policy-making has been achieved as a result of its influence on the conclusions and recommendations of the House of Lords Select Committee on Political Polling and Digital Media in 2017, for which Professor Sturgis served as Specialist Advisor. The resulting report published in April 2018 [ 5.10] made recommendations on the regulation of the polling industry including, notably, that a ban on the publication of opinion polls in the days leading up to an election should not be introduced. The Sturgis Inquiry is referred to extensively throughout the report and the Chair of the Committee, Lord David Lipsey said, " As chair of the House of Lords select committee on opinion polling, I find it hard to overstate Professor Sturgis's contribution to our work. His seminal report on polling error in the 2015 general election shaped our analysis, and brought an authority which immensely helped us when he served as specialist adviser to our committee." [ 5.11]. Lord Hayward, a member of the Lords Committee commented [ 5.12] that “as somebody who was present during the Lords inquiry, it fed into our report very well….identifying what caused the problems was very very helpful both for the polling industry and to the Lords inquiry”.
(iii) Public engagement and trust in polling
The research has made an important contribution to media and public understanding of political polling and the methodological challenges facing the industry. It has done so through providing an accessible and definitive account of the limitations and vulnerabilities of political polling in the UK, and by highlighting the transparency of the industry and its willingness to improve its practices. This has informed media coverage and enhanced public confidence in opinion polling. The conclusions of the Sturgis Inquiry and the House of Lords Select Committee were covered extensively by the national and international print and broadcast media. This included stories in all the national daily newspapers, including The Guardian, The Times, The Financial Times, The Daily Mail, The Week and The Telegraph and interviews on the Today Programme, ITV News, and BBC News [5.13]. Sturgis was interviewed by BBC, ITV, Sky News, Channel 4 and Channel Five regarding the interim findings of the Inquiry in January 2016. In an editorial the Guardian described the Sturgis report as “ the nearest thing to a definitive explanation”. Sturgis has written comment pieces on polling for outlets such as Prospect, The Guardian, The Huffington Post, The International Business Times and Sky News online [ 5.13]. The report has been downloaded 38,699 times from the UoS website (as of 31/12/20).
The interview series with pollsters [ 5.7] revealed further insights into the public influence of the Sturgis Inquiry:
Martin Boon, Co-founder and Director of Deltapoll said “It was measured in how it said, there’s unreasonable expectations on polling and pollsters to produce accuracy, the expectation of accuracy is beyond what polls are actually capable of doing within standard expectations, the application of fiscal tolerances and all of that. They (media) simply felt the polls should be better than they’re capable of being and I think one of the great things in the Sturgis inquiry was to say hang on a minute journalists, you’re overstating what your belief in polls says from the brink. And I think it was good in doing that”;
Adam Drummond of Opinium said that “in terms of the reputation of the industry and how its perceived by the media and other stakeholders and the public, it was extremely valuable in that it a) showed that we were being transparent and were committed to resolving the various issues and we weren’t just sort of doing things on a shoe string and you know putting our fingers in the air and taking a punt, you know we were actually taking things quite seriously and were committed to getting our various houses in order”.
Jane Frost, Chief Executive of the MRS, the UK’s leading professional body for market research highlights the broad importance of this programme of research to the £4.8 billion sector has noted that the Sturgis inquiry was ‘widely accepted’ for its results and independence, and was “ instrumental in moving the debate on polling onto more constructive shared ground” [ 5.14] .
5. Sources to corroborate the impact
5.1 The responses of the British Polling Council to the recommendations of the 2016 Inquiry Report http://www.britishpollingcouncil.org/2016/03
5.2 The British Polling Council statement 1 May 2018 on the introduction of a new rule on reporting opinion polls in response to the Sturgis Inquiry http://www.britishpollingcouncil.org/british-polling-council-introduces-new-rule-on-uncertainty-attached-to-polls
5.3 The report on how individual polling companies changed their methodologies in response to the Sturgis Inquiry http://www.britishpollingcouncil.org/how-have-the-polls-changed-since-2015/
5.4 YouGov blog post 6 May 2016 acknowledging the influence of the Sturgis Inquiry research on its approach to sampling https://yougov.co.uk/topics/politics/articles-reports/2016/05/06/election-polls-new-methods-working
5.5 Survation blog post 14 June 2017 acknowledging the influence of Sturgis Inquiry research on its approach to sampling: https://www.survation.com/survation-most-accurate-pollster
5.6 BPC statement 29 November 2019 on changes in the conduct and reporting of polls in the 2019 General Election http://www.britishpollingcouncil.org/principal-changes-in-the-conduct-and-reporting-of-polls-in-the-2019-general-election
5.7 BPC statement 13 December 2019 on the performance of the polls in the 2019 General Election http://www.britishpollingcouncil.org/the-performance-of-the-polls-in-the-2019-general-election
5.8 Transcript of semi-structured interviews with pollsters YouGov, Opinium, Ipsos MORI, Deltapoll, June-July 2019. Contact details of interviewer supplied.
5.9 Testimonial from Darren Pennay, Chair of the Australian (AMSRO) Polling Inquiry, 2 October 2020.
5.10 The House of Lords Select Committee on Political Polling and Digital Media report https://publications.parliament.uk/pa/ld201719/ldselect/ldppdm/106/106.pdf
5.11 Transcript of interview with Lord Lipsey, 16 July 2019. Contact details of interviewer supplied.
5.12 Transcript of interview with Lord Hayward, 10 July 2019. Contact details of interviewer supplied.
5.13 Media coverage of the findings of the Sturgis Inquiry and House of Lords Select Committee.
5.14 Testimonial from Jane Frost OBE, Chief Executive of the Market Research Society, 29 July 2020.