Skip to main

Impact case study database

The impact case study database allows you to browse and search for impact case studies submitted to the REF 2021. Use the search and filters below to find the impact case studies you are looking for.
Waiting for server

The Pupil Premium Toolkit: Evidence for Impact in Education

1. Summary of the impact

The Pupil Premium Toolkit is an evidence-based resource for schools seeking guidance on improving outcomes for learners, particularly those from disadvantaged backgrounds. It guides the work of the Education Endowment Foundation (EEF) and its funding strategy for the GBP200,000,000 being spent over 15 years to reduce inequalities in school outcomes in England. International use is growing with localised online versions for Scotland, Australia, Spain and Latin America (with Catalan, Spanish and Portuguese translations). The Toolkit has directly influenced UK Government policy and spending on education and the policy decisions of governments outside England. As the ‘What Works Centre for Educational Achievement’, it has also influenced the development of the ‘What Works’ centres in the UK. It is consulted by two thirds of all the head teachers in England.

2. Underpinning research

The Toolkit is a synthesis of research evidence from 200 systematic reviews, meta-analyses and other quantitative studies [R1], [R2], [R3]. It aims to support schools in spending their resources, especially their Pupil Premium allocation, more thoughtfully and more effectively. The Pupil Premium will account for an estimated GBP16,000,000,000 of education spending in England between August 2013 and July 2020, representing a significant element of resourcing for disadvantaged pupils.

The contribution of the Toolkit is that it provides estimates of the relative benefit of the impact of different approaches on pupils’ attainment, using effect size as a common metric [R1], [R2], [R3]. It also includes an estimate of the financial costs of adopting each of the different approaches. The resulting findings about the relative cost/benefit of adopting different educational approaches on attainment in schools provides highly valued support to schools [R4], [R5]. The initial research drew on data from over 60 meta-analyses and systematic reviews of educational approaches and interventions to improve learning in schools. Clear criteria for selecting the meta-analyses have been identified by the authors so that comparable studies with quantitative evidence from well-controlled experimental studies are included [R3], which is a further distinctive feature.

The research insight provided by this synthesis is that many of the approaches initially chosen by schools to support disadvantaged students are either ineffective, or unrealistic to implement, on the basis of the initial levels of funding made available to schools. It advises that some of the most popular uses of the Pupil Premium, such as appointing additional teachers or the general deployment of teaching assistants, are unlikely, on average, to be cost effective in increasing pupils’ attainment, based on the synthesis of research findings. The Toolkit identifies other approaches, such as providing feedback, or developing pupils’ skills in planning, monitoring and evaluating their own learning (metacognition and self-regulation), and identifies interventions that teaching assistants and others could be trained to carry out, which are all more likely to be successful [R3]. It recommends that schools use the information to inform their decision-making about the most effective support they can provide for disadvantaged learners, and encourages them to evaluate whatever they select.

The Toolkit was produced from funding awarded by the Sutton Trust to a team from Durham (Higgins, Coe & Kokotsaki) to undertake a new synthesis to create a ‘Pupil Premium Toolkit’. This research was undertaken between November 2010 and May 2011 and the resulting Toolkit was published in July 2011 [R1].

The research was adopted by the EEF who subsequently commissioned a team at Durham led by Higgins to extend and update the review annually [R3] over three successive three-year research contracts (between 2012 and 2021), and to use it as a basis to develop a methodology for the comparative analysis of EEF’s commissioned projects. The Toolkit is now presented as a public website and called the Sutton Trust/EEF Teaching and Learning Toolkit, with an overall synthesis of the findings, detailed information about each of the areas covered, references to the sources used and a summary of the synthesis of quantitative evidence used to estimate overall effects. It is routinely updated with new and emerging evidence. Other links aim to support take-up and implementation in schools and further publications have explored the theoretical rationale and methodological approach [R2], [R3], [R4]. A detailed manual sets out the procedures and processes for the synthesis.

The findings in the Toolkit draw on conceptual work undertaken at Durham for the ESRC-funded Researcher Development Initiative (RDI) ‘ Training in the Quantitative Synthesis of Intervention Research Findings in Education and Social Sciences’ (between April 2008 and March 2011). An extensive database of educational meta-analyses was produced for this project and enabled the preliminary analysis of effect sizes for the Sutton Trust.

Higgins has been a Professor in the School of Education since 2006. Kokotsaki is an Associate Professor. Coe was Director of the Centre for Evaluation and Monitoring (CEM) and a Professor in the School of Education until 2019.

3. References to the research

R1. Higgins, S., Kokotsaki, D. & Coe, R. (2011) Toolkit of Strategies to Improve Learning: Summary for Schools Spending the Pupil Premium and Technical Appendices. London: Sutton Trust.

This is the initial research report and summary for schools which sets out the aims of the Toolkit and the key findings with detailed technical appendices setting out the rationale and methodology for the analysis. This report was used in a previous REF 2014 Toolkit case study. It was published by the Sutton Trust, an important educational charity.

R2. Higgins, S. (2018) Improving Learning: understanding meta-analysis of intervention research in education Cambridge: Cambridge University Press https://doi.org/10.1017/9781139519618

This book is a synthesis and extension of a number of previously published chapters and journal articles which sets out the history of meta-analysis in education and the development of approaches which synthesise these reviews, based on the Toolkit research and its development. It has been rated 4* by an external peer reviewer. The book has 14 citations on Google Scholar.

R3. Higgins, S. and Katsipataki, M. and Villanueva-Aguilera, A.B. and Coleman, R. and Henderson, P., Major, L.E. and Coe, R. and Mason, D. (2016). The Sutton Trust-Education Endowment Foundation Teaching and Learning Toolkit. London: Education Endowment Foundation: http://educationendowmentfoundation.org.uk/toolkit/ .

This is an archived copy of the current online version of the Toolkit with additional entries, with further details of the review and synthesis methodology and linked references to the research and studies from which the effect sizes are calculated. The underpinning approach has been reviewed and developed in response to peer review from the EEF’s Evaluation Advisory Group and feedback from stakeholders. It has been rated 4* by an external peer reviewer. It has 226 citations on Google Scholar. The website is published and maintained by the Education Endowment Foundation, a charity dedicated to breaking the link between educational disadvantage and achievement in schools.

R4. Higgins, S. (2016) Meta-synthesis and comparative meta-analysis of education research findings: some risks and benefits Review of Education 4.1: 31–53. https://doi.org/10.1002/rev3.3067

This article sets out the case for the synthesis of data from different meta-analyses, and the limits of comparative inference, drawing on examples from the Toolkit. It appears in a leading UK review journal published by the British Educational Research Association. It has 13 citations on Google Scholar.

R5. Higgins, S. & Katsipataki, M. (2016) Communicating comparative findings from meta-analysis in educational research: some examples and suggestions International Journal of Research & Method in Education 39(3) pp 237-254 http://dx.doi.org/10.1080/1743727X.2016.1166486

This article explores some of the challenges in communicating research findings in education, exemplifies some of the challenges (such as the tension between accuracy and accessibility) and provides exemplification from the Toolkit. It appears in a leading UK research methods journal and has 14 citations on Google Scholar.

4. Details of the impact

4a.) Impact on Educational Understanding, Funding, and Practice

The Toolkit remains central to the work of the EEF who identify it as “transformational in communicating the key messages of educational research to teachers and senior leaders” [E1]. The current EEF Toolkit website receives approximately 28,000 unique page views per month, an increase from 6,500 per month in 2012 [E2]. The Toolkit also directly informs the work that the EEF funds. Since 2014, it has committed approximately GBP96,000,000 to test the impact of 160 high-potential programmes, using the evidence in the Toolkit as a guide when commissioning. Its reach has been extensive and has involved approximately 10,000 schools and more than 1,000,000 children and young people in England [E1], [E2]. The Toolkit has also underpinned an EEF and Northern Rock campaign costing GBP10,000,000 over five years (between 2016 and 2021) to improve literacy achievement in primary schools in the North-East of England [E1]. The Toolkit entries on the impact of 1-to-1 and small group tuition have been “pivotal” to the design of the National Tutoring Programme [E2].

The Toolkit is used directly by schools in England as a source of information for decision-making on pupil premium spending (approximately GBP16,000,000,000 of targeted funding between 2014 and 2020) [E3]. It has a direct influence on the spending decisions of schools in England [E1], [E4]. In 2015, the Toolkit was used by 64% of schools in England (an increase from 36% in 2012) according to the National Audit Office [E4]. By 2019 this had increased to 67% [E2]. Analysis of school websites indicates that more than 13,000 schools mention the Toolkit by name, as an influence their allocation of pupil premium resources [E5].

4b.) Impact on Education Policy

UK Government has acknowledged that the Toolkit is a key part of its policy on the education of disadvantaged children, with 4 citations in the March 2016 White Paper [E6]. It acknowledged that the Toolkit “is helping teachers to find and use evidence about the most effective teaching methods to improve standards for all children, including the most disadvantaged” and cites the Toolkit an example of the evidence-base which sets out “what works and what doesn’t, and which develops and evolves over time”. The Government committed to ensuring that the EEF will continue to have a role in improving the education evidence-base through its role as the designated What Works Centre for education [E6: 2.54; 2.58; 2.63].

The Toolkit is identified as a model for other UK ‘What Works’ Centres and has influenced the design of 4 other evidence comparison ‘toolkits’ in the network [E3], [E8]. The Toolkit is also cited in the media in the UK and US as an authority for education research [E7].

4c.) International Reach

The EEF has established partnerships with a number of global organisations [E8] based around the Toolkit and its potential development in different parts of the world. In 2014, they began working with school systems in Australia to develop a localised version of the Teaching & Learning Toolkit, taking the global evidence base that underpins the Toolkit and contextualising it with recent examples of local research to enhance its relevance for Australian teachers [E8], [E9]. In 2016, CommBank supported this work with AUS50,000,000 (GBP28,000,000 (09-2020)) and is working in partnership with Social Ventures Australia to commission three large-scale randomised controlled trials, which will provide further evidence of impact that will be added to the Toolkit analysis [E10].

The Australian work was extended in February 2017 by Education Scotland, with the development of a Scottish version of the Toolkit [E8] to strengthen the use of evidence underpinning the Scottish Attainment Challenge. In July 2017, the EEF entered into a new partnership in Latin America and the Caribbean with SUMMA (the Laboratory of Education Research and Innovation for Latin America and the Caribbean). This partnership has seen the translation of the Toolkit into Spanish and Portuguese, along with the integration of more than 250 recent local studies that help contextualise the evidence for Latin American and Caribbean teachers, and contribute to the global knowledge about the Toolkit strands [E8].

In June 2018, EEF, in partnership with BHP Billiton Foundation, announced a 5-year GBP9,800,000 fund aiming to improve learning outcomes for disadvantaged pupils across the world by building a global evidence network with the development of the Toolkit as a central aim [E1]. As a result of this development, the EEF entered into partnership with EduCaixa (Fundación Bancaria “la Caixa”) to develop a Spanish version of the Teaching & Learning Toolkit. This partnership has seen the translation of the global evidence base into Spanish and Catalan, providing teachers across Spain with access to robust evidence about effective educational approaches [E2], [E8].

In 2019 the EEF established a partnership with the Queen Rania Foundation which will see the translation and contextualisation of the Teaching and Learning Toolkit into Arabic for teachers and policymakers in the Middle East and North Africa region. A similar agreement with Effective Basic Services Africa (eBASE) will see the adaptation of the Teaching and Learning Toolkit for teachers in Cameroon, Nigeria, Chad, and Niger and will include a French translation [E2].

5. Sources to corroborate the impact

E1. Educational Endowment Foundation Annual Report 2018.

E2. Testimony from Becky Francis, Chief Executive Officer, The Education Endowment Foundation [28.08.20]

E3. What Works Network Report: (2018) The What Works Network: 5 Years On (2018)

E4. National Audit Office Report on Funding for Disadvantaged Pupils (2015)

E5. School Websites: advanced searching limited to the domain “.sch.uk” retrieves approximately 13,000 hits all mentioning ‘Toolkit’ and either ‘EEF’ or ‘Sutton Trust’ alongside ‘pupil premium spending’ [11.09.20].

E6. Department for Education Educational Excellence Everywhere White Paper 2016

E7. Compiled Media Evidence [The Economist, 11.06.2016; The Guardian, 21.07.2017].

E8. International Versions of Toolkit [Australian, Spanish, Portuguese, Scottish, 27.02.20].

E9. Evidence of impact report on behalf of Evidence for Learning (E4L) Australia [04.08.2020].

E10. CommBank Media Release [23.06.2015].

Additional contextual information

Grant funding

Grant number Value of grant
EEF: Toolkit01-2012 £131,125
EEF: ToolkitNPD02-2015 £452,819
EEF: ToolkitArchive04-2018 £881,739
Sutton Trust: Durham_2011_Strategies £13,750
Sutton Trust: Durham_2011_Improving learning £27,572