Impact case study database
Improving the use of empirical evidence to increase returns on public expenditure and investments aimed at improving local and national economic growth
1. Summary of the impact
Research conducted at LSE has affected local and national economic growth policies, particularly through the development and implementation of improved governmental appraisal and impact evaluation methods. Impact on specific policies and expenditure, and on the use of appraisal and impact evaluation within government, can be demonstrated at supra-national level for the EU and at national and local levels for the UK. The work has been especially influential in the areas of business support, transport investment, and the development of local economic and industrial strategies. Its direct influence on policy design and decision-making has delivered important knock-on impacts on the value for money of public expenditure.
2. Underpinning research
Understanding the determinants of local economic performance provides crucial guidance for investment intended to improve local and national economic growth. LSE researchers have developed ways to strengthen both pre-investment appraisal (e.g. of transport) and post-investment evaluation (e.g. of business support). The work has helped address concerns about appraisal and evaluation processes. Those concerns are outlined in academic work and in EU and UK Government reports, including the 2006 Eddington Transport Study and the National Audit Office (NAO) Review of Evaluation in Government, published in 2013.
Impacts described here are underpinned particularly by research by Professors Stephen Gibbons and Henry Overman. This was carried out at LSE since the mid-2000s with colleagues in the Departments of Geography and Environment and of Economics, and in the Spatial Economics Research Centre (SERC), the Centre for Economic Performance (CEP), and the What Works Centre for Local Economic Growth (WWG). Underpinning work focuses on understanding the causes of spatial disparities and developing and applying: (i) appraisal techniques that better capture the wider economic impacts of investments; and (ii) approaches to impact evaluation that use counterfactual methods to more clearly attribute economic outcomes to a specific policy. Three general contributions are relevant; these are outlined below.
1. Estimates of the extent to which skills, density, and connectivity explain area differences in productivity
Developing effective appraisal and evaluation methodologies for local economic policies requires an understanding of the factors underpinning spatial disparities. Extensive CEP and SERC work - conducted in the mid-to-late 2000s - considered two key factors: 1) agglomeration economies that explain area-level differences in productivity; and 2) differences in the level of individual education and skills across areas. Much of the existing literature focussed on the economic mass of places (e.g. their population or employment size) to explain agglomeration economies. The LSE research, however, emphasises the role of transport costs and connectivity. It also emphasises the ways in which agglomeration economies and education and skills interact - for instance because more educated and higher-skilled workers are attracted to places with good connectivity. Illustrative outputs include [1], which develops methods to decompose variance in wages into the contribution from individual and area-specific effects. Applying those methods to British data demonstrates that as much as 90% of the observed inequality in local area average wages can be explained by “sorting” (i.e. the spatial concentration of more educated and higher-skilled workers).
This insight was particularly important in work to understand the wider economic impacts (WEIs) of transport improvements, much of it initiated following the Eddington Review. Transport is viewed here as facilitating agglomeration economies that mean area productivity tends to increase with density and connectivity. Estimates of the size of these agglomeration benefits, and how they respond to improved transport connections, allow the productivity benefits of proposed transport investments to be quantified. This is discussed in [2], co-authored by Gibbons with Professor Daniel Graham (Imperial College London). The paper discusses the link between transport and agglomeration, calculation of the WEIs that arise via agglomeration, and the importance of controlling for “sorting” when calculating these. It outlines a three-step procedure allowing these findings to inform the appraisal of agglomeration impacts within standard cost-benefit analysis (CBA). These insights were essential to recommendations improving transport appraisal (see Section 4).
2. Ex-post evaluations of the economic impacts of infrastructure and other local economic growth programmes and projects
While ex-ante appraisal helps inform policy decisions before they are made, ex-post evaluation looks at what happens after a policy is implemented and what changed because of it. A broad body of LSE work has developed and applied quantitative impact evaluation methodologies to such ex-post evaluation. This research emphasises the value of secondary data sources and appropriate economic methods in evaluating the causal effect of policies. It particularly argues for the importance of constructing appropriate control groups to properly assess the economic impact of interventions.
An important strand of CEP and SERC work applies these techniques to understand the impact of infrastructure and other local economic growth policies. Relevant examples include work to assess the economic impacts of new roads [3] and an evaluation of the UK Regional Selective Assistance Scheme (RSA) [4]. The RSA evaluation [4] considered whether firms would have maintained or increased employment in the absence of this form of state aid, and thus whether it made a difference. It shows that the additional employment triggered by RSA was most significant for smaller firms. Larger firms, however, appeared to “game” the system by taking investment subsidies that did not substantively change their course of action. Methodological lessons drawn from this strand of the underpinning work are summarised in papers including [5].
3. Improving the theoretical and empirical basis for policy appraisal and evaluation
Gibbons and Overman have worked to bring insights from their research on ex-ante appraisal and ex-post impact evaluation to bear on specific areas of UK local and national economic policy. Relevant publications include [6], developed between 2007 and 2009 with the Department for Transport (DfT), to “translate” the academic work on connectivity and density (e.g. **[3]**) and recommend ways to include these wider effects in transport appraisal. A second report [7] was co-authored with their CEP colleague, Professor Sandra McNally. Commissioned as part of a wider NAO project (undertaken in 2013) on the use of cost-effectiveness evidence in the UK Government, it summarised and made recommendations for improving the use of impact evaluation methods.
Finally, the underpinning research also includes many evidence reviews conducted by the What Works Centre for Local Economic Growth, of which Overman is Director. These summarise lessons from existing ex-post impact evaluations, identify gaps in the evidence and make recommendations for improving future impact evaluations [8].
3. References to the research
[1] Gibbons, S., Overman, H. G., and Pelkonen, P. O. (2014). Area disparities in Britain: understanding the contribution of people versus place through variance decompositions. O xford Bulletin of Economics and Statistics, 76(5), pp. 745-763. DOI: 10.1111/obes.12043.
[2] Graham, D. J. and Gibbons, S. (2019). Quantifying wider economic impacts of agglomeration for transport appraisal: existing evidence and future directions. Economics of Transportation, 19(100121). DOI: 10.1016/j.ecotra.2019.100121.
[3] Gibbons, S., Lyytikäinen, T., Overman, H. G., and Sanchis-Guarner, R. (2019). New road infrastructure: The effects on firms. Journal of Urban Economics, 110, pp. 35-50. DOI: 10.1016/j.jue.2019.01.002.
[4] Criscuolo, C., Martin, R., Overman, H.G., and Van Reenen, J. (2018). Some Causal Effects of an Industrial Policy. American Economic Review, 109(1), pp. 48-85. DOI: 10.1257/aer.20160034.
[5] Gibbons, S., Overman, H. G., and Pattachini, E. (2015). Spatial Methods. In Duranton, G., Henderson, V., and Strange, W. (Eds.) Handbook of Regional and Urban Economics, Volume 5B (pp. 115-168). Elsevier. ISBN: 9780444595317. Chapter DOI: 10.1016/B978-0-444-59517-1.00003-9.
[6] Graham, D.J., Gibbons, S., and Martin, R. (2010). The spatial decay of agglomeration economies: estimates for use in transport appraisal. London School of Economics. Available at: https://personal.lse.ac.uk/gibbons/papers/agglomerationreport.pdf
[7] Gibbons, S., McNally, S., and Overman, H. G. (2013). Review of Government Evaluations: A report for the NAO. LSE Enterprise. Available at: https://www.nao.org.uk/wp-content/uploads/2013/12/LSE-Review-of-selection-of-evaluations-with-appendices1.pdf
[8] Evidence Reviews. What Works Centre for Local Economic Growth. Available at: https://whatworksgrowth.org/policy-reviews/
Evidence of quality: **[1]**- [4] are published in highly-ranked peer-reviewed general interest or field economics journals. [5] is published in the leading reference handbook series for economics. Project work that led to [6] was funded by DfT and [7] by the NAO. The WWG [8] is funded by the ESRC and UK Government departments.
4. Details of the impact
The LSE-led research outlined above has supported better-informed policy design and decision-making in local growth. It has encouraged reduced reliance on anecdotal and case study-based evidence alone, and facilitated policymakers’ greater use, instead, of quantitative data and the findings of econometric analysis and impact evaluations. This has had direct impacts on specific policies and wider influence on the selection and implementation of evaluation and appraisal methodologies guiding public expenditure decisions. These impacts have resulted primarily from the use of the work to: a) improve appraisal and evaluation methodologies supporting more efficient public investment; and b) support better decisions on feasible and proportionate appraisal and evaluation processes. Effects have been felt at supra-national, national, and local levels. Illustrative examples of impacts at each of these levels are provided below, though the effects of the work - particularly at local level - have been much more widespread and diverse than can be captured here.
Supranational impacts
The impact evaluation of the UK Regional Selective Assistance Scheme co-authored by Overman [4] directly influenced changes made by the European Commission Directorate-General for Competition (DG COMP) to regional aid eligibility rules on state aid to large firms [A]. Eligibility rules are structured around a set of ex-ante presumptions on allowable aid - the General Block Exemption Regulations (GBER) - and ex-post evaluation requirements. The impact evaluation [4] (which was first published in 2011) showed that regional subsidies have less of an incentive effect on large firms and that awarding such aid to large enterprises is ineffective and costly. It is cited in a 2014 explanatory note published by the European Commission as key supporting evidence justifying a shift toward promotion of regional investment aid for SMEs. Citing [4], the paper states:
“There is a strong body of evidence suggesting that regional investment aid is more effective and efficient when geared towards SMEs (i.e. it changes the behaviour of the aid beneficiary to undertake an investment that contributes to a common objective).” [B]
This shift was achieved by allowing the GBER on large firms to apply only to “initial investment in favour of new economic activities” [C].
It is not possible to accurately estimate the shift in expenditure to SMEs that has resulted from this new guidance, but the potential redirection of funds is at a large scale. EU Member States’ state aid expenditure is increasing year-on-year; EUR121 billion (representing 0.76% of GDP) was allocated in 2018. The study [4] and advice from the What Works Centre for Local Economic Growth (WWG) have helped inform guidance on ex-post evaluation of both current and future state aid in the EU (see, for example, [ **D]**), as well as UK guidance on the evaluation of structural funds [E].
National impacts
On a national level, research by Gibbons has informed the cost-benefit analysis (CBA) methodology used by the UK Department for Transport (DfT) and Transport Scotland. Specifically, insights published in [6] have input directly to the methods used by transport planners to account for the potential wider economic impacts (WEIs) of transport projects, in ways that support more accurate appraisal. The research published in [2] was developed in close collaboration with DfT specifically for this purpose, and this and other relevant outputs continue to inform future directions in transport appraisal methodology.
This direct impact is manifested most clearly in DfT’s Transport Analysis Guidance (TAG) [F], and in Transport Scotland’s Scottish Transport Appraisal Guidance (STAG) [G]. These cite [6] as providing the key parameters used to translate planned transport improvements into quantifiable productivity benefits. By informing specific aspects of the appraisal methods, [6] contributes to decisions about all publicly-funded transport projects in the UK that meet the threshold for TAG appraisal as part of central government approval. This represents hundreds of billions of pounds’ worth of public investment: the 2018 National Infrastructure and Construction Pipeline alone set out more than GBP400 billion of planned investment. Research by Gibbons and Overman [3] [4] is also informing the development of new approaches to the ex-post evaluation of transport investments [H].
In addition, the LSE research has influenced UK Government advice on evaluation more generally. Impacts here are underpinned particularly by Gibbons and Overman’s reviews of prior evaluations, including that produced for the National Audit Office (published in **[7]**), which made explicit recommendations to government about ways to improve its impact evaluation. As Director of WWG, Overman has overseen a suite of outputs and services that build on the work of early reports such as [7] and focus on improving evaluation methods for a wide range of policies. WWG has, for example, been signposted in guidance from the UK Government to support the creation of Local Industrial Strategies [I].
Local impacts
At local level, WWG has had a significant impact on the evaluation methods and activities of local authorities, including those relating to devolution deals associated with billions of pounds of spending. WWG work with Liverpool City Region Combined Authority (CA) provides an illustrative example.
Impacts in Liverpool include a decision taken by the CA in 2018 not to conduct a complex meta-evaluation of their devolution deal. This decision was taken following a discussion with the WWG, whose work suggested that pursuing the meta-evaluation would yield little insight at high cost [J]. The WWG instead suggested that the CA should identify aspects of the devolution deal for which in-depth evaluation would be a cost-effective and productive endeavour. The principle of focusing evaluation efforts and using counterfactuals (central to WWG evaluation methodology) was informed by meetings between Overman and lead officers from the CA in November 2016. This focus on “meaningful” evaluation has informed interventions, fund allocation, and scheme prioritisation across the CA. The CA continues to draw on the work of the WWG, for example in its Strategic Investment Fund Assurance Framework, which governs the management of devolved, regional, and national funding sources. Here, WWG is cited as a source for best practice on evidence-based approaches to policy and investment [K].
The close involvement of WWG in informing Liverpool CA’s evaluation methodology is not an isolated case. The approach advocated by the UK Government for devolution areas draws directly on the counterfactual principle proposed by Overman at WWG. This is demonstrated in a September 2017 Memo from the Head of Paid Service (who has overall responsibility for the management of the CA). The Memo corroborates the application of WWG advice by central government. It notes the importance of “conducting in-depth evaluation of a small number of initiatives, rather than complex and costly programme-wide evaluations, which can be meaningless in practice”. This approach: “ builds on ‘counterfactual’ principles advocated by Professor Henry Overman from the ‘What Works Centre for Local Economic Growth’ at the LSE” [L]. The Memo further notes that this advice is being “impressed upon” all devolution areas [L].
5. Sources to corroborate the impact
[A] For use of LSE research by DG COMP in setting eligibility rules for state aid to large firms, see presentation given by its Chief Economist at LSE (15 December 2018). See, especially, slides 11 (which cites **[4]**) and 15.
[B] European Commission Competition DG (2019), “ Explanatory note on the paper of the services of DG Competition containing draft regional aid guidelines 2014-2020 ” p. 4, n.2 for reference to [4].
[C] European Commission (2014), “ Guidelines on Regional State Aid”.
[D] European Commission (2014), “ Common methodology for state aid evaluation”. See p. 32, n.32 for reference to [4].
[E] European Regional Development, “ European Regional Development Fund Summative Assessment Guidance – Appendices”, June 2020. See pp. 13-14, 24, and 31 for references to WWG.
[F] Department for Transport (DfT), Transport Analysis Guidance (WebTAG):
Unit A2.1, “ Wider Economic Impacts Appraisal”, May 2019. See Box 7, p. 28.
Unit 2.4, “ Appraisal of Productivity Impacts”, May 2018. See pp.14 and 24.
Unit M5.3, “ Supplementary Economic Modelling”, May 2019. See p.19.
[G] Transport Scotland, “ Scottish Transport Appraisal Guidance (STAG), Section 9: Economy”. See References section for inclusion of [6].
[H] Ipsos Mori (2019), “ Economic Performance Impacts of Road Enhancements”. Report prepared for UK Department for Transport. See p. 28, n.9 for reference to [4], and p. 40, *n.*27 for reference to [3].
[I] BEIS (2019), “ Local Industrial Strategies: Policy Prospectus”. See section on Evaluation for reference to WWG.
[J] Liverpool City Region Combined Authority, “ Devolution Deal Monitoring and Evaluation Framework (submission to BEIS)”, September 2018. See p. 5.
[K] Liverpool City Region Combined Authority, “ Strategic Investment Fund Assurance Framework”, December 2018. See p. 21, “Use of evidence”, for reference to WWG as a key source of best practice in investment, for use by the CA “to shape calls and commissions”.
[L] Liverpool City Region Combined Authority, “ Report of the Interim Head of Paid Service: Monitoring and Evaluating the Devolution Deal”. See para. 4.2 for reference to Overman and the WWG in deciding the approach to be taken to evaluation; and para. 4.42 for the wider application of the WWG approach in all devolution areas.
Additional contextual information
Grant funding
| Grant number | Value of grant |
|---|---|
| ES/G005966/1 | £2,336,311 |
| ES/J021342/1 | £1,022,477 |
| ES/L003945/1 | £5,667,701 |
| ES/S000062/1 | £503,705 |
| ES/T007702/1 | £3,792,675 |