Impact case study database

Validation of predictive policing software

Mastrobuoni’s research on the economics of crime evaluates both reducing criminal activity through predictive policing but also the tendency of a convicted criminal to reoffend. His research on bank robberies to estimate individual-level disutilities of imprisonment, based on a collaboration with the Italian Banking Association (Associazione Bancaria Italiana), [R3] led to an ongoing collaboration with an Assistente Capo in the Italian Police who established an innovative start-up company ‘KeyCrime’, specialising in a predictive policing software called ‘delia®’.

The founder of KeyCrime approached Mastrobuoni to research the impact of data analysis on police investigations. The results of the long-run pilot in Milan alongside the academic evaluation of KeyCrime’s predictive policing software by Mastrobuoni [R1] created significant interest in the software (which was later named delia®), guided it’s development and enabled the founder to leave the police force in 2017 in order to run the company as a viable business [S1].

The evaluation by Professor Mastrobuoni guided the development of this predictive policing software from early on in the company’s inception, which subsequently influenced the efficacy of police investigations in Milan. On the collaboration the Founder and CEO of KeyCrime confirms: '… Research and the first independent evaluation of any predictive policing software, 'Crime is Terribly Revealing' by Professor Mastrobuoni enabled KeyCrime to validate its results [R1]...The software has produced validated results in the area of prevention and repression of crime which can be seen in the data made public by the system's users, the Polizia di Stato in Italy. Other institutions have also supported the use of KeyCrime such as the office of Prosecutors of Milan who, in their annual report cite it as a useful tool to support their investigation’. [S1]

The evaluation undertaken by Mastrobuoni [R1] also cemented the formal adoption of the software by law enforcement agencies in Italy in 2018 [S2]. It is embedded into Italy’s national police force, the Polizia di Stato, in the Province and city of Milan where they have reported a 58% reduction in commercial robberies in the city of Milan and 89% reduction in the province of Milan for bank robberies over the period that the software has been deployed [S3].

The Founder and CEO of KeyCrime states: 'The publication of the paper [R1] has, without any doubt, brought additional attention to the company KeyCrime s.r.l - an innovative start-up formed at the beginning of 2018 - as well as credibility due to the scientific and statistical validation detailed in Professor Mastrobuoni's paper. This validation is a fundamental element for us and has helped guide the direction of our investment in R&D to extend the approach that the paper described'. [S1].

KeyCrime now has 20 employees and has received millions of euros of finance from venture capital firms. In 2018 formal partnerships with companies sharing a common vision to help make communities safer were announced with IBM and sdg Group (one of the most important company in the Multinational Analytics Market) to propose delia® software to the global market for corporate use and to deliver a delia® based solution at the consumer level. While in 2020, PSS, an important Spanish system interrogator, has partnered with KeyCrime to resell delia® predictive policing software to the Spanish and Latin American markets [S3a].

Following the pilot phase after the evaluation of the software by Mastrobuoni [R1], a statement by Milan’s Head of the State Police in the Giornale scientifico a cura di O.N.A.P December 2014 confirmed the benefits achieved by the predictive policing software which was subsequently formally adopted

“The uses of KeyCrime, which was experimented in Milan’s Police headquarter, have largely met and overcame our expectations. It has proven itself not only capable of meeting our operative needs, but also of indirectly contributing to let the administration’s cost plummet, in a period in which the resources have to be rationalized and well managed in order to grant high security standards in our cities. Thanks to the use of this system, Milan’s Police headquarters succeeded in significantly reducing bank and commercial-related robberies, becoming a deterrent; this was also possible thanks to the success in identifying numerous perpetrators of such crimes”. [S4]

Reducing recidivism

Professor Mastrobuoni's research on evaluating the prison conditions that are more likely to aid rehabilitation and reduce criminal recidivism [R3] was used by the Commissioner for prison reform in Italy and the Guarantor of the rights of persons deprived of personal liberty in Milan to evidence the role that the condition of prisons plays in the tendency of a convicted criminal to reoffend. This was the only research cited in the House of Representatives by the Commissioner for prison reform in January 2018 in relation to the XVII legislature and recorded by the Bulletin of the Joints and Parliamentary Committees Justice [S5]. The Guarantor utilises Mastrobuoni’s research to evidence that prisons with a higher rate of condition, (such as Bollate), should be considered as a model for the entire penitentiary system to improve reductions of recidivism and positive reintegration of convicted criminals at the end of their sentence [S6]. Subsequent recommendations set out by the prison reform commission and the Guarantor, include a booklet published in 2018, ‘Norms and Normality’ which contains a set of standards for the incarceration of adults in Italy including that the standards of detention should not be just ‘minimum’ [S7].

The Guarantor of the rights of persons deprived of personal liberty by the Municipality of Milan’s (est. 2012), whose mandate includes the promotion of rights and opportunities of persons deprived of their liberty, cites Mastrobuoni’s research [R4] in its 2015 Annual Report [S8].

The European Commission (EC) commissioned the Study [R4] as part of the wider consultation of the revision of the framework offering a neutral evaluation of the Policy [S1]. The authors’ research record demonstrates that they are theorists with a research history that is entirely policy-focussed, relates to areas of work that were pertinent to this revision (competition policy issues in licensing and standard setting), integrates law and economics frameworks, and appeals to a multidisciplinary audience. Commercial confidentiality and the paucity of relevant legal cases reduce related empirical work, making theory the appropriate tool of analysis. This made the authors of the Study [R4] uniquely suited to the task.

Impact on legal frameworks

Essex research directly led to amendments to legal frameworks and regulations affecting consumers across the EU and entities doing business in the EU through the take up of recommendations revising the European Commission’s (EC) technology transfer guidelines and technology transfer block exemption regulation (TTBER), which came into force on the 1st May 2014. This Policy is revised once every 12 years and remains a legal requirement until 2026 [S1].

Impact on competition and benefits to consumers

The remit of this regulation covers any business of a sufficient size conducting activity that affects European consumers. That includes businesses outside Europe that sell into Europe, those located within Europe that sell within Europe or otherwise do business in Europe that involves technology transfer. The TTBER and guidelines (“Policy”) ensure that consumers have the quality, affordability, and variety of products needed where those products rely on technology access agreements, and that firms wishing to compete do not face exclusionary or anticompetitive behaviour.

On the revisions of the Policy the EC states: ‘ The goal of this revision is to verify that the Commission's competition policy as regards technology transfer agreements still reflects the right balance between providing effective incentives for competitors and non-competitors to enter into innovation and welfare increasing technology transfer agreements while ensuring that such agreements do not undermine economic welfare by unnecessarily distorting competition’ [S2].

Rebalancing the interests of consumers vs other stakeholders

Stakeholders (“submitters”) to the public consultation stated that they considered policy from a private (profit driven) perspective; however, the goal of EC competition policy is improving consumer welfare, including innovation concerns. The authors conducted the Study [R4] to fulfil those needs with the first comprehensive economic welfare analysis of the Policy in its history. Crucially, then, the Study recommendations counter-balanced the submissions by taking a consumer welfare perspective [S3]. Submitters read and reacted to the Study [R4] as part of the consultation, but comparing the EC Submissions Overview (“SO”) to the Study coverage and argumentation illustrates a clear distinction in priorities and argumentation between submitters and the Study [S3]. The EC Impact Assessment’s three main areas of focus cover two highlighted in the Study (grantbacks and patent pools) [S2] and one highlighted by submissions (termination clauses). Given the divergent recommendations of the Study and submitters, this made the adoption, at least in part, of 15 of the 16 Study recommendations remarkable. [S4, S5]

Innovative use of economics to evaluate a legal framework in this context

Evaluating legal frameworks from an economics standpoint is commonplace in North America, but is a more recent development in Europe. Furthermore, a theoretical perspective is not common in itself, as explicitly stated by submitters [S4, S5].

Reach of recommendations within and beyond the EU

The recommendations from the Study [R4] focussed on technology transfer agreements, agreements by which one party (licensor) authorises another (licensee) to use its technology for the production of goods and services. The recommendations of the Study [R4] relate to specific changes in the regulation of technology transfer agreements that allow consumer products to be manufactured and sold, affecting the design and market of such products. The impact comprised both implementing policy change with wide scope and influencing the debate leading to those modifications. The EU-wide and global scope is described in the EC’s own Impact Assessment as affecting, conservatively, 56,000,000 workers , which was about a quarter of those in employment in the EU, and €4.7trillion, which was almost 40% of the total GDP of the EU in 2014 [S2]. These figures do not consider indirect or international effects, which are evidenced in the international nature and comments of submitters to the consultation as well as the international scope of the regulation.

Impact on grantbacks

The starkest impact to policy is the tightening of the regulation of grantbacks, now removing all exclusive grantbacks from the regulation’s exemptions, and instead requiring individual assessment by the companies [S2 IA at 1.3]. Consistent with recommendations C12 and C13 of the Study [R4], the distinction between the treatment of severable and non-severable technologies was abolished, leading to an overall stricter approach [S2 IA at 1.2.3].

The Submissions Overview [S3 SO points 13 and 14] shows that this change was overwhelmingly opposed by stakeholders [S3]. The need to counter-balance these arguments and evaluate policy based on the EC’s stated consumer welfare criteria is clear. In contrast, the SO showed [S3 SO at 15 and 16] submitters’ full support for the recommended changes in patent pool treatment. The submitters’ views are consistent with the recommended non-inclusion of pass-through [S3]. Market share thresholds [S3 SO at 6-8] were a prominent concern for submitters but were not considered an issue for the Study, and subsequently were not addressed in the revision.

The 2014 Policy’s patent pool and grantback changes were consistent with the authors’ recommendations [S4, S5]. The dearth of work on mergers for innovative industries, clear in the Study [R4], pointed to this becoming a main research area for the EC, with two subsequent publications by the Chief Economist (DGComp) focussing on merger policy for innovation. The improved treatment of Termination Clauses in the updated guidelines had the effect of tightening cross licensing treatment, as recommended by the authors in the Study.

Influencing debate

The balanced arguments for change in the EC Impact Assessment draw on both the Study [R4] and consultation responses, illustrating that the Study influenced debate. The Study also broadened debate to issues that stakeholders would not otherwise have addressed, such as the unification of grantback treatment for different technology types. Submitters only discussed pass through and merger control where prompted to respond to the Study [R4].

The Deputy Chief Economist of Directorate-General for Competition in the EC receiving the report commented, “ The report [R4] had a significant impact on the revisions that were instituted, perhaps most visible with respect to so-called "grant-back" obligations. Here the report, based on original work developed for the report, argued that a specific distinction made in the previous rules between "severable" and "non-severable" improvements might be misplaced…[It] was a pleasure receiving a report containing original work that ended up influencing policy. In my humble opinion, many reports produced for the Commission do not really move the debate and are, perhaps, not the best use of taxpayers' money. This was clearly not the case here. This report [R4] made a difference.” [S6]

Changing the way UK Government and Parliament view the objectives and effectiveness of the free childcare policy

The main aims of the free childcare policy in England were to simultaneously support child development and encourage maternal employment, thereby creating a ‘double dividend’ (Cabinet Office, 2002). Our research constitutes the first causal evaluation of this important and costly government policy and we presented results to HM Treasury (HMT), Department for Work and Pensions (DWP), Department for Education (DfE), the Welsh Government, the Government Equality Office and others. Our finding that the policy had to a great extent crowded out private investments into early education with little behavioural change among parents and consequently very small and short-lived effects on children’s outcomes and modest increases in female labour force participation significantly changed the way that Government viewed the effectiveness of the free entitlement policy. In particular, our research clarified to Government and policy makers the trade-offs inherent in early years’ policy in the UK. Our written and oral evidence to the House of Lords Select Committee on Affordable Childcare is cited in their report multiple times, highlighting that the policy had little impact on child outcomes or maternal employment [S1]. The Treasury Select Committee on childcare policy and its influence on the economy cites our evidence and highlights that free childcare had not led to a significant impact on working patterns of parents [S2, S3].

Our research has aided policy makers to understand that the policy instead has a substantial transfer element and has displaced expenditure by parents to a significant degree. Department for Education (DfE) Chief Analyst wrote, “ *We have briefed our Minister, Sam Giymah on your research and it fed into discussions about what objectives we are really pursuing in early years.*” [S4]. Consequently, for the first time, the aim of helping families with the cost of raising children was included as a policy aim of the extension of the free entitlement from 15 to 30 hours for working families from September 2017 in England.

The research provided a unique evidence base forGovernment on the effect of free childcare on maternal employment. The finding of a larger effect of full-time rather than part-time subsidies has led the Government Equalities Office to take “ *a more nuanced approach to the policy changes we were seeking.*” [S5].

Improving understanding of third sector organisations and interest groups on ‘what works’ in Early Years Policy

We provided evidence on the Early Years policy features that are related to favourable outcomes for children. We demonstrated the limited importance for child development of having graduate teachers in childcare settings while documenting a significant role of settings that have been rated as Outstanding by Ofsted [R2]. This provided a strong challenge to third sector organisations and interest groups. For example, Save the Children was campaigning for graduates to be present in all childcare settings. They say “ Their findings (…) have challenged our thinking and led to us to consider what quality means in early education, how it influences children’s early development and what we can do to ensure all children have access to high quality settings.” [S6]. Ofsted was made “ *to think carefully about what the particular features of Outstanding nurseries might be, and to focus much more of [their] published statistical analysis on Outstanding settings rather than the combination of Good and Outstanding as [they] did previously.*” [S7]. The Executive Chairman of the Education and Policy Institute states that our research has made it “ clear at policy level that a more nuanced view of the link between qualifications and quality is needed.” [S8].

Influencing research strategies by funders and government departments

Essex has influenced research strategies by funders [S9] and government departments [S4]. The Nuffield Foundation launched their new Early Years research stream with a report on lessons from evidence and future priorities that drew heavily on the results of our research. Our work “ helped to shape our research priorities in this area and led to a new research programme on Early Years Education and Childcare.” [S10.] Further, Essex worked with Frontier Economics on a project for HMRC and DfE on how best to evaluate the tax-free childcare, and the extension of free childcare to 30 hours [R3]. We have advised the Early Intervention Foundation on research strategy, explaining the uses and limitations of existing data sets and fed back on their research plans. Essex submitted to a DfE consultation on surveys on childcare and early years in England contributing to changes of the focus of the surveys from ages 0-14 to pre-school aged children [S11].

Essex research has led to improved teaching evaluation practices in Higher Education with evidence of change from 14 institutions across Europe, Canada and the United States. Specifically, at the University of Alberta, University of Birmingham, Brown University, University of Colorado, Hertie School of Governance Berlin, Maastricht University, McGill University, Oregon University, University of Rochester, Sheffield University, Tilburg University, Toulouse School of Economics and Union College. Based on the number of teachers and students at these 14 institutions, we estimate that the resulting changes have directly affected approximately 25,000 teachers and 425,000 students as well as many more indirectly.

The research [R1] has led to (i) changes to how teaching evaluations are conducted at several institutions including use of the research within teacher training, (ii) changes to how junior teachers approach teaching and view teaching evaluations, (iii) informing policy debate around teaching evaluation and public debate through press coverage and online communication of the research to lay public readerships.

Changing how teacher evaluations are conducted in Higher Education Institutions

The research has impacted on a wide variety of Universities across the regions discussed in terms of the establishing of task forces and reviews and changing institutional policies and practices.

The research was instrumental in establishing a task force to consider teaching evaluations at Oregon University and at Brown University (both United States), as a result, teaching evaluation practices have changed to counter gender bias.

At Brown University the committee’s final recommendations include specific measures to mitigate the potential bias including changing questions from more holistic questions to questions about specific aspects of teaching quality and asking students to back up their scores with examples. Essex research [R1] is explicitly referenced in the final report including in the recommendations section [S1] and in the current teaching resources on how to interpret your course feedback [S1a]. Brown University piloted their new course feedback in spring 2018 and subsequently adopted these recommended changes in the 2019-2020 academic year [S1b].

In 2019, based on this research [R1], the University of Oregon (UO) set up a task force and subsequently introduced a holistic new teaching evaluation system that does more than simply replace problematic evaluation instruments so that they can help the UO community more effectively define, develop, evaluate, and reward teaching excellence [S2]. The task force originally convened to address the issues, set up a website with their results [S2a], and the Chair of the task force confirmed that Essex research was crucial for the establishment of the task force, as well as their results.

'Without this evidence [R1] we would not have been able to help our University Senate to understand the potential harm that is being caused to female faculty when evaluation of teaching relies predominantly on potentially biased student ratings.' [S2b]

The faculty council at Rochester University, Unites States, have decided to collect evidence from stakeholders (Deans, Heads of Departments) as to how they use teaching evaluations in direct response to the research [R1]. The chair of the curriculum committee at Rochester confirmed that the research was and remains crucial in this process.

“the Faculty Council at the University is going to establish a committee to study what teaching evaluations are actually used for by various stakeholders, such as deans, department chairs, faculty, and students.” [S3]

This research has been raised in connection with a motion to the General Faculties Council at the University of Alberta [S4], it has been extensively discussed in committees at Maastricht University and at Tilburg University. At Maastricht University the Dean for Education has asked for proposals of how to mitigate the problem [S4a].

At Hertie School of Governance a review is ongoing which was triggered by the research [S4b]. The McGill Tribune at Mc Gill University in Canada has published a piece urging their University to act upon our findings [S4c]. The magazine of Toulouse School of Economics in France also discussed our research [S4d].

Changing how teaching evaluations are approached and viewed

Mengel’s research has been used in teacher training and has also featured in a “MicroCPD” video and webpage developed for educational training at the University of Birmingham [S5, S5a]. It was used in a teaching programme at University of Colorado, Colorado Springs in 2017 [S6]. Junior teachers at the University of Birmingham, at Maastricht University, the University of Sheffield and Tilburg University confirm that the evidence made them reflect on both their teaching style as well as procedures in their University regarding teaching [S7].

Informing political and public debate around gender bias in teaching evaluations

The research has been used to support the case made by the Norwegian Academy of Young Researchers for changes to criteria for teaching competence at all Norwegian educational institutions. The research was cited specifically to support their recommendation that assessment of teaching quality should not give too much importance to student feedback [S8]. The German Economic Association has discussed the research in an email sent to its members and has set up a panel to discuss the issue of gender bias in economics where Mengel presented the research [S9].

Professor Mengel’s research has led to an increased understanding of gender bias generating a considerable amount of public discussion. The research has been tweeted about by more than 2000 tweeters (January 2021) located in Europe, North and South America, Asia, Australia and Africa with an estimated 6,506,927 of followers (Altmetric, January 2021). It has also been discussed in news articles in several major national and international newspapers and radio programmes with a combined readership of over 2.5 million including in The Economist, Sueddeutsche Zeitung (a German national newspaper), in a radio-show on Radio Bavaria (Germany), the Wire (India), Scientific America, Forbes, World Economic Forum , New York Times and others [S10].

Professor Alan’s research and expertise led to her involvement as a key academic advisor on the importance of socio-emotional skills and key non-cognitive skills for institutions, banks and policy makers in Turkey and Macedonia. Collaborating with ING Bank, the Turkish Ministry of National Education and the World Bank, Professor Alan developed training materials, implemented and evaluated unique and innovative educational interventions to foster key non-cognitive and socio-emotional skills as an indispensable element of children’s academic success.

Changes to the National Curriculum in Turkey

Alan’s research on the role of non-cognitive skills in success alongside the application of grit measurement methodology, the ‘Alan-Ertac Grit measure’ [R2] led to the implementation of a new education policy to restructure primary and secondary education in Turkey to foster socio-emotional and non-cognitive skills such as patience and grit through a nationwide change to the curriculum.

The research, impact evaluations and RCTs, which involved designing and implementing unique curricula and teacher training practices to improve key socio-emotional skills in socioeconomically disadvantaged elementary school children, shows that programmes which foster grit and instil a growth mind-set in children, significantly improve student attainment and behaviour [R2] [S1]. The intervention was a low-cost way of fostering non-cognitive skills in the classroom and thus offers a solution for reducing achievement gaps where educational policy actions aiming to enhance family inputs face challenges in engaging families of low socioeconomic strata. This research directly influenced the design and subsequent delivery of a new element in the curricula for primary and secondary schools to teach non-cognitive skills and prompted the Turkish Ministry of National Education to amend their overarching curriculum with the introduction of ‘Design-Skill Labs’ [S2].

The Turkish MoNE sets out in its 2023 vision that the curriculum is restructured as a series of flexible and modular structures, linked with skill sets, in an integrated manner across all levels. Based on Alan’s research and intervention [R2] a key part of this is the national rollout of the Design-Skill Labs, which form part of the teaching in classrooms and affect the built environment with schoolyards re-designed to transform them into spaces with real life experiences [S3]. The amendment to the curriculum is set to continue from primary into secondary education, Goal 2 of the 2023 Vision: Design-Skill Workshops will be established at schools so that the knowledge gained can be translated into living skills [S4].

Turkey's Education Vision 2023 published by the Ministry of National Education states:

‘Design-Skills Labs will be established at all schools for the development of the interests, talents, and character of our children. The curriculum will emphasize production, doing, interaction and deepening instead of revolving around testing and lecturing. Design-Skill Labs will serve as the instrument of such a curriculum approach. The Design-Skill Labs began as a pilot during 2019-2020. They will help students experience a process of self-revelation in which they can take time to think, design, and produce.’ [S4 p25]

Improved financial literacy through the ING Bank Orange Drops programme in Turkey

Essex research on fostering key non-cognitive skills such as patience and self-control [R3] led to Professor Alan’s involvement with ING Bank’s Orange Drop financial literacy programme for children aged 8 and 9. Alan led the development and implementation of this financial literacy programme, which educates Turkish schoolchildren about the importance of saving and aims to make positive change in communities [S5].

Alan’s research [R5] identified and developed a number of key skills around ‘patience’ and ‘grit’ and she led an interdisciplinary team of experts who focused on each of these skills in isolation to develop a novel curriculum for each.

Following the initial Orange Drop intervention led by Professor Alan a change in behaviour was reported. The ING Annual Report of 2017 states:

• Students became 21% more patient after the program.

• Students learned to be future-oriented and to wait for bigger gains.

• While boys were less patient than girls at the beginning of the program, a stronger change in behaviour was observed in boys afterwards, and boys became 23% more patient.

• A positive relationship was established between patience in spending and academic success.

• The impact on relatively poorly achieving pupils in school was a more pronounced 36% [S6].

This programme is the first single financial literacy education and social responsibility project by ING for Turkey's elementary school students, and encourages positive behavioural changes addressing low personal rates of saving and the lack of financial literacy. Since ING has been running Orange Drops, 1,213 teachers have taught the programme in 329 schools, benefiting 38,835 schoolchildren [S7, S8].

Cultivating non-cognitive skills in students in North Macedonia

Professor Alan was invited to be an academic advisor for the second phase of a World Bank funded project in North Macedonia (titled ‘Learning By Doing’) cultivating socio-emotional skills in 8^th and 5^th grade students. Building on existing research [R1, R3], as well as the successful interventions on grit and perseverance in Turkey, Professor Alan contributed to the design of a unique curriculum on socio-emotional skills which concentrated on expanding the existing program and introduced a unique mentorship component in the second phase [S9]. Alan generated intervention materials to be used in schools in North Macedonia that focused on socio-emotional skills developed to close gaps in education and labour markets [R4, R6]. Professor Alan designed and ran the evaluation, the measurement instruments and analysis of the intervention programme between February 2019 and May 2019. The objective of Learning By Doing was to implement a mind-set intervention as part of the current high school curricula to cultivate growth mind-sets and students ability to set goals and act on their intentions amongst 8^th and 5^th grade students in a subset of schools in North Macedonia.

The World Bank confirm: ‘In light of her expertise and success implementing interventions at scale, Professor Alan was invited to collaborate with the World Bank team on the second phase of a socio-emotional skills program in North Macedonia…Initial results show significant and positive impacts of the intervention on deliberate practice beliefs, growth mindset, curiosity and frustration tolerance for the average student…Results led to the government to start a third phase of work (postponed due to Covid-19) focussing on teachers practices’ [S9].