Impact case study database

The research of the Imperial team informed public and government understanding of the epidemic worldwide, was used by governments to justify maintaining restrictions, and had direct and extensive impact on policy decisions, particularly in Scotland (from April 2020) and New York State (from May 2020) and in the imposition of stronger restrictions in England in December 2020.

Public understanding and awareness: The research attracted strong public attention: At least 484 news stories from 332 outlets (e.g. BBC, Economist, Times, Financial Times, National Public Radio (US), Boston Globe, Washington Post) reported on [1-7], see [A]. The Altmetric score of [1] is greater than 6300, putting it amongst the top 250 articles of more than 16 million articles tracked on Altmetric [A]. The website reporting updates from [6] had more than 750k visitors from 142 countries up to the end of 2020 [A]. The news coverage was helping public understanding by picking up several results from the research, e.g.: results from [1] were used to show how many lives had been saved through the interventions (Daily Telegraph, **[A]**), that more individuals than previously thought has been infected (Economist, **[A]**), and to describe the state of the epidemic around the time of the first lockdown and to compare it to other European countries (The Times, **[A]**) ; local newspapers used [6] to inform about the likely development of the epidemic in their local area (Google News search, **[A]**).

Policy makers and health authorities used [1,4] to assess and evidence the effectiveness of interventions:

The Scottish government's assessment of lockdown effectiveness highlights our impact within this area. The Chief Statistician of Scotland states that “[…] *these methods have been crucial in our assessment that the lockdown in March in Scotland was effective, giving us confidence to sustain it and to be able to communicate our decisions to the general public.*” [C]

Members of the French government (the Health Minister, the Directeur Général de la Santé, President Macron) used [1] to demonstrate that lockdowns were working and to encourage the public to stay at home, using the slogan “one life saved every 8 minutes” [D].

The State of Michigan cited [4] in a legal case defending the legal basis of their public health measures, as well as on social media and in a press release, in the face of vehement opposition from a fringe (and violent) minority [E].

Impact on policy decisions, the general population and saving lives

The Scottish Government used [1,5,6] extensively. The Chief Statistician of the Scottish Government writes [C]:

*“Based on the source code released from Report 13, we developed a model for Scotland […]. This has been key in our decision making process throughout the epidemic. This model has been run on a weekly basis since the end of April 2020 […] Outputs from our model have also been central to the Scottish Government’s regular COVID-19 modelling updates, of key interest to members of the public. […] This has been a crucial input into ministerial decisions on advice for the Scottish population, including on whether to relax or to impose government restrictions.*”

From 29 October 2020, the Scottish government reports our local area estimates [6] and has been using them as criteria for areas transitioning between lockdown tiers [C]; 3 of the 5 criteria used are based on [6].

Overall, there has been strong impact in Scotland; quoting **[C]:

“[…] the methods developed by Imperial have been crucial in the Scottish Government’s response to the COVID-19 pandemic. These methods have formed a crucial part in many of our decisions during the pandemic and thus have had a profound impact on the entire Scottish population, helping to save many lives.”

Based on [1,4], the Imperial team developed a model for New York State (population ~19 million),

*“whose results were discussed weekly […] in the state’s determination of when regions should begin easing restrictions […] the modelling results have subsequently been used to assist in identifying hotspots and inform additional control strategies. The Imperial college team has been an important partner … in informing the state’s COVID response.*” [F]

New York State appointed a team of experts, including Dr Bhatt, “who, before giving a region the green light to move from one phase to the next, would review our data and then advise whether it was safe to continue the reopening of that region of the state” (Cuomo, **[F]**).

A model based on [1] has been influential in the State of Tennessee: “ We were able to construct our first model in one day, which we would never have been able to do without the work you shared. […] This model helped solidify the value of such measures, and safer-at-home in particular, for several leaders at a critical point in local decision making.” [G]

Tier 4 restrictions in England: On 19 December 2020, the UK government imposed stricter restrictions in several areas in England, due to the spread of a novel SARS-CoV-2 variant. The Prime Minister said: “NERVTAG’s early analysis suggests the new variant could increase R by 0.4 or greater. Although there is considerable uncertainty, it may be up to 70% more transmissible than the old variant.” [H] (NERVTAG is the “New and Emerging Respiratory Virus Threats Advisory Group” advising the UK government.)

The figures in the PM’s statement are the result of analysis by Imperial scientists (a group including Professors Ferguson, Gandy, Drs Mishra and Volz), which was presented to NERVTAG on 18 December 2020 [H] and later published as [7]. The Imperial analysis was based on genomic data as well as on linking local estimates of the reproduction number [6] to estimates of the proportion of the new variant present. Based on these results plus two further pieces of evidence from other scientists concerning the amount of virus present in samples of the new variant (PCR Ct values, viral load from genomic analysis) NERVTAG reached its conclusion [H].

By 30 December, twenty million more people in England were subject to the toughest level of tier restrictions. Further consequences of these results were travel bans imposed e.g., by France and Germany, to stop the spread of the new variant of SARS-CoV-2.

Further Policy Impact: Policy documents from the African Union, the European Union and the World Bank cited [1], see Altmetric, [A]. Prof Gandy gave evidence in the UK House of Commons based on [6] regarding the local spread of COVID-19 in the inquiry on “Lessons learnt from Covid-19” [B]. Prof Gandy and Dr Bhatt were appointed to the SPI-M subgroup of SAGE (Scientific Advisory Group for Emergencies) from 30/10/2020, contribution to consensus statements, for example through modelling outputs about the effectiveness of the first UK tier system.

Demand for air travel (despite the current pandemic) will continue to increase [A, B], with low drag airframes forming a cornerstone of design philosophy. Airbus continues to invest in LFC for the next generation of narrow body, hydrogen fuelled and battery–powered aircraft [C]. LFC technology offers up to 5% lower CO2 emissions corresponding to 3,000 tons of CO2 saved per aircraft annually [D]. Airbus states that:

“Wing design and production is a key capability for the UK. The supply chain impact of Airbus to the UK economy is worth £5 billion annually.” [E]

Imperial research contributed to key new technologies for the optimal design of laminar-flow wings for Airbus’s next generation aircraft programmes. This is a long-standing collaboration which resulted in co-authored publications, both in the underpinning research [2,3,4,6] and applied studies [F]. Airbus is using these advanced theoretical and computational tools in the industrial process of laminar wing design optimisation:

“Our collaborations with Imperial College have been ongoing for some time, but the period from 2015-onwards has been the most significant for us regarding its impact on our research direction in the field of Laminar flow wing design. This is due to the progression of the Imperial research from crucial fundamental aspects into an enhanced suite of theoretical tools that we are using in an industrial wing design setting.” [E]

The quality of methods is rated highly by Airbus

“The measure and scope of the work has been truly impressive due to (i) the robustness of Imperial software models enabling investigations of practical engineering problems, and (ii) the advanced physics fidelity of the Imperial software.” [E]

These tools allow Airbus to conduct investigations that have been previously impossible.

The Imperial state-of-the-art modern transition prediction methods for industrial design, have enhanced Airbus’s laminar flow analysis capability enabling the solution of previously intractable problems. ” [E]*

The new tools also aid LFC tests in industrial transonic wind-tunnel testing [G] led by the Aircraft Research Association (ARA).

Airbus invested €200 million towards the LFC BLADE project where the outer sections of an A340 aircraft had specially designed laminar flow wings [H]. This was part of the €4 billion EU Clean-Skies II initiative to reduce noise and emissions between 65-90% by 2050. Methods developed at Imperial were used in the analysis of the test flights:

“The computational suite comprising the 3D surface marching PSE algorithm (PPM-PSE3D) developed at Imperial and delivered to Airbus, was used by Airbus to analyse and interpret the data of its laminar flow wing demonstrator project using a A340-300 MSN001 “flight lab” aircraft. The flight tests (in 2017) were part of the Breakthrough Laminar Aircraft Demonstrator in Europe (BLADE) project sponsored by the EU Clean-Skies II initiative.” [E]

The impact of the analysis is the furnishing of a high-fidelity design tool for Airbus:

“The position of the laminar-turbulent front on the wing was predicted exceptionally well by PPM-PSE3D, making the software a high fidelity design tool that can be used alongside traditional empirical methods.” [E]

A major industry-wide barrier in predictive laminar wing performance has been the inability of modelling small-scale manufacturing and environmental uncertainties. Viability of LFC technology requires extremely tight control of manufacturing finishing standards and maintenance of tolerances during lifetime operation of aerodynamic surfaces. Imperial delivered to Airbus the MiPSecR receptivity software suite. The development has allowed several key areas (stochastic receptivity, steps–gaps, surface-roughness) to be investigated [2,3,4,6] and incorporated into advanced transition prediction software. Airbus state:

“Imperial’s receptivity software suite MiPSecR allows us to utilise its adjoint framework to simulate thousands of stochastic realisations and use uncertainty quantification to predict transition due to tolerances and imperfections.’' [E]

Capabilities facilitated by Imperial’s research offers Airbus significant time and cost savings in the development of optimised LFC aircraft (for both Hybrid LFC and natural LFC). This is achieved by facilitating faster design cycles using the developed advanced tools, and reducing the need for unnecessary and costly wind-tunnel tests, as corroborated by the supporting evidence:

“It is vital for Airbus to simulate such features in a controlled manner, so that the tolerances for building a laminar wing can be understood. The BLADE demonstrator flew over a hundred hours and collected such data. Imperial’s receptivity software suite MiPSecR allows us to utilise its adjoint framework to simulate thousands of stochastic realisations and use uncertainty quantification to predict transition due to tolerances and imperfections. Such state-of-the-art theoretical capabilities offer the potential to incorporate knowledge of manufacturing tolerances and real world limitations into future wing shape optimisation processes prior to expensive experiments and flight tests, again saving time and money.’' [E]

The Imperial research has contributed to the competitiveness of Airbus and its UK operations, as well as the broader position of the UK in advanced technologies and manufacturing:

“Each contribution to the excellence of the product enables Airbus to compete on the global stage and as such the Imperial College research is vital for the continued success of Airbus in the UK.” [E]

The process of developing and building a new aircraft is long, complex and costly. For example, the Airbus A380 “superjumbo”, announced in 1990, was delivered in 2005 with a cost of €25 billion. The design and manufacturing pipeline relies on fundamental knowledge and new technologies to provide a crucial competitive edge, and that is exactly what the research from Imperial has delivered and will continue to do as part of Airbus' plan to include LFC as a defining technology of a potential next-generation aircraft from the late 2020s.

WinBUGS has had wide-reaching and very significant impact. Here we describe the continuing impact of WinBUGS since August 2013: how WinBUGS underpins the most popular tools for Bayesian data science, enabled the wide-spread dissemination of Bayesian methods, and how this had led to increasing impact to the economy, public health and conservation, in tandem with increasing use of Bayesian methods since 2013.

WinBUGS underpins globally used Bayesian inference software tools. Each day, data scientists across the globe utilize general-purpose software to approximate posterior distributions numerically and perform Bayesian inference. The most widespread software tools during the impact period have been WinBUGS, Stan, JAGS, NIMBLE and OpenBUGS, which are freely available including for commercial purposes. It was WinBUGS that first stimulated the development of this ecosystem of software solutions, which all started their development after WinBUGS (Stan was first released in 2012, NIMBLE in 2015, JAGS in 2007 and OpenBUGS in 2004 [A]), and continue to evolve (Stan last updated in 2021, NIMBLE: 2020, JAGS: 2015, OpenBUGS: 2013 **[A]**).

From the Stan Development Team [B]:

“Stan is based on similar principles as BUGS. […] We could not have done any of this without the groundbreaking work by the BUGS developers, both in terms of technology and community. … WinBUGS was way ahead of everything else in providing (1) a simple language to describe Bayesian models in shareable model files, and (2) technology to derive a generalized […] sampler for automated inference for many Bayesian models.”

From the co-lead developer of NIMBLE [B]:

*“BUGS has inspired many of the software packages that followed over time, including the very popular JAGS and Stan packages, as well as the NIMBLE package […]. It is safe to say that NIMBLE would not exist without the great success of WinBUGS […].*”

From the JAGS user manual [B]:

“*Many thanks to the BUGS development team, without whom JAGS would not exist.” *

It is difficult to track how many data science developers and analysists rely on any one of the freely available software solutions that WinBUGS inspired, though for Stan alone the number of users was estimated above 100,000 globally in 2020 [C], and the rstan, nimble, rjags, R2OpenBUGS and R2WinBUGS packages have been downloaded from R’s package distribution network CRAN more than 3.6 million times between 2013 and 2020 [C].

WinBUGS led to the global acceptance of Bayesian data analysis. WinBUGS not only met demand, but also created a data science market. The software helped professionals without a statistics degree to become well-versed in formulating and applying advanced statistical models in their domains, which in turn drove methodological development and better software. Indeed, WinBUGS has been directly linked to the explosive adoption of Bayesian methods across the world, and has become the de-facto standard for data analysis in some industries:

Prof. Lawson, wrote in this book “Bayesian Disease Mapping” in 2018 [D]:

“the development of the BUGS package and its Windows incarnation WinBUGS have had a huge effect on the dissemination and acceptance of [Bayesian] methods. A brief search for recently published papers referencing WinBUGS turned up applications in food safety, forestry, mental health policy, AIDS clinical trials, population genetics, pharmacokinetics, paediatric neurology and other diverse fields, indicating that Bayesian methods with WinBUGS indeed are finding widespread use”.

Professor McElreath, Director of the Max Planck Institute for Evolutionary Anthropology, wrote in November 2017 [E]:

“BUGS started a revolution in efficient, desktop Bayesian computation…It has done more than any other initiative to promote and advance applied Bayesian data analysis.”

The Director and Principal Statistician for systematic reviews and meta-analysis at Evidera PPD states in on 26 October 2020 [F]:

*“The majority of our work is conducting Bayesian network meta-analyses (NMAs), generally to support health economic modeling. As you may know, the methodological leadership in the area comes from NICE, and their guidance exemplar codes are in WinBUGS.*”

Impact in the Economy: WinBUGS and the many data science software solutions that it stimulated are providing substantial economic value, estimated to be in millions of US dollars. For example:

The Director and Principal Statistician for systematic reviews and meta-analysis projects at Evidera PPD, writes [F]:

*“WinBUGS and its descendants has greatly facilitated our application of the most advanced statistical methods to the challenging problem of indirect comparisons between treatments [of pharmaceuticals]. Providing such inferences and evidence is mandatory in more and more settings, and the commercial value to Evidera PPD for those projects has been in the millions of dollars, leading to many submissions (and approvals) for our clients.*”

The Head of Advanced Methodology and Data Science (AMDS) group, Novartis Global Drug Development, writes [G]:

“Especially over the last five years, we have streamlined and now routinely use robust and automated statistical inference software in drug development, and WinBUGS played a pivot role in this development. Thus, WinBUGS and related projects have undoubtedly had an enormous impact on the efficiency, cost-effectiveness and economic value of drug development benefiting the pharmaceutical R&D process at Novartis but more importantly benefiting patients across the world.”

Impact in public health: The recommendations of the National Institute for Health and Care Excellence (NICE) aim to optimally allocate resources and maximize the quality of life of people in the UK based on cost-effectiveness and are typically taken up in care [H]. WinBUGS or JAGS were used in 26% (n=37) of the clinical guidelines that were published between 2013/08 and 2017/12 and had the full guidelines text available online [H], illustrating how these tools have helped drive effective and affordable healthcare in the UK. For example, the NICE impact mental health report describes the early intervention pathways that were implemented across NHS England as a result of guideline CG178, benefiting approximately 550,000 people diagnosed with severe mental illness in 2017/18 in the UK [H]. CG178 was published in 2014; WinBUGS was the main tool used for the statistical evaluation of the evidence [H].

Impact in Conservation: The ability of WinBUGS to fit geospatial models, and the same ability of the later software tools that BUGS inspired, have been frequently used in quantitative ecology, and have contributed to conservation management and policy.

One exemplar is the illegal ivory trade, for which JAGS was used annually between 2013-2018 to characterize global trends for the United Nations Convention on International Trade in Endangered Species of Wild Fauna and Flora (CITES) [I]. The estimates underpinned situation reports, informed CITES policy, numerous press releases, and national ivory action plans in Central Africa [I].

There are hundreds of similar examples, for example on wildfire management or the growling grassfrog, an endangered amphibian mainly found on the outskirts of Melbourne, for which national conservation management plans were created based on OpenBUGS. Co-author Professor McCarthy of the University of Melbourne wrote to us [J]:

*“This model is now being used by the Victorian government to plan management actions that are worth tens of millions of dollars, helping to ensure that this money is spent as efficiently as possible.*”

Imperial College research has been incorporated into Microsoft Defender, which is part of all modern Microsoft Windows versions, and into cyber-analytic software from EY.

Global critical national infrastructures face a growing and evolving threat from cyber-attacks, and our modern networked economy relies heavily on maintaining secure environments for exchanging, storing, and protecting business and consumer data and intellectual property. There is now a consensus amongst government and industry experts that statistical and machine learning techniques have an important role to play in current and future security defences. Providing statistical cyber-security solutions to industry effectively strengthens the UK and world economies. Cyber-security is therefore an emerging and strategically important field in statistics and the Statistical Cyber-Security Research group at Imperial is at the forefront, working closely with both government and industry, developing methods tailored to the most pressing security problems.

A single cyber breach, depending on the country and industry, has been estimated to cost on average $3.86 million, rising to an average of $8.64 million for the US [A]. The WannaCry attack of 2017 indiscriminately targeted Microsoft Windows operating systems that were not patched with respect to a known exploit. In the short period the attack was active, an estimated 230K computers were compromised globally [B]. Besides causing financial losses reaching approximately $4bn, WannaCry impacted the computer systems of NHS hospitals and surgeries, disrupting care for those in urgent need.

The Credential Analytics software of EY uses research from papers [1,2], which provide statistical models for detecting the actions of an intruder on an enterprise work using authentication and computer event logs. The EY PathScan analytic uses research from paper [3], concerned with detecting anomalous network traversal.

To enable this, three joint patent applications (two granted [C, D], one pending) have been filed by Imperial in collaboration with Los Alamos National Laboratory, a US research lab funded by the US Department of Energy. Two pieces of statistical software utilising these patents have been licensed by EY, for commercial distribution in their cyber-security platform called PathScan, which has been deployed within large international corporations. Quoting the letter of support from EY [E],

“EY have used the research from the Imperial-authored papers and patents listed below to develop two of our offerings in Cyber Analytics: PathScan and Credential Analytics […] In the 4+ years since, we have invested roughly 27 person-years to develop a scalable, sustainable product. We have clients across the globe, and have found particular success in APAC and the Middle East. EY is [also in] active discussions to license the IP associated with Credential Analytics. Again, with our LANL partners, we invested 4 person-years in 2018”.

Microsoft Defender Antivirus claims to deliver “comprehensive, ongoing, and real-time protection against software threats like viruses, malware, and spyware across email, apps, the cloud, and the web” [F]. This software is deployed worldwide on all devices running modern versions of the Microsoft Windows operating system, including Windows 10, Windows 7 and Windows Server. To quantify the scale of this provision, it should be noted that there are more than 1 billion devices running Windows 10 [G].

Microsoft have implemented solutions in their security product Microsoft Defender Antivirus using anomaly detection techniques from [2,4,6] for respectively modelling behavioural patterns, performing change detection with adaptive forgetting-factors, and latent feature modelling of hosts and users on enterprise networks. Furthermore, the anomaly scores from the range of analytics being run by Microsoft Defender are combined into a single measure of surprise using results in meta-analysis published by the Statistical Cyber-Security Research Group at Imperial [5].

Quoting the letter of support from Microsoft [H],

“Various methods described in these papers have been implemented in our products, including:*

• P-value combination, which we use as part of our lateral movement detection, leading to over 2 million detections of malicious behavior per month.

• Adaptive forgetting factors, which influence many of our streaming algorithms, and which represent approximately 20% of our overall detection portfolio.”

In the above, “P-value combination” refers to [5] and “Adaptive forgetting factors” refer to [4]. Microsoft’s letter continues:

“The scale at which we operate (again, over 6 trillion signals analyzed daily) means that any new production algorithms must be implemented in the face of enormous compute, memory, and storage challenges. The fact that research and development from Imperial has successfully been transitioned into production is a testament to the importance we place on the cutting-edge work coming from the College.”

The creation of a monitoring tool using the underpinning statistical research with the commercial partner Dr Foster Ltd has helped the NHS improve care and save lives since 2007. This case study describes the impact since 1 August 2013, but we first give some important background.

Before the current REF impact period, our hospital monitoring system identified high mortality at Mid Staffordshire NHS Trust. The alert letters sent by Imperial to the trust and the national regulator, the CQC, helped trigger CQC inspections, which led to a public inquiry (Francis Inquiry). The resulting report [A], published in February 2013, notes Imperial’s and Dr Foster’s contribution: *“There is no doubt that, without the work of the Dr Foster Unit and Dr Foster [Ltd], comparative mortality statistics would not have been published as quickly, or as fully, as they now are.*” One of the consequences of the Francis Inquiry was the investigation of other high mortality hospitals, which led to the Keogh Mortality Review [B], published in July 2013. The Review used in part our monitoring tool’s HSMRs (measures of hospital death rates) to spot high-mortality hospitals, and 11 out of 14 high-mortality hospital trusts were put into special measures. In 2013, the 11 trusts had 570,000 admissions and 21,000 deaths.

This resulted in improvements in the quality of care, mostly in the period after August 2013. The CQC assessed the 11 hospital trusts in August 2014 and found many improvements [C]. For example, one of those hospitals, Sherwood Forest, put in place regular audits and sufficient handover time between shift changes. With higher-than-expected mortality in 2014, they proceeded to transform sepsis care as a direct response to our mortality figures. Improvements continued over the coming years – see Dr Foster online case study [D]: “ The Trust has seen a significant reduction in its mortality rates, with deaths below the expected level since April 2016....for the first time since 2010”. They have around 50,000 admissions and nearly 2,000 deaths each year; NHS Digital’s SHMI web pages report their relative risk (the risk-adjusted rate compared with the national average, known as the SHMI) as 1.10 for Jan-Dec 2012, falling to 0.95 for Jan-Dec 2016.

To investigate the impact of the monitoring system beyond those 11 trusts, in 2018 Imperial staff evaluated the monitoring tool in a peer-reviewed report to NIHR [E], including interviews with and quotes from staff at site visits plus a national survey. Although the study time period straddled August 2013, it provides a robust summary of the monitoring system. The study included 532 alerts sent to 139 hospital trusts; 71 trusts responded to the survey, and 11 hospitals were visited as case studies. The evaluation found:

• Quality of care was cited as a factor in 70% of investigations into the alerts

• On average, the relative risk of death fell by 58% during the 9-month period immediately following an alert and then levelled to a slow decline

• Full action plans were created in 77% of trusts investigated, and several hospitals reported positive changes in culture. The tool changed how some hospitals respond to information such as death rates, leading to changes in their processes. For example, before receiving our alert regarding heart attacks, one site reported poor communication, and one doctor had been disciplined for speaking out about poor care. As a direct response to our alert letter, a new mortality committee was set up, reviewing every death. A chest pain pathway was introduced across the trust, and new coronary care facilities and more consultants were approved.

This impact had significant reach between August 2013 and December 2020: we sent out 1,312 alert letters to hospitals, covering nearly 41,000 deaths during this period.

Many hospitals report the HSMR for specific patient groups in their annual Quality Accounts and set targets for the following years: the source of these figures is our monitoring tool. An example is sepsis at Oxford University Hospital in 2017/8, with reports of improvements in both care processes and the HSMR for sepsis in 2018/19 [F]: the trust had 1,503 inpatients with sepsis in 2018/19.

Besides the public health impact, there are financial impacts. Dr Foster benefit financially from the underpinning research, which they sell, together with further analysis and support undertaken by their staff, to NHS organisations and others: in June 2018, their annual turnover was £6.5m, with 67 employees [G]. Telstra Health acquired them in 2015, a deal reportedly worth 25-30 million Australian dollars [I]; the monitoring system developed with Imperial and the expertise provided by the Imperial unit were important elements of that deal [H]. Dr Foster were the first organisation worldwide to publish hospital-level death data (as HSMRs) in national newspapers, working with hospitals to understand and improve their performance. As UK pioneers of healthcare analytics, Dr Foster Ltd led the way in the use of HES data and statistical methods, which was then followed by competitors such as CHKS and HED and helped create a market now worth $2.7bn in Europe alone [J].

Our work directly impacts upon civilian nuclear inspections and safety as evidenced in [C]; it also impacts the UK nuclear deterrent by ensuring that the UK submarine capability remains safe and available for operation; it ensures the health and safety of personnel; and by delivering improved estimates for component lifetimes and defect characterisation yields savings in inspection development costs [C, A]. Focussing in on improving the inspection capability for critical components, that are hard to access in confined space, has provided substantial cost savings [A]. These advances also have broad reach and significance as many of the UK, and international, nuclear facilities are ageing and the accurate inspection of difficult to access components is critical to ensuring that they continue to operate safely [B].

Underpinned by the research funded in [i-iv] we developed the first characterisation of scattering by rough cracks within reactor pipework -- previous algorithms assume smooth flat cracks and generate overly cautious estimates. Our work revises these estimates and is implemented by our industry partners [6, A, C].

EDF Energy and Rolls-Royce Submarines

Both Rolls-Royce Submarines and EDF actively deploy the Finite Element Method for modelling of ultrasonic wave-defect interaction [A,C,i,iv,6], aiming to provide a capability to model the reflection of ultrasonic waves from small and geometrically complex flaws typical of the types found in nuclear plants due to manufacturing, fatigue, or service [A, C]. Both Rolls-Royce and EDF highlight the cost-savings associated with accurate simulation versus the " notoriously expensive" standard approaches and that simulation is " invaluable", " as it allowed small regions of a component to be modelled using the FEM without wasting computing resources on unnecessary and currently un-solvable models" [A]. The economic benefits to Rolls-Royce and EDF are most keenly felt as cost savings as " The production of component safety justifications requires thousands of man hours that represents millions of pounds of cost. A small benefit provided to this effort will remove significant cost from the submarine enterprise" [A].

Our impact draws upon a suite of specialised software some of which is commercialised: DISPERSE software is used by [redacted from public version] different organisations/companies and we evidence its impact via [D] where Disperse is a key capability underpinning the R&D development of that company. Additionally, our hybrid method is now standard in industry partners as evidenced in [A]; this has facilitated Pogo a finite element solver designed for GPU computation.

Software

• the latest version of software DISPERSE contains the algorithms in [4]. DISPERSE is licensed by Imperial College Consultants. Since 2013, [redacted from public version] different companies/ organisations purchased licences, across [redacted from public version] countries, generating a revenue of [redacted from public version] [E].

• Customers include [redacted from public version] [E].

• The hybrid [5] method we developed with sponsorship from Rolls-Royce, Amec now Wood, EDF, BAE Systems and National Nuclear Laboratory is now a standard tool for these sponsors and other industry partners. [A, C].

• Pogo is a high-speed finite element package for ultrasound simulation that we have combined with the hybrid method [1,2]. This combination of PoGo/Hybrid method is now used internationally by industries operating in nuclear, oil and gas, and to [redacted from public version] for space exploration. Since 2017, [redacted from public version] licences have been supplied, with [redacted from public version] of these to private companies and [redacted from public version] to universities giving a total revenue of [redacted from public version] [E].

Acoustics Network

The industry-academic relationships we built in NDE/ultrasonics [iv], and in metamaterials [3, vi], led, through workshops and community events, directly to the EPSRC UK Acoustics Network [G] (RVC is Co-Director). Starting in Nov 2017 UKAN has rapidly grown to >1200 members (500+ from industry) and extended to an EPSRC Network Plus [v], with significant industry input, in 2020; there are special interest groups in both NDE and metamaterials.

This extensive network has broad impact, here we focus on a single exemplar of the impact that UKAN has on training industry staff in one organisation, DSTL [F]. Staff were trained at a UKAN workshop in machine learning (ML) for acoustics and “ *Dstl has been able to use its new expertise in ML to contribute to a number of high impact projects that support the RN operational advantage in the underwater battlespace.*” UKAN has become “ *an important part of the ecosystem being used by Dstl to maintain and develop internal capability in underwater acoustics.*” DSTL is just one of the many companies and external organisations (including Thales, GlaxoSmithKline, AECOM, DEFRA, Meridian Audio, Precision Acoustics, QinetiQ) that have been actively involved in UKAN. Using its industry connections, the network is actively involved in policy work for acoustics with the report [H] highlighting its value to industry, the economy, and society.

Impact in France:

The work of NJ/AF in [3] led to a member of their team being called before a French national commission (the ‘ Concertation Citoyenne sur la Vaccination’, see below) [A] to report their findings. Combined with press coverage of [3], this testimony led to the commission issuing a report that resulted in the French Ministry of Health substantially increasing the number of state-required vaccinations and so markedly reducing numbers of infections.

In September/October 2016, there was a ‘ Concertation Citoyenne sur la Vaccination’ in France (initiated by the health minister). This citizen consultation had two committees: one composed of medics and the other composed of the public. Heidi Larson, the joint first author (with AF -- Imperial) of [3], was called as a witness (8 Sept 2016) to the medical committee and presented the team’s results one day before they were released online. The transcript of this extensive presentation can be found in [A, p262-268] of which the below is an excerpt:

“Madame LARSON –…the timing is very good ... tomorrow we will be releasing an e-biomedicine in 67 countries on vaccine confidence. You are the headlines. … France … was, not by a small margin, the absolute least confident country in vaccine, particularly safety, but also its effectiveness. …

*Monsieur FISCHER, President. - Thank you very much, this was really a performance. A lot of information, not good for us, but, at least, it is a further reinforcement that the work of our group is more than needed. There is a lot to do everywhere in the world, but, unfortunately mostly in France.*”

The (medical) committee then went on to discuss details of the breakdown of opinions expressed by survey respondents in [3], including by gender, and asked for the analysis in NJ/AF’s paper itself. (See [A] p268).

Paper [3] was subsequently reported on in two articles in Le Monde [B], had coverage in La Tribune, and remains heavily referenced in the French (Le Monde, L’Express) and international media [J].

The report produced by the French commission, “Rapport sur la Vaccination” (30/11/2016 **[C]**) cites paper [3] in its opening paragraphs (p5), in particular stressing the high levels (41%) of vaccine hesitancy (i.e., low confidence) in France. In fact, the majority of the document covers vaccine confidence (rather than merely vaccine coverage). The consultation recommended that all childhood vaccinations be made mandatory (state-required).

On 5 July 2017, the French Health Minister announced the extension of mandatory vaccination, explicitly citing the consultation [D]. Refs [E, F] explain how the minister addressed the committee's principal recommendation (i.e., mandatory vaccination) as a direct consequence of report [C]. A recent analysis [G] of the effects of this mandatory vaccination (which cites paper [3] and its follow-up for the EU **[4]**) stated:

“…the extension of vaccination mandates on vaccination coverage is encouraging. It shows an increase in VC [Vaccine Coverage] of infants concerned by the extension of the vaccination mandates … vaccine coverage for the first dose of meningococcal C vaccine increased from 39.3% to 75.7% ... This sharp increase in MenC VC translated into a dramatic decrease in the number of invasive MenC disease cases notified in infants through the mandatory notification system”

Ref. [4] further notes that the number of invasive Meningococcal infections dropped six-fold in one year: from 3 per 100,000 (infants <1 year) in 2017 to 0.5 in 2018. MMR vaccine coverage levels increased by 3% in 2017/18 (MMR is a key vaccine for public health and target for concerns regarding vaccine safety). The number of HPV vaccines paid for increased by ~120,000 from 2016/17 to 2018 (from ~30,000 to ~40,000 per month).

NJ/AF’s work thus directly led to changes in French Vaccination policy, in vaccines delivered, and in infections averted. After French steps to increase mandatory vaccination, Germany recently passed a law that made measles vaccinations mandatory from March 2020 [H].

Impact on WHO:

NJ/AF’s track record on vaccine analytics enabled them to win support to improve the WHO’s reporting of coverage [2]. As a consequence of their efforts the WHO are developing probabilistic estimates of coverage [I]. WHO vaccine coverage estimates are highly significant since institutions like the Gates Foundation and GAVI tie their financial aid to WHO estimates of coverage levels.

Impact on EU and Wellcome Trust:

Consequent on paper [3], both the European Commission and Wellcome Trust commissioned large-scale surveys of the EU and World, respectively. These new surveys used the survey questions from the eBioMedicine paper [3] and sought to track progress from 2016. The vaccination part of the Wellcome Trust survey generated widespread media attention in its own right. Released in June 2019, it was picked up across the world (BBC/Reuters/CNN/Nature/ Science and [5] is in the top 1000 of papers scored by Altmetric out of 17M).

Media Impact:

AF has shown that vaccine confidence has recently rebounded in France, with a ~25% increase in the number of French respondents strongly agreeing in vaccine safety (2015 survey compared to 2019 survey) [5]. Beyond influencing perceptions in France the work of NJ/AF [1,3-5] has impacted global awareness of vaccine confidence as a key issue and hence the need to tackle confidence: the research has been covered in >200 press articles in over 130 media outlets worldwide including: UK (Financial Times, Daily Mail, BBC), US (Newsweek, NYT), and European (Le Monde, El Confidencial) news, TV and Radio news (Al Jazeera, Europe 1, CNN, Fox, NPR), and online venues like Vox and The Conversation. It has also been reported widely in the generalist science journals Science, The Lancet and Nature (appearing in 1 world-view, 1 feature, 3 perspectives, 2 news items), and popular science magazines (Scientific American, New Scientist). Paper [3] is in the top ~0.01% of all papers scored all-time by Altmetric [J]. With its nearly 500 citations, NJ/AF’s work is now used when people appeal to international challenges with both vaccine confidence and coverage.