Impact case study database

We present impacts in the area of creativity, culture and society on a network of collaborators: curators; composers; and a diverse group of media and other professionals who themselves influence public culture and society, together with impact on understanding, learning and participation through contributing to the public commemoration of Ada Lovelace.

Impacts on creativity, culture and society: museums and libraries

In 2014, Oxford Mathematics were invited by Oxford’s Bodleian Library, which holds extensive archives of the Lovelace family, to collaborate in marking Ada Lovelace’s 2015 Bicentenary. Following this invitation Hollings, Martin and Rice were given access to the archive during 2014-15 in order to analyse the manuscripts described in section 2. The transcription of the letters [3] was made available online in late 2015. The early (pre-publication) results of this striking research underpinned two events: Oxford’s Lovelace Bicentenary Conference (8 Dec 2015, 400 attendees); and a display of Lovelace’s mathematical papers co-curated by Martin and Hollings at Oxford’s Bodleian Library (Oct-Dec 2015) [A] [text removed for publication]. In parallel, material from the Lovelace letter archive was loaned to the Science Museum in London for their exhibition on Ada Lovelace (Oct 2015-Mar 2016) [A]. A Lancet review of the two exhibitions praised a “nuanced picture of an original thinker” [C]. The publicity surrounding the conference, exhibition and publication of the transcripts led to enthusiastic approaches from a number of potential collaborators [text removed for publication].

Oxford’s authoritative work [1, 2, 3] led to further approaches from curators keen to collaborate to present Lovelace, having previously been hampered by the absence of scholarly research to inform curatorial judgement of this controversial figure. Leading on from the Bodleian exhibition, Martin was invited to co-curate a display of facsimile material at the world’s largest computing museum, the Computer History Museum in Mountain View, located next to the Google campus at the heart of Silicon Valley. Thinking Big: Ada, Countess of Lovelace opened in late December 2015, with major sponsorship from Google, and ran for 4 years [text removed for publication] [D, B]. A substantial outreach programme attracted new visitors, especially from under-represented groups: it included a lecture by Martin and Rice to an audience of 600 in person and 6,000 online [G]; weekly “Women in Computing” museum tours for families and the general public; and monthly workshops for local schools targeting girls and minorities [F]. Prior to COVID-19, a version of the display was “on tour”, most recently at Facebook’s Silicon Valley campus.

For the Royal Holloway Exhibition Space in Egham, Martin co-curated 200 Years of Becoming Digital (Sept-Nov 2018), winner of the 2018 Great Exhibitions Prize of the British Society for the History of Science, who praised how it “excellently portrayed the overlooked contribution of women...and captured the audiences’ imaginations” [H]. Advice on using material from the Lovelace archive has been given to curators of other Oxford exhibitions, most notably Sappho to Suffrage (Mar 2018-Feb 2019), and to Hinckley and District Museum (Oct 2019), Guildhall Museum (May-Oct 2018), and Gunnersbury Park Museum (June 2018). The impact on museum professionals was assessed through professional impact evaluation [F, G] commissioned by Oxford Mathematics, including in-depth interviews with 13 collaborators, evaluation of sample events, and wider contextual analysis, and is summed up by the Project Evaluator [E]: “The robustness and novelty of the research has empowered curators, cutting through controversy and enabling new activities which use history to provoke audience reflection on current and future cultural issues. In a change to current practice, curators now recognise the value of future such collaborations with mathematicians.”

[Text removed for publication]

Impacts on creativity, culture and society: composers

Emily Howard is a British composer with regular commissions from leading international orchestras and concert halls. She presented a 2011 song-cycle inspired by Lovelace at Oxford’s 2015 conference, and as a result of discovering the new Oxford research into Lovelace’s mathematics she invited Martin to collaborate on a project on Lovelace and AI for London’s Barbican Concert Hall. Martin worked with Howard and other composers to interpret Lovelace’s mathematical writings, and to recast algorithmic aspects of her work, particularly the functional algebra of [2], in terms of modern AI for use in composition. Howard’s resulting composition, But then, what are these numbers?, was a setting of a Lovelace text proposed by Martin, who also advised rising young composer Robert Laidlow on the novel Lovelace percussion instrument he built for his AI-inspired piece Alter. These and other works were premiered by the Britten Sinfonia on 2 November 2019 in a concert Ada Lovelace Imagining the Analytical Engine, part of the Barbican’s 2019 AI Festival [I]. The Guardian reviewer enjoyed a “gratifying sense of theatrics”, and audience evaluation [G] evidenced enthusiasm for the new concepts (typical comment: “it gave my mind a real work-out”). The significance of the impact on composers, as summarised by the Project Evaluator, was “its catalytic role in the creative processes of composition, in stimulating both well-received new work, and new approaches to composition expected to be of increasing future importance” [E] .

[Text removed for publication]

Impacts on understanding, learning and participation: contributing to commemoration of Lovelace and making mathematics accessible

As visibility grew, numerous potential collaborators got in touch, seeing the potential for Oxford’s research to be a catalyst for devising their own activities which, in turn, influence others in public culture and society. Many of these approaches were associated with Ada Lovelace Day, a global celebration of women in STEM held every October since 2009: its 10th anniversary was marked by a US Senate resolution, drafted with Martin’s advice [J].

Science writer Georgina Ferry approached Oxford to collaborate on a two-part BBC Radio 4 dramatisation of Lovelace’s letters (‘The Letters of Ada Lovelace: The Poetry of Mathematics’; 14 Sept 2015 + 3 repeats) [text removed for publication] [F, K]. Invited presentations at literary festivals included Hay (2016, 2018), Edinburgh (2018) and Oxford (2018); media work included Radio 3’s the Verb (20 April 2018, audience 60,000); BBC Newsnight (11 April 2018, audience 600,000) interviewed Martin about Lovelace’s mathematical learning and her ‘connectedness’ with scholars and others in the society of her time [K]. Oxford Mathematics was approached [text removed for publication] to co-create a Lovelace special issue of its cs4fn magazine (Autumn 2015, 20,000 copies distributed to 2000 schools) [L]. COVID-19 disrupted planned Continuing Professional Development for computing teachers, using Lovelace’s work, due for pilot and evaluation from May 2020.

Between 2015 and 2019, the Oxford team accepted approximately 40 invitations to work with schools, companies, maths and computing organisations, and local history groups, creating bespoke activities which used the past to stimulate thinking about the present and future, for example presenting Lovelace in the context of technology, local history, or today’s women in mathematics. These have reached a total live audience of approximately 7,500, often amplified online, for example as part of the free open lecture programme at Gresham College (150 live + 12,000 online) [G]. Professional evaluation of the audience response to events held in 2019 found that they changed perceptions of participants, who found mathematics “more practical, more socially engaged than I thought”; were surprised and sad at “ the erasing of women from history”; with the account of 19C mathematicians as a “community providing support for its members so they can achieve” providing a “more inspiring, more hopeful view” [G].

A general interest book by Hollings, Martin and Rice, Ada Lovelace: the making of a computer scientist, 2018, (translated into Spanish as Ada Lovelace: la formación de una científica informática, 2020) enhanced the accessibility of the research of [1, 2] to non-mathematicians, by-passing the technicalities of Victorian algebra to explain the mathematical content and context at roughly GCSE level: for example, Lovelace’s striking diagrams of the Bridges of Königsberg problem are used to illustrate her algorithmic thinking. As of 1 June 2020 the book had sold 2,520 copies and received positive reviews in mainstream, literary and educational publications [M]; comments include “Dusty archives dance into life” (New York Review of Books), “admirable clarity” (Women’s History Review), “mathematics explained clearly and in detail” (London Mathematical Society Newsletter) and “The one I’d recommend ... much excitement, of a mathematical flavour” (Association for Women in Mathematics, in a survey for educators of books about Lovelace).

Impacts on creativity, culture and society: other collaborators

The Project Evaluator’s summary reveals that the significance of the impact on the professionals involved in such collaborations was [E]: “The confidence generated by the calibre of the research, and the additional insights provided through working with the Oxford team, empowered collaborators to incorporate Lovelace material in their own professional work. This in turn influenced others, through changing perceptions of the current and future importance of mathematics, and of women’s contributions”.

[Text removed for publication]

Ofcom’s Communications Market Report 2020 states that the retail mobile telecoms market in 2019 was worth GBP13,400,000,000 [A]. Billmonitor’s Business Mobile Report [B], released in 2018, estimated a total overspend on UK mobile phone contracts of more than 50%, totalling billions of pounds per annum. A follow-up report on SMEs later in the year, An Investigation into the B2B Mobile Provider ‘Wild West’ [C], suggested that the cost difference between the cheapest and most expensive provider was 16.3% for an average SME, and that 38,000 SMEs could save over GBP10,000 per year.These reports received extensive coverage in the specialist business press (including in MoneyWeek, Computing, Business Matters,The Register, Total Telecom, Bdaily News, ITProPortal and CityAm) [D]. In 2019, Billmonitor estimated that, in a single year, the mobile phone networks overcharged UK businesses and consumers approximately GBP7,600,000,000 [E].

The impact claimed in this case study is economic. During the period from 2014 to 2019, Billmonitor analysed the mobile phone contracts of nearly 150,000 private customers and nearly 700 business accounts (with nearly 25,000 connections) identifying combined total savings of GBP37,000,000 [F]. The NHS and other health and public sector organisations account for around 40% of all business accounts analysed [F], so the main beneficiaries are individuals, businesses, public sector organisations, and the SME Optimor, for which Billmonitor is the sole product.

Pathways to Impact

The impact is achieved through the SME Optimor, which launched the Billmonitor price comparison website in 2009. The Managing Director of Optimor has confirmed [F] “The research done by Professor Chris Holmes and Professor Nicolai Meinshausen of the University of Oxford was an essential component which has enabled us to build an accurate algorithm for predicting an individual's future mobile phone usage and we have used this to develop Billmonitor”. The statistical methodologies in Billmonitor analyse the mobile phone bills of both individual customers and customers with business accounts to predict their future usage. Price comparisons based on the predicted patterns of usage identify the best pricing plans available. This allows users to understand their pattern of usage and so make informed decisions and save money.

Once prototyping was completed by Meinshausen, using the statistical programming language R, the front-end user interface of Billmonitor was designed by engineers at Optimor, with Holmes and Meinshausen advising on the graphical displays of information. Holmes and Meinshausen were directors of Optimor (Holmes from April 2008 until January 2012 and Meinshausen from April 2008 until September 2011) and Holmes continues to act as a scientific adviser, for example overseeing the statistical model used in the report into overcharging of SMEs [C]. The notes in the company accounts (year ending 31 March 2020) [G] report that Optimor employed 9 staff members (Headcount: 9; FTEs: 9).

The financial model used by Optimor allows Billmonitor to provide completely unbiased comparisons: Optimor receives a small affiliate commission if users buy packages via links on the site, but the user will pay exactly the same price as if they had purchased direct from the provider.The links are non-biased; advice is given exclusively on the basis of identifying the plan which provides the best deal for the user’s projected usage.

Having provided their price comparison service to private customers since 2009, in 2015 Optimor began to apply the Billmonitor technology and expertise to help those with UK business accounts to analyse their mobile phone contracts. The Managing Director of Optimor writes [F] “Chris’s and Nicolai’s work has been instrumental in helping us to launch our business-to-business service, which commenced in 2015.” Since April 2016, MoneySuperMarket.com has offered Billmonitor services to corporate clients.

Impact on private customers

As confirmed by the Managing Director of Optimor [F] in March 2020, between January 2014 and December 2019, 149,847 private users have provided details of their mobile phone usage through the Billmonitor website [E]. Savings have been identified for around 70% of these users, with an average of GBP295.16 savings per account for a 24-month contract. The total savings identified during the period are approximately GBP31,000,000.

Satisfied customers have taken to Twitter [H]: Thanks to @billmonitor I saved over £700 [GBP700] per year this morning! After a 2014 recommendation from Martin S Lewis of the Money Saving Expert (MSE) website, a user tweeted [H]: Wow @MoneySavingExp wish you’d highlighted @billmonitor before! Look at the saving, followed by a screenshot showing projected savings of GBP1359.61. A recommendation from MSE in 2017 led to so much interest that the website was overloaded [H]: @MoneySavingExp@MartinSLewis has sent #billmonitor into a frenzy! #MSE. Even those who do not change providers as a result of using the site can use the information from Billmonitor to negotiate with their existing providers [H]: @MoneySavingExp I had to push a little, they came down in stages but we got there! £21 [GBP21] to £10.14 [GBP10.14] :-) Knew the deal I needed via #billmonitor.

Impact on businesses

Once Billmonitor expanded to cover business accounts in 2015, the scale of overspend was even more startling: “ where consumers overpay by 66% for their required level of service, businesses overpay by 96%. Such huge savings…[have been]… *found by so few.*” [B].

Since 2015, Billmonitor has analysed nearly 700 business accounts with nearly 25,000 connections, identifying savings of over GBP6,000,000. The MoneySuperMarket.com website [I] advises: “ The average saving over the contract duration for a small business...[is]… £3,360 [GBP3,360] (based on an average savings per line of £336 [GBP336], *from savings calculations from over 10,000 lines under business contracts from companies that analysed their bills via Billmonitor for Business April 2016)*” . The year-on-year business savings, provided by the Managing Director of Optimor [F], are shown here:

Testimonials

Around 40% of the business accounts analysed have been for the health and public sectors [F].

[Text removed for publication]

Billmonitor is also used by intermediaries to optimise their service to clients. On the Billmonitor website [L], the Founder and Director of independent mobile phone agent Oxford Street Connections Ltd. states “ *Billmonitor saves me a lot of time. Its detailed analysis of Data, Voice, Roaming and International Calls by users allows me to gain a comprehensive understanding of a client[’s] profile and their key issues. I can analyse and prioritise then plan recommendations to find one that meets my client[’s] needs, offer[s] good savings and allows a meaningful handset fund.*”

Other testimonials on the Billmonitor website [L] include:

• “ AMS Aircraft saved 58% on out-of-allowance costs ”

• “ London-based Investment Management Firm saved £92,000 [GBP92,000] ”

• “ Amius saved 37.5% through improved tariff and bundle selection ”

• “ City & Guilds realised instant cash flow savings of 30% with projected combined savings of 60% ”

• “PLMR realised total savings of 65% over a new 24-month contract”

• “Fishawack Health realised combined cash savings of 23%”

Ofcom accreditation

In 2009, Billmonitor was the first mobile phone contract price comparison tool to be accredited by Ofcom, and it has been regularly re-accredited at 18-month intervals ever since, most recently in November 2019 [M]. This required it to meet the terms of a rigorous independent audit, which tests whether the information provided is comprehensive, clear and easy to understand. The Director of Ofcom's Consumer and External Relations Group said [M]: “Comparison websites like Billmonitor provide crucial information that helps consumers to choose the best products and deals for their needs. By ensuring this information is accurate, transparent and up to date, our accreditation scheme means consumers can navigate the market with confidence.” Billmonitor is one of only three Ofcom-accredited providers of advice for mobile phone users, saving consumers, public sector bodies, and businesses millions of pounds on their mobile phone costs.

Antibodies are the most important and successful class of bio-therapeutics with over 100 approved for clinical use in the US, and hundreds more in the pipeline [A]. In 2018 the global market was estimated at USD135,000,000,000, with further growth to over USD200,000,000,000 anticipated [B].

However, the generation of novel antibody therapeutics is currently still primarily a very time-consuming and expensive experimental endeavour. Pharmaceutical companies have therefore begun using databases and computational tools to accelerate the process of engineering an antibody to bind to a particular target or identifying antibodies that are markers for exposure or measures of vaccine efficacy. The SAbDab-SAbPred suite contains leading-edge tools that have provided this capability rapidly and efficiently. The resulting economic impact is realised by over 100 companies that have exploited this suite of tools.

Pathway to Impact

Deane took an early decision to make the SAbDab-SAbPred suite freely available under an open source licence in order to maximise the usage and impact of the tools. Funding from an EPSRC Impact Acceleration Award was used in 2016 to hire a Research Software Engineer (RSE) to further refine and integrate the software tools, making them easily accessible through a web server, and also to assist pharmaceutical companies who wished to bring the tools in-house and wanted help in adding new custom features. In 2020, to make it even easier for companies to use the tools, a second RSE was hired and the SAbDab-SAbPred suite was packaged as a virtual machine, named SAbBox, that can be downloaded as a single encapsulated licensed product.

9 pharmaceutical companies (GSK, UCB, Roche, AstraZeneca, Lonza, Iontas, Kymab, Tikcro, and Agenus Bio) currently have custom installations of the SAbDab-SAbPred suite. Five companies have already purchased SAbBox licences: GlaxoSmithKline plc (GSK), Kotai Biotech, Gritstone Oncology, Cenmed Enterprise, and Dragonfly Therapeutics. These collaborations have resulted in staff from AstraZeneca, Roche, GSK and UCB being involved as co-authors in much of the underpinning research [1,3,4,5,6].

Impact of SAbDab-SAbPred

The SAbDab-SAbPred suite has been utilised by over 100 pharmaceutical companies across the globe as part of their antibody therapeutic development pipelines [C]. These companies include very large multinationals such as Abbott, Medimmune/AstraZeneca, Bristol-Myers, Genentech, GSK, Novartis, Roche, Sanofi, and UCB as well as many smaller companies including Just Evotec Biologics and Voyager Therapeutics.

Components of SAbDab-SAbPred and/or the underlying research papers have been cited in 71 different patents (including 3 from Roche and 1 from UCB) in the area of therapeutic development [D], with some patents being filed in multiple regions (e.g. Europe, USA, Japan, China). Source [D] also includes the full text of 4 of these patents; two by Roche and UCB cite the use of SAbDab, while the other two by Seattle Genetics and Humabs BioMed cite ANARCI. The impact of using SAbDab-SAbPred by pharmaceutical companies has been to enable faster and more accurate results than commercial tools, thereby significantly reducing development costs and avoiding unnecessary expenses. The benefits that have been realised are illustrated below by company representatives of two large multinationals, GSK and UCB, along with two SMEs, Just Evotec Biologics and Voyager Therapeutics. Each outlines the particular impact that the suite of tools has had on their activities.

GlaxoSmithKline plc (GSK) is a British multinational pharmaceutical company with around 100,000 employees. It has collaborated with OPIG from its inception in 2008. A GSK employee was a co-author on publication [5], and GSK has paid for a custom installation of SAbDab-SAbPred and also a licence for SAbBox.

The use by GSK of these University of Oxford tools has been of great economic benefit.

A GSK Scientific Leader & GSK Fellow in Protein Design and Informatics, Data & Computational Sciences [E] states: “ We use ABodyBuilder as our main antibody homology modelling tool, which in-house benchmarking has shown to be an order of magnitude faster, and of higher accuracy, when compared to the antibody modelling tool in [the software package] MOE. High quality homology models are required for many subsequent activities, such as the investigation of experimentally determined developability risks and identification of residues for mutation. ABodyBuilder is also embedded in our in silico assessment pipeline, as homology models are a pre-requisite for our in-house post-translational modification and isomerisation prediction methods… the prediction and mitigation of developability risks during discovery reduces the cost of development by an estimated £500,000 [GBP500,000] to £2,000,000 [GBP2,000,000] in wet work [experiments] and FTE [staff] per campaign depending upon the number of liabilities.” GSK indicate [F] : “unfortunately the number of molecules [campaigns] we put through to development is not something we would disclose externally.”

UCB is a multinational bio-pharmaceutical company headquartered in Brussels, with around 7,500 employees. Like GSK, it has been involved in OPIG from its inception with a paid custom-installation of SAbDab-SAbPred. UCB has co-authors on publications [1,3,4,5,6].

At UCB, SAbDab and SAbPred have become an integral part of their antibody therapeutic discovery pipelines and have provided them with significant savings. The Senior Director, Head of Global Computer-Aided Drug Design at UCB [G], says of SAbDab : “ This comprehensive antibody structure database provides the data foundation for UCB’s computational antibody design. The antibody design team within my department would not function without this data source. Should my team lose access to SAbDab, we will have to recruit a full-time contractor for data curation and database maintenance. The expected cost would be £150,000 [GBP150,000] per year.” Concerning SAbPred, they say [G]: “The toolbox is used routinely by the antibody design team in my department to support every antibody therapeutic project at UCB. This toolbox is indispensable to our antibody optimization workflow and, in our hands, produces more informative results than commercial toolkits offered by two well-recognized software companies specialized in computational drug discovery. Each of the commercial toolkits would cost over £250,000 [GBP250,000] annually for the size of UCB.”

With their database, tools and an understanding of company needs, OPIG were ideally placed to respond when UCB commissioned a study to rapidly evaluate software developed by Massachusetts Institute of Technology. The Senior Director states [G]: “ In July 2020, UCB wanted to evaluate the feasibility of adopting an open-source AI software developed by Massachusetts Institute of Technology and published in Science. Within a month UCB received from your team not only an annotated, enhanced version of the open-source software, but also a report detailing components/functions that are missing from the software. This analysis led to UCB’s conclusion that it will take £450,000 [GBP450,000] and 1 year to adopt this AI software. Therefore, UCB invested in an alternative solution which costs £130,000 [GBP130,000] and 2 months to implement. The savings of £320,000 [GBP320,000] and 10 months, thanks to the Impact Software Engineer Service which costed UCB only £40,000 [GBP40,000] …”.

Just Evotec Biologics is a Seattle-based biotech company focussed on bio-therapeutics, with approximately 200 employees. Just Evotec Biologics rely on tools from the SAbDab-SAbPred suite, using them to assess developability of antibodies on important development campaigns including an anti-HIV antibody. Use of these tools has led to the company realising huge savings. A Principal Scientist at Just Evotec Biologics [H] states: “…we rely on tools such as those provided by the Oxford Protein Informatics Group (OPIG)… Optimization with respect to developability has been shown to have a profound beneficial impact on conformational stability… Additionally, well behaved, developable antibodies can be used… saving tens of thousands of dollars [USD10,000] and months of process development time during the development process alone and ultimately leading to lower drug costs for patients. Most recently, this work benefited the optimization and process development of an anti-HIV bnAb with a potential global impact”.

Voyager Therapeutics is a Boston-based biotech company with around 200 employees. They have replaced commercial tools with ABodyBuilder, TAP and ANARCI to realise significant financial savings. A Senior Scientist in Vector Engineering, Antibody Discovery / Design at Voyager Therapeutics [I] states they have used ABodyBuilder, TAP and ANARCI: “…to predict antibody homology models which saves substantial cost (>$100K/Ab [more than USD100,000 per antibody] ) towards experimental structure determination…[and have]… replaced huge licenses costs incurred from Schrodinger, Chemical Computing Group and others ($50K/yr [USD50,000 per year] ).”

In summary, research from OPIG has enabled and accelerated antibody therapeutic discovery in pharmaceutical companies, both large and small, saving both time and money, reducing the costly experimental effort required, and bringing treatments to patients faster.

The research by Giles has enabled the major "Tier 1" investment banks to much more efficiently compute the sensitivities ("Greeks") they require for risk management and satisfy increasingly stringent regulatory checks without greatly increasing their already huge computational facilities. This saves both money and energy, with corresponding environmental benefits.

The scale of the savings is indicated by a 2020 industry survey [A] which estimates the worldwide High Performance Computing (HPC) market at about USD39,100,000,000 per year in 2019, with the Banking, Financial Services and Insurance sector approximately 12% of the total. Servers largely devoted to Monte Carlo simulations comprised a significant fraction of this 12%.

In addition, a UK SME, the Numerical Algorithms Group (NAG), has gained financially from developing and selling mathematical software to the banks to support them in their implementation of the new adjoint approach.

From research to impact

The "Smoking adjoints" paper [1] was recognised immediately as a significant advance in the state-of-the-art, addressing a very important industry need. In January 2007 the paper was voted by the finance industry readers of Risk as the best paper of 2006, with Giles and Glasserman being jointly named "Quant of the Year" by the magazine [B]. The paper was re-published in 2016 as a chapter in a book on ‘Landmarks in XVA’, edited by two leading finance industry experts Kenyon and Green, indicating its continuing importance.

Another indication of the significance of paper [1] in stimulating work within banks is given by Capriotti, who writes [D]: “ As you know, when I joined the modeling team at Credit Suisse in 2005, coming from my background in theoretical physics, one of the first things I was asked to do ... was to read your [Risk] paper and understand the methodology. … I was then able to compute first order sensitivities at a greatly reduced computational cost for a range of applications far beyond what [was] initially presented in the paper, including products with complex and path dependent payoffs. This led to a decision to incorporate the methodology into our main pricing engine, and since this would be critical to our future software developments we protected our legal position by filing a US patent application in 2008 which was finally granted in 2015: (US Patent number US9058449B2). … Credit Suisse's pioneering role in the use of AAD was also mentioned in a number of editorials in Risk Magazine, including the one covering the 2013 Credit Derivative of the House Award.”

Giles and Capriotti manually applied the Adjoint Algorithmic Differentiation (AAD) ideas to produce highly optimised adjoint code for the test cases presented in [1,4,5,6]. This manual process is error-prone when tackling very large legacy code-bases. To address this limitation and accelerate the take-up of adjoints within the finance industry, in 2009 NAG worked with computer scientist Professor Uwe Naumann from RWTH Aachen University to develop the software package for AAD called dco. This package largely automates the development of adjoint codes, although at the cost of producing less efficient code due to the technical details (operator overloading and “taping”). The 2018 online article [E] by Naumann refers to the “seminal paper titled ‘Smoking Adjoints: fast Monte Carlo Greeks’ published in Risk Magazine in January 2006” and discusses progress in AAD in the 10+ years since then, including his development of the dco software which is supported and marketed by NAG. Naumann’s work has further encouraged the take-up of AAD within the sector.

The transfer of adjoint ideas to the finance industry has also been facilitated by a one-day course on "Adjoint Methods for Option Pricing" given by Giles, Naumann and Capriotti at the leading international conference for quants and other researchers in the finance industry, QuantMinds (formerly Global Derivatives). The course has been delivered annually from 2013 to 2019, with 10-20 attendees each year with, at most, two from the same company.

Nature and extent of the impact

At the time of the REF2014 submission, the VP of Global Markets at NAG estimated that approximately 20% of Tier 1 banks had adopted the adjoint approach for computing sensitivities for certain classes of financial products. They now estimate [C] that 70% of the Tier 1 banks are using AAD, while Capriotti [D] believes that all of the Tier 1 banks are now using it. As explained in both [C] and [D], this growth has been partly driven by regulatory changes after the financial crisis of 2008. Capriotti [D] in particular writes:

“What has greatly increased the importance of adjoints has been the development of XVA, i.e., the stream of Valuation Adjustments (VAs), which started with CVA (Credit Valuation Adjustment) but now includes lots of other adjustments, such as those associated with own credit (DVA), funding (FVA) and capital costs (KVA). Some of these XVA calculations are a very important part of the Basel III regulatory environment introduced [in 2013] by the international Basel Committee on Banking Supervision. They all require a portfolio level calculation involving the simulation of many, often, hundreds of risk factors. The calculation of the sensitivities of such VA, required for hedging, is often impractical without AAD or a very large amount of computational power; it would often require 100x more computation using traditional bumping methods [which use a finite difference approximation to the sensitivity].

The current importance of adjoints to the finance industry can be seen in many ways: the continuing popularity of the course which the two of us teach with Uwe Naumann at the annual QuantMinds conference; the high proportion of talks in the main part of the conference which talk about AAD for XVA; the number of articles which discuss the use of AAD; the number of citations of my adjoint papers.”

In a video interview [F], the Head of Quantitative Research at Danske Bank, which uses NAG’s dco in developing their software, says: “ The purpose of AAD in finance … is real-time intra-day risk management for derivatives and XVA. Without AAD it couldn’t be done.” Naumann also expands on the importance of XVA, saying in a second video interview [F] “ With developments like XVA and FRTB [another new regulatory requirement from the Basel Committee] the number of sensitivities is growing … so that AAD is becoming essential”.

The significance of Giles’ research on adjoints to the banking industry is reinforced in a letter by a Managing Director at Scotiabank who says [G]:

“I am writing to highlight the importance of adjoint algorithmic differentiation (AAD) to the calculation of Derivative Valuation Adjustments or XVAs, and the lasting influence of your paper on AAD, ‘Smoking Adjoints: fast Monte Carlo Greeks’. All banks with derivative portfolios are required by accounting standards and banking regulation to perform XVA calculations, and in my experience most banks seek to actively hedge the market risk sensitivities of XVAs. As you know XVA calculations are amongst the most computationally demanding in modern finance, requiring the valuation of the banks derivative trades many thousands of times inside a Monte Carlo simulation just to obtain the basic XVA values. For those banks that seek to risk manage XVAs and for those that seek to use the forthcoming FRTB-CVA regulatory capital framework, sensitivities to market data inputs must also be calculated. XVA depends on thousands of market data inputs and traditional finite difference techniques would lead to thousands of recalculations of the XVAs to obtain sensitivities. In practice banks that do not have access to AAD are forced to either use a vast compute grid to perform these calculations or severely limit the accuracy or number of sensitivities that are calculated.”

NAG’s webpages [H] detail its extensive AAD offerings, including consultancy services to assist banks in developing adjoint software. The VP Global Markets at NAG states [C]:

[text removed for publication]

The VP continues:

“It seems the banks are still expanding their computing facilities, and they were already huge; it’s hard to know how much bigger they would need to be if they didn’t have adjoint capabilities.

Ultimately, I think all of this can be traced back to your Smoking Adjoints paper, either directly through its publication in Risk and the talks you gave a few years ago at industry conferences, or indirectly through the follow-on work, publications and conference presentations of people like Luca Capriotti.”

In the context of Machine Learning, adjoint methods are known as "back propagation". The source code files for the adjoints of the high level numerical linear algebra functions of the three major Machine Learning packages TensorFlow [I], PyTorch [J] and Theano [K] collectively cite just 13 papers and only reference [3] is cited by them all. These packages are used many millions of times each day and further demonstrate the wide reach of Giles’ research on adjoints.

The filtration models developed by researchers at the University of Oxford have made important contributions to applications in a range of fields where filters are of critical importance. In each case, there was a pre-existing product for which the University of Oxford researchers developed a mathematical model. This model was validated against experimental data, and then used to improve or further develop the product, saving a considerable amount of time and expense compared to existing methods based on extensive experimentation.

4.1 Arsenic removal from groundwater

Background and pathway to impact

The Ganges–Brahmaputra Delta is a global hotspot for arsenic groundwater contamination affecting Indian and Bangladeshi populations. High levels of naturally occurring arsenic, far above the World Health Organization (WHO) safe level of 10µg/L [A], are consumed by people drinking water drawn from shallow wells. This has created a major public health issue in West Bengal and Bangladesh, described by WHO in 2000 [A] as “ the largest mass poisoning of a population in history”. WHO [A] also states: “ it has been estimated that in West Bengal the number of people exposed to arsenic is 1.5 million, and one estimate of the number of patients with arsenicosis exceeds 200,000”, highlighting the urgent need for cost-effective solutions to remove arsenic from water supplies and motivating work on this aspect of the impact.

In 2006, an experimental team led by Prof De from the Chemical Engineering Department at IIT Kharagpur in West Bengal developed a filter for removing arsenic from contaminated water, using naturally abundant laterite soil and flow driven by gravity, and demonstrated that this was ultra-low cost and highly effective at producing water with arsenic below the recommended WHO level [B]. Independently, in 2014, Griffiths began a Royal Society University Research Fellowship project, “21st Century Fluid Dynamical Challenges in Water Purification”. Part of this project concerned mathematical modelling of arsenic removal and he appointed an ex-student of Prof De, Sourav Mondal, as a PDRA in 2015 as part of this work. Griffiths then instigated a collaborative programme between Oxford and IIT Kharagpur, including appointing a second PDRA on a Global Challenges Research Fund grant. Prof De has received several prestigious awards, including national recognition by India in 2018, for the societal benefits of this collaborative work [B,C].

The model of Professor De's filters developed by the University of Oxford team addresses two key requirements: it enables prediction of the lifetime of the filters [D] (beyond which the filter medium becomes saturated with contaminants), and it resolves the question of how they can be upscaled from single home filters to much larger filters for schools and communities. Prof De [B] states “ *Ian and his team derived a mathematical model that addressed both of the key requirements. The quantification provided by the mathematical models has allowed us to have confidence as we deploy these filters on a national scale.*” The lifetime prediction, based on the system parameters (flow rate, filter size and arsenic concentration) removed the need for manual inspection, speeding up the deployment of filters in homes, schools and communities as well as increasing the cost-effectiveness of the filter. Prof De [B] indicates that “ this work has enabled us to provide filters with a service life of more than seven years”.

IIT Kharagpur first licensed the technology to major manufacturers Vas Bros Pvt. Ltd, in 2015, and subsequently Mondal Precision Pvt. Ltd, in 2016, to manufacture these filters [B]. (Note that this Mondal is unrelated to the two PDRAs.) Prof De [B] indicates that “ the mathematical models … have provided additional guidance to Vas Bros and Mondal Precision and confidence to the Government. This filter invention has been approved by the Department of Science & Technology,..., Ministry of Sanitation and Drinking Water [see [E]], ..., and the West Bengal Arsenic Task force and UNICEF. The number of community scale filters...deployed by our industry partner are about 130 by Mondal Precision Pvt. Ltd and about 12 by Vas Bros till now [22 Dec 2020]”. The Director of Mondal Precision reports [F] “ We anticipate that this [Mondal Precision’s 130 filters] caters to the daily needs of more than 150,000 affected people”.

Major improvements in public health

The severe health effects of arsenic in the Ganges–Brahmaputra Delta are not well quantified [G]. However, a WHO Report [A] states that “ it has been estimated that the lifetime risk of dying from cancer of the liver, lung, kidney or bladder while drinking 1 litre a day of water containing arsenic at this [50μg/l] concentration could be as high as 13 per 1,000 persons exposed” . For the 150,000 people served by the new community-scale filters, this fraction equates to a total of 1,950 premature deaths that have been avoided.

*Financial and economic benefits

The upscaled filters are very cheap to run. The Government of India Department of Drinking Water and Sanitation, reporting these laterite filters as a Recommended Technology [E], states this system has a “ lifecycle cost of 31 Rs/Kl [GBP0.0003 per litre]”). These filters have therefore been financially viable for providing clean water to the affected communities of West Bengal.

There have also been clear consequential improvements in economic activity as a result of fewer early deaths, as well as improvements in general wellbeing and quality of life. Specific savings from these filters are difficult to estimate. However, a Bulletin of WHO [G] states: “ We estimated the economic losses resulting from the arsenic-related mortality burden by calculating lost productivity in terms of per capita gross domestic product (GDP). According to estimates by the International Monetary Fund, the per capita GDP for Bangladesh in 2009 was 1,465 purchasing power parity dollars…could lead to a loss of US$12.5 billion [USD12,500,000,000] , provided arsenic exposure (>10 μg/L) remains the same as in 2009”. Performing a similar calculation for the 1,950 premature deaths avoided in West Bengal, based on the same per capita GDP of USD1,465 per year for deaths avoided, results in gains of 1,950 x USD1,465 per year = USD2,900,000 per year.

[Text removed for publication]

4.5 Conclusion

In summary, the models developed by researchers at the University of Oxford have been used to understand and quantify the behaviour of filtration processes in a multitude of areas and the impact has included: saving many lives, significantly reducing development costs, and dramatically reducing the timeline for new product development.

The impact is primarily economic, with HYDRA being part of the technology which significantly contributes to the success of Rolls-Royce, one of the UK’s premier engineering companies and a world leader in gas turbine engines for aircraft and naval propulsion, as well as for power generation. According to its 2019 Annual Report [A], published in March 2020, Rolls-Royce Holdings plc (the official name of the company) achieved annual revenue of over GBP15,000,000,000 with approximately 51% coming from Civil Aerospace, 22% from Power Systems, 20% from Defence, and 7% from other sources [A, page 2]. Rolls-Royce employs around 50,000 people in 50 countries [B], with approximately 24,000 of those being in the UK [B], and the company estimates that they contribute GBP12,000,000,000 annually to the UK economy and support over 135,000 jobs [B]. As well as contributing to these economic benefits, HYDRA has contributed to very substantial benefits to the environment and to airlines due to the fuel savings achieved.

In addition to other key software tools for combustion analysis, structural analysis and heat transfer, the ability to simulate the flow of air through the engines is an important part of Rolls-Royce’s design capability, as engines are now designed almost exclusively through computer simulation, with experimental testing carried out afterwards to verify the performance of the final design.

Pathway to impact:

The initial development of HYDRA was carried out in Oxford up to 2002. The initial code was then transferred to Rolls-Royce, and, over the following years, was further developed internally by Rolls-Royce and, until 2006, by Giles’ group in Oxford. It was also used by university research groups at Cambridge, Loughborough, Southampton, Surrey and Sussex who were also funded by Rolls-Royce, with these groups adding additional features in some cases, such as for the flow through internal cooling passages. This history is partially documented in paper [C], written in 2004 by the Rolls-Royce engineer leading the further development of HYDRA within the company, which states: “HYDRA-CFD is a unique suite of steady, unsteady, harmonic and adjoint solvers for turbulent CFD, built on a common input, output, multi-grid acceleration, parallelization and visualization core.”

HYDRA was gradually introduced into service within Rolls-Royce from 2008, becoming the primary corporate CFD code in 2012. Rolls-Royce’s Chief Design Systems Architect [D] states: “I can confirm that HYDRA has been the company’s primary aerodynamic CFD code for the whole REF period 2014-2020. It has been used extensively in the design of all recent engines; and, today, is run over 1,000,000 times per year.”

The many uses of HYDRA within Rolls-Royce are illustrated in this diagram which is used with permission from Rolls-Royce.

Nature and extent of impact:

The greatest impact within Rolls-Royce has been in the design of its gas turbine engines for major civil aircraft manufactured by both Airbus and Boeing. At the beginning of the REF impact period in August 2013, there were two main civil aerospace engines [E] which had been designed using HYDRA, the Trent 1000 for the Boeing 787 and the Trent XWB for the Airbus A350. As mentioned above, all further development has also used HYDRA, with new variants of the Trent 1000 being certified in 2015 and 2016 [F], and a new variant of the Trent XWB certified in 2017 [F].

The Rolls-Royce Chief Design Systems Architect’s letter [D] states: “HYDRA contributed directly to the technology developed for the Trent XWB, the world’s most efficient aero engine which delivered a 15% improvement in fuel consumption relative to the original Trent engine and provides USD2,900,000 savings per year per aircraft in fuel alone. HYDRA specifically contributed to the aerodynamic design of the fan, compressor and turbine – the fan design delivered world-beating levels of performance and the turbine has the highest efficiency of any Trent engine.”

In July 2020, Rolls-Royce data on the Trent XWB [E] shows that over 1,600 engines have been purchased or are on order, powering more than 800 Airbus A350 aircraft. This corresponds to total airline fuel savings of more than USD2,300,000,000 per year. In addition, the 15% improvement in fuel consumption provides huge environmental benefits in reduced CO₂ emissions. The global aviation industry produces 2-3% of global CO₂ emissions and IATA has targeted a 1.5% improvement in fuel efficiency per year over 2009-2020 [G].

An entirely new engine, the Trent 7000 [E], has also been designed using HYDRA. The Trent 7000 powers the Airbus A330neo and replaces the Trent 700 used on earlier A330 models; as of 31 December 2020, 57 Trent 7000-powered A330neos have been delivered, and 331 are on order [F].

HYDRA has also been used to develop other Rolls-Royce products, such as ground power installations for electricity generation, and the Pearl 700 engine for the Gulfstream G700 business jet. In reference to this engine, the Chief Design Systems Architect’s letter [D] also says: “HYDRA is a global system used by engineers in the UK, US, Germany and India, and has therefore been used on a wide range of engines including, for example, our recently [November 2019] unveiled Pearl 700 engine for the Gulfstream G700 ultra-long-range business jet. This engine has 3.5% less fuel burn and 5% greater efficiency than the previous BR725 engine for

the G650.”

Impact of parallel computing implementation:

The Rolls-Royce website [E] describes the design process for the Trent XWB as: "The most intense/comprehensive development programme ever undertaken by Rolls-Royce. Six times the computing power applied than the previous generation."

The emphasis on the computing power is significant because of the huge computational resources required for large-scale simulations. In reference to this, and the more recent HYDRA research on improving its parallel performance on modern computing architectures, the Chief Design Systems Architect’s letter [D] states: “Simulation and modelling, enabled by High Performance Computing (HPC), have transformed the way our products are designed and engineered and will continue to do so. In 2018, we invested [text removed for publication] in an HPC upgrade based on the strategic importance of simulation and modelling to the company. HYDRA continues to have world class parallel performance which is a testament to your vision in designing the code.”

Conclusion

HYDRA is a key part of the suite of computational engineering tools which has been used by Rolls-Royce to design its gas turbine engines, helping Rolls-Royce to maintain its position as one of the UK’s premier engineering companies, and has provided downstream additional benefits to airlines and the environment due to improved fuel efficiency.

The research into Rough Path signatures carried out at the University of Oxford has resulted in economic impact for Sogou, a major Chinese corporation, and has enabled an estimated 75,000,000 people in China to daily input accurately Chinese characters into their smartphones by drawing on the screen. This has resulted in societal benefit particularly to older Chinese users by enabling them to fully embrace the internet through their smartphones.

Sogou is a Chinese internet company which was spun off from a company called Sohu.com in 2010 [A]. It is best known for two things: its search engine, which is number two in the Chinese market with roughly a 20% market share compared to about 70% for Baidu [B], and its Mobile Keyboard (previously known as Sogou Input Method) which is a utility for mobile phones through which users can input Chinese characters [A]. The great success of the keyboard product came from Sogou’s development of an interface for Pinyin, a Romanization of the Chinese characters based on their pronunciation; this is extremely popular amongst younger users and made the Mobile Keyboard the market leader in this area [A]. Sogou is a Beijing-based public company. Until recently it was listed on the New York Stock Exchange but in September 2020 it was acquired by Tencent, one of China’s largest internet companies, for more than USD2,000,000,000 [B].

Pathway to impact

In 2012, Lyons had discussions with Ben Graham (then at University of Warwick, now at Facebook) who was pioneering the development of sparse neural network techniques. Graham also had an interest in Chinese character recognition, and Lyons’ recent work on signatures led Graham to combine signature annotation of pen strokes with his sparse neural net methods. With this combination, Graham won one part of an international competition on Chinese handwriting recognition as part of the IEEE 12th International Conference on Document Analysis and Recognition (ICDAR). His accompanying 2013 paper (“Sparse arrays of signatures for online character recognition”) cites four of Lyons’ papers, including [1].

The successful combination of signatures and neural nets for Chinese character recognition immediately attracted the attention of Prof Lianwen Jin, a computer scientist at SCUT who in 2015 was Dean of Information Engineering in SCUT’s School of Electronic and Information Engineering, and also head of the HCII (Human-Computer Intelligent Interaction) lab. Jin led a team working on handwriting recognition using other methods, but he was also aware of machine learning techniques. As Jin explains [D]: “ We have for many years been a leader in the technology of document analysis and optical character recognition, and particularly online entry of finger drawn Chinese handwriting. It was a breakthrough when Ben Graham introduced his prize winning technology in this area, combining path signatures with sparse convolutional neural networks. We very quickly introduced and further developed use of the combination of signature and traditional features in our approach. The successful application of signature feature in our work was inspired by the wonderful work and publications by Prof Terry Lyons and his team at the University of Oxford.” The introduction of signatures, combined with the considerable experience Jin and his team had in the development of mobile phone apps, led them to develop a new version of their gPen handwriting recognition app.

Jin demonstrated (again) the benefits of using signatures to represent the features of the input finger strokes in a paper presented at the IEEE 13th International Conference on Document Analysis and Recognition (ICDAR) in August 2015. Along with Weixing Yang, Jin visited the University of Oxford, also in August 2015, in order to start a collaboration with Lyons and Hao Ni, a PDRA employed on Lyons’ ERC grant. Subsequently, Yang and another researcher from SCUT visited the University of Oxford at the beginning of 2016 for 3 months, and Lyons visited SCUT twice. This collaboration led to further improvements in Jin’s handwriting recognition software (as well as the joint publications [5,6] described in Section 2).

The impact from the University of Oxford’s research into Rough Paths has been achieved in two stages:

• Signatures were incorporated into SCUT’s free gPen handwriting recognition app, that has been subsequently downloaded by over 1,000,000 users

• the licensing of SCUT’s software to Sogou, who have incorporated it into their market-leading Mobile Keyboard for the input of Chinese characters on smartphones, with 75,000,000 users estimated to use the SCUT-derived software each day in China.

Impact part 1: gPen app

Prior to the collaboration with Oxford, Prof Jin’s research group already had considerable experience with Chinese handwriting recognition and had an existing mobile phone keyboard app called gPen [C], with versions 1 and 2 released in 2011, and version 3 in 2013. The incorporation of signatures as a key pre-processing step in the version 4, released in 2014, significantly improved the accuracy of the app and enabled character recognition in real time, and additional performance improvements from the collaboration with the University of Oxford were incorporated into version 4.3. As Jin [D] explains: “ This new technology allowed us to develop a fast and compact mobile phone app gPen for IPhone, and Android. Version 4.0 of our software, released in October 2014 was the first mobile phone app to use signatures and neural nets for online Chinese handwriting recognition. Extensive tests showed it to be state-of-the-art. When compared with the other main competitors it was clearly the most accurate, and the tests also showed that the use of signatures was an important part of this success.” In the interview below, Jin states that version 4.0 and the subsequent versions were downloaded over 1,000,000 times for Android phones.

Left: image of gPen app being

used to enter

a Chinese character

Right: Sogou Mobile Keyboard handwriting module acknowledgement of SCUT core software [F].

Impact part 2: Sogou Mobile Keyboard

In 2015, Jin’s team developed an open source testing tool to objectively evaluate the 6 most popular handwriting input methods then available, including the handwriting recognition software of Sogou. This showed that gPen had the best recognition accuracy on different test sets, and also again demonstrated the importance of using signature information. This led to Sogou approaching SCUT to license their software. Unfortunately, the licensing deal between SCUT and Sogou prohibits Jin from writing about it officially, but the story is captured in an interview given by Jin in July 2020 (edited Google Translation [E]):

[Interviewer] “ From 2015 to 2019, the handwriting setting copyright notice column of the Android version of Sogou mobile input method shows that the technical support is provided by South China University of Technology. Can you share your experience with us?”

[Jin] “ After we added some data augmentation and preprocessing techniques, we made a recognition effect far superior to traditional methods. […] We compressed and accelerated the model, transplanted it on the mobile phone, and realized the first deep learning-based handwritten Chinese character recognition method on the mobile phone. In 2014, we released this handwriting recognition engine on the Google market, and it has been downloaded more than 1 million [1,000,000] times. Once, a Sogou researcher saw our app and evaluated it. We not only have a high recognition rate, but also make the model small and fast. On the mobile phone side at that time, it took about 20ms to complete the entire process of processing a character recognition without a GPU. On the CPU side of the server, it was about 4~6ms. This speed was amazing. The CNN [convolutional neural net] model that supports more than 10,000 types of characters is less than 3MB in size. Therefore, under comprehensive consideration, Sogou chose to cooperate with us.”

In 2015, the software in gPen was licensed to Sogou to improve the marketability of their Mobile Keyboard software package by incorporating the gPen methodology and software for handwritten character recognition in place of their own software (which had been shown to be inferior to gPen). The initial software, based on gPen 4.2, was transferred to Sogou at the end of 2015. SCUT continued to develop and update gPen, releasing version 4.3 in May 2016 and a final version 4.3.1 in February 2017 [C].

Although the terms of the licensing deal prevent Prof Jin from revealing most of the details, as mentioned in the interview [E] it included an undertaking from Sogou to publicly confirm, from 2015 to 2019, the use of SCUT’s software within the app in the Settings page for the handwriting input module (see the screenshot of the app on the previous page). According to Google Translate, the acknowledgement says “ Handwriting core technology provided by South China University of Technology” [F].

The Statista website estimates that there were approximately 750,000,000 smartphone users in China in 2019 [G] and, at the end of 2019, Sogou’s Mobile Keyboard had over 450,000,000 daily active users [H]. According to the Google Translation of a 2019 survey of mobile phone users by one of China’s leading media data analytics companies [I], 61.8% of users prefer a Pinyin interface (either “Jiugongge” or full keyboard), 16.8% prefer handwriting input, 10.0% prefer voice input, and the remainder prefer a variety of other alternatives. The use of handwriting input is also anecdotally described in a Chinese blog article from April 2020 [J], the Google Translation of which says: “ No one writes by hand anymore? But in fact, it's not [the case] ! With the increasing size of smartphone screens, in addition to pinyin and stroke input, the space available for handwriting input to stretch out also expands. According to the March information of China Internet Network Information Center and Sogou Input Method, as of June 2019, the number of Internet users in my country reached 854 million [854,000,000], of which more than 100 million [100,000,000] *people use handwriting daily, mainly middle-aged and elderly people over 40 years old and small town residents, and handwriting input is an important tool for this part of the user group to embrace the Internet.*”

Based on 16.8% of mobile phone users preferring handwriting input, approximately 75,000,000 people are using Sogou’s handwriting module to interact effectively with their mobile devices on a daily basis, benefitting from SCUT’s technology that utilises the mathematical ideas of Rough Path signatures developed at the University of Oxford to represent the key features of their handwritten input.

The primary impact of the statistical research undertaken by Donnelly has been on improved public health policy and actions in England and Scotland during the management of the COVID-19 pandemic. In addition, there are wider public health contributions in Europe and the US, and economic benefits are identified in one specific area through the provision of advice directly to theatre leaders.

Pathway to Impact

The reports and papers written by the COVID-19 Response Team and the REACT Study Team have been very widely read by key policy-makers in the UK, Europe and the US. Throughout the pandemic, the information flow to SAGE was facilitated by Donnelly’s colleagues who were/are members of SAGE and SPI-M, the SAGE subgroup on modelling (including Prof Neil Ferguson until May 2020 and Prof Steven Riley throughout).

Impact: UK public health

Paper [1], in its Imperial College pre-print version known as Report 13, and the accompanying computer model, have been used extensively by the English and Scottish governments to model the development of the pandemic and the impact of different public health measures, such as lockdowns. Report 13 is reviewed on a webpage of the UK Parliamentary Office of Science and Technology [B], and a SPI-M report from 16 April 2020 [C] cites both Report 13 and the pre-print version of [2] in discussing the role of children in COVID-19 transmission, and the implications for possible school closures.

The impact on public health in Scotland is indicated by the model and computer code from [1] being used by the Scottish Government in a sequence of 32 reports in 2020, the first of which [D] says: “ Scottish Government uses the publically available Imperial model as reported in their Report 13 to help understand the Covid-19 epidemic in Scotland over the longer term and what the reproductive rate at a point in time (Rt) is for Scotland.”

The REACT programme [4,5,6] has been used extensively to inform the deliberations for SAGE and its sub-committees during 2020. According to the UK Government press release on 29 April 2020 [E i)], REACT “ forms part of Pillar 4 of the Government’s COVID-19 testing strategy, to conduct UK-wide surveillance testing to learn more about the spread of the disease and inform the development of new tests and treatments.” At the launch, Health Minister Lord Bethell said [E i)]: “ Understanding more about the current spread of coronavirus and the prevalence of antibodies is a vital part of our ongoing response to this pandemic.”

The REACT results [4], summarised in a Government press release on 11 Sept 2020 [E ii)], showed that infections in the UK were doubling every 7 to 8 days. These results led to the introduction of the so-called rule-of-six, with Health Secretary Matt Hancock saying in the same press release [E ii)] “ It’s so important that everyone abides by the law and socialise in groups up to 6, make space between you and those outside your household, get a test and self-isolate if you develop symptoms and wash your hands regularly.” Following this, on 21 Sept 2020 in a Downing Street briefing, Sir Patrick Vallance said [F]: " The challenge therefore is to make sure the doubling time does not stay at seven days. There are already things in place which are expected to slow that, and to make sure that we do not enter this exponential growth and end up with the problems that you would predict as a result of that. That requires speed, it requires action."

The later REACT results [5] from testing over 365,000 volunteers, the world’s largest home SARS-CoV-2 antibody testing programme, were summarised in a Government press release on 27 Oct 2020 [E iii)] in which Health Minister Lord Bethell said: “ This study … is a critical piece of research, helping us to understand the nature of COVID-19 antibodies over time, and improve our understanding about the virus itself. We rely on this kind of important research to inform our continued response to the disease, so we can continue to take the right action at the right time.”

The most recent REACT results [6] were summarised in a Government press release on 15 Dec 2020 [E iv)] which stated “ The study findings demonstrate a rise in infections among secondary school age children. To tackle this rise in London and surrounding areas, additional mobile testing units will be deployed in or near schools for staff, students and their families ...” Two days later a report to SAGE on children, schools and transmission [G] cited [6] saying “ *REACT-1 data between 13th Nov and 3rd Dec … show the highest prevalence in children aged 13-17 years (high confidence)*”. On 30 Dec 2020, the Government in a press release [E v)] “ responded to rapidly rising case rates due to the new, more transmissible variant of coronavirus by triggering the education contingency framework and pushing back the staggered return for secondary schools and colleges by one week”. Having previously committed to keeping schools open, the Prime Minister explained at the 30 Dec 2020 coronavirus press conference [H] “ *But we must face the reality that the sheer pace of the spread of this new variant requires us now to take even tougher action in some areas and that does affect schools.*”

Impact: Commercial

The COVID-19 Response Team was contacted by various industry and commercial groups for advice and guidance. For example, Donnelly was asked for advice by a group of leading theatre producers, directors and CEOs representing major theatres in the UK and US, and directly communicated the implications of her findings for indoor live performances in the UK and the US. They were able to anticipate that the ban on full-capacity indoor live performances would last for many months and, as explained by the group [I]: “ Thus, we used Prof Donnelly’s research to inform production schedules and investments to reduce avoidable losses incurred by preparing productions that then could not go ahead. … The size of the risks to subsidised and commercial theatre production of making the wrong decision as to when to reopen are, within each specific context, immense. In the case of a major art house theatre such as Sadler’s Wells or The National Theatre with an annual turnover in the 10s of millions [GBP10,000,000 per year], had they gone into production with the aim of re-opening with a full season in July, or even September, and then had to cancel for a second time, this might have brought the company to the verge of bankruptcy.”

Impact: Europe and US public health

Within Europe, the European Centre for Disease Prevention and Control [ECDC] plays a coordinating role in issuing advice to member states during pandemics. ECDC reports between April and September 2020 drew extensively on [1, 2]. In the sample of six reports provided [J], three cite [1],and five cite [2]. Public health policy documents from both the EU Commission and individual European governments cite both the ECDC reports and [1, 2] directly. Source [K] contains three examples: a) "Joint European Roadmap towards lifting COVID-19 containment measures" from the EU Commission (17 April 2020) cites [1]; b) "Plan de Respuesta Temprana en un Escenario de Control de la Pandemia por COVID-19" ("Early Response Plan in a COVID-19 Pandemic Control Scenario") from the Spanish government (16 July 2020) also cites [1], and c) "Operational guidance for the management of SARS-CoV-2 cases and outbreaks in schools and kindergartens" from the Italian government (28 Aug 2020) cites [2].

In the US, leadership on the public health response to COVID-19 has been provided by the state governors. New York State was particularly badly hit, and Governor Cuomo and his officials paid close attention to [3] on the state-level tracking of COVID-19 within the US. One of the authors of [3] was involved in a virtual press conference on the day of its publication, at which the governor discussed plans to re-open the state and drew attention to the state’s use of the modelling and analyses undertaken in [3] saying [L]: “ In a State like New York, what the people did dramatically changed that curve so it affected the projections. The Imperial College model, as we've been following this for weeks, was the best, most accurate model. And therefore, I think Dr. Bhatt [co-author of [3]] deserves all our thanks because they really helped us all through this to date, and I want to thank him very much for taking the time to advise us, not just on how we constructed our model to date but what happens going forward as we increase the economic activity and we start to see numbers change.”