Impact case study database

Introduction

CBD models [3.1, 3.2] have been used to assess the potential impact of longevity risk on the future solvency of insurance companies writing annuities and pension funds in the UK, Europe and the US. Use of the models helps these institutions to (a) assess more accurately their capital requirements (including regulatory) to ensure a secure future for policyholders, and (b) price more accurately longevity risk transfers between institutions. In turn, this provides policyholders and pensioners with greater confidence that their promised benefits will be covered for the next 30, 40, 50 years.

Members of the CBD family are often referred to by practitioners by their “M” numbers, e.g. M5, M7 (see e.g., [5.1, 5.2, 5.3, 5.4]).

Impacts

Software

Growing interest in the CBD family from insurance companies led the specialist actuarial software provider Longevitas to incorporate the CBD family into its Projections Toolkit [5.1]. This software is used by UK and US insurers (e.g. [text removed for publication] [5.2]) and its embedded suite of models facilitates insurance company compliance with the UK’s Prudential Regulatory Authority (PRA) guidance on the use of stochastic mortality models.

Other stakeholders, including [text removed for publication] [5.5] and the European insurance regulator EIOPA [5.6, p 65], made use of an open-source R stochastic mortality modelling package, StMoMo, [5.4], first released in 2015 that builds on [3.4]. A significant proportion of StMoMo is devoted to the CBD family (see [5.4, pages 1, 3, 6-7, 20, 27-35, 39, 49-50]).

Regulation of European Insurers

Insurers in the EU have, since 2016, been subject to the Solvency II regulatory environment. This governs how much capital insurers are required to hold to cover their future uncertain liabilities. Required capital consists of the market consistent value (MCV) plus the Solvency Capital Requirement (SCR). The SCR is an additional amount over the MCV to cover the risk that, alongside other risks, mortality rates fall at a faster rate than anticipated. The SCR can be calculated using a simple stress test or through use of a stochastic internal model. Larger insurers often prefer to use the latter, as these allow for a more accurate, company-specific analysis of all major risks.

Two distinct but related regulatory impacts can be identified.

1. For insurers using stochastic internal models, the UK’s Prudential Regulatory Authority (PRA) recommends use of the CBD family [5.7, p 8] alongside three other families of model (with the use of distinct families to address model risk) to assess capital requirements for longevity risk. Out of these families, [text removed for publication] recommends CBD-M9 as being one of only two models suitable for a wide age range [5.8, slide 68].

2. For insurers using the simpler longevity stress test, the CBD model was used by the EU insurance regulator, EIOPA, alongside one other model, to verify the suitability of the 20% longevity stress [5.6, pages 60-73] using the StMoMo software package [5.4]. Responses to the EIOPA consultation (including institutions in the Netherlands, France and Ireland and pan-European professional bodies) indicate that the CBD model is in widespread use around Europe.

The PRA approach to the assessment of model risk and the range of models employed reflects the influence of Cairns et al. (2009) (see e.g. [5.5]). As a result of the PRA guidance, CBD models and the model-risk framework of Cairns et al. (2009) are widely used by insurers in the UK and elsewhere [5.9, 5.10]. Quoting UK and EU insurers, [3.2] “ was cited in the governance of our [text removed for publication] *Solvency II internal model and helped to mould our approach to the assessment of longevity risk.*” [5.9], and “ has impacted significantly on our work in three ways: their proposal of a number of new and innovative stochastic mortality models; their finding that model risk can be quite significant; and their approach to model selection using multiple criteria” [5.10].

Derisking of Defined Benefit (DB) Pension Funds

Many DB pension funds are seeking to reduce their exposure to longevity and investment risks. Derisking is often achieved, in part, through longevity hedges such as longevity swaps and bulk buy-outs with insurers in the UK or overseas (such as [text removed for publication] [5.2]). Insurers receiving the risk from pension funds use CBD models (a) to assess a best estimate price, (b) to assess the degree of longevity risk embedded within each portfolio of pensions and then (c) to determine a risk premium to be charged for acceptance of the risk. UK insurers (e.g. [5.9]) taking on such risks are then subject to the Solvency II regulations discussed above.

Longevity Risk Transfers

There has been “ approximately $200 billion [USD] of longevity risk transfer since 2014” ([5.2]). [text removed for publication] states [5.2] “ Over the period 2014-2017, we used the CBD models for pricing all of our global transactions, covering a total transfer of liabilities of approximately $40 billion [USD]. This included the [text removed for publication] longevity transaction [text removed for publication] , covering [text removed for publication] of pension liabilities in the [text removed for publication] pension scheme in [text removed for publication] *. … The use of the CBD models was central in establishing confidence in the pricing of all of these transactions.*”

Actuarial Consultancies

CBD models and their descendants are used by actuarial consultancies in the UK and elsewhere in their advisory and development work [5.5, 5.3]. [text removed for publication] [5.5] states that [3.2] is the go-to, fundamental reference that has brought order and rigour to what had, at times, been a somewhat naïve longevity risk-transfer market. [5.5] also states that [3.5] and e.g. [3.6] are fundamental works in helping his firm and clients understand and model socio-economic mortality differentials.

Professional Education

Recognising the role of CBD models and model risk in longevity risk management, the German Actuarial Association (DAV) recommends [3.2] as part of the DAV’s reading for its Chartered Enterprise Risk Actuary qualification (CERA) [5.11].

Impact on Great Britain Capacity Market

A reliable electric power supply underpins almost all of modern life (see https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/48129/2176-emr-white-paper.pdf) – thus ensuring an appropriate level of security of supply is critical for society. This societal significance is confirmed by testimonial letters from National Grid, and from a former board member at Ofgem (the GB energy regulator) who was responsible for this area [5.1, 5.2]. The direct cost of capacity procurement (i.e. the cost feeding to consumer bills) in the main 4 year ahead capacity auctions has ranged from GBP290,000,000 (for delivery in 2022/23) to GBP1,180,000,000 (delivery 2020/21) [5.3].

The Maxwell Institute team of Dent, Wilson and Zachary have been the main academic consultants to National Grid (NG) on risk modelling and decision analysis methodology since 2011, and thus for the entire REF period. This is evidenced by the specific points of impact below. The ongoing broad significance of the decisions based on our research is confirmed by NG’s 2019 Capacity Assessment report [5.4a]: on pages 22-25, this references in broad terms the significance of the Maxwell Institute’s research-based advice (references to unnamed ‘academic consultants’ refer to us).

The reach of the impact is GB-wide. The Maxwell Institute work underpins the decision support approach of NG by the Department of Business, Energy and Industrial Strategy (BEIS) in this nationally vital area of procuring capacity to ensure electricity security of supply.

The specific items of GB impact in this study fall into two strands:

1. Choice of scenarios considered and decision analysis approach used. Our decision analysis work [3.4] was used as described in National Grid’s 2017 Capacity Assessment Report [5.4b] to confirm the choice of methodology for supporting the decision on volume of capacity to procure in 2017/18 auction, for delivery in 2021/22, an auction worth GBP423,000,000 [5.3]. The 2019/20 auction for delivery in 2023/24, with design based similarly on our work, was worth GBP699,000,000. In particular, pages 17, 39 and 87-97 of the 2017 report [5.4b] describe how our work guided the choice of scenarios (particularly extreme optimistic and pessimistic scenarios) used in decision support for capacity procurement, through the demonstration in [3.4] of how the LWR analysis outcome depends on these extreme scenarios.

BEIS had been concerned about whether sufficient account was being taken of very pessimistic scenarios of planning background by the LWR approach, and we were asked to investigate whether including more extreme scenarios with an appropriate weighting scheme could improve decision making. We showed that the outcome was not very sensitive to addition of further extreme scenarios, and thus BEIS decided to go forward with their incumbent analytical approach given appropriate choice of scenarios. The significance to BEIS of this confirmation of performance of their approach is confirmed by BEIS’s Panel of Technical Experts [5.5, page 46] who note that ‘ The analysis for National Grid by Wilson and Zachary formalises the concerns we have previous expressed about the LWR methodology’ and ‘ we were grateful that National Grid did conduct a trial run to assess the impact of adopting such a hybrid approach’.

National Grid’s 2016 Electricity Capacity report [5.4c] confirms (page 30) that our work [3.1] on the statistical relationship between electricity demand and available wind capacity formed the basis for inclusion of a ‘low wind sensitivity’ in their decision analysis, due to historic data suggesting the possibility of wind power availability typically being slightly poorer at times of highest demand. This resulted in a need for an additional 0.8 GW of other capacity to give the appropriate level of security of supply, equivalent to an extra large gas-fired power station, and at a clearing price of GBP22.50/kW giving a direct additional expenditure of about GBP18M. This is an example of where improved analysis can demonstrate the need for increased up-front expenditure in order to give an appropriate level of security of supply.

The scale of this impact on decision support is the whole monetary value of the capacity market, and the nationally significant issue of procurement of generating capacity for security of supply, in that the methodology directly supports the top level decision of volume of capacity to procure.

2. Determining how to treat energy storage in the capacity market. In 2016 it was necessary to update at short notice the approach to crediting battery storage in capacity auctions. Under the existing rules, batteries that would only discharge at maximum output for up to 1 hour had been credited with 95% of their capacity, in analogy to the nearest existing technology of pumped storage hydro (which can discharge at maximum capacity for much longer, and cover the whole duration of the early evening demand peak). Based on our work in [3.2] and [3.3], National Grid and the government changed the risk index used for valuing storage from duration-based to energy-based, so as to make a fair comparison between storage and other resources as described above. They also made the specific decision to use the marginal value of the resource, when added to the rest of the portfolio procured, to credit emerging technologies such as storage and wind energy within the calculations.

The executive summary of [5.6] states “ We also acknowledge the benefits of discussions with academic experts from The University of Edinburgh on a number of high level issues, notably in relation to the choice of risk metric and the definition and calculation of the storage EFC”, with the Maxwell Institute report based on research in [3.2] and [3.3] included as Appendix 3. Following implementation of the recommendations in this report, battery units have been credited with as little as 10% of their capacity, versus the previous 96% [5.6, page 29]. The capacity of battery and similar storage awarded contracts in the 2019/20 auction for delivery in 2023/24 is 0.18 GW [5.3, page 3 of that year’s report]. Without the revised capacity credit factors based on our research this would have been over-credited by more than 0.1 GW, a direct overpayment to storage of approx. GBP2,000,000 at the market price of GBP16/kW/year. The supply capacity available to the system to support reliability would effectively have been lowered by this amount.

More importantly than these figures for the 2019 auction is the basis our work provides in supporting inclusion of storage in the capacity market on an ongoing basis, which in turn supports national plans for increased use of storage to the benefit of consumers while maintaining security of supply. Thus the annual capacity and monetary figures will grow considerably as storage becomes more significant on the scale of the whole system.

Verification of GB impact through testimonial letters. The EMR Modelling Manager from National Grid [5.1] commented “ *What the research has enabled is the ability to quantify the statistical relationship between wind and demand at times of high demand, improving the robustness of our modelling, which had previously used an assumption of independence.*” A former Board Member of Ofgem [5.2] said “ The work of Dent, Wilson and Zachary in this field has been significant in promoting … statistical techniques…. and has informed the assessment of capacity requirements in the capacity market and the assessment of capacity adequacy”. The letters also contain their own statements of the importance of reliable electricity supply and the significance of our research in underpinning analysis supporting the capacity market.

Impact on development of IEEE Standard 859

Dent was chosen to chair the recent revision of IEEE Standard 859 on “Standard Terms for Reporting and Analyzing Outage Occurrences and Outage States of Electrical Transmission Facilities”, published as IEEE 859-2018 [5.7]. IEEE 859 is widely used in the N American industry, in particular underpinning the continent-wide Transmission Availability Data System run by the North American Electricity Reliability Corporation (NERC), which records reliability data from the majority of utilities, and by other industry organisations internationally.

According to the letter [5.8], based on the data collected under guidance of IEEE 859, “ Industry and regulators use the resulting transmission outage statistics to determine best investments, and societal impacts resulting from transmission performance”. In NERC’s own geographical scope, this supports reliable supply to nearly 400 million people. In the USA alone, the significance of the use of data collected based on this standard is shown by annual investment in electricity networks at USD45,000,000,000 annually in 2016 and on an increasing trajectory [5.9]; and the US Government estimating annual costs of weather-related outages (see point (ii) below) standing at between USD18,000,000,000 and UDS33,000,000,000 annually in the decade to 2012 [5.10]. The US Energy Information Agency explicitly credits IEEE standards including 859 as underpinning relevant statistics [5.9] (“Many of the standards in reporting these metrics were initially developed by the IEEE” – IEEE standards 762 and 859).

As well as his general leadership based on research and industry experience, Dent made a number of specific contributions based on the above research [5.7]:

1. Note after 4.2.2.2.2 “Multiple Independent Outages”: clarifies, based on insights from [3.1-3.3] above, the concept of statistical (in)dependence in circumstances where failure rates are elevated (e.g. due to bad weather). This has been a point of confusion in the power system research and industry literature for many years.

2. Note in 6.3 “Weather” and Note 1 in 6.3.1 “Adverse Weather”: clarifies how the classification of different weather states (normal, adverse etc) in the standard should be used in data recording, and how care should be taken using indirectly relevant data (again based on insights from [3.1-3.3]).

3. Note in 7.3 “State Probability Indices” clarifies the important conceptual point that the indices defined are empirical probabilities, i.e. “the proportion of a set of historic trials in which a particular event occurred”. This is closely related to the use of historic wind resource and demand data described in [3.1].

This section of the impact study is verified by the testimonial letter of the Chief Reliability Officer at the North American Electricity Reliability Corporation [5.8]. The letter further describes the international significance of IEEE 859, and the contribution of our research in underpinning the revisions in IEEE-859-2018 described above.

The blending of simple raw materials to manufacture food is an industry that is worth approximately a hundred billion USD per annum, and is dominated by the petfood market, where small percentage savings are significant. The competitive position of the world’s leading supplier of food formulation software, Format Solutions, depends on techniques and software developed by the University of Edinburgh Optimization group. This software is used to identify raw material blends and production strategies [text removed for publication]. With the solution of optimization problems being the overwhelmingly dominant computational cost of Format Solutions’ software, post-2000 research and development by the Edinburgh group has led to its performance improving by more than an order of magnitude (independent of the computing platform).

Manufactured food for both farmed animals and domestic pets, as well as some human food, is produced by simple blending and binding of fundamental raw materials. A particular food mill will produce multiple recipes for different animals, with shared use of the same raw materials. However, the nutritional properties of the resulting food are important, particularly when raising farm animals to provide optimal yields of meat, milk and offspring. Thus, the amounts of raw materials that are combined to produce a particular recipe must satisfy many linear constraints on its nutritional properties, and the total amounts of raw materials used must not exceed their availability. Food manufacturers wish to manufacture their products at minimum price. This extension of the classical single diet problem is an example of the optimal decision-making problem that, mathematically, is referred to as linear programming (LP). The problem size increases with the number of recipes and raw materials, and larger companies wish to consider raw material usage over a number of mills. A secondary, but potentially very profitable enterprise is to consider substituting materials when it is more valuable to trade ownership or purchase options for them on global commodity markets. To obtain the information for all this manufacturing and economic activity requires the solution of large-scale LP problems.

To place the post 2013 impact in context, it is necessary to consider the involvement of Hall, McKinnon and Grothey of the University of Edinburgh Optimization group with Format International (as it was known at the time), that began in 1997. The company was seeking to improve the quality of its implementation of the simplex algorithm for solving the LP problems generated by its animal feed formulation software. It was immediately clear that Hall’s simplex solver, EMSOL, was far superior. For problems with a large matrix size, EMSOL returned the optimal solution up to 15 times faster than Format’s own version, so Format bought a copy of EMSOL for incorporation in their software. In the early 2000’s, Hall’s development of techniques to exploit hyper-sparsity in the type of problems Format needed to solve provided a further major performance enhancement. The result of adopting EMSOL had significant impacts on the company’s growth. It was able to win new business with larger clients [text removed for publication].

The SLP solver developed for Format by McKinnon and Grothey gave the company the unique ability to solve pooling problems, in which production requirements dictate that ingredients are used to produce several pre-mixed combinations, before final blending into products. Such problems are often found in industries such as pet food and dried milk production. The development therefore gave Format a significant commercial advantage over its competitors, who simply were unable to solve these types of problems. The solutions presented by the new solver enabled Format clients to streamline production and supply chains resulting in significant savings to them [text removed for publication]. Format was therefore able to open new markets and win new business, particularly in the pet food industry. This included several of the world’s leading pet food manufacturers [text removed for publication].

In 2015, Cargill Inc., one of the world’s leading animal feed manufacturers and a long-time customer of Format, acquired Format International, renaming it Format Solutions. This purchase was made following a review of the future direction of their formulation software solutions across the world-wide business. The Cargill review identified that Format’s own software developments, coupled with its optimization capabilities and the talent in the Format team were far ahead of Cargill’s own in this area, and to match them would have taken many years and expense. The purchase of Format International, bringing with it the ability to acquire the software, its I.P. and solvers, was assessed as a far better option than developing their own solution. The Format-Edinburgh relationship spanned many years of highly rewarding interaction and resulted in some industry significant developments. Format’s turnover and profitability during the period grew four-fold [text removed for publication].

Format Solutions is now (2020) the world’s leading supplier of food formulation software, so is dependent on high performance optimization software for the reliability and performance of its products. As the modelling capabilities of its software have developed over the years, so has the size of the underlying optimization problems that must be solved. Continued collaboration with the Edinburgh Optimization Group has given Format Solutions the necessary enhancements of its optimization solvers. The most significant advance in this respect has been the performance improvement offered by HiGHS, that has been used by Format since 2017. In the case of LP problems, Format’s customers are now able to “optimize larger feed formulation problems, and at significantly greater speed” [5.2]. The Format product for pooling problems that requires the SLP solver “enables customers to significantly improve on the cost and quality of their feed-mix products and solve new classes for problems that were not tractable before” [5.2]. This has “contributed to Format’s world market-leading position and allowed them to gain new important clients” [5.2]. A significant illustration of this is the fact that Format’s software is used to formulate petfood for the world’s top manufacturer [text removed for publication] who chose Format specifically because of this solution capability [5.2]. HiGHS offers Format Solutions a tenfold performance improvement over EMSOL, and will contribute significantly to maintaining Format’s world market-leading position in feed formulation software into the future.

Beneficiaries and Types of Impact

The results of this research are being used by or have had an impact on companies and institutions in the UK, US and France: [text removed for publication]; the Continuous Mortality Investigation of the Institute and Faculty of Actuaries (CMI); actuarial consultancies advising insurers and pension funds; UK pension funds; insurers and reinsurers; the Office for National Statistics (ONS). CBDK [3.1] has impacted on: professional practice; published mortality tables; pricing of billions of pounds of longevity transactions, with a documented/verifiable pathway to impact in each case.

Pathway to impact

The authors’ original and longer report was completed in December 2013, and was followed by a series of presentations by Cairns and co-authors to key stakeholders to kickstart generation of impact from the research. Presentations included US practitioners (New York), the US Social Security Administration (Baltimore), United Nations Population Directorate (New York), members of the Continuous Mortality Investigation (London), mortality experts at the Office for National Statistics (Titchfield), actuaries’ Life Conference (Birmingham), and the International Mortality and Longevity Symposium (Birmingham).

[text removed for publication]

The impetus for CBDK originated from [text removed for publication] who were concerned about the accuracy of ONS population data. The results in CBDK gave [text removed for publication] the confidence to revise their mortality tables and reduce prices. Since publication of the paper, [text removed for publication] have used the results in all of their UK transactions. [text removed for publication] supporting letter [5.1] states: “ *As a direct result of this study, my team revised [their] mortality tables … [and] have reduced the price charged by [text removed for publication] by nearly 1%*”. She continues to note that [text removed for publication] l have executed transactions worth [text removed for publication] since 2016, resulting in savings to UK pension funds and insurers of approximately [text removed for publication]. “ *Moreover, we believe that the entire market has taken these revised tables into account … [with] estimated savings … nearing £1 billion.*”

Office for National Statistics

The ONS conducted a methodological review in 2016 of official high-age population estimates. One of the key drivers for the review was CBDK [3.1]. (See ONS, 2016, page 4; [5.2].)

Continuous Mortality Investigation (CMI)

The CMI conducts mortality investigations on behalf of the UK actuarial profession and produces mortality tables and forecasts of mortality improvements that are extensively used by UK pension funds and insurers when they value their liabilities.

Methods and results in CBDK have been used extensively in two key work streams (mortality projections and high-age mortality) of the CMI that are clearly documented in the CMI working paper (WP) series.

CMI Working Paper 91 (WP-91) [5.3] (pages 4, 5, 11, 56, 57) and its successors concern the CMI’s approach to projecting mortality improvements. These require use of a table of historical mortality rates. The results of CBDK convinced the CMI that anomalies could have a material impact on mortality projections and so, inspired by CBDK, they developed a simplified method to adjust for anomalies. The CMI’s flagship projection tool itself [5.4] gives users the choice of whether or not to use the adjusted or unadjusted population data, but use of the adjusted data is the default.

A further impact of CBDK on the work of the CMI is evidenced in WP-100 [5.5] (pages 3, 13, 15, 18, 19, 39-49, 73-78 including references to “CBDK” diagnostics). This documents the CMI’s work to improvements in single age mortality rates at very high ages, and makes extensive use of the CBDK diagnostic tools 1, 2A and 2B. [5.5] builds on the conclusions of CBDK that the ONS data have a small discontinuity at age 90, and developed their own methodology to correct this.

The CMI projections model, including its high age methodology are used extensively by life insurance and pensions actuaries in setting best estimate mortality improvement assumptions (e.g. see the USS 2018 Valuation [5.6], page 12). These assumptions impact on the valuation of trillions of pounds of insurance and pension liabilities. A decrease of only 0.1% per annum (quite common in recent years) to the mortality improvement rate in CMI_2018 will take tens of billions off these liabilities.

[text removed for publication]

[text removed for publication] is among the world's largest providers of actuarial services. Their supporting letter [5.7] [text removed for publication discusses the impact CBDK has had internally and how it has impacted on their clients.

• The work of CBDK “initiated more than four years of research in [text removed for publication]”;

• Work with a major client, [text removed for publication], led the client “to update its own mortality tables, …. leading to a change in [their] capital calculation modelling.”

• CBDK more generally has had a strong impact on consulting, including increased awareness of data quality issues, and, through CBDK-inspired adjustments to data, have had a significant impact on the calibration of stochastic mortality models: calibrations that have an impact on regulatory capital requirements for insurers.

The supporting letter [5.7] also emphasizes the impact of CBDK on the Human Mortality Database (HMD) (a much-used source of mortality data by multinational insurers) which revised its exposures methodology in 2018.

The research has had an impact on the administration of justice, leading to a better use of evidence and accompanying judicial and economic benefits. This is split into three main areas:

(i) The procedures developed in [3.1-3.3] are now routinely used for forensic casework internationally.

This is confirmed by various forensic experts from across Europe: “The work of Aitken and Lucy … is the basis of our method for e.g. calculating LRs in glass and is still used routinely” [5.1]; “ His [Aitken’s] research papers and books are used as teaching material, [and] reference material to justify approaches in practical works such as forensic reports” [5.2]; “ Results of Aitken’s research, …are used in my daily forensic practice as a forensic expert in microtraces, i.e. in cases involving the analysis of glass fragments. I use it (… LR model) for the evaluation of the evidential value of results of glass analysis… The method is used in about seven cases per year…, about one hundred cases in total to date” [5.3].

A software package in the statistical programming language R known as ‘comparison’ which follows [3.1] was developed by Lucy and has been downloaded around 25,000 times since 2014, indicating the acceptance of the methods by the applied community.

(ii) Research on cocaine traces on banknotes in [3.5-3.6] has been used to support the accreditation of a UK forensic science laboratory and to support expert evidence delivered in UK court cases.

The Scientific Director, Mass Spec Analytical Ltd (MSA) confirms this [5.4]: “ *The work … is routinely referred to in the supporting material [in] every court statement sent out by [MSA]…. As such, it is frequently subject to cross examination in Court, and the expert witnesses make reference to the peer reviewed papers regularly. It is difficult to quantify, but the experience of the court attending witnesses is that ensuring that our offering is on a sound scientific footing has greatly reduced the ‘attacks’ on our methodology.*” and “ *the work demonstrating that the general circulation background samples do not vary greatly across the country has proved to be enormously useful following adverse criticism of the sampling strategy in the trial of Rashid et al in Sheffield Crown Court in 2015. This work is referred to in talks given to police officers for marketing purposes, and is included in the “In-house method validation” documentation provided to UKAS [the United Kingdom Accreditation Service] at Accreditation Assessment visits (the most recent being February 2020).*”

More broadly, the research in [3.5] has changed the way in which forensic scientists at MSA think about the presentation of their forensic evidence, resulting in improvements in the administration of justice [5.4]. MSA provided evidence for around 200 cases per year in the UK (most of the UK cases featuring this evidence type). Key Forensics purchased the banknotes part of the business in January 2020 and have continued using the research as described above. Further to the above, Aitken has given evidence as an expert witness based on research in [3.6] under oath in two trials: R. v. Hussain and others (Snaresbrook Crown Court) and R. v Parry and others (Liverpool Crown Court).

(iii) The research has influenced the framework in which scientific evidence is presented in court and is widely used to train forensic scientists and lawyers.

The book [3.4] is a well-cited authority on the role of statistics in the evaluation of evidence in forensic science. One of the main methodologies set out in the book is that of [3.1]. The influence of the book on forensic casework is illustrated in [5.1]: “ The Bayesian framework explained in the book is the basis for evidence interpretation and evaluation in the casework of the NFI” and [5.6]: “ The well-known text book… is the fundamental literature in this field”. This is further supported by the inclusion of the methodology set out in the book (e.g. paragraph 2.4 on p6) in the European Network of Forensic Science Institutes (ENSFI) Guidelines [5.7] which sets out a framework for reporting statistical evidence in forensic science. The book and methodology contained in it is used consistently by laboratories in ENFSI to train forensic scientists and lawyers [5.1, 5.2, 5.3, 5.6].

The practitioner guides [3.7] are on the communication and interpretation of statistical evidence in the administration of criminal justice. These set out the likelihood ratio methodology in [3.1-3.4] for legal practitioners, e.g. Sections 2.17-2.19 in the first guide and Section 2.21 in the fourth guide. The influence of [3.1], [3.4] and [3.7] on forensic casework is corroborated in [5.3]: “ *The general ideas on application of LR approach in forensic sciences as expressed in [3.1] and [3.4] (and disseminated in the practitioner guides of the Royal Statistical Society and the ENFSI guidelines) are used routinely by me in the evaluation of evidential value of results of blood pattern analysis.*” The practitioner guides have been used in court cases (e.g. in the Kentucky Supreme Court - Ivey v. Commonwealth in 2014), indicating that they add value to the presentation of expert evidence in court (also see [5.4]). The Royal Statistical Society and the Inns of Court College of Advocacy published an introductory guide to statistics for barristers and advocates [5.5] describing the general approach to evidence evaluation set out in [3.4] and [3.7] (e.g. Section 1.7 and p66-68). As the Chair, Aitken led the contribution from the Statistics and Law Section of the Society and Wilson contributed as a committee member. This guide was recognised as a useful resource for training lawyers and judges in a House of Lords Science and Technology Committee inquiry in 2019 on “Forensic science and the criminal justice system: a blueprint for change” [5.8, paragraph 132].

As evidence of the overall impact of the research in the forensic and legal communities, Aitken was awarded the Howard Medal of the Royal Statistical Society in 2018 [5.9] for work that is an “ outstanding example of how a statistician can integrate with those in a substantive area”. Reasons for the award included the research on cocaine on banknotes, [3.5, 3.6]. Further to this, [5.2] states “ *I can testify that Professor Aitken’s research in general has deeply influenced the way a scientist approaches the evaluation of evidence and the way he/she presents evidence in a written report or during a testimony in front of a Court of Justice.*”

Impact on Red Squirrel Conservation Policy in Scotland

The mathematical research in (3.2,3.3) led to commissioned reports [5.1,5.2] undertaken by Prof White for Scottish Natural Heritage and the Forestry Commission Scotland which delivered management advice for the protection of red squirrel populations exposed to squirrelpox virus. Prior to this work, conservation policy had focussed on control in regions where grey squirrels tested positive for squirrelpox. This had reduced grey numbers in local areas but had not prevented disease spread or increased red squirrel numbers. The work showed that it is extremely challenging to prevent the spread of the squirrelpox into areas where grey squirrels are already well-established and, importantly, that squirrelpox will not persist in red squirrel (only) populations. Saving Scotland’s Red Squirrels (SSRS), who are responsible for red squirrel conservation in Scotland, state [5.3] that Prof White’s work “ had impact both on policy through inclusion in the Scottish Strategy for Red Squirrel Conservation (2015) [5.4] and practice as SSRS recognised the difficulty in containing squirrelpox as the trigger for a radical change of strategy that protects remaining red squirrel populations in Priority Areas for Red Squirrel Conservation (PARCs) in southern Scotland”. Furthermore, SSRS indicate that our “ mathematical modelling work adds a scientific underpinning and rationale for the use of grey squirrel control to protect red squirrel population” which is coordinated by local communities in and around PARCs and has led to an increase in red squirrel distribution and abundance [5.3]. The modelling work continues to direct red squirrel conservation policy in Scotland through commissioned research undertaken by Prof White for Saving Scotland’s Red Squirrels in 2017 [5.5] to determine the level and location of grey squirrel control to prevent further spatial expansion of the grey squirrel distribution beyond the Highland protection line in Scotland and for the Forestry Commission Scotland in 2018 [5.6] to examine the level of grey control required to protect red squirrels in Southern Scotland. The modelling research (3.2,3.3),[5.1,5.2,5.8] was promoted by stakeholders and led to extensive coverage on BBC radio, BBC online and in national newspapers (The Times, The Herald, The Scotsman) [5.7].

Impact on Forest Management Practice in Northern England

Prof White’s modelling research (3.2,3.3) was extended to answer specific red squirrel conservation issues faced by forest management practitioners. Working with Stakeholders in the Forestry Commission (FC), future felling and restocking scenarios for Kidland and Uswayford forest (2962 ha of forest in north-east England) were used to drive the mathematical model and led to predictions of red squirrel population viability (3.4). Quoting from the Planning and Environment Manager for the North England Forest District of the Forestry Commission [5.8] “ *The findings in the report (3.4) highlighted how the (original) proposed forest design plans could lead to a risk of red squirrel extinction (particularly in Uswayford). As a result alternate felling and restocking regimes were developed by FC and tested in the model. The model results had a direct impact on the final forest design plans for the Uswayford and Kidland forest (for 2017 - 2041) which are outlined in the Cheviot Forest Plan (FC 2015) and which represent the optimum in terms of silviculture and red squirrel viability. The combined FC and Maxwell Institute modelling study therefore played a key role in the red squirrel conservation strategy in these forests.*”

Impact on Grey Squirrel Control Strategy on Anglesey

Prof White further developed the model to include a realistic representation of grey squirrel control effort by fitting the model to field observations for the removal of grey squirrels and reintroduction of red squirrels on the Isle of Anglesey, Wales (3.5). The model showed that the most effective strategy to remove grey squirrels would focus the initial trapping effort on high density regions and then spread the trapping effort to all regions where greys are present. The model also showed that the best strategy to prevent grey squirrel re-invasion of Anglesey would focus trapping efforts on the mainland side of the Britannia bridge and use high levels of targeted trapping on Anglesey when grey squirrel sightings are reported. The research findings were adopted by the Red Squirrel Survival Trust (RSST) in 2017 as the strategy to protect red squirrels on Anglesey who state [5.9] that “ the modelling work continues to be used to determine the optimal strategy to prevent the re-invasion of Anglesey by grey squirrels and has therefore had an impact on the RSST conservation practice to protect red squirrels on Anglesey – which are still thriving. The modelling work also played a key role in providing evidence-based research to convince the public of the ongoing need to control grey squirrels”. The applicability of our mathematical research is emphasised by Dr Craig Shuttleworth of the Red Squirrel Survival Trust who states [5.9] that “ the collaboration between ecologists, conservation practitioners and mathematical modelling provided a template for how interdisciplinary research can provide solutions to real world problems for the protection of endangered species and therefore the maintenance of biodiversity in the UK.”

Impact on Forest Management Practice in Scotland

Forest and Land Scotland (formally the Forestry Commission Scotland) commissioned Prof White to adapt the model (3.4,3.5) to determine the viability of 19 designated red squirrel strongholds in Scotland [5.10,5.11]. The model compared red squirrel viability under stronghold forest management (SHM) plans (where tree species composition is altered to discourage grey squirrels) and UK forest standard (UKFS) management plans. The research showed that some designated strongholds were unsuitable for red squirrel protection, that other regions may provide natural strongholds (where reds are protected under UKFS plans) and that north of the Highland red squirrel protection line there was no benefit to red squirrel abundance under SHM compared to UKFS. Forestry and Land Scotland state [5.12] that “ reconciling the SHM policy with other management objectives, fluctuating timber markets and wind-blow events, requires significant additional management input and can affect income from timber sales. Furthermore, the single-species focus of the SHM policy has consequences for other environmental work that aims to increase ecosystem diversity. The results of the research by Professor White are therefore important for Forestry and Land Scotland and will have a direct impact on management across large parts of the national forest estate. The research findings will allow a future focus on natural strongholds under the UKFS approach and this will afford more flexibility to conserve red squirrel populations whilst simultaneously delivering other forest management objectives.”

Mycoplasma bovid is a bacterial disease that affects dairy and beef cattle, causing severe illness with major impact on production. It is a disease of major significance to the New Zealand farming industry, which represents the second largest export market for the country. The New Zealand epidemic was first detected in 2017, following which a major eradication programme was announced in May 2018 [5.4]. The total cost of the programme was estimated to be NZD886,000,000 (05-2018) [around GBP470,000,000], including NZD16,000,000 in lost production and NZD870,000,000 of response costs needed to fight the cattle disease over a 10-year period [5.4]. According to [5.4], had the epidemic been allowed to proceed unchecked the cost would have been approximately NZD1,300,000,000 (05-2018) [around GBP689,000,000] over 10 years, with ongoing productivity losses across the New Zealand farming sector.

As part of a wider programme of epidemiological work that included support (approximately AUD70,000) from New Zealand’s Ministry of Primary Industries (MPI), Firestone and colleagues developed a computer package BORIS (Bayesian Outbreak Reconstruction Inference and Simulation) [5.2, 5.3] which incorporated the methods of [3.2] with the additional capacity to accommodate contact data and farm-level covariates. According to the team who developed BORIS, “The main benefits provided by your algorithm derive from its capacity to fit epidemic dynamics and pathogen phylodynamics within an integrated framework, to represent multiple introductions, and to cope with partial sampling scenarios – essentially allowing the state of unobserved nodes in a network to be inferred” [5.5].

The BORIS package, code for which is now freely available from GitHub [5.3], has subsequently been applied within the M bovis eradication programme since July 2018 [5.2, 5.6] as one of a suite of analytic tools, to analyse epidemiological and genomic data in order to inform the government of the timing of likely introduction of M bovis into New Zealand, whether there were multiple introductions, and the extent to which the case network could be determined. Specifically, BORIS is used to infer timing of infections, infectious periods for farms, and to identify likely transmission routes (who-infected-whom) for the ongoing epidemic. Such insights serve to identify possible infector-infectee links, and corresponding time periods, that should be scrutinised in order to understand the potential risk factors, or failures in biosecurity, that may lead to transmission of the disease. Unlike competing genomic methods, BORIS has the capacity to impute chains of infection that include farms for which genetic data on the pathogen may not be available. Results of these analyses are key to providing confidence to MPI of their understanding of the outbreak [5.5, 5.6]. Since the eradication programme was initiated, the epidemic has been successfully controlled to the extent where, as of July 2020 (resp. November 2020), there were only 4 (resp. 6) properties (out of more than 20,000 dairy and beef farms) [5.5] in New Zealand where the disease was known to be active, having been successfully cleared from 246 premises by that time. BORIS continues to be used as a decision-support tool for guiding surveillance teams within the programme as it moves towards eradication of the disease [5.6].

More widely, the techniques of [3.2] have been promoted through a training programme delivered by Firestone on the use of the BORIS package to Australian government veterinarians at the Department of Agriculture, Water and the Environment (Canberra, November 2019). To date 20 scientists have received training in its use [5.2]. BORIS has also helped improve the understanding of spread of foot-and-mouth disease in Japan [5.7], for example demonstrating the high transmissibility of the disease from farms holding predominantly pigs.

STACK is a contemporary assessment software for mathematics. It is installed on the learning management system of over 1300 registered websites with groups of up to 1500 students [5.1]. Extrapolating, we estimate 1.2 million students worldwide are benefitting from this methodology. Having been translated into 20 languages, STACK is used in over 30 universities in Germany, every university in Finland and many other countries including Japan, Israel, Kenya, and the USA [5.2, 5.3, 5.4]. The abacus consortium, established to share STACK and other mathematics assessments, currently has 38 partners [5.5], further extending the reach of STACK.

In comparison with other online assessment software, STACK offers unique additional functionality that changes the way that both learners and teachers engage with learning and assessments. Students enter mathematical expressions and STACK will assess the properties of these expressions; teachers author their own questions; STACK can generate questions with random variables, to reduce copying of answers; STACK allows teachers to give partial marks and tailored feedback depending on the different mathematical properties of the students’ answers. STACK supports multipart questions, enabling teachers to write step-by-step questions; STACK automates the “question testing” process enabling robust questions with long-term support. This means that there is almost double the assessment per student using half the staff time, increasing the efficiency of staff while improving student learning [5.3]. The research described in [3.4] has led directly to uptake for physics textbooks by the commercial textbook publisher, Physics Curriculum & Instruction, and more generally wider use in STEM subjects. The publishers write “ Instructors using our system have stated that students have greater success with solving physics problems, and gain deeper understanding into problem-solving strategies, when compared to traditional homework environments of working alone without computer aided assistance” [5.6].

Combining online assessment with human marking saves institutions thousands of hours of work each year: the University of Edinburgh estimates the School of Mathematics saves over 8,300 hours of work annually, through 7,500 accounts, equating to over GBP200,000 of savings year on year. This is replicated for STACK users across the world, for instance “STACK is used for both summative and formative assessment. Over 1.2 million STACK questions were answered by Open University students in the 2019/20 academic year. This saves a significant amount of staff time each year. Marking this volume of questions, even if only taking 5 seconds per question would take approximately 1 year of staff time each year” [5.7].

As a result, STACK enables institutions to redeploy human resource and maintain quality despite growing student numbers. The University of Edinburgh consolidated online assessment in most year 1 and 2 mathematics and general science courses with the support from a dedicated learning technologist [5.8]. The University of Durham established a similar post in 2020 [5.9], and University College London appointed two people to 3-year posts to create STACK questions. This has changed the way courses in mathematics and science are delivered across these institutions. The Head of Department, Mathematics, UCL writes: “The impact of this change of assessment on our year 1 students, most of whom will end up studying the whole year remotely, is immense. They get much more detailed feedback than they would have had in a normal year; and this is vital when we're exactly at the time when the support structures (such as informal contact with tutors and peers) that would have formed the academic safety net have been damaged by the pandemic” [5.10].

The novel work described in [3.3] extends assessment capabilities to entire mathematical arguments in a well-defined class by assessment of students’ working itself. Feedback is provided to increase students’ understanding of mathematics, improve students’ experience, and raise their competence. The Open University comments “ Whilst it is difficult to isolate the effect of STACK ….. modules within Mathematics and Statistics enjoy some of the highest student satisfaction ratings within the University. The use of STACK questions within modules is often praised by students in unsolicited comments within end-of-module feedback” [5.7].

In 2020, to mitigate the effects of COVID-19 on teaching, the Maxwell Institute ran 12 practical workshops on question authoring, with over 350 academic staff attendees from across the world, and presented 15 online talks about STACK, the underlying research, and how to use this technology in teaching [5.11]. These “ seminars and conference talks have guided and challenged colleagues’ thoughts on the nature of assessment and how to assess different mathematical topics, particularly for distance-learning students. The most recent example of this was Chris Sangwin’s online workshop on the assessment of mathematical proof, given in September 2020” [5.7]. As a further response to COVID-19, STACK was used in December 2020 to replace university examinations, in Edinburgh reducing the marking time in one module alone from 35 person days to 22 person days with a combination of automatically marked STACK assessments and human marking of more complex proofs.

In addition, the research underpinning STACK has influenced the design of other successful online assessment systems, particularly the open source NUMBAS system and “Learning Algebra” from Haese Mathematics, [5.12].

Standard Life Aberdeen PLC is a multinational investment company. It operates in asset management through its subsidiary ASI with equities, multi-asset, fixed income, real estate, and private market funds. Based on the outcomes of this research, a fund was launched in November 2019 by Aberdeen Standard Investments (ASI) under the name 'Standard Life Investments Global SICAV – Macro Systematic Dimensions Fund (Bloomberg ticker: SLMSDDU LX Equity)'. The fund was developed as a result of the joint research effort of ASI and the Maxwell Institute (MI) team, and was co-funded by ASI. The Macro Systematic Dimensions Fund was launched in July 2019, and had USD33,608,897 under management as at 31 December 2019 [5.1].

The holdings of the Fund are not selected with reference to a benchmark index, but the performance of the Fund is compared with the Secured Overnight Financing Rate ("SOFR") [5.2].

The investment team applies a systematic approach to portfolio construction. Their primary focus is to use evidence-based, data-driven quantitative approaches to identify investment ideas within four categories (i.e. equity and credit, interest and inflation rates, relative value and others, e.g. the volatility of investment markets). Such portfolios receive substantial investment from institutional players, including large pension funds, as a way for them to achieve a healthy growth in their balance sheets. Their objective to invest in well-diversified (and thus resilient) portfolios, which can cope in times of financial turbulence, is of utmost priority for them. According to the Head of Macro Systematic Strategies Research at ASI “ The Multi-Asset Solutions team within Aberdeen Standard Investments considers this methodology to be a unique selling point, both as a diagnostic tool as well as a differentiated way to think about portfolio construction and diversification. The feedback from our clients has been very positive” [5.3]. The importance of the aforementioned methodology as described above highlights the significance of this research for the company and its clients.

Moreover, and according to ASI, the research has helped the company to develop an investment philosophy which underpins other products (funds/segregated mandates/sub-exposures in existing funds) and, importantly, to differentiate from the competition. ASI confirmed “ The experience in early 2020 when financial markets experienced some of the most extreme and unusual behaviour since the great depression and the financial crisis of 2008, has re-enforced the interest in risk management, portfolio diversification and risk mitigation. For those funds that deployed the methodology developed during the collaborative research agreement, we have retrospectively compared the portfolio construction methodology with other common approaches, such as Risk Parity or Minimum Variance and found the Dimensionality approach to be superior” [5.3].

The joint research article [3.1] was co-presented (by ASI and Sabanis) at one of the most prestigious conferences for industry and academia in Financial Mathematics, namely the SIAM Conference on Financial Mathematics & Engineering (FM19) which took place in Toronto [5.4]. The joint research efforts were highlighted by ASI at the CBOE (Chicago Board Options Exchange) Risk Management Conference which took place in Munich [5.5].

The fund was awarded the Fund Launch of The Year at the 5th annual EQDerivatives Awards (the event was postponed due to the current global health emergency). According to the award committee, “ Not only is The Macro Systematic Dimensions Fund unique compared to peers when you take into consideration the innovative portfolio construction methodology and strategy, it is also extremely relevant now as global institutional investors we have spoken to are increasingly seeking systematic strategies that integrate macro views given the backdrop of an evolving market landscape. What was also a key differentiating factor has been your own contribution to expanding education among institutional investors across the globe, not only in a practical context, but in academia also. What has also been very positive is the feedback we have received on the 'Dimensionality' methodology - it truly is unique and how it taps into four unconnected return streams” [5.6]. Thus, the significance of this research is recognised by independent experts both in academia and also in industry.