Impact case study database

IBEX Innovations Limited was created in 2010 to develop and commercialise an innovative X-ray detector technology. It has been collaborating with Durham since 2015 on Bayesian approaches to improving X-ray imaging [ E3]. The collaboration has greatly enhanced the commercial capability of the patented technology and ‘ allowed a small North East company to compete on a world stage’ [ E3, E4, E5]. Specifically, Durham research underpins their patented multi absorption plate (MAP) technology and their Trueview® product which uses the Scatter Reassignment Method (SRM) [ E9]. IBEX has raised almost GBP6,000,000 in funds for product development and to secure international customers [ E2], securing a total of 14 Jobs in the North East of England (gross value added for full time equivalent jobs to the North East is GBP662,144) [ E3, E4].

The MAP technology developed by IBEX in collaboration with Durham University enabled inferences, not possible with standard X-ray methods, about the material composition of an object being imaged with the MAP equipped detector [ E3]. A core technology at IBEX, it is at the base of a number of their products as outlined below.

-  The first product developed was for security, which enabled portable X\-ray examination equipment capable of extracting accurate information about the materials being imaged. Portable X\-ray examination systems are key for safety when objects being examined cannot be moved. For example, one may want to X\-ray image a suspicious bag in an airport without moving it. IBEX signed a Joint Development Agreement \(JDA\) with 3DX\-ray, \(the main trading subsidiary of Image Scan Holdings plc\), to incorporate IBEX technology into 3DX\-ray’s next product on an Innovate UK assisted project. \[ **E1b, E3**\]. JDAs give a route to market for IBEX’s technology and increase their revenues. 

-  IBEX are also combining the MAP technology with their food inspection products. Recently, chocolate bar manufacturers have had to recall bars due to possible plastic contamination, therefore, an accurate plastic contamination system could save chocolate manufacturers the cost of recalling products and the negative media publicity associated with it. IBEX signed a JDA with Mettler\-Toledo, a manufacturer of precision instruments, to develop a methodology for i\) plastic contamination detection and ii\) bone in chicken breast detection, on an Innovate UK assisted project \(IBEX GBP109,163, Mettler\-Toledo GBP24,775\) \[ **E6**\]. IBEX expect this product to generate revenues of GBP1,000,000 per annum and the JDA has already secured 3 jobs within the company \[ **E4**\]. 

In addition to the research related to the MAP, our research supported the development of the SRM which is core IP at IBEX and forms the basis of their software patented product ‘Trueview®’ [ E3]. SRM offers benefits of i) “really good image detail”, ii) with “no additional equipment to reduce scatter, reducing the number of radiographs taken” (Nicola Hind, Consultant Radiographer, Royal Victoria Infirmary, Newcastle) [ E1] and iii) improved diagnostic measures in medical X-ray images. IBEX obtained a software patent to protect the novel approaches developed in collaboration with Durham University (WO2016051212A1, GB2563115A) [ E5]. The proof-of-concept helped IBEX receive numerous grants and venture capital funding to market their technology [ E1-E4, E6, E7]. Current impact around Trueview® includes:

1. The Trueview® software has undergone clinical trial at The James Cook University Hospital in Middlesbrough [ E8]. It measures a patient’s bone health from a standard X-ray and initial trial results indicate that it provides an accurate early warning of osteoporosis, and therefore a patient’s risk of potentially fatal fragility fractures. The study which was led by the orthopaedic research team, collaborating with colleagues in radiology and rheumatology, “compared images from the new software to results from 130 patients who had attended appointments for a DEXA bone scan at James Cook with positive results” [ E8]. IBEX Trueview® offers early detection, early intervention and better outcomes. Poor bone health represents a substantial global healthcare challenge. In the UK, the NHS spends around £1bn per annum on the diagnosis and treatment of hip fractures, but sadly it remains the largest single cause of accident-related death. IBEX Trueview® provides opportunistic early screening for bone health of all patients referred for a Digital Radiography scan, enabling timely detection of early stage osteoporosis before debilitating hip fractures occur.

2. IBEX have signed an agreement with CurveBeam to integrate the Trueview® product into their products [ E2, E3]. Leading cone-beam computed tomography systems (CBCT) manufacturer, CurveBeam LLC have chosen Trueview® to deliver unrivalled image quality for their weight-bearing CT products, including the new flagship HiRise system. CurveBeam President, Arun Singh, says that it produces “the best bilateral foot renders I have ever seen” [ E10a]. CurveBeam specialises in weight bearing CT imaging [E10b]. The Curvebeam HiRise system has attained FDA approval with Trueview® incorporated so will be on sale shortly. Two HiRise systems were installed for investigational studies prior to FDA clearance at University of Iowa Carver School of Medicine’s Department of Orthopedics and Rehabilitation in Iowa City, IA and at Tennessee Orthopaedic Clinics (TOC) in Knoxville, TN. Researchers at the University of Iowa have used HiRise for multiple research studies, including examining hip dysplasia infunctional position and evaluating wrist injuries in gymnasts. “The HiRise promises to revolutionize our biomechanical understanding of the entire lower extremity the same way previous generations of CurveBeam’s weight bearing CT systems enabled better investigation into the foot, ankle and knee,” said Dr. Cesar de Cesar Netto, MD, PhD, Assistant Professor of Orthopedics and Rehabilitation at University of Iowa [ E10c].

3. IBEX have signed a licensing agreement to integrate Trueview® into Planmed’s Clarity 2D, Clarity 3D and Clarity S mammography systems [ E2, E7]. The company is part of Finnish Planmeca Group, a well-known company in the medical and dental field which provides tools for healthcare professionals in over 80 countries worldwide. The target release for this product has been delayed by Covid-19. Trueview® enables better image quality, improved workflow and better patient outcomes. It has been demonstrated to give better contrast in 2D and 3D mammograms – with up to 50% lower patient dose and up to 25% less breast compression [ E7]. Of the collaboration Mr Jan Moed, Managing director of Planmed says “We are constantly listening to our customers and developing our product lines accordingly. We feel that this collaboration will bring added value, not only for the users but ultimately for the patients which is what really matters,” [ E7].

4. A grant has been obtained by IBEX to see how Trueview® can improve AI algorithms. This resulted in one new hire. The grant number is 44808 and the award was for GBP288,136 and a continuity grant 74463 was obtained for GBP100,000. [ E6].

In summary, Durham research underpins the technology that is enabling IBEX to grow as a company, receive funding and take products to market [ E3, E4]. Ian Wilson, who leads Mercia’s team in the North East of England, which provided funding via the North East Venture Fund to IBEX, says: “IBEX is the only one in the market that can achieve this quality of images. The company continues to make progress in both developing new products and new commercial relationships. Mercia is delighted to support IBEX once again and help roll out this new system which will be an important step forward in breast cancer detection.’’ [ E2].

**

Atom is a digital challenger bank based in the City of Durham, which started trading in 2016, offering an innovative and efficient mobile-only app-based personal and business banking experience. One of the reasons Atom chose Durham as its base was to take advantage of the research output of the University. *‘It quickly became apparent that the department’s specialisation in applying Bayesian techniques to financial and other systems [e.g. **[R3, R4]*] offered the opportunity to create a digital twin of some of the bank’s key allocation decisions.’ [E1]. The collaboration between Atom bank and Durham University resulted in the creation of the Atom Bank Digital Twin using methods [ R1-R5] developed within the Mathematical Sciences department. This has allowed Atom Bank to re-invent financial planning in financial services enabling it to 1. Optimise business activity so that it generates the maximum return from each set of investment decisions and 2. Understand in full the consequential impacts of any business decision on each business area. This enables the bank to link its resource planning accurately to product and pricing decisions [E2]. Atom says that this was ‘critical to building a competitive advantage that can be shared between investors and customers’ [E1]. The total estimated savings, detailed below, are currently [REDACTED].

The Atom Bank Digital Twin is a novel decision support suite, which is being used to support major business decisions such as financial planning, resourcing, product pricing and funding. Atom Bank now has the unique opportunity to mathematically model the organisation and manage the business according to the inputs and outputs of that model. The direct impact of the collaboration with Durham University (which originally centred around a Knowledge Transfer Partnership) on Atom's financial performance can be summarised in three categories as taken from E1:

1. Improved management of short-term liquidity requirements [E1]

Banks are required by regulation to hold sufficient liquid assets to cover their forthcoming cash flows. Many aspects of inflows and outflows, such as the quantity of loans being provided to customers, and the savings deposits received by customers, are uncertain. The models created with Durham research [R1-5] have greatly increased the accuracy of Atom's ability to forecast these cash flows and so reduce the overall liquid assets the bank is obliged to hold (which are an overall cost to the business). The main items are:

1. Improved modelling of loan completions allowed the liquidity held against future loan completions to be reduced by approximately [REDACTED] at all times, leading to around [REDACTED] reduction on average in liquid assets held, with an annualised cost reduction of around [REDACTED]. This benefit was first fully realised in 2018 and a similar annual benefit remains today.

2. Improved ability to predict the behaviour of savings customers has allowed Atom to reduce the volume of liquid assets needed to ensure that deposit holders can access their funds upon maturity. It is estimated that around [REDACTED] lower liquid assets were needed in 2018 and 2019, leading to a reduction in cost of liquidity of around [REDACTED] in each year. In 2020, a different business strategy meant that the modelling reduced the risk of accidental breaches of regulatory limits, which cannot easily be measured in terms of financial value, but is of critical reputational importance.

3. Advanced uncertainty analysis (R) on the above models allowed the business to reduce buffers (designed to minimise the risk of regulatory breaches) on liquidity, leading to another [REDACTED] of reduced liquidity requirements in 2019. Again, in 2020, this methodology has led to risk reduction rather than financial improvements.

Banks, particularly fast-growing banks such as Atom, are often constrained by available capital. They are obliged to hold certain levels of capital for each loan sold. Therefore, maximising the efficiency of this limited resource is key to profitability. Working with Durham University has principally provided benefits in the following areas:

1. Quality assurance work carried out on Atom's internal lending models is estimated to have led to at least a [REDACTED] improvement on margin. This equates to around [REDACTED] per year on loans originated in 2019.

2. Improved long-term financial planning from the summer of 2017 onwards will have led to a more efficient business plan. This benefit is significantly harder to estimate than any of the others, but financial planning experts quantified this as at least [REDACTED] improvement on the net interest margin of the entire book, which equates to around [REDACTED] per year based on Atom's current book.

Another use of a bank's capital is to be held against risks to implementing the business plan successfully. Collaborating with Durham University meant a number of operational risks were reduced. The highlights were:

1. Quality assurance on regulatory mortgage calculations.

2. Reduction in model risk, i.e. the risk that Atom's internal models give erroneous or misleading forecasts.

[REDACTED]

Embedding the use of the Atom Bank Digital Twin has had a substantial impact on processes and policy within Atom Bank. Atom’s data and analytics team has expanded from [REDACTED] employed in the North East of England to deliver this project and its extensions [E2] (gross value added based on North East 2017 figure is **[REDACTED]**) and a total of 30 staff have been trained in using the model output as well as in the R language to allow model development and maintenance to continue [E2]. By embedding expertise in the open-source R programming language, the business has been able to avoid costly licensing for commercial data analysis software. Based on quotes provided to Atom, the impact of this is a saving of around [REDACTED] per year [E2]. Atom says that the collaboration ‘ has also shifted expectations and perceptions amongst the senior team at Atom, many of whom had not had or seen the benefit of deep engagement with university researchers in their previous roles’ [E1]. Atom Bank have continued to win industry recognition and have recently been included in the Tech Nation Future Fifty business community [E5]. They continue to increase the number of products offered and [REDACTED] [E6].

Impact on customers of Atom Bank

Atom bank is a retail business and so every efficiency improvement also has a benefit to customers. ‘ *Reducing the friction between teams, increasing the allocative efficiency of our capital, and responding to the constraints on optimisation that are a necessary part of being a regulated bank all mean that customers get better prices for their savings and lower costs for their lending than we could otherwise afford to offer. Atom continues to be recognised for its excellence in the savings and lending markets’ [E1] (as recognised by various media organisations **[E3]**) *‘and behind all of these sits an engine that is refined and tuned by the work done with Dr Caiado and colleagues.*’ [E1]. Atom is Trustpilot’s most trusted UK bank and consistently achieves Net Promoter Scores in excess of +75 [ E4, 2020 annual report, page 15].

In summary Durham University have helped to develop an end-to-end banking model that provides a live and interactive overview of the business with a strategic understanding of the relationships between the bank’s key components including customers and products. This is unique within the banking sector and the change in practice has given Atom Bank the opportunity to update both the simulation and the optimisation in real time as the Bank evolves and grows, increasing sustainability and providing rigorous calculations supporting the mortgage lending process that minimise the risk of operational damage or regulatory breaches from calculation error. All these benefits are passed on to the customers as savings.

.

Roxar AS, which owns ENABLE, is an international provider of products and associated services for reservoir management and production optimisation in the upstream oil and gas industry. Working with the Durham Statistics Group has helped Roxar to develop a more successful product, leading to substantial commercial success (see table below). It has also helped Roxar secure long-term technical jobs in its development team and delivered significant impact for its customers. The company is committed to both its support for ENABLE as its flagship product and its continuing work with Durham Statistics Group.

This is how Roxar currently describe the role of ENABLE [E4, E5]:

“ENABLE provides mathematical support to reservoir engineers in their use of reservoir simulation software. This support allows engineers to complete tasks like history matching much more quickly than using the simulator on its own and also provides a more rigorous approach to predicting future reservoir performance or optimising field development.”

Roxar is headquartered in Stavanger, Norway and operates in 19 countries with around 900 employees. Roxar offers software for reservoir interpretation, modelling and simulation, as well as instrumentation for well planning, monitoring and metering. Roxar was acquired by Emerson Electric Company in April 2009 and is now part of the Emerson Process Management Group.

The ENABLE product has been very successful. The current list of ENABLE clients includes [E1]: [REDACTED].

This has led to substantial and sustained financial impact. Roxar/Emerson report [E1] the following total Tempest ENABLE annual sales figures in USD for 2014 to 2019. Results are reported to Emerson’s financial year which runs from 1st October to 30th September.

The ENABLE product has also maintained several long term Roxar employment positions [E1]:

*“During the period 2014 to date the core Tempest ENABLE development team in the UK is 6 highly technical people (5 PhDs). Additional sales, services and support jobs are supported by ENABLE’s success.”*

Roxar remains committed to the continuation of their support for ENABLE, as evidenced by them integrating it within their flagship TEMPEST simulator package now called TEMPEST-ENABLE, the development of which was part funded by STATOIL [E3, E6, E8].

The implementation within ENABLE of more advanced algorithms resulting from recent research work, by the Durham statistics group, from 2012 onward has led to substantial improvements in the code performance, and kept the ENABLE software highly efficient and competitive in an aggressive market since Oct 2014. The Roxar team have written that [E1]:

“The runtime performance of Tempest ENABLE is critical to the commercial viability of the product. During the period 2012 – 2016 many improvements were made to the underlying algorithms in Tempest ENABLE which brought an order of magnitude improvement in performance. These improvements included more advanced Bayes linear emulator construction strategies, improved experimental and sequential designs, robust correlation length estimation and robust active inputs selection. New Bayes linear emulator diagnostics were introduced which demonstrated, in a visual manner, the improvements in the algorithms. These improvements were based on Bayes linear emulation and history matching methodologies developed in the following papers by Durham [R1-R4] . These advances greatly improved the quality of fits of the emulator resulting in substantial improvements in history matching.”

Roxar are committed to continuing their relationship with Durham University Statistics Group, funding three consecutive consultancy contracts [E9]: June 2012 – May 2015 (GBP135,000), June 2015 – May 2018 (GBP150,000), June 2018 – May 2021 (GBP157,200), to fund Dr Vernon to help implement improvements to ENABLE. They have also funded an iCase studentship (Oct 2018 – Sept 2022, GBP55,000 in addition to full EPSRC contribution) to develop further impact related statistical methodology [E1].

ENABLE facilitates a detailed analysis of oil reservoirs including the quantification of uncertainties vital for forecasting, decision making and hence managing oil assets [ E4-E8], and therefore has substantial impact on the oil companies who use it. Examples are [E1]:

“Tempest ENABLE is also a key piece of Big Loop and was instrumental in avoiding a potential loss of $5M in renewal fees at a large international oil company. Big Loop is gaining traction in the market and is being successfully adopted at major national and international oil companies.” (Big Loop combines geological and reservoir models in the inferential framework, but again uses Tempest-Enable at its core).

Another example is given in [E2], a 2017 press release of a successful long-term joint project by Statoil and Emerson, again using Tempest-Enable at its core. ‘‘The program focused on how E&P companies can improve history matching, uncertainty management and quantification across the entire reservoir characterization workflow through Roxar Tempest ENABLE, Emerson’s industry-leading history matching and uncertainty estimation software solution.’’

.

The research at Durham University had a key role in the Met Office UK Climate Projections 2009 (UKCP09) and UK Climate Projections 2018 (UKCP18). These projections had a fundamental impact upon Government, public and private planning for adaptation to climate change. These impacts are as follows.

The UKCP18, an updated version of the UKCP09, is the Met Office’s climate analysis tool. It provides projections for UK and global climate change throughout the 21st century using the latest climate science and analysis. It is designed to assist decision-makers by enabling them to assess climate change risks and is used by both public and private users to plan for a wide spectrum of environmental impacts. UKCP09 provided authoritative climate forecasts which were widely used in climate adaptation planning, within a statutory legal framework that we will describe below, until they were replaced by the enhanced forecasts arising from UKCP18. Thus, they continued to have impact throughout the current REF impact period until the updated forecasts took over.

The science and methodology used to construct UKCP09 are described in the Met Office report [ E1] and in [ E2] for UKCP18. Each report emphasises the importance of the careful treatment of uncertainty in the climate projections. Here is an indicative quotation from [ E1]. "Uncertainty in climate change projections is a major problem for those planning to adapt to a changing climate. Adapting to a smaller change than that which actually occurs (or one of the wrong sign) could result in costly impacts and endanger lives, yet adapting to too large a change (or, again, one of the wrong sign), could waste money..... The 2008 projections are the first from UKCIP to be designed to treat uncertainties explicitly ... This means that probabilities are attached to different climate change outcomes, giving more information to planners and decision makers." [ E1, p19]

The methods developed at Durham University play an important role in the uncertainty analysis throughout each report and Durham University research is referred to explicitly in the UKCP09 and UKCP18 documents. For example, for UKCP18 "These results support the chosen approach, allowing Strand 1 to provide an updated product that combines evidence from HadCM3-based PPEs and CMIP5 models in a consistent manner, retaining the Bayesian statistical framework used for UKCP09 (Goldstein and Rougier, 2004) [ R2]." [ E2, p11] and "Prior pdfs ... are then produced using the Bayesian method of Sexton et al (2012) , based on the general framework of Goldstein and Rougier (2004) [ R2]." [ E2, p17]. Our contribution is amplified in Sexton et al (2012) [ E3], which explains in detail the uncertainty methodology used by UKCP09 and later by UKCP18. For example, in [ E3] section 3 “Here we describe the general steps in Goldstein and Rougier (2004) [ R2] necessary to determine a probability distribution of some aspects of climate change that we want to predict." while section 6.2 begins "...Goldstein and Rougier (2004) [ R2] gives us several key advantages. ...First, the multivariate nature of this probabilistic framework allows us to have more than one prediction variable. Predicting joint probabilities provides us with important information on how uncertainty is related across different climate variables....."

With the Climate Change Act (CCA) 2008, the UK constructed a legally-binding long-term framework to cut greenhouse gas emissions and a framework for building the UK’s ability to adapt to a changing climate [ E4]. The Act requires a UK-wide climate change risk assessment (CCRA) to take place every five years. The CCRA constructs sector reports describing a wide range of potential risks in each of the following sectors: Agriculture; Biodiversity & Ecosystem Services; Built Environment; Business, Industry & Services; Energy; Floods & Coastal Erosion; Forestry; Health; Marine & Fisheries; Transport; Water. The CCRA facilitates the Climate Change Act mandated National Adaptation Programme, (NAP) produced in 2013 [ E6] (covering the period of 2013 to 2018) and updated in 2018 [ E7]. The NAPs set out the Government’s objectives, proposals and policies for responding to the risks identified in the CCRA as detailed in [ E6, p8] “In developing the NAP for England, we have taken the highest order risks from the CCRA and working in partnership with businesses, local government and other organisations, have developed objectives, policies and proposals to address them.''

The original 2012 CCRA [ E4], and thus the 2013 NAP, draws heavily on the uncertainty analysis in UKCP09 [ E4, p9]. The impact of that report is still being felt heavily through this REF impact period. The second report was published in 2017 [E5]. The CCRA used the UK Climate Projections (UKCP09) for three – 30-year periods centred on the 2020s, 2050s and 2080s. The CCRA attempted to monetise the most important risks to the UK, and concluded that the results indicated that the net economic costs to the UK are of the order of tens of billions/year by the 2050s (in current prices). UKCP09 therefore played a key role within the Government's statutory responsibilities for assessing and responding to climate change throughout the current REF period and the updated UKCP18 is continuing this role. The ministerial foreword to the updated version, NAP 17 [ E7], links the various strands of activity as follows.

"Later this year we will be launching a revised set of UK climate projections (‘UKCP18’), replacing the current 2009 set and providing the most up to date and scientifically robust estimations of climate scenarios out to the end of this century. These projections are a key tool to enable everyone to future proof policies and activities to ensure our resilience to possible future climate change. ... so that we take back control of our fisheries and agriculture, restore nature and care for our land, rivers and seas. Using a natural capital approach ensures that we take account of all the many benefits our environment provides, and we will develop policies that ensure our seas and lands are healthy and productive and resilient to climate change."

When the UKCP18 was launched, Environment Secretary Michael Gove said: "This cutting-edge science opens our eyes to the extent of the challenge we face and shows us a future we want to avoid. By having this detailed picture of our changing climate, we can ensure we have the right infrastructure to cope with weather extremes, homes and businesses can adapt, and we can make decisions for the future accordingly.” [ E9]

PUBLIC AND PRIVATE SECTOR IMPACT OF UKCP09, UKCP18

In addition to the statutory role, the Met Office has worked with a wide range of public and private sector organisations to use UK climate impacts to inform decisions on investment amounting to billions of pounds to future-proof projects against climate change.

The UKCP09 website [ E8] contains case studies showing how the UKCP09 products have been used, with topics including: national assessment of river flows, climate change and pollution, strategic planning for flood management, changes in flood damages at a catchment scale, emergency planning, defining land capability for agriculture specifications, assessing potential vulnerabilities to climate change, future-proofing design decisions in the buildings sector, investigating coastal recession & shore profile development, storm surge and sea level rise and assessing impacts of climate change on tourism in South West England.

As early evidence of the ways in which the Met Office are rolling out the effects of UKCP18 to user groups, the document [ E10] details a set of six project leaflets. Each leaflet describes how UKCP09 has been used in the area and how UKCP18 will improve adaptation planning for climate change. The titles show some of the range of areas of application that UKCP18 affects. (i) Assessing climate change risk in Yorkshire; (ii) Future surface water flood hazard risk; (iii) Coastal cliff recession under climate change; (iv) Thermal performance of buildings; (v) Forests for the future; (vi) Water resources and drought planning.

The impact of Craig’s work should be seen in the context of the critical role played by the European Food Safety Authority (EFSA), the European Union (EU) agency established in 2001 by the European Parliament and Council to develop and manage risk assessment policy in the EU regarding food and animal feed safety. EFSA’s scientific advice helps to protect consumers, animals and the environment from food-related risks. EFSA publications are used as the basis for policy making and implementation by the European Commission and Parliament and by individual EU member states. By contributing to a step-change in the way that EFSA addresses uncertainty, Craig’s research is crucial to decisions that have serious consequences for the health of people, animals and the environment.

In food safety risk assessment, scientists must assess the safety of a new food, pesticide or food-borne bacteria. As evidence or knowledge is always incomplete, it is important to explain how uncertainty may affect conclusions and the implications for decision-making. In quantifying these uncertainties, Craig’s methodology and tools have significantly improved EFSA’s assessments of risk and benefit in relation to food production, packaging and consumption.

One EFSA role is overall supervision of EU risk assessment for pesticides. In that role, EFSA produces and publishes risk assessment peer reviews of individual pesticides which support decisions made by individual EU member states in response to applications from the pesticide industry for authorisation for use. Risk assessment for pesticides covers the risk to humans from eating food containing pesticides, the risk to humans from exposure via the skin (for example when spraying pesticides or working with plants which have been sprayed) and the risk to other species exposed due to the use of pesticides in agricultural production.

In the period for consideration of impact for REF 2021, the ecotoxicology sections of EFSA peer review reports for 5 pesticides used the geometric mean approach proposed and justified in [R4]. For example, [E1] refined the acute risk assessment for fish species in this way for the pesticide acrinathrin. The benefit of the geometric mean approach was that applicants for authorisation were encouraged to provide test results for more species than required by legislation. Previously the test result for the most sensitive species would be used and this was effectively a disincentive to test additional species. The benefit for applicants of the changed approach was knowledge that the geometric mean would be used, thereby making authorisation more likely, while the reliability of the assessment benefited from the greater knowledge provided by additional test results

The EFSA 2018 [E5] guidance document on dermal absorption (for pesticides) made two changes to guidance on the basis of research reported in Appendix B: (i) to the calculation to allow for uncertainty about absorption on the basis of limited in vitro data; (ii) to the default values to be used in the absence of in vitro data for the pesticide to address the uncertainty arising from the absence of the data. The first of these changes means that uncertainty due to limited sample size is now taken properly into account in the assessment and the second means that the new default values are based on a transparent analysis of a large dataset taking into account uncertainty about absorption for an untested pesticide. In May 2018, the EU Standing Committee on Plants, Animals, Food and Feed recommended [E6] that the guidance be used, from August 2018, by applicants for authorisation and by EU members states in peer reviews of pesticides. The result is improved treatment of uncertainty and greater transparency about the basis for decisions.

Guidance document [E2] provides explicit direction on how to carry out uncertainty analysis in scientific assessments. The guidance was adopted by the EFSA Scientific Committee in December 2017 [E3a] and is applicable to all areas of EFSA’s work [E3b]. The most significant new feature of this guidance is the requirement that assessors should assess the overall impact of uncertainty on conclusions and that they should express the uncertainty using the mathematical language of probability. This new approach is fundamentally rooted in the subjectivist view of probability and associated methodology, areas of Craig’s research expertise. During development of the guidance, Craig was one of two lead drafting authors and had responsibility for the quantitative sections [E3b].

Following the adoption of the guidance, EFSA created a standing cross-cutting Working Group (WG) on Uncertainty [E4] and Craig has been a member since its inception. As well as experts in uncertainty, the cross-cutting WG has members from four of EFSA’s eight Scientific Panels and is mandated to support the Panels in applying the guidance in their outputs. Examples of EFSA outputs following the new guidance and containing substantial uncertainty analysis are:

• the scientific opinion of the EFSA Panel on Nutrition, Novel Foods and Food Allergens on dietary reference values for sodium [E7], produced at the request of the European Commission. Sodium is an important element in human diet but also a source of health risk, and both risk and benefit are uncertain. The scientific opinion provides a transparent account and quantification of uncertainties affecting the conclusions as required by [E2].

• the EFSA Panel on Plant Health Guidance on quantitative pest risk assessment [E9] which is used routinely by the Panel in subsequent assessments of risk of entry, spread and consequent environmental and economic damage for plant and plant-based pests.

• The scientific opinion of the EFSA Panel on Biological Hazards on control options for Campylobacter in broilers at primary production [E10], produced at the request of the European Commission. Campylobacter is a significant contributor to food-poisoning resulting from consumption of contaminated poultry.

• EFSA Scientific Reports on cumulative risk from presence of multiple pesticides in human diet, produced by EFSA in the context of the EU regulations requiring that (a) decisions about the maximum levels permitted in food should take into account cumulative effects as and when methods for doing so become available and (b) pesticides should have no harmful effects – including cumulative effects – on humans. The first such report is [E8]. Previously, risk to consumers from the presence of pesticide residues in food was only assessed substance by substance despite the possibility of multiple substances contributing cumulatively to a particular harmful effect.

Uncertainty assessment requires expert training both for assessors and for decision-makers who use assessments. To support implementation of the uncertainty guidance, EFSA commissioned a series of training courses (for example [E11]). Craig was one of two tutors who prepared and delivered the training for four courses on general application of the guidance, delivered to EFSA scientific staff and external members of the ten scientific panels, and three courses each tailored to the needs of a single scientific panel and attended by all panel members and staff from related EFSA units. Each 1.5-2 day course was attended by 25-30 scientists full-time. Craig was a facilitator at a two-day 2017 EFSA workshop reviewing 13 case studies covering a wide range of EFSA activities and conducted during a one-year trial period for the draft guidance. Participants were relevant risk managers (decision-makers) from the European Commission and the scientific experts who worked on each case study.

Further impacts derive from the influence of EFSA outputs on other international groups involved in food safety, e.g. the US Environmental Protection Agency, the Organisation for Economic Cooperation and Development (OECD), the Joint Food and Agriculture Organization of the United Nations (FAO)/World Health Organization (WHO) Meeting on Pesticide Residues, and the WHO. During the period for consideration of impact for REF 2021 and following from his work with EFSA: (i) Craig joined an OECD working group on international harmonisation in relation to dermal absorption of chemicals; (ii) Craig delivered training on uncertainty to the EU Joint Research Centre, the EU Scientific Committee on Health Environment and Emerging Risks, the UK Food Standards Agency and Finland’s food safety agency EVIRA.

.

Practice and policy impact

The work in [R1] to [R4] shaped guidelines and procedures concerning the choice of models and the quantification of uncertainty for count data biomarkers. Citing the letter from Public Health England, ‘the methods … have been incorporated into the retrospective dosimetry elements of the EU radiation emergency response plans under the RENEB network’ [E1]. RENEB (Running the European Network of Biological and retrospective Physical dosimetry) is a major network of public health organizations and research institutes/laboratories funded by the 7th EU framework EURATOM Fission Programme [E1, E2], with the mission to provide ‘rapid, comprehensive and standardised methodology for individualised dose estimation in case of large-scale radiological events in Europe and beyond’ ( http://www.reneb.net/). As a further activity linked to RENEB, ‘as a collaboration between Universitat Autònoma de Barcelona (UAB), Bundesamt für Strahlenschutz (BfS), Durham University (DU), Institut de Radioprotection et de Sûreté Nucléaire (IRSN), Universidad de la Rioja (UdR), and Public Health England (PHE)’ [E4], dosimetry software for multiple biomarkers, including the code produced for [R1], have been developed into a ‘BioDose Tools’ package [E2]. This tool not only provides an easy-to-use web-applet (available at https://aldomann.shinyapps.io/biodosetools-v3/) for biological dosimetry laboratories worldwide [E8], but also serves to ‘simplify and standardize uncertainty estimation in biological dosimetry’, hence making an important contribution to the ‘international community of biological dosimetry’ [E2].

In ISO standard 20046:2019(en) [E3], Subsection 12.1.3., [R2] is given as the reference for methods to derive confidence limits for overdispersed count data. That subsection discusses how to correctly derive the sampling distribution of the biomarker count under violation of the Poisson assumption, and, exclusively referring to our work, which distributions should be employed by laboratories in this case. Standards play a crucial role to ‘harmonize the procedure of biological and retrospective physical dose assessments’, and ISO standards are for instance used by RENEB for ‘certification of the laboratory/department/institute’ or ‘accreditation/certification of one or several techniques according to standard ISO’ (Gregoire et al, 2016).

While the technological feasibility of the gamma-H2AX histone as a radiation biomarker was established in the relevant literature about a decade ago, its practical implementation by laboratories had so far been hampered by a lack of methodology to transfer the biomarker measurement into a dose estimate. Our promising initial results [R3] using H2AX biomarker data, which were previously collected by PHE but had been left unpublished (due to lack of an obvious methodology to analyse them), convinced PHE that this is an area which required further resource and research, and led ‘to a revised assessment of dose estimation techniques and biomarkers’ [E1]. One of the actions arising from this was a research visit of Felix Kaestle from the German Federal Office for Radiation Protection (Bundesamt für Strahlenschutz; BfS) to PHE in the summer of 2017 in order to carry out a substantial experimental study using this biomarker [E1]. These data, along with other existing data sets, were used in DU to devise a new methodological framework for radiation dose estimation from this assay, including quantification of uncertainty [R4]. This methodology is not yet part of the BioTools package, but we have produced an applet ‘DoseEstimateH2AX’ alongside publication [R4] which is available online (since November 2018). The methods are now included in PHE’s standard operating procedures [E1]. The letter [E1] also expresses that this collaboration had been ‘hugely important and influential’ and of ‘direct benefit’ to PHE.

We are in contact, through LD-RadStatsNet, the BioDose project, and via RENEB, with essentially every laboratory or public health institution in Europe which is able to carry out gamma-H2AX-foci analyses (many of these are co-authoring [R2].). So, the reach of our methodology, across Europe, is close to 100%. The letter by the Bundesamt für Strahlenschutz [E2] exemplifies the significance of our work to public health institutions, and the explicit mentioning in a NATO report [E7] its visibility.

Societal and economic impact

This body of work has contributed to an increased appreciation in the field that dose estimation from biomarkers requires sophisticated statistical methodology [E1, E5]. Specifically, [R1] has raised awareness that the inferences obtained from cytogenetic radiation biomarkers will depend on the assumed response distribution , and, by extension, that the resulting dose estimates carry uncertainties, which can lead to incorrect triage [R2], hence urgently requiring methods to quantify these uncertainties [E1, E5]. However, towards the middle of the last decade, there was a clearly identifiable skills gap in the field of dosimetry, with ‘the proportion of individuals with formal mathematical and statistical training [in laboratories or public health institutions] relatively low’ [E5]. PHE and DU were involved in several joint activities to fix this skills gap, including the organization of a workshop aimed at Statisticians (2015, Barcelona) [E5, E6] and the creation of a network (LD-RadStatsNet) [E5, E6], which in turn was also involved in planning training activities [E6]. These activities have contributed to establishing biodosimetry as a statistical discipline, as evidenced for instance by a dedicated session ‘Statistical Methods in Radiation Research’ at the CMStatistics 2018 conference ( http://www.cmstatistics.org/CMStatistics2018/), and several students working on (statistical) PhD projects in this field (some based at public health institutions, such as BfS, and some in academia, including DU), thereby producing a generation of skilled researchers in statistical biodosimetry.

According to [E1], the methods developed in the framework of the collaborative work with PHE ‘are now used by PHE for commercial biological dose estimation’ in PHE’s Cytogenetics and Pathology Group, which runs the UK’s commercial Chromosome Dosimetry Service ( https://www.phe-protectionservices.org.uk/cds/). This unit has responsibility for emergency preparedness in retrospective biological dosimetry for triage purposes in the case of a large- scale radiation accident or incident.

The impact of this work, especially [R4], on the general public is a better preparedness in the case of a mass radiation casualty scenario, by providing methodology for the production of meaningful dose estimates and uncertainty bounds, hence enabling effective triage in a much shorter time than through previously existing methods. While the full impact of this innovation will hopefully never be realised, recent events have shown that emergency response preparedness, and in particular testing capability, is vitally important. This holds both for the biomedical technology on which such diagnostics are built, and the computational and statistical routines which give meaning to, and allow principled decisions based on, the raw results of the biomarker. The importance of statistical biodosimetry for emergency response preparedness is, for instance, highlighted in [E8] (page 19, 1^st column, of the 2018/19 report), referring to potential large-scale irradiation scenarios caused by terror attacks or nuclear weapons.

References

Gregoire E., et al. (2016). The harmonization process to set up and maintain an operational biological and physical retrospective dosimetry network: QA QM applied to the RENEB network, International Journal of Radiation Biology, 93, 81-86, https://www.tandfonline.com/doi/full/10.1080/09553002.2016.1206232