Impact case study database

When Prof. Loke’s 2007 AE framework was first published, there were just under 80 systematic reviews in 2007 that focused on evaluating AE. By the year 2014, this had seen a more than four-fold jump with 348 systematic reviews focusing on AE. Since publication of the PRISMA-Harms checklist in 2016, more than 75 systematic reviews have cited PRISMA-Harms as an underlying basis for the methods in their review. This includes a very recent review on COVID-19-related adverse events (e.g., miscarriage, foetal mortality) in pregnant mothers.

Methodological advances in the process of evidence synthesis for detailed assessment of harms: New technology or process has been adopted by the US Government Agency for Healthcare Research and Quality (AHRQ) [Source A]

Professor Loke’s work led to new AHRQ methodological guidance to prioritize and select harms in their evidence-based reviews of healthcare interventions. This ensures that balanced assessments are available for those who commission and use information from systematic reviews in the US and internationally.

The AHRQ is part of the Department of Health and Human Services and is the lead US Federal department charged with improving the safety and quality of America's health care system. It has a USD451,000,000 budget to develop the knowledge, tools, and data needed to improve the health care system and help patients, clinicians and policymakers make informed health decisions. The AHRQ’s overarching objectives are to produce evidence to make health care safer and of higher quality and they have produced more than 700 systematic evidence-based reviews in the past 20 years. Notable current examples in progress include masks for prevention of COVID-19, and no-touch disinfection modalities.

In Feb 2016, the AHRQ contacted Prof. Loke to arrange an interview and provide written expert advice into their methodological guidance for assessing adverse effects. The AHRQ emphasised that the selection and prioritization of harms:

“was identified as an important area given the potentially large number of harms that could be assessed in many reviews.”

[Source A page 23]

Here, the AHRQ had recognized the incomplete benefit:harm coverage in their evidence-based reviews, particularly when they became aware of Prof. Loke’s published data regarding selective and incomplete reporting on harms (2,4). Hence the AHRQ set up interviews with key experts/informants by invitation and incorporated Prof. Loke’s expertise into their Methods guide to explicitly identify harms, and to select specific harms for evaluation in AHRQ evidence-based reviews.

This led to the publication in Feb 2018 of the updated AHRQ Methods Guide for Comparative Effectiveness Reviews: Prioritization and Selection of Harms for Inclusion in Systematic Reviews [Source A].

Methodological recommendations on best way to search for and identify reports of adverse effects (European Medicines Agency, Australian National Health and Medical Research Council) [Sources B and C]

The European Medicine Agency (EMA) set up the European Network of Centres of Pharmacoepidemiology and Pharmacovigilance (ENCEPP) in 2007. This network gives methodological guidance to public institutions and commercial research organisations across all of Europe on the standards they should adhere to when conducting pharmacoepidemiology studies with safety outcomes. Prof. Loke’s systematic framework and terms used in the search (5) are explicitly cited and form the basis for ENCEPP Methodological Standards on how to conduct comprehensive reviews of safety [Source B]. Similarly, the Australian National Health and Medical Research Council has drawn up methodological standards on identification of evidence [Source C], and they cite Prof. Loke’s research (4) into missing adverse effects data as the basis for explicit recommendations to search for and include data from grey literature.

Empirical use of this new benefit:harm methodology has led to policy changes in recommendations of treatment by the World Health Organization [Source D]

Surgical site infections are a major burden internationally affecting up to 300 000 patients per year in the USA, costing between USD3,500,000,000 and USD10,000,000,000. Previous WHO 2016 guidelines made a Strong Recommendation for high inspired oxygen to reduce wound infections. However, this was widely challenged and publicly debated by the clinical community who had major reservations about safety of oxygen and incomplete evaluation of benefit:harm balance. Prof. Loke was invited by the WHO to take part in the conduct of two new systematic reviews (one on benefit, the other focusing on harm). The two reviews yielded a far more complete understanding of the clinical effects of oxygen and the limitations of the underlying data in terms of bias and inadequate measurement of outcomes. Both reviews (with Professor Loke as co-author) generated new and more comprehensive benefit:harm evaluations for the December 2019 re-written WHO Global Guidelines for the Prevention of Surgical Site Infection. The two reviews are specifically listed as the evidence-base for the more carefully nuanced updated recommendations. Here, incorporation of Prof. Loke’s new methodologies for assessing benefit and harm led to the WHO down-grading the recommendations for high oxygen delivery in surgical patients, particularly in developing countries where oxygen is scarce [Source D]. Now that WHO is no longer making a strong recommendation for high oxygen concentrations in surgical patients, the clinicians in developing countries are able to prioritize scarce supplies oxygen for use in COVID-19 patients.

UK government reviewed and revised their definition of COVID-19-mortality in August 2020 following publication of Prof. Loke’s evaluation [Sources E, F and G]

Prof. Loke’s analysis of COVID-19 -deaths triggered a review of PHE’s method of data collection which was ordered by the Secretary of State for Health on that very same day [Source E]. The review of PHE's method of data collection was reported globally thousands of times in just two days between 17-19th July [Source F]. The review reported its findings on 12 Aug 2020, with PHE revising its method of measurement to report both: (i) deaths within 60 days or if the death occurred after 60 days, COVID-19 is listed on the death registration and (ii) deaths in laboratory-confirmed positive individuals where the death occurred within 28 days. The four Chief Medical Officers across the UK also agreed on the standardised measurement of COVID-19-related deaths to a defined time-period of 28 days within a positive test allowing direct comparisons of mortality rates. The death toll in England was revised down by 5,377 to 41,329 [Source G]. The greatest impact of Prof. Loke’s recommendation was seen for mid-July, where the death toll for England was drastically revised down by 75% from 442 to 111 [Source H, table 2]. The standardised definition of COVID-19 deaths also allows compatibility with WHO international datasets: England is no longer an outlier (the WHO definition specifies “There should be no period of complete recovery from COVID-19 between illness and death”). A quote from the Director of Health Improvement at PHE emphasises the importance of the revised method:

“Our analysis of the long-term impact of the infection now allows us to move to new methods, which will give us crucial information about both recent trends and overall mortality burden due to Covid-19”

[Source G]