Impact case study database
Transforming the assessment of global health risks from chemical exposures through the development and implementation of systematic review methods
1. Summary of the impact
There is divergence in international regulatory policies on the safety of chemicals and the risks associated with their use or exposure. Lancaster University Researchers have adapted systematic review methods (SRMs) into chemical risk assessment protocols to improve chemical risk policy and reduce uncertainty. This was achieved through a consensus-building process with scientists and stakeholders to foster adoption of these methods followed by the development of protocols, methods and standards. Our work has been used to improve the hazard protocol of the European Food Safety Authority (EFSA) for bisphenol-A, a widely-used chemical substance (1.25Mt per annum in the EU), and the regulatory requirements for endocrine-disrupting chemicals in the EU’s Biocidal Products Regulation (528/2012). Our SRM training programmes resulted in changes to the assessment of scientific evidence in the development of global evidence-based health guidelines by the United Nations World Health & International Labour Organisations (WHO/ILO) responsible for establishing the global burden of disease from occupational environmental exposures. Our quality appraisal tools (COSTER/CREST), which ensure best practice for implementing SRMs in environmental health, have now been used in over 50 published reviews by over 1000 researchers worldwide and have shaped journal publishing practices.
2. Underpinning research
There is controversy surrounding the chemical risk assessment (CRA) of glyphosate (‘Roundup’), bisphenol-A (BPA) and other ‘everyday’ chemicals, with reputable scientific organisations disputing their health risks – even with access to the same evidence. Initial research into how risk assessment methods used by the European Food Safety Authority (EFSA) for BPA compared to 'gold-standard' methods utilised in medicine suggested systemic shortcomings in appraisal practices. These rely on narrative reviews that are inconsistent, in that: review objectives are not sufficiently clearly stated; methods for locating data are not consistently given; the criteria for selecting data for analysis are incompletely stated; how studies are evaluated for quality appears to be neither transparent nor consistent; and, the synthesis and presentation of results is unclear. These shortcomings could explain why such differing opinions on the health risks of BPA and other chemicals exist within the scientific community.
A policy document authored by Paul Whaley called Systematic review and the future of evidence in chemicals policy (2013) provided the first technical analysis comparing European chemical risk assessment practices to the ‘gold-standard’ systematic review methods (SRMs) used in medicine. Subsequently, in 2013, Paul joined Professor Crispin Halsall’s research group as a PhD student to investigate how the current shortcomings in CRA could be overcome by the adoption of a systematic review methodology. The review methodology approach ensured that the appraisal of evidence is conducted in a systematic way so as to reduce bias, recognise uncertainty and provide, where possible, unambiguous answers based on all of the evidence available.
In 2014/15, with the support of the Royal Society of Chemistry, we gathered leading experts from regulatory agencies, NGOs and industrial and academic sectors to develop a strategic framework for introducing systematic review practices to CRA [3.1]. This multi-stakeholder exploration of the application of SRMs to CRA developed a consensus view on the potential benefits of adopting SRMs and provided strategic recommendations for promoting SRM uptake. This is relevant in the wider context of utilising SRM for the appraisal of scientific information by expert committees, who provide governmental or regulatory guidance for risk management policy in environmental health.
The adoption of SRMs in the appraisal of primary studies on endocrine (hormone system) disrupting chemicals (EDCs) resulted in a robust framework for assessing evidence from multiple streams of research (e.g. in-vivo, in-vitro, epidemiology) in the assignment of a substance as an EDC [3.2]. Research into SRMs and their application to the environmental sciences resulted in the development of a set of reporting standards (Reporting Standards for Systematic Evidence Syntheses – ROSES) that ensure that meta-analyses and systematic reviews are reported consistently and to a very high level of detail [3.3]. In turn, these reporting standards ensure reliable synthesis of often disparate and growing bodies of evidence (e.g. epidemiology vs. in-vitro assay data) required for evidence-informed decision making. ROSES is designed to accommodate the diversity of methods applied to a wide-variety of review subjects and reflects the heterogeneity and inter-disciplinarity of topics such as the conservation and environmental management field.
The application of systematic evidence mapping as a technique for evaluating the ‘evidence landscape’ with regard to chemical exposure and toxicity has been demonstrated [3.4]. This method provides a step change in evidence-gathering by providing a comprehensive, queryable summary of a large body of policy-relevant research to aid chemical risk management. Systematic evidence maps (SEMs) provide a broad and comprehensive overview of an evidence base and facilitate the identification of trends which can be used to inform more efficient systematic reviews or more targeted primary research. Evidence mapping applied to CRA is a technique that draws into consideration all data which are relevant to chemicals policy and risk management, which leads to large, interconnected but heterogeneous databases. Locating, organising, and evaluating all relevant data is challenging when the quantity of that data is very large and growing exponentially. Therefore, to make full use of these data Halsall and his team have applied knowledge graphs which offer a flexible, schemaless, and scalable model for systematically mapping the toxicology and environmental health literature [3.5].
Research on systematic review methods for application to CRA has centred on the development of reporting standards and codes of practice for conducting and reviewing systematic reviews in the chemical exposure, toxicology and environmental health fields. The idea of creating toolkits for authors and editors to appraise the conduct and reporting of systematic reviews in the environmental health and toxicology fields (akin to those used in biomedical fields) was developed. For example, we have developed a detailed code of practice for systematic reviews in toxicology and environmental health research, called ‘COSTER’ [3.6]. In essence, these codes set quality standards to ensure that systematic reviews in the field of chemical risk are of high quality and comparable to Cochrane reviews undertaken in the biomedical sciences.
3. References to the research
3.1 . Whaley P., Halsall, C. J., et al. ( 2016) Implementing systematic review techniques in chemical risk assessment: Challenges, opportunities and recommendations. Environment International 92-93, 556-564. https://doi.org/10.1016/j.envint.2015.11.002 (citations: 30)
3.2. Vandenberg, L. N., Ågerstrand, M., Beronius, A., Whaley, P., et al. ( 2016). A proposed framework for the SYstematic Review and INtegrated Assessment (SYRINA) of endocrine disrupting chemicals. Environmental Health, 15(1), 74. https://ehjournal.biomedcentral.com/articles/10.1186/s12940\-016\-0156\-6 (citations: 46)
3.3. Haddaway, N.R., Macura, B., Whaley, P., and Pullin, A. ( 2018) ROSES RepOrting standards for Systematic Evidence Syntheses: Pro forma, flow-diagram and descriptive summary of the plan and conduct of environmental systematic reviews and systematic maps. Environmental Evidence 7, 7. https://doi.org/10.1186/s13750-018-0121-7 (citations: 60)
3.4. Wolffe, T., Whaley, P., Halsall, C., Rooney, A., and Walker, V. ( 2019) Systematic evidence maps as a novel tool to support evidence-based decision-making in chemicals policy and risk management. Environment International 130, 104871. https://doi.org/10.1016/j.envint.2019.05.065 (citations: 8)
3.5. Wolffe, T. A. M., Vidler, J., Halsall, C., Hunt, N., Whaley, P. ( 2020) A survey of systematic evidence mapping practice and the case for knowledge graphs in environmental health and toxicology. Toxicological Sciences 35-49 https://doi:10.1093/toxsci/kfaa025
3.6. Whaley P., Halsall, C., et al. ( 2020) A code of practice for the COnduct of Systematic reviews in Toxicology and Environmental health Research (COSTER). Environment International 143, 105926. https://doi.org/10.1016/j.envint.2020.105926
4. Details of the impact
The incorporation of systematic review methods in chemical risk assessment and the application of our SYRINA, COSTER and CREST tools (see https://crest\-tools.site/) has directly influenced risk assessment protocols and regulations within the EFSA and the European Commission. Whaley and Halsall have provided training to the WHO/ILO, which led to a change in their methodologies for assessing scientific evidence in the development of global evidence-based health targets and guidelines. Additionally, our codes of practice for conducting and reviewing systematic reviews in environmental health and toxicology have changed publishing practices on SRMs in leading health science journals. These four areas of impact are detailed below.
4.1. Changing chemical risk assessment methods at the European Food Safety Authority (EFSA)
In 2016/17, EFSA sought to re-evaluate, through public consultation, its hazard assessment protocol for BPA, a high production volume chemical with some 1.25Mt produced in Europe each year. Amongst other uses, BPA is used in food contact materials with BPA-based epoxyphenolic resins used in protective linings for food and beverage cans as well as in polycarbonate food and liquid containers. EFSA reviewed scientific evidence from 2012 onwards to investigate whether the currently indicated tolerable daily human intake (TDI) of 4 µg/kg body weight/day was appropriate. During the consultation period, Whaley and Halsall provided written comments on the shortcomings of the initial assessment protocol, particularly with regards to the lack of SRMs. In 2017, we were invited as SRM experts to give a presentation at a BPA public meeting hosted by EFSA [5.1], attended by ~30 stakeholders from regulatory, food safety, NGO and industry sectors. An analysis using the COSTER/CREST framework [3.6] led to a significant and documented improvement in the EFSA's hazard assessment protocol [5.2]. The EFSA considered the inclusion of SRMs to provide a more transparent, less biased and overall more accurate risk assessment. This provided confidence in the current TDI of BPA but is considered temporary until a further evaluation is conducted using a protocol that now incorporates SRM.
4.2. Impacting international regulatory requirements for identification of Endocrine Disrupting Chemicals (EDCs)
Preventing public exposure to harmful EDCs is estimated to have a median annual cost saving of €163 billion1 across the EU in terms of disease reduction and associated healthcare costs. On the basis of SYRINA (an international SRM-based framework for the identification of EDCs) Whaley and Halsall coordinated scientific comments on the draft EDC criteria from the EU Commission. This took the form of letters to the EU Commissioner for Food & Heath Safety and in proposed redrafts of regulatory provisions during public consultation periods in 2015. The team also received direct feedback on the proposals from the Commissioner, who confirmed changes to specific criteria in the draft regulation [5.3].
A quote from this letter states: “You suggest that best practices should be followed in evidence gathering, appraisal and integration. Under the proposed revised criteria, information must now be gathered and analysed using a weight-of-evidence approach and according to systematic review methods”.
The primary outcome was a change in how scientific evidence is assessed in evaluating whether a chemical should be classified as an endocrine disruptor under the Biocidal Products Regulation (BPR; Regulation EU 528/2012), which concerns market distribution and use of biocidal products. The proposed changes were adopted by the European Parliament and the EU Commission in 2017. This resulted in changes to several BPR provisions to specifically reference SRMs [5.4]. This has substantial implications for the approximately 40 countries that follow these EU and international regulations, in particular manufacturers producing chemicals suspected as EDCs (e.g. BPA, phthalates, etc.).
1 Andrology (2016) 4, 565–572
4.3. Influencing global health policy impact assessment methods at two UN agencies
In 2016, following a request from one of the World Health Organisation (WHO) systematic review teams, Whaley and Halsall provided training in SRMs based on SYRINA [3.3] and the COSTER [3.6] code of practice. The emphasis of the training was on the value of adopting a protocol (that describes the conduct of a systematic review prior to the systematic review being undertaken) and the uncertainty of the validity of current methods in light of COSTER [5.5]. This led to the WHO pre-publishing and externally peer-reviewing the systematic review protocols later that year. Additional training was provided in 2017 at the International Labour Organization (ILO) in Geneva on bias in risk assessment methods to 40 global scientists leading the various review teams tasked with estimating the work-related burden of disease and injury for different factors and/or exposures. This led to radical improvement in the proposed approaches and development of a new risk of bias tool for prevalence studies. Further training was provided in 2019 prior to the finalisation of the completed SRMs.
These training programmes resulted in changes to the assessment of scientific evidence in the development of global evidence-based health targets and guidelines by the international UN agency responsible for global health. Specific WHO/ILO projects (15 internationally-relevant areas of concern) exploring the use of SRMs to establish the global burden of disease from occupational environmental exposures are now provided in a Special Issue of Environment International, with an editorial overview provided by Whaley [5.6] who had a fundamental role in in designing the review methods and training the teams of authors involved (see [5.5]).
The Special Issue in turn led to improved understanding of the impact of environmental factors on global public health and, specifically, to occupationally-exposed populations (i.e. workers) in developing countries with particularly high chemical exposure burdens. Following UN guidance should therefore result in improved health in those countries with exposed populations, with the WHO and ILO, along with our research team, ensuring compliance through audits. The Systematic review has now been adopted as the methodology with which to assess progress of the UN’s Sustainable Development Goals, most notably for Goal 8 - to promote health and well-being in the working environment [5.7].
4.4. Changing publishing practices at multiple leading environmental health journals
Lancaster research into appraisal tools (e.g. COSTER [3.6] and systematic evidence mapping [3.4, 3.5]) and hosting of an international workshop [3.1] on strategy for mainstreaming SRMs in CRA (2014) led to a request from Environment International to edit the first-ever Special Issue on SRMs for Chemical Risk Assessment, coordinated by Whaley and Halsall.
The success of the special issue (relative citation ratio >2.5) led to Whaley being appointed as the world’s first specialist SRM editor for Environment International, a leading environmental health journal (2018). As a result of this appointment, Whaley has been able to implement the COSTER/CREST framework [3.6] into the journal's workflow and has been invited to speak on more than ten occasions (since 2018) about SRM expectations and new processes, including informal trainings and consultations with submitting authors. In this role he has delivered workshops (along with other editors) to disseminate codes of practice and standards for systematic reviews.
These positions and the integration of SRMs into journal practice have changed how researchers assess existing evidence of health risks from chemical exposures, ensuring objective analysis and better reporting. To date (2021), there have been some 200 manuscripts submitted (1500-2000 submitting authors) who have followed and benefitted from the COSTER [3.6] and CREST frameworks. Two scientific journals have provided consensus statements agreeing to the use of SRMs [5.8a and b]. ROSES (see [3.3]) has been endorsed by Nature Climate Change, Environmental Evidence and Environment International, providing, for the first time, an explicit minimum standard expected for a systematic review submitted to these journals [5.9 a, b and c].
5. Sources to corroborate the impact
Changing chemical risk assessment methods at the EFSA
5.1. EFSA Workshop on the BPA Hazard Assessment Protocol (2017) - agenda highlighting Lancaster University’s contribution ( https://www.efsa.europa.eu/en/events/event/170914)
5.2. Documents from EFSA corroborating (point by point) the changes to its BPA Hazard Assessment Protocol (2017).
Impacting international regulatory requirements for identification of EDCs
5.3. Letters from the EU Commissioner for Food & Heath Safety (2014-present) corroborating our contributions in defining scientific criteria for the regulation of EDCs
5.4. Public list of Biocidal Products Regulation changed as a result of the EDC review (2017): https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32017R2100&from=EN
Influencing global health policy impact assessment methods at two UN agencies
5.5. Letter from a Professor at Radboud University Medical Centre commenting on the value of the Lancaster University training, its contribution to the decision to publish protocols and the consequences for introducing SRMs (2016).
5.6. Link to Special Issue featuring the WHO/ILO projects as SRM ‘protocols’ (2020)
5.7. Evidence of the incorporation of SRMs in assessing progress of the UN WHO/ILO Sustainable Development Goals “ Implementation of the 2030 Agenda for Sustainable Development” (specifically for Goal 8) [WHO A72/11 Rev. 1, 2019] with Whaley instructing on SRM in WHO expert meeting 1 (2017) and 2 (2019).
Changing publishing practices at multiple leading environmental health journals
5.8. Policy change statements from the journal editors of: a) Environment International ; b) Toxicology Sciences (2018) and Letter from Associate Editor, Toxicological Sciences.
5.9. Published guides for manuscript authors/reviewers and endorsement of ROSES for the journals: a) Nature Climate Change ; b) Environmental Evidence; c) Environment International (2018)