Impact case study database
Improving the Ways in which Evidence of Mechanisms is Used in Carcinogenicity Assessment and Guideline Development
1. Summary of the impact
Jon Williamson and Michael Wilde have argued that evidence-based medicine can be improved by systematically assessing mechanistic studies alongside clinical studies when evaluating causal claims related to health. This research has led to:
Concrete improvements to the evidence-based methods for assessing carcinogenicity employed by the International Agency for Research on Cancer (IARC);
Concrete improvements to the evidence-based methods for developing guidelines for clinical practice employed by the National Institute of Health and Care Excellence (NICE).
This represents far-reaching impact, since NICE’s methods inform guidelines for clinical practice on all health topics for England and Wales, and IARC’s methods inform assessments of carcinogenicity on behalf of the World Health Organisation, which represents the 193 member states of the United Nations. The research has also led to greater understanding amongst medical researchers, clinicians, and guideline developers of the need for change in how evidence of mechanisms is evaluated in evidence-based medicine.
2. Underpinning research
How do medical researchers go about evaluating a causal claim; for example, that a particular substance is a cause of cancer, or that a particular pharmaceutical causes recovery from disease? Evidence-based medicine (EBM) recommends scrutinising evidence from comparative clinical studies, such as observational studies or randomised controlled trials. EBM downplays evidence from mechanistic studies, which explore features of the mechanisms linking cause and effect. Against this, Jon Williamson, Michael Wilde, and collaborators have argued that evidence from mechanistic studies is in fact crucial for establishing causal claims in medicine. These collaborators include the University of Kent postdoctoral researchers Phyllis Illari, Veli-Pekka Parkkinen, Federica Russo, Yafeng Shan, Samuel Taylor, and Christian Wallmann. They argue that poorer medical decisions are made unless researchers scrutinise mechanistic studies alongside clinical studies. These arguments led them to advocate a novel approach to evaluating evidence in medicine, which recommends the explicit and systematic evaluation of both mechanistic and comparative clinical studies to best inform medical decisions.
Since joining the Department of Philosophy at Kent in 2005, one strand of Williamson’s research has involved an examination of the epistemology of causality. Since 2015, this research has been conducted together with Michael Wilde. The main findings of this research can be summarised as follows:
Evidence of mechanisms is crucial for evaluating causal claims in the health sciences: in order to establish a causal relationship in the health sciences, one normally needs to establish both that the putative cause is appropriately correlated with the putative effect, and that there exists a mechanism linking cause and effect that can explain the extent of this correlation [R1]. This thesis, put forward in 2007, became known in the literature as the ‘Russo-Williamson thesis’ (RWT) or 'evidential pluralism'. R2 offers a recent restatement and defence of this thesis. R3 (2017) provided a defence of RWT in the context of carcinogenicity assessment at IARC.
Evidence of mechanisms is also crucial for the design and interpretation of clinical studies, for drawing inferences from animal experiments to humans, for transferring causal claims to new populations, and for personalising claims to particular individuals. R4 argued that evidence of mechanisms plays a crucial role in drug approval, for example, at NICE. R6 explored the relevance of RWT to transferring causal claims to new populations.
These findings have the following implications: The methodology of EBM should be extended in a principled way to admit and evaluate not only comparative clinical studies, but also evidence of mechanisms obtained by other means, e.g., basic lab research, biomedical imaging, and simulations. This extended methodology is called EBM+. The 2014 paper [R5] argued that present-day EBM erroneously overlooks evidence of mechanisms. The EBM+ monograph [R6] (which has had more than 31,000 downloads) developed the positive EBM+ proposal in detail, and gave explicit guidance on how to systematically evaluate mechanistic evidence alongside clinical studies.
3. References to the research
[R1] Russo, Federica, and Williamson, Jon (2007), ‘Interpreting Causality in the Health Sciences’. International Studies in the Philosophy of Science 21(2): 157-170. ISSN 0269-8595.
[R2] Williamson, Jon (2019), ‘Establishing causal claims in medicine’. International Studies in the Philosophy of Science 32(1): 33-6. ISSN 0269-8595.
[R3] Wilde, Michael, and Parkkinen, Veli-Pekka (2017), ‘Extrapolation and the Russo–Williamson thesis’. Synthese 196: 3251-3262. ISSN 0039-7857. https://doi.org/10.1007/s11229-017-1573-y*
[R4] Aronson, Jeffrey, La Caze, Adam, Kelly, Michael, Parkkinen, Veli-Pekka, and Williamson, Jon (2018), ‘The use of evidence of mechanisms in drug approval’. Journal of Evaluation in Clinical Practice 24(5): 1166-1176. ISSN 1356-1294. https://doi.org/10.1111/jep.12960*
[R5] Clarke, Brendan, Gillies, Donald, Illari, Phyllis, Russo, Federica, and Williamson, Jon (2014), ‘Mechanisms and the Evidence Hierarchy’. Topoi 33(2): 339-360. ISSN 0167-7411.
[R6] Parkkinen, Veli-Pekka, Wallmann, Christian, Wilde, Michael, Clarke, Brendan, Illari, Phyllis, Kelly, Michael, Norell, Charles, Russo, Federica, Shaw, Beth, and Williamson, Jon (2018), Evaluating Evidence of Mechanisms in Medicine: Principles and Procedures. Springer Netherlands. ISBN 978-3-319-94609-2.
https://link.springer.com/book/10.1007%2F978-3-319-94610-8.*
(* Gold open access)
Principal Grants
[G1] Causality and the interpretation of probability in the social and health sciences, British Academy, 2006, PI: Jon Williamson, postdoc: Federica Russo, £7,436.
[G2] Mechanisms and causality, Leverhulme Trust, 2007-10, PI: Jon Williamson, postdoc: Phyllis Illari, £119,367.
[G3] Levels of causality, British Academy, 2008, PI: Jon Williamson, postdoc: Federica Russo, £6,580.
[G4] Causality across the levels: biomedical mechanisms and public health policies, British Academy 2009-11, PI: Jon Williamson, postdoc: Federica Russo, £119,541.
[G5] Mechanisms and the evidence hierarchy, AHRC, 2012, with Brendan Clarke, Donald Gillies, Phyllis Illari, Federica Russo, PI: Jon Williamson, £14,218.
[G6] Grading evidence of mechanisms in physics and biology, Leverhulme Trust, 2015-18, PI: Jon Williamson, postdoc: Veli-Pekka Parkkinen, PhD: Stefan Dragulinescu, £222,096.
[G7] Evaluating evidence in medicine, AHRC, 2015-18, with Brendan Clarke, Donald Gillies, Phyllis Illari, Federica Russo, PI: Jon Williamson, postdocs: Christian Wallmann and Michael Wilde, and collaborators in medicine and public health, £750,859.
[G8] Evidential pluralism in the social sciences, Leverhulme Trust, 2019-22, PI: Jon Williamson, postdocs: Yafeng Shan and Samuel Taylor, £244,183.
4. Details of the impact
(A) Improving methods for carcinogenicity assessment
Williamson and Wilde’s research has led to concrete improvements to the methods for assessing carcinogenicity employed by the International Agency for Research on Cancer (IARC). IARC is the United Nations body responsible for evaluating the carcinogenicity of various agents. Since 1971, IARC has evaluated more than 1,000 agents, of which more than 400 have been identified as potential carcinogens. IARC’s findings result in restrictions on the use of these carcinogens across the world, by agencies such as the European Chemicals Agency and the US Environmental Protection Agency.
In the period 2015-18, Williamson and Wilde visited IARC to observe their methods for carcinogenicity assessment and to discuss improvements to these methods with Kurt Straif, then Head of IARC Monographs. In 2017, Straif stated: ‘the EBM+ programme has already led us to think in new ways about the nature and kinds of evidence of carcinogenicity and to scrutinize our own approach’ [a]. In 2018, Wilde recommended improvements to IARC’s methods. He identified problems arising from giving mechanistic evidence a subsidiary role and failing to provide guidance on how to review mechanistic evidence alongside evidence from comparative studies [b]. Wilde appealed to the underpinning research to suggest that all of these problems could be addressed by treating mechanistic evidence on a par with evidence from comparative studies, and by providing more guidance on how to systematically review mechanistic evidence alongside evidence from comparative studies [b]. In November 2018, Williamson reiterated these points when he sat on the panel responsible for rewriting the IARC Preamble, which documents IARC’s methods [e].
As a result, IARC’s methods now treat mechanistic evidence in accordance with the recommendations of EBM+. Williamson’s contribution ‘led to significant changes in the way mechanistic evidence is evaluated in the new Preamble, with mechanistic evidence now treated on a par with epidemiological studies on humans and with animal studies’ (Kate Guyton, current Head of IARC Monographs) [c]. An updated Preamble reflects this improvement to IARC’s methods, and refers to the systematic review methods put forward in R6 [d]. This Preamble is likely to be in operation for many years (the previous version was used for 13 years), and key improvements in methodology are retained in subsequent versions of the IARC Preamble. Thus, this research will continue to lead to wide-ranging improvements to public health.
(B) Improving methods for developing guidelines for clinical practice
Williamson and Wilde’s research has led to concrete improvements to the methods for developing guidelines employed by the National Institute for Health and Care Excellence (NICE), which aims to improve public health outcomes in England and Wales by providing evidence-based guidelines on a variety of health topics, including guidelines on how to best manage certain health conditions.
Throughout 2015-18, Williamson and Wilde observed guidelines development meetings at NICE. Based on their research, they made a number of recommendations for improving the way in which evidence of mechanisms informs these meetings [k], after discussing these recommendations with NICE [f]. In April 2018, several of these methodological recommendations were incorporated into a draft revision of the NICE methods manual. In May 2018, Wilde made further suggestions for changes to this draft to better reflect the methodological improvements [g]. Ultimately, this led to the following changes in the final methods manual, released in October 2018 [h]:
An explicit recognition of the importance of evidence of mechanisms in identifying different sub-populations. (Section 2.3 of the updated methods manual.)
A move away from the terminology of ‘pathophysiological basis’ towards the more precise terminology of ‘mechanism of action’ (ibid. Section 4.3 and 6.4).
An acknowledgement that an evidence review of the effectiveness of an intervention should in some cases include review questions on the intervention’s mechanism of action, and that it may be necessary to combine any resulting evidence of the intervention’s mechanism of action with other types of evidence for effectiveness; for example, evidence from randomised trials (ibid. Section 4.3 and Appendix A).
An explicit recognition that the link between a surrogate outcome and a clinical outcome needs to be justified by mechanistic evidence (ibid. Section 7.6 and Appendix A).
These improvements have all been retained in a subsequent (2020) update to the NICE methods manual. NICE state: ‘We are grateful to the team for proposing these important changes to the NICE guidelines manual. We look forward to receiving further feedback from the team to inform future revisions’ [h].
More widely, Williamson and Wilde’s research has increased awareness among clinicians, medical researchers, and guideline developers, among others, of a need for change in how evidence of mechanisms is evaluated in evidence-based medicine (e.g. through a letter published in the BMJ EBM journal, signed by many philosophers and health scientists **[i]**). Professor Sir Michael Rawlins, chair of the UK Government Medicines and Healthcare Products Regulatory Agency (MHRA), notes that the EBM+ monograph [R6] is ‘not just a timely reminder of the importance of mechanisms. It is also a wake-up call to the evidence-based medicine movement to incorporate mechanisms in their evaluation of “evidence”. EBM+ comes of age’ [j].
5. Sources to corroborate the impact
[a] Testimonial letter from the then Head of IARC Monographs. This letter corroborates the impact of the research on the methods for assessing carcinogenicity employed by IARC.
[b] Public Comments Form in Response to IARC Consultation. This form provides evidence of concrete recommendations made on the basis of the underpinning research to improve the methods for assessing carcinogenicity employed by IARC.
[c] Testimonial letter from current Head of IARC Monographs. This letter corroborates that Williamson’s contribution to the recommendations to revise the Preamble was significant.
[d] Summary statement describing the improvements to the IARC Preamble. This statement corroborates that the improvements made by the Advisory Group strengthened the emphasis on mechanistic evidence.
[e] This list of members of the Advisory Group. This list corroborates that Williamson was a member of the Advisory Group to Recommend an Update to the IARC Preamble.
[f] Testimonial letter from then Senior Technical Advisor at NICE. This letter corroborates that the EBM+ group helped to examine the value and use of different types of evidence (including evidence of mechanisms).
[g] Consultation form for NICE guidelines manual. This form details the further recommendations made on the basis of the underpinning research to improve the methods for developing guidelines at NICE.
[h] Testimonial letter from Programme Director at NICE. This letter corroborates ‘the involvement of the “Evaluating evidence in medicine” project team in the recent revisions to the NICE guidelines manual’. It also describes some of these revisions.
[i] Letter to the British Medical Journal Evidence-Based Medicine (BMJ EBM). This letter corroborates that the underpinning research has increased awareness of a need for change in how mechanistic evidence is employed in evidence-based medicine.
[j] Foreword to the EBM+ monograph. This foreword corroborates that the underpinning research demonstrated the importance of the evidence-based medicine movement.
[k] Response from NICE to proposed changes. Wilde submitted recommendations for improving the methods for developing guidelines at NICE. This document, from NICE, contains their preliminary responses to these recommendations.
Additional contextual information
Grant funding
| Grant number | Value of grant |
|---|---|
| SG-41924 | £7,436 |
| F/00 236/L | £119,367 |
| SG- 47884 | £6,580 |
| BARDA 53292 | £119,541 |
| AH/M005917/1 | £750,859 |
| RPG-2014-181 | £222,096 |
| AH/J011185/1 | £14,217 |
| RPG-2019-059 | £244,183 |