Impact case study database
Influencing the uptake and reporting of adaptive designs in clinical trials
1. Summary of the impact
In order for a new treatment or therapy to be used in medical practice, clinical trials must be run during the research and development process, often taking years to complete and costing millions of pounds. Approximately 30,000 new clinical trials worldwide are now registered annually, with approximately 300,000 clinical studies registered globally since 2000. To improve efficiency of the clinical trial process, such that treatments can be identified and ruled out more quickly and cost-effectively, research at Reading has made important statistical contributions to the field of multi-arm multi-stage adaptive designs. Reading’s research combines methodological work on novel statistical design and analysis approaches with practical research to facilitate implementation of statistical methods in the healthcare sector. Reading’s research has underpinned clinical trials and has been instrumental in the development of software resulting in time savings, economic benefits and advantages to patients. The research has also fed into the development of two sets of trial reporting guidelines, recently published in the British Medical Journal (BMJ) and the Journal of the American Medical Association ( JAMA), that serve as checklists for authors who wish to publish the results of clinical trials, making studies more transparent and easier to interpret, ultimately benefiting clinical practice.
2. Underpinning research
Clinical trials are the primary tool for evaluating the efficacy and safety of new treatments in health-related research. The timelines and costs associated with conducting clinical trials are increasing due to such factors as: the complex nature of many modern therapies; the price of supplies and equipment; intricacies associated with recruiting and monitoring participants; and stringent regulations. It takes at least 10 to 12 years for a new treatment to reach clinical practice, and two thirds of that time is spent conducting clinical trials. It is estimated that the average cost of research and development associated with successfully bringing, for example, a new drug to market is approximately USD2.6bn (November 2014), whilst less than 12% of drugs are ultimately approved for use. Consequently, industry, the public sector and regulators are keen to reduce development costs and to accelerate the availability of new treatments for patients.
Traditionally, clinical trials have used a fixed sample size statistical design, where data are collected from all participants in a study before any analyses are performed. Most often these studies consist of two treatment arms, comparing the new treatment under investigation with a control treatment, which is either the current standard of care or a placebo. As costs and timescales escalate, there are clear benefits to be gained from improving the efficiency of the whole process, in particular the efficiency of the clinical trials element. Multi-arm multi-stage adaptive designs compare several experimental treatments with a control treatment and include opportunities during a clinical trial for the accumulating data to be examined and used to inform how the trial will progress. For example, treatment arms can be stopped before the end of the trial if interim data show them to be less effective compared to other treatments being tested, making the trial more efficient in terms of resources.
Reading’s statistics researchers were pioneers in this field and published one of the earliest papers on treatment selection in multi-arm multi-stage adaptive designs [R1]. The research generalised the standard two-arm group sequential design to a multi-arm setting in which, at the first of a series of planned interim analysis, the best of several competing experimental treatments is selected to continue in the trial along with the control arm. The key idea uses the distribution of the maximum of several test statistics to generalise the two-arm group sequential stopping boundary.
Based in part on the research described in [R1], a short course on adaptive trials was held at the University of Reading in 2006. Through a collaboration with one of the participants, the Multiple Sclerosis (MS) Society funded joint work, between 2008 and 2009 for researchers from Imperial College Healthcare NHS Trust and the Universities of Reading and Warwick, to look at statistical methodology for multi-arm multi-stage adaptive designs in Secondary Progressive MS. The Reading team worked specifically on how to use data to inform development of appropriate statistical hypotheses in the proposed adaptive design; and conducting simulations to evaluate the methods. The MS work yielded novel statistical aspects that could be applied beyond the MS context. First, the team proposed an enhanced framework for the evaluation of statistical design and analysis approaches for clinical trials, allowing the assessment of competing strategies via simulations [R2]. The refinement proposed new categories within the three elements of assumptions, options and metrics, which should be considered when working within the framework. Second, the researchers proposed a method formulated from combination tests for adaptive designs and the closure principle for multiple testing, for using early outcome data to choose between competing treatments at the first stage of the study whilst protecting the familywise error rate of the trial [R3].
Most statistical developments in the field have focussed on the design of adaptive trials and there has been less research considering the analysis of these trials after they have been completed. A long-standing collaboration between researchers at Warwick and Reading has considered estimation following a multi-arm multi-stage adaptive design [R4]. The methods developed provide estimators for the difference between selected treatment arms and the control arm on completion of the trial which are unbiased, conditional on treatment selection.
Whilst conducting the MS work, the Reading team became interested in the question of how to influence the uptake of adaptive designs, particularly in the public sector— the intended setting of the MS trial. Professor Todd subsequently co-supervised three National Institute for Health Research Doctoral Research Fellowships held by two statisticians at the University of Sheffield and one at the University of Leeds. In all cases, Professor Todd’s role in the supervisory team was to bring detailed knowledge of the methodological aspects of adaptive designs. One project considered why adaptive designs are underused in the public sector via both qualitative and quantitative surveys. It explored potential facilitators to the appropriate use of adaptive designs and proposed recommendations to improve their uptake. These included addressing the lack of knowledge and experience of these designs and transparent and accurate reporting of adaptive trials [R5]. Another project looked at statistical methodology for the design and analysis of multi-arm trials, particularly those which seek to add a new treatment arm. Both analytical investigations and simulations were used to explore the consequences for error rates of different multiple testing strategies for multi-arm trials [R6]. This resulted in a structured decision process for determining the requirement for multiple testing adjustment in multi-arm studies.
Over many years, Reading’s research in adaptive designs has been dual-layered in nature, rooted in developing novel statistical methods for designing and analysing trials [R1, R3, R4], but going beyond this to conducting research which facilitates the use of these methods in practice [R2, R5, R6].
3. References to the research
[R1] Stallard, N. and Todd, S. (2003). Sequential designs for phase III clinical trials incorporating treatment selection. Statistics in Medicine. 22, 689-703. https://doi.org/10.1002/sim.1362
[R2] Friede, T., Nicholas, R., Stallard, N., Todd, S., Parsons, N., Valdés-Márquez, E. and Chataway, J. (2010). Refinement of the Clinical Scenario Evaluation framework for assessment of competing development strategies with an application to Multiple Sclerosis. Drug Information Journal. 44, 713-718. https://doi.org/10.1177/009286151004400607
[R3] Friede, T., Parsons, N., Stallard, N., Todd, S., Valdés-Márquez, E., Chataway, J. and Nicholas, R. (2011). Designing a seamless phase II/III clinical trial using early outcomes for treatment selection: an application in multiple sclerosis. Statistics in Medicine. 30, 1528-1540. https://doi.org/10.1002/sim.4202
[R4] Kimani, P.K., Todd, S. and Stallard, N. (2013). Conditionally unbiased estimation in phase II/III clinical trials with early stopping for futility. Statistics in Medicine. 32, 2893-2910 . https://doi.org/10.1002/sim.5757
[R5] Dimairo, M., Julious, S. A., Todd, S., Nicholl, J. P. and Boote, J. (2015). Cross-sector surveys assessing perceptions of key stakeholders towards barriers, concerns and facilitators to the appropriate use of adaptive designs in confirmatory trials. Trials. 16, 585. https://doi.org/10.1186/s13063-015-1119-x
[R6] Howard, D. R., Brown, J. M., Todd, S. and Gregory, W. M. (2018). Recommendations on multiple-testing adjustment in multi-arm trials with a shared control group. Statistical Methods in Medical Research. 27(5), 1513-1530. https://doi.org/10.1177/0962280216664759
Evidence for the body of work meeting minimum 2* threshold:
Research resulted from competitive funding applications made to industry, government funders, research councils and charities. All publications appear in peer-reviewed, international journals. Outputs which develop statistical methodology meet or exceed 2* quality definitions in that they contribute new techniques and results to the field. Outputs which facilitate use of methodology meet or exceed 2*quality definitions in that they have made meaningful contributions to advancing the use of the statistical techniques in practice either via influencing processes and practice or enhancing user engagement with methods.
4. Details of the impact
Reading’s statistical research into the design, conduct and analysis of clinical trials which implement adaptive designs has helped shape the current landscape of their usage. In the last seven years, Reading’s contributions to research in the field have: (1) enabled advances to be made in the clinical research being conducted, in particular therapeutic areas; (2) influenced the development of software which allows more potential users of the approaches to access methodology; (3) aided in the production of international reporting guidelines which improve the transparency of clinical trials and how they are interpreted and used.
Use and influence of the research in three public sector clinical trials
Reading’s research has influenced three recent public sector studies, resulting in more efficient trials, with better decision making, to the benefit of both patients and trial sponsors. First, the design described in [R3] and an extension of the analysis methodology described in [R4] have been used in the PROVE trial, a multicentre, adaptive, three-arm trial in osteoporotic vertebral fractures. The trial began recruitment in November 2013 and was completed in September 2017. Professor Todd was a member of the Data Monitoring Committee for the study. It was acknowledged in the Health Technology Assessment (HTA) monograph containing the full results of the trial [S1] that the adaptive design gives the opportunity for “ the research question to be answered in less time with less cost, randomising fewer participants to a less effective treatment” which allowed the trial investigators to “ maximise the efficiency of the trial”.
Second, the research in [R2, R3] was the starting point for a funding application to run a trial in MS. During the application process, modifications and simplifications were made to the trial design, such that it retained its multi-arm element, but lost the other adaptive features. The trial was launched as the MS-SMART trial in December 2014, reporting in October 2018. The MS Society website details the results of the trial, [S2] saying: “This is the first time a clinical trial in MS has tested multiple drugs at the same time. This method delivers answers up to ten years earlier than a standard clinical trial.” Furthermore, the impact of the trial on future research within the MS community is clear: “We’re now better placed to identify and rule out treatments that won’t work, and have demonstrated how to run faster, cheaper trials successfully – meaning we can test more potential drugs, quicker.”
Third, the decision process developed in [R6] underpins the statistical rationale for a protocol amendment to allow addition of a new treatment arm to the ongoing FLAIR trial in patients with chronic lymphocytic leukaemia. The amendment application was submitted to Cancer Research UK in November 2015 for review by its Clinical Research Committee. The process incorporated an international peer review by four reviewers. One commented: “ As the availability of novel agents increases across all types of cancer, studies such as this can be looked at as a model for efficiently answering key questions in a field” [S3] . All comments, including those concerning the increased complexity of the amended design were addressed to the satisfaction of the committee and the amendment was approved. The new treatment arm was opened to recruitment in July 2017 and the trial is ongoing.
Use of the research in the development of software / consultancy business
Methodology for the design and analysis of adaptive trials can be complex and software is needed to allow the approach to be widely accessible to potential users. Reading’s research underlies some recent developments in the commercial software package ‘East,’ developed by the company Cytel Inc, which has the dominant market share worldwide in software for this specialist field. Cytel is the world’s largest biometrics Contract Research Organisation and ‘East’ is their flagship software package “ which has sold multi-user licenses to more than 600 sites worldwide including brand name and generic pharmaceutical companies, biotechs, hospitals, universities and research institutes” [S4] . In addition, the Food and Drug Administration (FDA), which regulates the development of new drugs in the US, “has a site license for East that is used by its reviewers to verify the validity of the designs submitted for scientific review by pharmaceutical sponsors” [S4]. In 2018, the latest version of ‘East’ was released, which contains a new module for multi-arm multi-stage designs. Reading’s research has been fundamental to the development of this module. The President of the company states [S4]: “That research <[R1]> has had a significant impact at Cytel on both our internal research and our software development. To be specific, Cytel's East software system was able to expand its offering from the classical two-arm group sequential design to the multi-arm setting in which several treatment arms are compared to a common control arm with possible early stopping and treatment selection. This development was made possible by our careful study of the pioneering paper on this topic by Stallard and Todd (2003).” In addition to software sales, Cytel also offers a consulting service. The President of the company further says [S4]: “Since the implementation of the East modules that were influenced by Professor Todd's work, Cytel has seen about a 10% increase in our strategic consulting business.”
The work described in [R2] has specifically been influential in the development of the free-to-download Mediana R package, produced by a development team of industry experts from different companies. The package gives a general framework for conducting clinical trial simulations which is applicable beyond adaptive trials to aid decision-making and strategy selection across many areas of clinical development. The latest version of the package was released in May 2019 and the developers say that the package “ has been successfully used in multiple clinical trials to perform power calculations as well as optimally select trial designs and analysis strategies” [S5]. Download statistics for the package are approximately 650 per month, placing the package in the top 20 to 25% of most downloaded packages from the Comprehensive R Archive Network (CRAN), one of the main repositories for sharing computer code developed in R.
Development of reporting guidelines
Incomplete and poor reporting of the results of clinical trials means that the findings of studies where this is an issue can be difficult to interpret, reproduce and synthesise. This can hamper a study’s ability to inform clinical practice and to shape future research. Approximately USD85bn (January 2019) of medical research funding is wasted each year because its outcomes are not appropriately published [S6]. The CONSORT (CONsolidated Standards Of Reporting Trials) checklist, first published in 1996 and subsequently updated, exists for guiding the reporting of ‘standard’ two-arm trial designs, but this does not give enough guidance for more complex trial designs. To improve the reporting of trials with more complex designs, CONSORT extensions have been developed. Research in [R5] and [R6] has been integral to the development of two separate CONSORT extension guidelines: one for Adaptive Trials and one for Multi-Arm Parallel-Group Trials, respectively. Professor Todd was part of an international cross-sector team of experts from academia, industry and regulatory authorities who developed the Adaptive CONSORT Extension statement which has been co-published in the BMJ and Trials in 2020 [S7]. The Multi-Arm Parallel-Group Randomised Trials Extension has also been published in 2019 in JAMA [S8]. Both sets of guidelines were developed with the involvement of the CONSORT group and are registered with the CONSORT Initiative [S9] and the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) network [S10]. Both are international initiatives which seek to improve the value of published health research through better reporting of trials. The original CONSORT statement is endorsed by prominent medical journals and leading editorial organisations and influences how clinical trials are reported. The CONSORT extensions are being widely consulted ( JAMA 2019: approximately 5,000 pdf downloads between April 2019 and December 2020; BMJ 2020: approximately 1,500 pdf downloads between June 2020 and December 2020), meaning that when more complex designs which include adaptive or multi-arm elements are being undertaken in the future, authors will adhere to the guidelines to publish their results, thus strengthening clinical trials research and reducing research waste.
The need to improve efficiency of the clinical trials process has meant that statistical research into novel study design and analysis methods is crucial. The field of multi-arm multi-stage adaptive designs is one of the most rapidly expanding areas of such research. Reading's contributions have had far-reaching influence on individual trials, take up of the methodology via software and reporting of studies sector-wide, benefiting patients, trialists and research funders.
5. Sources to corroborate the impact
HTA monograph for the PROVE trial. HTA monograph: NIHR: three-arm PROVE RCT Reference to [R3] is on page 14 and Data Monitoring Committee membership is on page 114. Reference 58 extends [R4] to give uniformly minimum variance conditionally unbiased estimators.
MS Society website discussion of the MS-SMART trial: First multi-drug clinical trial in MS successfully completed
PhD Thesis, Chapter 6 describes the process of amending the FLAIR trial and gives the quotes from reviewers: The process of amending the FLAIR trial; subsequently published: Howard et al. "A platform trial in practice: adding a new experimental research arm to the ongoing confirmatory FLAIR trial in chronic lymphocytic leukaemia" Trials (2021) 22:38 – the later publication is but a peer-reviewed subset of the PhD Thesis.
Letter from the President of Cytel Inc, September 2019
Mediana R Package: Mediana R package
a. REFORM article: Medical Research Waste: Expensive to ignore, cheap to fix.
b. Link within to the underlying BMJ article which discusses the figures: Is 85% of health research really "wasted"?
BMJ, 2020, co-publications: The Adaptive designs CONSORT Extension (ACE) statement BMC Trials, June 20: BMC trials link
JAMA, April 19: Reporting of Multi-Arm Parallel-Group Randomized Trials
CONSORT statement: Extensions of the CONSORT Statement
Enhancing the QUAlity and Transparency Of health Research network: Equator network reporting guidelines
Additional contextual information
Grant funding
| Grant number | Value of grant |
|---|---|
| ENV15-1 | £32,478 |
| ENV15-2 | £12,017 |