Impact case study database
UoP03Ethics: Transforming health research governance and ethics in the UK National Health Service and Ministry of Defence
1. Summary of the impact
This Impact Case Study demonstrates how original research undertaken at the University of Portsmouth has directly benefited the National Health Service (NHS) and Ministry of Defence (MOD) research ethics and governance processes. The work was highlighted in the House of Commons Science and Technology Select Committee’s 2018 report on Research Transparency. It was the basis for the Health Research Authority’s ‘Make it Public’ Transparency Strategy, and a major revision of the MOD Joint Service Publication 536 (JSP536) governing defence human participant research. Updated policies have led to measurable improvements in the transparency of research and efficiency of the research approvals process, specifically cutting average review times by one third within the MOD. It has also enabled the MOD to comply with clinical trial regulations, directly impacting researchers and research participants across a wide range of medical and military related topics, and covering many millions of research investment.
2. Underpinning research
Building on a background in medical science, Dr Simon Kolstoe’s research focuses on the efficiency and suitability of research governance, ethics and integrity processes, and the role of Research Ethics Committees (RECs) in promoting research transparency and data sharing. The body of research that underpins this case study has used primary research, as well as novel, peer-reviewed, published audit and service evaluations, to create an evidence base for directing policy renewal in two UK government departments.
Improving research transparency through improvements in governance
Rigorous, reliable and trustworthy publication and dissemination practices are essential for both the transparency and efficiency of research, especially given that it is estimated that up to 50% of research is wasted due to non-publication ( The Lancet, 2009, 374(9683): 86-89). In 2014, Kolstoe used projects submitted through an NHS REC to compare project outcomes as reported to ethics committees with those stated in subsequent publications, and to assess if ethics committee records could be used to accurately determine publication rates within a health-related context ( G1). This novel analysis established a publication rate of 32% (in peer reviewed journals), along with a 57% inconsistency in reported outcomes between original ethics committee applications and final published papers ( R1). This was the first time this had been directly measured for NHS research. This research evidenced the scale of publication and outcome bias in a UK health research context and, importantly, demonstrated the potential for ethics and governance processes to support higher levels of transparency in UK research.
Clinical trials play an important role for the development of health information and research for public health care. Since 2013, it has been a UK policy condition of a favourable REC opinion that all clinical trials are registered on a publically accessible database. In accordance with the Declaration of Helsinki, this should occur before the first participant is recruited. Un-registered, non-reported and mis-reported clinical trials risk hiding unsuccessful or adverse results, distort both the scientific literature and the evidence used for clinical decision making and can represent a significant waste of public and private research funding. In 2017/18, Kolstoe examined the levels of public registration of clinical trials using data from the Integrated Research Application System (IRAS). The IRAS system covers applications for permissions and approvals for all health and social care research in the UK. This was the first study to look specifically and comprehensively at UK trials using records held by the Health Research Authority (HRA) and included a primary follow-up of non-registering chief investigators. This research identified the ‘true’ trial registration rate and demonstrated that, despite researchers and sponsors being reminded that registration of clinical trials is a mandatory condition of the REC favourable opinion, one-fifth of clinical trials had either not been registered, or their registration could not easily be found, 14 to 20 months after receiving the favourable opinion letter ( R2).
The use and mining of large scale datasets is rapidly becoming central to medical research. Consequently, many researchers are establishing databases and gaining consent from patients and the public to store data for long periods, with the promise of it being shared for use in multiple research projects. Indeed, there is an implicit assumption that research databases will generate many more publications than a normal research project. In 2018, Kolstoe turned his attention to larger medical and health datasets constructed using NHS patient data in order to test this assumption and to benchmark UK performance with other national studies. He analysed the extent of data sharing and number of publications arising from a total of 354 UK research databases listed on the HRA’s Assessment Review Portal ( G2). This study showed that only a third of databases registered with the NHS had shared their data, and only 40% had produced a publication ( R3).
Research Ethics Committee efficiency and consistency
The HRA has adopted a consistency improvement plan, including a process called “Shared Ethical Debate” (ShED), where multiple committees review the same project. In 2016, Kolstoe compared the consistency of outcomes of two ShED exercises and with a “mystery shopper” exercise, where the research team presented the same study to twelve NHS RECs ( G3). This work devised an original scoring method to quantify inconsistencies between committees and demonstrated the central role that RECs play in monitoring research transparency ( R4).
To pursue the ideas developed through this novel research, Kolstoe has actively engaged with national ethics and governance systems as Chair of the Hampshire A NHS REC (July 2014 to present), Chair of the MODREC (April 2016 to present) and as a member of the HRA Confidentiality Advisory Group (December 2018 to present). He is also a member of the HRA Emergency COVID-19 REC considering all human infection challenge vaccine studies, and chair of Public Health England’s Regulation and Governance Group (REGG). These national positions have provided opportunities to access and engage with policy makers and to support the delivery of significant policy change.
3. References to the research
3.1. Research outputs
R1 . Begum, R., & Kolstoe, S. (2015). Can UK NHS research ethics committees effectively monitor publication and outcome reporting bias? BMC Medical Ethics, 16, 51. https://doi.org/10.1186/s12910-015-0042-8
R2 . Denneny, C., Bourne, S., & Kolstoe, S. E. (2019). Registration audit of clinical trials given a favourable opinion by UK research ethics committees. BMJ Open, 9(2), [e026840]. https://doi.org/10.1136/bmjopen-2018-026840
R3 . Trace, S., Bracher, M., & Kolstoe, S. E. (2020). Determining the level of data sharing, and number of publications, from research databases that have been given a favourable opinion by UK research ethics committees. BMJ Open, 10(9), e039756. https://doi.org/10.1136/bmjopen-2020-039756
R4 . Trace, S., & Kolstoe, S. E. (2017). Measuring inconsistency in research ethics committee review. BMC Medical Ethics, 18(1), [65]. https://doi.org/10.1186/s12910-017-0224-7
3.2. Evidence of the quality of research
These outputs are a representative selection of related work. All employ robust design, appropriate research techniques and are published in respected peer-reviewed academic journals that are relevant to the discipline. R1 has been cited in UK Parliamentary debate and R1 and R4 have been cited internationally (Canada, Finland, New Zealand, Singapore and USA).
3.3. Relevant grants
G1. Kolstoe, S. Audit of Scientific Publications by Projects given favourable opinions by the Southampton A REC. Funded by Health Research Authority, 09/2013 - 09/2014 (GBP8,000)
G2. Kolstoe, S. Audit of Publications Resulting From HRA Approved Research Database. Funded by the Health Research Authority, 01/2018 - 12/2018 (GBP10,000)
G3. Kolstoe, S. Consistency in ethical review: analysis of ShED19 and ShED20. Funded by the Health Research Authority, 03/2016 - 02/2017 (GBP8,333)
G4. Kolstoe, S. Consultancy for MODREC. Funded by the Ministry of Defence, 05/2016 - 03/2017 (GBP14,300)
G5. Kolstoe, S. Ministry of Defence Ethics and Governance. Funded by the Ministry of Defence, 08/2017 - 07/2019 (GBP44,720).
4. Details of the impact
It is widely reported that 85% of health research is wasted because ‘ it asks the wrong questions, is badly designed, not published or poorly reported’ ( The Lancet, 2009, 374(9683): 86-89). Research ethics committees (RECs) review research proposals and give an opinion about whether the research is ethical. Reviews by RECs normally occur after funding has been granted and protocols have been developed, but prior to study recruitment and data gathering. Failure to achieve a “favourable ethics opinion” can prevent a study progressing. This makes ethics committees a key gatekeeper of research, with the opportunity to influence research conduct so as to reduce research waste and enhance transparency. National Health Service (NHS) RECs consider and approve the ethical acceptability of research involving NHS staff and/or patients, as well as other health-related research. There are more than 60 NHS RECs across the UK and they review around 6,000 project applications each year. The Health Research Authority (HRA), an arm’s-length body situated in England’s Department of Health, takes a leading and coordinating role for managing RECs throughout the UK. The HRA’s mission is to promote and protect the interests of patients, harmonise and streamline regulation, and promote transparency in health and social care research.
4.1. Enhancing the transparency of research in the United Kingdom
Research transparency is central to ethical research practice. Health and social care research studies should be registered and the results made public, so that participants are protected from unnecessary studies, use of research funding is maximised and the greatest public benefit is delivered. Non-publication of research is the main cause of research waste (50%). Kolstoe’s research on reporting bias and non-registration of clinical trials has led to government policy change, specifically a significant extension of the mandate and role of the UK Health Research Authority in relation to monitoring clinical trials transparency. Following the primary publication ( R1), the topic of whether RECs should police reporting bias featured in a BMJ “Head to Head” debate ( https://www.bmj.com/content/356/bmj.j1501, 27/03/2017). Subsequently, Kolstoe submitted written ( S1) and oral evidence ( S2) to the 2017 House of Commons Science and Technology Select Committee (STSC) inquiry on research integrity. This evidence:
identified reporting bias as a research integrity issue;
highlighted a case where non publication of clinical trials results had led to significant wasted public expenditure (GBP424,000,000 spent on Tamiflu, based on incomplete evidence);
proposed that the Health Research Authority was best placed to support research ethics committees in monitoring publications arising from projects and ensuring clinical trials transparency.
In view of this evidence, the STSC considered the issue of clinical trials transparency to be so significant for public health that it recommended it be considered separately and an additional report, ‘ Research integrity: clinical trials transparency’ was published in October 2018 ( S3). This report repeatedly referenced Kolstoe’s research alongside recommendations that the HRA establish a national audit programme, modelled on Kolstoe’s work. The HRA was also instructed to publish information on trials that have received ethical approval but are not registered in a publicly accessible register, and introduce a system of sanctions to drive improvements in clinical trials transparency.
In February 2019 and coinciding with the publication of R2, the HRA established the Research Transparency Strategy Group, with Kolstoe as a member. Following drafting and public consultation, the HRA approved the HRA ‘ Make it Public’ Transparency Strategy, which was published in July 2020 ( S4). The HRA ‘ Make it Public’ Strategy extended the role of the HRA in relation to monitoring clinical trials transparency and directly addressed the Science and Technology Committee recommendations by committing the HRA to:
assume responsibility for registering clinical trials on behalf of the sponsor, using data that provided for study approval;
require the submission of a final report within 12 months of study completion; and
use information in the final report to measure research transparency and take action in cases of non-compliance.
As such, the ‘Make it Public’ Strategy ‘ extends the role of the HRA in relation to monitoring clinical trials transparency and represents a significant advance in our ability to support high-quality health and social care research and to promote the interests of patients and the public’ ( S5).
The HRA also published their commitment to conduct ongoing audits of clinical trials registration, using the method developed by Kolstoe, to monitor legal and policy compliance ( S6). Additionally, in response to the results of R3, the HRA have incorporated data access arrangements into a revision of their ethics review form for Research Databases ( S5).
4.2. Improving the consistency and efficiency of NHS ethics committees
In response to the findings from R4, the Director of Approvals Service at the HRA halted the Shared Ethical Debate (ShED) consistency exercises in 2019, and reinstated them in a new format in 2020, with a new focus on using the exercise as a learning opportunity and encouraging members to explore decision-making ( S5), following suggestions published in R4.
4.3. Enhancing the capability and efficiency of research in the UK Ministry of Defence
The Ministry of Defence Research Ethics Committee (MODREC) ensures that all research involving human participants undertaken, funded or sponsored by MOD meets nationally and internationally accepted ethical standards. It is recognised by the United Kingdom Ethics Committee Authority (UKECA) to review clinical trials of investigational medicinal products involving subjects who are UK Armed Forces personnel recruited in a military setting, as well as Phase 1 trials in healthy volunteers conducted by the MOD. It is also recognised as an Appropriate Body under the Mental Capacity Act 2005 for review of research involving UK Armed Forces personnel who are unable to consent for themselves (commonly research in emergency/battlefield situations).
On the basis of his research and experience of NHS RECs , Kolstoe was appointed chair of the MODREC in 2016, and was tasked by the Surgeon General with an evidence based modernisation of the committee, with the goal of streamlining research governance and approvals within the MOD ( G4). Kolstoe rapidly identified a gap in the MOD’s research governance arrangements in relation to the legal requirements for ethics review and, subsequently, represented the MOD in negotiations with the UKECA. These negotiations ensured that the MOD was able to continue to review and approve studies falling under the Clinical Trials Regulations 2004, Mental Capacity Act 2005 and Human Tissue Act 2004. Without this intervention, the MOD would have not been authorised to approve and conduct these types of research and the disruption to high profile and time-sensitive research would have been significant. For example, in 2014 the MOD directed GBP20,000,000 to a five-year, physiological research programme to understand the health effects of combat roles on men and women. This delivered significant changes within the British Army: data from the ‘Women in Ground Close Combat’ research team underpinned the Defence Secretary’s decision to open Ground Close Combat (GCC) roles to women in July 2016, and new role-related, rather than gender- or age-specific, Physical Employment Standards for GCC roles were introduced in April 2019. Any lapse in the ability of the MOD to conduct research with human participants would have jeopardised this, and other, research programmes of national significance.
The agreement brokered by Kolstoe required the subsequent harmonisation of the MOD human research governance policy (JSP536) with two national policies: the ‘UK Policy Framework on Health and Social Care Research’ and ‘Governance Arrangements for Research Ethics Committees’. A contract was awarded to the University of Portsmouth to complete this critical work ( G5, S7), and the new policy was published in December 2019 ( S8). The revised MOD policy was introduced in January 2020 and provided a Proportionate Review Service which allows for research proposals that present ‘no material ethical issues’ to be reviewed and approved via an executive sub-committee. ‘This revision has hugely improved and clarified the MODREC application process for researchers’ ( S7). As a result, the MODREC approval process has become more streamlined and the average time for MODREC project approval has reduced by 35%, from 32 days to 21 days. In addition, the capacity of the committee to review studies has increased by 40% ( S9) with no additional cost. As the annual governance budget is ~GBP350,000, this could be considered as a GBP140,000 efficiency saving for the department in 2020 alone, compared to the previous three years.
5. Sources to corroborate the impact
S1. ‘ Ethics committees, managed though the Health Research Authority (HRA), are key for monitoring research integrity’. Written evidence submitted by Dr Simon Kolstoe to House of Commons Science and Technology Committee on Research Integrity, 03/2017 (RIN0022) http://data.parliament.uk/writtenevidence/committeeevidence.svc/evidencedocument/science-and-technology-committee/research-integrity/written/48484.html
S2. UK Parliament House of Commons Science and Technology Committee Oral evidence: Research integrity, HC 350, 04/12/2017. Questions 277- 360. Written transcript of witness statements http://data.parliament.uk/writtenevidence/committeeevidence.svc/evidencedocument/science-and-technology-committee/research-integrity/oral/75580.html
S3. UK Parliament House of Commons Science and Technology Committee. (2018) Research integrity: clinical trials transparency. Tenth Report of Session 2017–19: report, together with formal minutes relating to the report, ordered by the House of Commons to be printed 23 October 2018. [HC, Session 2017-19] https://publications.parliament.uk/pa/cm201719/cmselect/cmsctech/1480/1480.pdf Six direct references in the body of the report to Dr Kolstoe’s underpinning research.
S4. ‘ Make it Public: Transparency and openness in health and social care research’. Health Research Authority Transparency Strategy, 07/2020 https://s3.eu-west-2.amazonaws.com/www.hra.nhs.uk/media/documents/8828_transparency_strategy_2020_V4.pdf and https://www.hra.nhs.uk/planning-and-improving-research/policies-standards-legislation/research-transparency/make-it-public-transparency-and-openness-health-and-social-care-research/. Dr Kolstoe named as a co-author of the new HRA strategy.
S5. Statement from Director of the Approvals Service, Health Research Authority, confirming Kolstoe’s key role in the establishment of the MiP strategy and value of SK’s research to the HRA in discharging its responsibilities for supporting research transparency, 10/12/2020.
S6. Health Research Authority’s commitment to ongoing audits of clinical trial registration: https://www.hra.nhs.uk/planning-and-improving-research/research-planning/research-registration-research-project-identifiers/.
S7. Statement from Surgeon-General of the British Armed Forces, Defence Medical Services, confirming Kolstoe’s involvement in re-drafting JSP 536, 11/12/2020.
S8. ‘Defence research involving human participants’ (JSP 536), UK Ministry of Defence, Directive and Guidance, Version 3.1, 03/2020: https://www.gov.uk/government/publications/defence-research-involving-human-participants-jsp-536
S9. Data on MODREC project approvals: 2017 - 2019 vs 2020 (pre- and post- JSP 536 revision) from MODREC Annual report, 01/2021.
Additional contextual information
Grant funding
Grant number | Value of grant |
---|---|
N/A | £8,000 |
N/A | £10,000 |
N/A | £8,333 |
N/A | £14,300 |
N/A | £44,720 |