Impact case study database
Evaluating medical revalidation to shape policy and improve doctors’ experiences of regulation
1. Summary of the impact
Research by the Collaboration for the Advancement of Medical Education Research and Assessment (CAMERA) team investigated the implementation of medical revalidation, which requires all doctors practising in the UK to periodically demonstrate that they remain up to date and fit to practise, combining continued professional development with regulatory oversight. CAMERA’s findings changed the General Medical Council’s (GMC) guidance for the revalidation of approximately 250,000 doctors from 2018 onwards. GMC revisions to the guidance for both the doctors and medical leaders involved in the process responded to critical challenges that the research identified, including: doctors’ difficulties in collecting supporting information for their annual appraisals; organisations and appraisers setting additional requirements locally; and variation in local leaders’ decision-making processes. In response to our research, the GMC clarified its requirements and supported the use of high-quality evidence within revalidation to improve consistency of the process, and its effectiveness as a tool for both regulation and professional development.
2. Underpinning research
Research into the implementation of medical revalidation has been conducted at the University of Plymouth by Julian Archer and colleagues in CAMERA since 2010, as part of a programme of research on medical regulation.
The introduction of revalidation in December 2012 represented a major change to the regulation of the medical profession in the UK. Early research led by Archer identified tensions in how policy aims were conceptualised by policymakers and professional leaders, suggesting a lack of clarity and raising questions over whether revalidation, based around annual appraisal participation, could raise standards while also identifying poor performers [3.1]. Follow-up research during the policy pilot phase found that requirements for revalidation were impacting upon formative appraisal processes, with doctors and appraisers concerned that the process was more focused on ‘box ticking’ than on professional development [3.2].
A realist review led by Dr Nicola Brennan theorised that effective appraisal – leading to behaviour change – would achieve its desired effects through three key mechanisms: dissonance, denial, or self-affirmation [3.3]. The skills of the appraiser to provide appropriate feedback and preparation for the meeting were identified as important contextual factors in this programme theory [3.3].
Later CAMERA research into revalidation built upon these findings, and centred on two major collaborative mixed methods evaluations of the nationwide implementation of revalidation. Between 2015 and 2018, Archer led an evaluation of revalidation commissioned by the General Medical Council (GMC), the UK Medical Revalidation Evaluation coLLAboration (UMbRELLA). The research also involved Dr Marie Bryce and Dr Samantha Regan de Bere, at UoP, and collaborators at University College London, University of Manchester, NHS Education for Scotland, Health Improvement Scotland, the Belfast Health and Social Care Trust, and the Wales Deanery. Through statistical analyses of existing data, surveys, interviews, and documentary analyses, the study found that the requirement to submit information across six categories had caused the collection of evidence to be prioritised as a goal in itself and had consequently shifted the focus away from the quality of the information and its usefulness in supporting professional development through effective appraisals [3.4]. Furthermore, the research identified significant scepticism about the value of the written reflections that doctors produce for their appraisal portfolios [3.4]. Some groups of doctors experienced difficulties collecting some types of supporting information, such as psychiatrists [3.5], who often found collecting requisite patient feedback challenging. The research found that revalidation requirements were unclear to some doctors, raising questions about the clarity of guidance provided by the GMC, and revealing that employing organisations and appraisers sometimes added their own requirements or interpretations of the guidance to those of the regulator, causing confusion for doctors.
Additionally, CAMERA researchers (Archer, Bryce, Price) contributed to a second collaborative evaluation of the organisational impacts of revalidation led by Professor Kieran Walshe, University of Manchester, also collaborating with the University of York. This research identified that revalidation had caused organisations to strengthen governance processes around doctors’ performance, including using appraisal for performance management, and also highlighted the importance of Responsible Officers – who make revalidation recommendations – as emergent medical leaders while identifying some variations in their approaches to decision-making [3.6].
3. References to the research
CAMERA’s research has been presented at key international conferences to both academic audiences (OTTAWA Conference on the Assessment of Competence in Medicine and the Healthcare Professions, 2018) and to policymaker and practitioner audiences (International Association of Medical Regulatory Authorities conference, 2018; International Society for Quality in Healthcare conference, 2017).
The research has been published in prestigious journals, including the two highest ranked medical education journals Academic Medicine (IF 5.354) and Medical Education (IF 4.57).
Archer J, Regan de Bere S, Nunn S, Clark J, Corrigan O. 2015. "No one has yet properly articulated what we are trying to achieve": a discourse analysis of interviews with revalidation policy leaders in the United Kingdom. Academic Medicine, 90/1: 88-93. https://dx.doi.org/10.1097/ACM.0000000000000464
Archer J, Nunn S & Regan de Bere S. 2017. 'The McDonaldization of appraisal? Doctors' views of the early impacts of medical revalidation in the United Kingdom.' Health Policy, 121/9: 994-1000, http://dx.doi.org/10.1016/j.healthpol.2017.07.006
Brennan N, Bryce M, Pearson M, Wong G, Cooper C , Archer J. 2017. ‘Towards an understanding of how medical appraisal produces its effects: A realist review.’ Medical Education, 51: 1002-1013, http://dx.doi.org/: 10.1111/medu.13348
Wakeling J, Holmes S, Boyd A, Tredinnick-Rowe J, Cameron N, Marshall M, Bryce ME & Archer J. 2019. 'Reflective practice for patient benefit: an analysis of doctors' appraisal portfolios in Scotland', Journal of Continuing Education in the Health Professions, 39/1: 13-20, http://dx.doi.org/10.1097/CEH.0000000000000236
Baines R, Zahra D, Bryce M, Regan de Bere S, Roberts M, Archer J. 2019. ‘Is collecting patient feedback ‘a futile exercise’ in the context of recertification?’, Academic Psychiatry, 43: 570-576, http://dx.doi.org/10.1007/s40596-019-01088-w
Bryce M, Luscombe K, Boyd A, Tazzyman A, Tredinnick-Rowe J, Walshe K , Archer J. 2018. ‘Policing the profession? Regulatory reform, restratification and the emergence of Responsible Officers as a new locus of power in UK medicine.’ Social Science & Medicine, 213: 99-105. https://doi.org/10.1016/j.socscimed.2018.07.042
4. Details of the impact
.
CAMERA’s large programme of research has provided empirical evidence of how medical professionals experienced revalidation during its first cycle of operation, including identifying what worked well but also the aspects that had been challenging. These findings influenced revisions made by the GMC to its guidance about revalidation processes, with updated guidance for the UK’s quarter of a million doctors issued in May 2018 [5.1], for Responsible Officers in March 2018 [5.2], and further changes to guidance for organisations and in other related materials [5.3]. UMbRELLA’s findings were reported by professional and policy-focused media outlets, including the BMJ and HSJ, informing debates about the merits of revalidation. The Chair of the GMC used UMbRELLA’s findings to emphasise the importance for doctors of collecting and reflecting on patient feedback, citing our finding that ‘ where patients submitted feedback, a majority of doctors found it to be the most helpful type of supporting information to help them reflect on their practice’ [5.4].
However, UMbRELLA’s main impact has resulted from its findings on the challenges faced by doctors when collecting supporting information for appraisal and revalidation, including difficulties for some groups in collecting patient feedback. The GMC has publicly cited UMbRELLA’s findings as having informed the revision of its guidance for doctors on the supporting information requirements for revalidation. In a press release in May 2018 entitled ‘ GMC updates guidance to help doctors with appraisals and revalidation’ [5.5], the GMC recognised that ‘UMbRELLA's findings, in ‘Evaluating the regulatory impact of medical revalidation’ [5.6], included a concern about ‘supporting information’. The GMC’s guidance was changed in several ways in response to specific UMbRELLA findings. For example, following the finding that some groups of doctors found collecting patient feedback difficult, the GMC ‘created five new patient feedback case studies [5.7] which provide practical advice and show how doctors in certain situations collected feedback successfully’ [5.3].
UMbRELLA’s impacts on GMC policies and processes were achieved through direct engagement with policymakers. Throughout the study, emerging findings were regularly reported to the GMC and the Department of Health through a joint Revalidation Research Advisory Board. As a result, the research influenced the review and redevelopment of aspects of revalidation processes and procedures, notably the guidance provided to doctors by the GMC.
UMbRELLA’s interim findings informed an independent review of revalidation policy led by Sir Keith Pearson, to whom CAMERA researchers provided verbal and written testimony [5.8]. Consequently, the Pearson review’s final report [5.9] contained multiple references to the interim findings of the UMbRELLA research [5.10]. One key recommendation for the GMC from the Pearson review was that the medical regulator should ‘ Update guidance on the supporting information required for appraisal for revalidation to make clear what is mandatory (and why), what is sufficient, and where flexibility exists. Ensure consistency and compatibility across different sources of guidance’ [5.9]. Just as this recommendation was influenced by UMbRELLA research, so too was the GMC’s response as it produced revised revalidation guidance. In order to address the concerns about burden identified in UMbRELLA research, the revised guidance emphasises the importance of quality over quantity, and proportionality [5.3]. This revision sought to ensure that doctors feel able to identify supporting information that can aid their reflective activities as part of appraisal and that informs their professional development.
Furthermore, in response to the UMbRELLA finding that local organisational requirements can influence doctors’ experiences of requirements for collecting supporting information and that these can go beyond the GMC’s own requirements, the GMC has included a specific section clarifying the relationship between local expectations and mandatory national regulatory requirements [5.1, 5.3]. The new section distinguishes the GMC’s regulatory requirements and explains that while employing organisations may set other additional appraisal or contractual requirements, these should not influence revalidation outcomes except in exceptional circumstances [5.1].
Both revalidation evaluation studies found that the ways in which Responsible Officers made decisions about doctors locally varied and the recommendations they make to the GMC about individuals’ revalidation outcomes may be impacted by this variation in resources, information and local systems. The GMC testimonial notes that they have improved their protocol for Responsible Officers on making revalidation recommendations [5.2], plus further guidance for organisations, in order to address the issues identified through our programme of research [5.3].
Our research has therefore shaped the national medical regulator’s actions in developing several aspects of its flagship policy. The GMC has revised its guidance in order to clarify its expectations and to improve the experience of collecting supporting information and participating in appraisal and revalidation for the c.250,000 doctors licensed to practise in the UK who are required to participate in this important regulatory process.
5. Sources to corroborate the impact
5.1. General Medical Council. 2018. Guidance on Supporting Information for appraisal and revalidation.
5.2. General Medical Council. 2018. The GMC protocol for making revalidations recommendations: Guidance for responsible officers and suitable persons. https://www.gmc-uk.org/-/media/documents/Responsible_Officer_Protocol.pdf_56096180.pdf
5.3. Testimonial from GMC on impact of UMbRELLA research.
5.4. Professor Sir Terence Stephenson, Chair of the General Medical Council 2015-2018, blog.
https://blogs.bmj.com/bmj/2018/01/12/terence-stephenson-doctors-shouldnt-fear-patient-feedback/
5.5. General Medical Council, news release. https://www.gmc-uk.org/news/media-centre/media-centre-archive/gmc-updates-guidance-to-help-doctors-with-appraisals-and-revalidation
5.6. Archer J, Cameron N, Lewis M, Marshall M, O'Hanlon J, Regan de Bere S, Walshe K, Baines R, Boyd A & Bryce M et al. 2018. UMbRELLA: Evaluating the regulatory impacts of medical revalidation – final report. General Medical Council. https://www.gmc-uk.org/-/media/documents/umbrella-report-final_pdf-74454378.pdf
5.7. GMC. Revalidation resources – case studies for doctors with patients who can’t give feedback or who have limited patient contact. https://www.gmc-uk.org/registration-and-licensing/managing-your-registration/revalidation/revalidation-resources#revalidation-patient-case-studies
5.8. Revalidation Advisory Board, minutes. https://www.gmc-uk.org/-/media/documents/minutes-of-the-meeting-on-9-june-2016_pdf-68986152.pdf
5.9. Pearson, K. Taking revalidation forward. https://www.gmc-uk.org/-/media/documents/Taking_revalidation_forward___Improving_the_process_of_relicensing_for_doctors.pdf_68683704.pdf
5.10. Archer J et al. 2016. UMbRELLA interim report. General Medical Council. https://www.gmc-uk.org/-/media/documents/UMbRELLA_interim_report_FINAL.pdf_65723741.pdf
Additional contextual information
Grant funding
Grant number | Value of grant |
---|---|
n/a | £123,908 |