Impact case study database
Avoiding unintended consequences of healthcare interventions: A large-scale evaluation halted the national rollout of a predictive risk stratification tool in primary care in Wales
1. Summary of the impact
Predictive risk stratification tools to identify those at highest risk of emergency admission to hospital and thereby reduce the level of those emergency admissions have been widely promoted in UK and international policy. The national rollout of a web-based risk stratification tool (PRISM) for use by general practitioners (GPs) was halted following the results of Swansea University’s study: PRISMATIC. This large-scale trial in the general population (n = >230,000) showed that the introduction of the PRISM software led to unexpected increases in emergency admissions, days spent in hospital and costs, without benefit to quality of life. Close collaboration between Swansea University researchers and policy makers in Wales prevented the implementation of PRISM, which did not have an evidence base of effectiveness and had unintended adverse consequences for patients and the NHS in practice. Non-implementation of PRISM avoided additional emergency admissions (27,690 annually) and hospitalisation days (75,815 annually) for patients in Wales and resulted in cost savings for the NHS (GBP201,000,000 annually).
2. Underpinning research
Although an evidence-based approach is the ideal model for planning and delivering healthcare, barriers exist to using research evidence to implement and evaluate service change. In 2013, Swansea University’s Health Services Research (HSR) group conducted a national email survey of health service commissioners at the most devolved level of decision-making in Wales (Local Health Boards – LHBs) followed by in-depth interviews with representatives of LHBs, purposively selecting five to reflect geographic and economic characteristics. This programme of work included qualitative work to understand barriers and facilitators in implementing policy and research evidence in chronic conditions management at local and regional levels. The research exposed a gap between evidence-based aims of national health policy and how health services are commissioned, implemented, and evaluated at local level (R1).
It was against this background that the Predictive Risk Stratification Model (PRISM) tool was introduced in Wales, designed to allow GP staff to be able to view risk scores for patients across the spectrum of risk of emergency admission to hospital during the following year1. Routine data used to generate scores included inpatient, outpatient and general practice data, alongside a deprivation index. The full PRISM intervention comprised the software, a user friendly handbook, 2 hours of practice-based training, clinical support through two locally appointed GP champions’ and a ‘help desk’ accessible by telephone or email. At the time, there was research evidence on the accuracy of emergency admission risk prediction tools but there was a lack of evidence regarding how well predictive risk tools work in supporting the management of patients. A study was designed by the HSR group to provide information on costs and effects of PRISM; how it was used in practice, barriers and facilitators to its implementation; and its perceived value in supporting the management of patients with and at risk of developing chronic conditions (R2).
From the outset, the research group aimed to understand what might be needed to bring PRISM into effective use by exploring clinician’s and practice managers’ attitudes and expectations about using it. The group conducted 4 focus groups and 10 interviews with a total of 43 primary care doctors and colleagues from 32 general practices. The researchers found that policy imperatives and the pressure of rising demand meant respondents were open to trying out PRISM, despite underlying uncertainty about what difference it could make (R3).
Between 2012 and 2015, the HSR group conducted a randomised trial to evaluate the PRISM intervention (PRISMATIC, ISRCTN55538212). The trial included the outcomes of 230,099 participants registered to 32 general practices in the Swansea area who received the intervention in random clusters over 52 weeks. The approach was innovative in two ways:
the use of routine data from the Secure Anonymised Information Linkage (SAIL) databank to compare services delivered to patients (emergency, acute, primary, community and social care) across the spectrum of risk between intervention and control practices. SAIL includes routine Welsh hospital data such as emergency admissions secondary care as well as GP practice data. The linked data approach allowed for the inclusion of routine outcomes for everyone registered for participating general practices without their explicit provision of consent based on a privacy protection methodology developed at Swansea University by Professor Lyons (R4).
the use of a progressive cluster randomised stepped wedge trial design, ensuring that all participating practices had the opportunity to implement and use the intervention during the study period. As the trial progressed, the number of intervention practices increased and the number of control practices fell (Fig . 1).
Figure 1 Randomised multiple interrupted time series study design overview (R2)
The trial outcomes were wholly unexpected, with the introduction of PRISM increasing emergency episodes, hospitalisation and costs across, and within, risk levels without clear evidence of benefits to patients, findings that were contrary to the views of practitioners and UK policy (R5, R6).
1 Wales predictive model, final report and technical documentation. D Wennberg, M Siegel, R Stephens - Prepared for NHS Wales, Informing healthcare, 2008 https://www.yumpu.com/en/document/view/32003147/wales\-predictive\-model\-final\-report\-and\-technical\-documentation
3. References to the research
All papers represented are published in peer reviewed journals and have been supported by NIHR and Welsh government. R5 has been submitted to REF2021. PRISMATIC findings were published in both an NIHR Journal Series monograph and BMJ Quality and Safety (R5 and R6). The Editor’s choice article prompted the publication of an editorial and the paper was ranked number 2 in their “Top 10” articles of 2019.
R1. Evans BA, Snooks H , Howson H, Davies M . How hard can it be to include research evidence and evaluation in local health policy implementation? Results from a mixed methods study. Implementation Sci 8, 17 (2013). doi:10.1186/1748-5908-8-17 cited 72 times 17.11.2020 https://implementationscience.biomedcentral.com/articles/10.1186/1748\-5908\-8\-17.
R2. Hutchings HA, Evans BA, Fitzsimmons D, Harrison J, Heaven M, Huxley P, Kingston MR, Lewis L, Phillips CJ, Porter AM, Russell IT, Sewell B, Warm D, Watkins A and Snooks HA. Predictive risk stratification model: a progressive cluster-randomised trial in chronic conditions management (PRISMATIC) research protocol. Trials 14, 301 (2013) doi:10.1186/1745-6215-14-301.
R3. Porter A, Kingston MR , Evans BA, Hutchings H , Whitman S, Snooks H. It could be a 'Golden Goose': a qualitative study of views in primary care on an emergency admission risk prediction tool prior to implementation. BMC Fam Pract. 2016 Jan 6;17:1. doi: 10.1186/s12875-015-0398-3.
R4. Lyons RA , Jones KH, John G, Brooks CJ, Verplancke J-P, Ford DV, Brown G, Leake K. The SAIL databank: linking multiple health and social care datasets. BMC Medical Informatics and Decision Making. 2009;9(1):3 doi: 10.1186/1472-6947-9-3.
R5. Snooks H , Bailey-Jones K, Burge-Jones D, Dale J, Davies J, Evans BA, Farr A, Fitzsimmons D, Heaven M, Howson H, Hutchings H, John G, Kingston M, Lewis L, Phillips C, Porter A, Sewell B, Warm D, Watkins A, Whitman S, Williams V, Russell I. Effects and costs of implementing predictive risk stratification in primary care: a randomised stepped wedge trial BMJ Quality & Safety 2019;28:697-705. doi:10.1136/bmjqs-2018-007976.
R6. Snooks H , Bailey-Jones K, Burge-Jones D, Dale J, Davies J, Evans B, Farr A, Fitzsimmons D, Harrison J, Heaven M, Howson H, Hutchings HA, John G, Mark Kingston, Leo Lewis, Ceri Phillips, Alison Porter, Bernadette Sewell, Daniel Warm, Alan Watkins, Shirley Whitman, Victoria Williams, and Ian T Russell. Predictive risk stratification model: a randomised stepped-wedge trial in primary care (PRISMATIC). Health Serv Deliv Res 2018;6(1). doi:10.3310/hsdr06010.
Grants supporting the underpinning research at Swansea University:
G1 PI: Helen Snooks, Swansea University,” Implementation of the Framework for Research and Evaluation related to the Model of Chronic Conditions Management in Wales” Welsh Government, 2008 – 2012, GBP241,129.
G2 PI: Helen Snooks, Swansea University, “How do people with chronic conditions experience care in Wales?”, Wales Office of R&D (WORD), 2008, GBP13,300.
G3 PI: Helen Snooks, Swansea University, “Chronic Conditions Management Research and Evaluation Advice and support”, Wales Office of R&D (WORD), 2009 – 2012, GBP9,956.
G4 PI: Helen Snooks, Swansea University, “Baseline study of CCM in Wales – supplementary reports” WORD and Caerphilly and Rhondda Cynon Taf Local Health Boards, 2009, GBP10,512.
G5 PI: Helen Snooks, Swansea University “Predictive risk stratification: impact on care for people with or at risk of chronic conditions” 09/1801/1054, NIHR, HSDRP 2010 – 2015, GBP691,101.
G6 PI: Mark Kingston, Swansea University “Emergency Admission Risk Prediction survey” Abertawe Bro Morgannwg University Health Board, 2014 – 2016, GBP11,000.
G7 PI: Mark Kingston, Swansea University “Emergency Admission Risk Prediction Qualitative study” Abertawe Bro Morgannwg University Health Board, 2016 – 2018, GBP7,000.
4. Details of the impact
Tackling the increasing burden of emergency hospital admissions is a major policy goal in the UK and internationally. Ageing populations, the increasing prevalence of chronic diseases and risk-averse practitioner behaviour underlie unmanageable increases in emergency admissions, leading to increased costs, risks associated with inpatient stays and difficulties with patient flow. As part of a move from the inefficient provision of care within ‘silos’ of medical disciplines, such as cardiac, respiratory, or gastrointestinal medicine, stratification of general practice populations by risk of emergency admission has been widely promoted. This new approach allows proactive assessment and care for the whole person, rather than the reactive treatment of patients following crises. Implementation and use of predictive risk stratification in primary care has been incentivised in UK policy through targets and payments. The aim of this policy has been to enable the delivery of targeted interventions for people at high risk of emergency admission to hospital and thus to reduce these admissions, and also reduce pressure on the acute sector.
From 2008, Professor Helen Snooks led a programme of policy research and evaluation support commissioned by the Welsh Government and established a close working relationship with colleagues responsible for the development and implementation of the Chronic Conditions Management policy in Wales (G1 – G4). This programme of work led to the development of a successful application for research funding led by Snooks, in partnership with the Welsh Government and local NHS collaborators, to evaluate the introduction of predictive risk stratification in primary care in one area of Wales (G5). At the time of the initiation of this evaluation, a national roll out of the PRISM software to all GPs across Wales had been planned for April 2010 but was paused in 2011 pending the PRISMATIC trial outcomes (C1). Media interest was high throughout the study, including a feature on BBC Wales (C2).
The trial findings (R5, R6) showed that the implementation of PRISM was associated with increases in:
emergency hospital admissions by 1%,
emergency department (ED) attendances by 3%,
outpatient visits by 5%,
proportion of days with recorded GP activity by 1%,
days spent in hospital by 3%.
NHS costs per participant of GBP76.00 per year.
The unintended consequences of the PRISM intervention highlighted by the PRISMATIC study were hugely significant in terms of NHS usage and costs.
When the findings were released, the roll out was halted altogether in Wales. In 2020, the Deputy Director of Primary Care in the Welsh Government stated “The trial results indicated that effects were unanticipated and in the opposite direction to those sought. The work concluded that caution needs to be exercised in using predictive risk tools at an individual patient level to support clinical decision making. This is a useful piece of research for consideration in decision making and planning. As a result, the PRISM tool was not rolled out more widely in the Welsh health system” (C3).**
The 2020 follow-up survey results show that the impact of the research in Wales has been high, with only 14% of general practices having access to emergency admission predictive risk tools in Welsh Health Boards compared to over 80% across the UK (C4).
PRISMATIC has had significant impacts on health policy, patient care and NHS costs in Wales. The research findings showed that the introduction of predictive risk stratification in primary care had the opposite of the intended effect, increasing emergency admissions to hospital (primary outcome) and the use of emergency, primary and outpatient services. Extrapolating the study findings to the population of Wales, it is estimated that the non-implementation of PRISM in Wales has avoided approximately 30,000 admissions to hospital and 76,000 hospitalisation days per year, thus avoiding incurred annual costs of GBP2,000,000.
Note: A workshop/seminar had been planned in 2020 to disseminate the findings across the other nations of the UK but it was cancelled due to the COVID-19 pandemic.
5. Sources to corroborate the impact
C1 Letter from the Medical Director, Department for Public Health and Health Professions, NHS Wales, confirming the halt to roll-out of PRISM, Oct 2011.
C2 BBC Wales “Preventing emergencies: Swansea team pioneer trial of new NHS system” Nov 2013.
C3 Confirmation letter from the Deputy Director, Primary Care, Welsh Government on the influence of the PRISMATIC findings on health policy in Wales, 14.2.2020.
C4 Kingston M, Hutchings H, Griffiths R, Porter A, Russell I, Snooks H. Emergency Admission Risk Stratification in UK primary care: national survey of access and use. Peer reviewed article published in British Journal of General Practice 21 September 2020; DOI: 10.3399/bjgp20x712793.
Additional contextual information
Grant funding
| Grant number | Value of grant |
|---|---|
| JPR559 - Internal ref | £241,129 |
| 206/2003 A4/1 - Internal ref JPR541) | £13,300 |
| JPR555 - Internal ref | £39,956 |
| JPD329 - Internal ref | £10,512 |
| 09/1801/154 - Internal ref JPR566 | £691,101 |
| Internal ref JPR598 & JHR1071 | £10,049 |
| Internal ref JHR1071 following on from JPR598 | £7,750 |