Impact case study database

Pathway to impact: The NAP5 report [3.1] included recommendations for clinical practice, such as better monitoring of neuromuscular blockade [from finding H], and two specific instruments: the ABCDE handover checklist [3.4] to reduce anaesthetic errors during transfer to theatre [G] and the awareness support pathway [3.2] to guide anaesthetists’ management of AAGA cases [based on findings B-E]. To maximise reach, Andrade and others presented the findings and recommendations in September 2014 at the Royal College of Medicine and Association of Anaesthetists.

Impact 1: Improving professionals’ understanding of AAGA

Finding [A] that AAGA leads to long-term psychological harm has made anaesthetists more aware of AAGA as a serious complication [evidence 5.1]. A consultant anaesthetist testified to the impact of findings B and H: “ We now always make sure that neuromuscular blockade is reversed at the end of the operation. I am much more aware, because of NAP5, of the possibility that emergence [from anaesthesia] with residual paralysis could be perceived by the patient as awareness during surgery. I know my colleagues are too.” [5.2]

To ensure that anaesthetists understand the implications of AAGA and its prevention, RCoA introduced NAP5 findings and recommendations into its Fellowship examination curriculum in 2015. Approximately 900 candidates take the examination each year. Exam preparation textbooks have been updated accordingly [5.3].

Impact 2: Improving information for patients

RCoA updated their patient information leaflet “Accidental Awareness during General Anaesthesia” in 2017 [5.4], to incorporate findings on incidence [E], risk [G,H], subsequent operations [D] and awareness support [A,C]. An RCoA survey the same year found that the leaflet was already being used by a quarter of preassessment anaesthetic leads [5.4]. RCoA also updated its main leaflet, “You and Your Anaesthetic”, in 2015 to incorporate finding E on incidence of AAGA [5.4]. Regional NHS trusts subsequently updated their own versions of the RCoA leaflet [5.4]. Finding F led to RCoA issuing a new patient leaflet, Sedation Explained, in 2018 to explain the differences between sedation and anaesthesia, including their impacts on memory [5.4].

Impact 3: Helping anaesthetists deliver safer anaesthesia.

The Association of Anaesthetists published a new 5.5]. It recommended using the ABCDE handover checklist [reference 3.4] and monitoring neuromuscular blockade to ensure paralysis has worn off at the end of surgery as ‘patients interpret unintended paralysis in extremely distressing ways’ [findings B,H]. A consultant anaesthetist at Royal United Hospitals Bath testified that the checklist is embedded in practice: “ ABCDE check is used by every anaesthetist in all hospitals in Bath for every case” [5.6}. Although direct impacts on AAGA can only be tested in another very large study, there are signs that practice has improved. Since NAP5, monitoring of neuromuscular blockade in obstetric anaesthesia has increased to 53% in 2018, from 38% in 2013, and use of drugs to reverse paralysis has increased to 88%, from 68% in 2013 [5.12].

Andrade, Cook, Pandit and Wang responded to NAP5 findings [A-E] by developing an awareness support pathway to reduce likelihood that AAGA causes lasting psychological harm [3.1a, 3.2]. A consultant used this pathway when their patient experienced awareness: “ I found it incredibly helpful having a written series of steps to go through to make sure the patient was followed up and everything was done to help her” [5.2].

Findings on specific AAGA risks [H] led to new Association of Anaesthetists guidelines on Peri-operative management of the obese surgical patient (2015), Guidelines for the safe practice of total intravenous anaesthesia (2018), and S afer pre-hospital anaesthesia (2017) [5.5]. In March 2019, RCoA and AAGBI published the NAP5 Handbook [E.9] to make concise guidance on communicating and responding to AAGA available to anaesthetists and healthcare directors [based on findings A-F]. It incorporates recommendations on neuromuscular blocker monitoring [B,H], and the awareness support pathway.

International reach is shown by the American Association of Nurse Anesthetists’ 2016 position statement on Unintended Awareness during General Anesthesia, which cites NAP5 in support of its guidelines on anaesthesia, including the recommendation on neuromuscular blockers [H; 5.11]. A US anaesthesiologist commented that, by raising awareness of psychological sequelae, NAP5 has “enabled an approach to caring for awareness patients by referring them to psychologists” [5.12]. An anaesthesia update by the Australian and New Zealand College of Anaesthetists in 2015 [5.12] discussed NAP5’s findings [G,H] and recommendations on reducing drug errors in obstetric anaesthesia [3.4].

Updating the standard:

Three connected committees (AAMI 60601-1-8, IEC 60601-1-8 and a joint alarms working group), each comprising members across five continents, were tasked in 2015 with updating the standard AAMI/IEC 60601-1-8. The committees co-opted Prof Edworthy to their membership, and recognized that all alarm-related activity needed to be transparent, documented, and backed up by research to be found in the public domain, as well as requiring regular updates and presentations to the committee by Prof Edworthy, in order to win approval from all quarters. Prof Edworthy recommended quite early on in this process that auditory icons (those developed and tested in the research papers) would be the best alarms to adopt for the update of the standard. Convincing stakeholders that a change to sounds which will ultimately dramatically change the soundscape of all clinical areas, even with overwhelming evidence, required considerable time and attention.

The impacts claimed are policy change, as well as changes to understanding and process. Some parts of this project are immediately accessible to a broad range of beneficiaries, for example downloadable alarm sounds are available to all without further development from stakeholders. Other parts of the project, such as the guidance and the table discussed below, are useful for medical device companies, and known experts in this field who have now been provided with tools which provide greater understanding of the design and evaluation process.

The impacts:

1. Publication of the standard

Updating a worldwide standard takes years of work, the pace of which is dictated by the various agreements, procedures, votes, and interactions between the relevant committees and authorities. The final, voting draft of the standard was voted on in July and August 2019 and resulted in favourable votes from 17/18 countries and 19/19 countries respectively across the two key alarms committees. The updated standard was then published in July 2020. The updated standard differed from the previous version in two important ways in addition to having much improved alarms: the updated alarm sounds are downloadable, and the standard contains extensive guidance and help for researchers, designers and manufacturers in terms of developing and evaluating alarms.

2.  Updating and improving the audible alarms in IEC 60601-1-8

The updated alarm signals are available to download from the standard body’s website for immediate implementation by manufacturers and other stakeholders such as medical human factors and medical engineering specialists [5.1]. These are of direct benefit to manufacturers as they can now, for the first time, incorporate the alarms directly as .wav files into their equipment. In the longer term, the alarms will benefit patients, clinical staff, and guests, as the alarms are less aversive and can be played more quietly; clinicians find them more memorable and localizable, and less fatiguing, than the old alarms (all evident from the research, section 2).  Prof Edworthy wrote the majority of the narrative, tables and figures in the standard relating to the acoustic and temporal specifications of the new alarms [5.2]. The website Prevention.com (upwards of 5M monthly unique users per month (October, 2020)) selected the development of the new alarms as one of its ten health breakthrough awards for 2019 [5.3].

2. Improving the audible alarm guidance in IEC 60601-1-8

For the first time, the standard contains information allowing medical device manufacturers to develop their own alarm signals, as well as their own alarm and risk categories. Annex H (pp.100-105) in the standard) is mostly written by Prof Edworthy.^b  A  table  in Annex H contains all relevant metrics concerning the alarms incorporated into the new standard, as follows:

• A table (H1) delineating learnability, localizability and other performance metrics for the alarms specified, generated from the published papers, for manufacturers to compare with their own designs (as they must demonstrate comparable performance if they wish to implement their own alarms)

• Detailed advice describing how in-house designs might be developed and tested for learnability, localizability, detectability, and performance in simulation, based on simplified versions of some of the procedures used in the relevant published papers

• Advice as to how new or different categories of risk might be generated in a meaningful and empirical way (based on an AAMI-funded study involving Prof Edworthy, directed by Dr Melanie Wright of Trinity Health, Boise, ID, US).

The beneficiaries of this improved guidance are medical device manufacturers, human factors and auditory specialists, sound designers, testing houses, and other writers of clinical safety guidance documents and policy (see 3 and 4 below).  Dave Osborn, the chair of the three interconnected committees, says ‘ This standard affects ALL medical equipment worldwide that is either patient connected or controls the flow of energy into or out a patient in all environments …Professor Edworthy has driven all of this work, from the project’s initiation to the recruitment of other key scientists in the area in order to carry out both the formative and the summative testing of the candidate alarm signals in increasingly realistic settings, such as simulated, task sharing environments. Professor Edworthy has also written significant parts of the standard’ [5.4]. In his letter he also summarises the likely long-term beneficiaries of the update to the standard, including patients.

3. Working with industry (policy change, tools, understanding, process)

Prof Edworthy has worked extensively with two medical instrument companies, Masimo Inc (based in Irvine, CA, USA) and Kestra Inc (based in Seattle, WA, USA) and has less formal connections with other companies. Masimo makes non-invasive sensors which are currently used by over 100 million patients each year ^e Prof Edworthy has worked with a UK sound designer (Henry Daw, London) also working with Masimo, from 2018 onwards to develop alarms in improving audible alarms which comply with the standard. They have developed a new version of the ‘general’ alarm sound specified in the standard to make it distinctive to Masimo.  Nicholas Barker, VP of Design at Masimo, says ‘ Professor Edworthy’s expertise has allowed our company to prepare for the publishing of the update of IEC 60601-1-8 during the year 2020…The insights and advice Professor Edworthy have provided allow Masimo to enhance its leading role in the industry and to meet its high standards for consumer satisfaction and patient safety [5.5].

[text removed for publication] [5.6].

1. Working with key designers and research teams

A key benefit of the new standard is that it allows manufacturers to develop new categories of risk and new types of alarms, and provides them with benchmarking data with which to compare their own designs, if they wish to do more than download the sounds already provided (see 1 above).  Prof Edworthy has worked with various design and research groups to expedite this process. For example, a project is underway within the Human Factors group at Lisbon University, who say ‘ In our project funded by the Portuguese Science and Technology Foundation, we are designing clinical alarms with the goal of creating a library of sounds that will be made available and that ultimately would benefit manufacturers who would be able to choose them for their devices. For this we are using the IEC 60601-1-8 as design tool and its alarms as best-case scenario baseline.’[5.7].

Protecting marine environments, marine litter and microplastics have previously been the focus of the natural sciences. However, with plastics playing such a detrimental role in modern life, changing human behaviour is essential for mitigating further harm. UoP research into perceptions and behaviour change has resulted in policy action towards acknowledging the role of society and the realisation that these complex problems require interdisciplinary integrated analysis. Furthermore, their evidence has supported local governments and charities with their strategic aim of reducing marine litter. UoP research has put the human dimension and role of behavioural science approaches firmly on the agenda in the context of these marine environmental issues.

Integrating Psychology and Behavioural Approaches into policy

1. United Nations

Pahl’s research has informed policy debates on finding effective solutions to environmental challenges. In 2015 and 2016, Pahl was one of two psychologists representing behavioural sciences in the Working Group 40 (WG40) on microplastics of the Joint Group of Experts on Scientific Aspects of Marine Pollution (GESAMP) which is sponsored by several United Nations Agencies. The first GESAMP WG40 2015 report included Pahl’s review of risk perception and communication processes and the 2016 GESAMP WG40 report additionally included the role of behaviour change based on Pahl’s research [1]. It concluded with the recommendation that the field needed behavioural science input in order to find effective solutions that integrate the human dimension into this environmental problem [5.1]. Pahl’s review findings within the reports informed policy debate through the Food and Agricultural Organisation of the UN whose recommendations included strengthening education and awareness measures on marine litter by providing educational and outreach materials targeted to specific interest groups and range of ages to promote behavioural change [5.2]. In addition, the report informed a G20 Insights policy brief report [5.3] and Seafish, the UK public body which supports the seafood industry, who concluded that from a seafood industry perspective there was a need to influence human attitudes and behaviours to meaningfully reduce the sources of microplastic litter [5.4].

At the UN Environmental Assembly in Nairobi (May 2016), Pahl’s research on the behaviour perspective was integrated into the resolution on marine plastic litter and microplastics, which was agreed by the 193 UN member states. Policy recommendations included improved governance at all levels as well as behavioural and system changes such as the promotion of measures to reduce plastic material use to ensure a more sustainable production and consumption pattern and encouraging governments to promote change in individual and corporate behaviour [5.5].

Pahl was invited back to the GESAMP group in 2020 and contributed a summary of the role of human decisions and behaviours in the plastics system [5.6]. This resulted in her leading a stocktaking exercise on all actions addressing marine plastic litter and microplastics funded by the UN Environment Programme (UNEP). This is the first time UNEP has run such an ambitious stock take using quantitative social science methods.

Pahl’s research resulted in the development of targeted messages to help reduce human impacts on the world’s coral reefs. In 2017, Pahl contributed to a workshop on how to harness citizen behaviour to aid the protection of coral reef ecosystems, hosted by UN Environment World Conservation Monitoring Centre in partnership with UN Environment, Pierre-Yves Cousteau (CEO of Cousteau Divers), and Reef World Foundation. This workshop served to inform the UN Environment global campaign of citizen behaviour change and the International Coral Reef Initiative.

1. European Commission, the G7 and UK national level policy

Previously, social and behavioural sciences had not been valued or considered within the European actions required to protect the marine environment. Pahl was appointed as vice chair to the European Commission’s Science Advice for Policy by European Academies (SAPEA) group (2015-2019) and co-led an evidence review which resulted in recommendations that research efforts be directed towards the study of individual and collective behaviour. This report clearly emphasised the value of the Social and Behavioural Sciences and informed the scientific opinion of the EU’s Chief Scientific Advisors.  In addition, Pahl’s contributions to the G7 Roundtable on Microplastics (Oct 2019) resulted in a strong recommendation that the field of behavioral sciences should be mobilized [5.7]. At a UK national policy level, UoP research was included in a Houses of Parliament POSTnote (2016) that stressed the important role of prevention and the co-benefits of pro-environmental action such as taking parts in beach cleans.[5.8]

The impact of this work is evidenced through changes in national health policy [see below 1, 2, 3, 4] and improvements to the delivery of health care in the UK [1, 2, 5, 6] and overseas [7] by changing organisational, professional and patient behaviour using the P3C toolkit. These changes have created the conditions for improved patient experience of care [5] and improved health and wellbeing by elevating the collective ‘patient voice’ as a mechanism to challenge and improve care services [5, 7].

Policy Impact:

Using the P3C toolkit, Lloyd et al. have contributed to National Health Service for England (NHSE) policy changes for UK General Practice [1, 2 & 3]. The P3C toolkit has also featured in policy guidance and strategic recommendations by the Social Care Institute for Excellence (SCIE) [4].

National Health Service England (NHSE) General Practice Policy

Over a period of three years (2015-2018), Lloyd et al. worked with the Somerset healthcare system to facilitate a system-level NHS policy change aimed at relaxing the requirement for GPs to report pay-4-performance statistics (Quality Outcomes Framework (QOF). The Somerset Practice Quality Scheme (SPQS) was established as an alternative to QOF because GPs, Commissioners, and the Local Medical Committee believed that QOF was a barrier to P3C for people with LTCs because it incentivised care delivery on a single diseased based model. The P3C toolkit, supported by NHSE, evaluated SPQS across 55 general practices, serving approximately 440,000 people.

The evaluation revealed that relaxing QOF requirements did not result in adverse outcomes such as increased hospital admissions for people with long-term conditions. The P3C-EQ revealed positive care experiences. The P3C-OCT revealed statistically significant improvements in person-centred coordinated care in in a cohort of 2636 people with long-term conditions whose GP practices signed up to the new scheme [5]. Care delivery was improved by creating stronger agreements across and between general practices to create informal work networks to enhance multidisciplinary working, share resources and change the structure and timings of GP appointments to better support patient need. Importantly, the SPQS scheme leveraged time savings and reduced administrative burden from QOF which enabled practices to develop improved care for people with LTCs by developing complex care hubs [5]. The Associate Director of Primary Care in Somerset, Michael Bainbridge, reported that “the Somerset scheme (with the three other alternative pilot sites) led to significant changes to the national requirement for QOF. These were taken directly from SPQS”. The subsequent contractual amendments to QOF at a national level made a formal requirement for GPs to undertake quality improvement activities to deliver person-centred and coordinated care across the UK [1, 2]. Furthermore, national QOF indicators were reformed to include ‘a personalised care adjustment’ to reflect the following three broad categories: care described as unsuitable for patient because [medication intolerance or allergy], patient chose not to receive care [e.g. after a shared decision making discussion], or patient did not respond to offers of care [2]. This reform in indicator response is significant as it acknowledges the role of patients in decision-making discussions about their care with GPs. This indicator response did not exist prior to work of Lloyd and others. Under the old QOF scheme, patient decisions and preferences were not counted, thus not incentivised as an expectation of GP care. In summary, the P3C toolkit as used in Somerset provided the necessary quality assurances and evidence of P3C implementation to justify the above changes in policy.

Social Care Institute for Excellence (SCIE) & National Health Service for England (NHSE) Policy

Lloyd et al’s work has informed SCIE and NHSE policy recommendations for the measurement of person centred coordinated care. In an email to Dr Lloyd, Deborah Rozansky (SCIE) outlined how the work of the P3C team had informed national policy recommendations: “The research you led about measuring person-centred coordinated care influenced SCIE’s work with the Department of Health and Social Care and a cross-government Integration Board (of Senior policymakers) during 2018-19”. She further added “ The online compendium of P3C measures influenced SCIE’s and DHSC understanding of available measures, the international evidence underpinning them and tools that could be used to generate real-time data”[4]. Web links to the P3C compendium and website developed by Lloyd et al were included in reports and presentations and the P3C-EQ was recommended for capturing information about people’s experiences of integrated care in social care settings. SCIE also produced a series of webinars and seminars about how to measure the outcomes of integrated care, featuring the P3C tool. The two full day events on measuring the outcomes of integrated care had 25 attendees each; the two Better Care Fund Support Team’s regional events had 40 attendees each; and the two webinars had up to 100 attendees representing integration leads/managers and commissioners from local authorities and the NHS from across the UK [4].

The NHSE have also used the work produced by Lloyd et al to inform their national policies on personalised care and care for people with Long-term conditions [3]. Prof Alf Collins, Clinical Director for Personalised Care, NHS England and NHS Improvement commented that “.the quality of their (Lloyd et al’s) outputs and the relevance and importance of these means that they are one of a small handful of research organisations that I follow. Measurement in person centred approaches has become much more important over the last 5-10 years and NHS England and NHS Improvement have been guided by the outputs of the P3C team when considering evaluation, monitoring, accountability and incentive frameworks”[3].

Impact on Health Care Delivery: Implementing P3C

The P3C toolkit has been used to develop P3C interventions through changing professional behaviour that drives interventions to improve care experiences and outcomes for people with multiple long-term conditions in the UK [6, 7] and Australia [8].

UK Regional Implementation of P3C: Test and Learn Complex Care Hubs

In July 2014 three complex care hubs were launched in Somerset to implement person centred and coordinated care for people with three or more long-term conditions. This was made possible by the policy changes to the QOF contract facilitated by Lloyd et al described above. These hub-based models aimed to link networks of services across health and social care in the South (Yeovil), North East (Frome and Mendip) and the West of the county (Taunton). The results of the final evaluation [5, 6] demonstrated how changes in professional practice driven by an increased understanding of P3C improved patient care. For example, staff engaged in more person centred care planning and goal setting with patients. Patients reported better care coordination and more person-centred care planning, including the identification of personal health goals [6]. Approximately 1500 patients received care through the hubs during 2015-2017 during which time Lloyd’s team supported the implementation and development of the hub’s P3C models. The hubs are now fully developed and operational. Michael Bainbridge commented “ the conceptual model of person-centred co-ordinated care was and remains a helpful structure as we try to develop more integrated care”[1].

International Implementation of P3C: Australia

In 2019, the Australian government commissioned general practice to implement care coordination in the Melbourne region. The P3C toolkit was used to assess system performance and implementation over a 1-year period. The P3C-OCT has been used to implement care across 30 practices and the P3CEQ has been used to collect data on 1078 patients to evaluate care delivery from their perspective. Results show a correlation between OCT scores and P3CEQ scores with practices scoring higher on relational care. Higher scores are driven by efficient use of information systems and technology [8]. Evaluation lead Dr Jennifer Hester reported that GP practices are changing care processes in response to using the toolkit for implementation [8]. For example, after completing the P3C-OCT a practice manager reported “We are definitely at the beginning.. but it (the toolkit) has helped me realise how far we have to go. We will improve our documentation and templates regarding patient decision making/goal setting/care planning” [8].