Impact case study database
Making music with brain signals to improve the quality of life of severely motor-impaired individuals
1. Summary of the impact
Locked-in syndrome is a rare neurological disorder characterised by complete paralysis of voluntary muscles. Despite physical paralysis, cognitive function is unaffected, but communication is extremely limited. Professor Eduardo Miranda’s Brain-Computer Music Interface (BCMI) technology has profoundly changed the quality of life of four individuals with the condition, enabling connection and expression. This bespoke technology enables an individual or groups of individuals suffering from severe motor impairment to create and perform music benefitting their mental health and well-being. In addition, it has raised awareness of the issues facing patients and changed medical practice, and international media coverage of the work has raised public awareness of both the condition and the technology.
2. Underpinning research
A brain-computer interface (BCI) is a system that interacts directly with the user's brain by picking up tiny electrical impulses of neurons in an electroencephalogram (EEG). Other teams worldwide have proposed BCI systems for music but had so far failed to make a positive effect upon severely motor-impaired persons who are paralysed because they have been unable to manage the fundamental problem of voluntary control. Previous technology has been limited to simply reading the brain signal and translating it to music. The user is not able to control the music because they cannot control their brain signals. In this ‘passive’ system, the user simply “listens” to his/her signal. This technology is unsatisfactory for paralysed individuals as they yearn for ability to control the system now that they are no longer able to control their limbs
University of Plymouth research provided a unique solution to this problem to enable the user to be able to control the music with their brain signals [3.1]. This revolutionary research is based on a neurological phenomenon known as SSVEP (Steady State Visually Evoked Potentials). These natural responses to visual stimulation at specific frequencies produce signals that can be detected in the EEG. Different flickering lights activate individuals’ brains differently. When a person with sensors placed on their scalp looks at different flashing lights at specific frequencies, it shows up in their EEG, and a computer can be programmed to infer at which icon they are staring. It then associates the different brain activities with commands to produce different melodies [3.1 & 3.2]
Miranda and Williams created musical algorithms that translate specific EEG signals associated with the light frequencies of different icons into distinct musical processes. Looking at one icon sounds a certain note or produces a certain rhythm: staring at another changes its pitch, and so on. This enables individuals to use their minds to play music.
Professor Miranda and his team (Williams and PhD student J. Eaton) subsequently worked in collaboration with biomedical engineers at the University of Reading (Daly (University of Essex from 2016), Hwang, Malik, Weaver and Nasuto), and the Royal Hospital for Neuro-disability (RHN), London, to understand the needs of staff and patients by interviewing doctors and nurses to assess practical requirements in the field. RHN staff highlighted the necessity for the technology to be easy to use, quick to set up, reliable and not requiring constant technician input. Based on this feedback, a proof-of-concept system was developed alongside a framework to conduct research engaging directly with motor-impaired patients in the hospital setting. The system was subsequently trialled with hospital staff and a patient who had locked-in syndrome after suffering a severe stroke. The success of this trial has meant the technology is now integrated into the hospital’s work-based practice and custom-made systems have been developed to support three other patients [3.3 – 3.6].
Subsequently, the work of Miranda and his team has enabled severely motor-impaired individuals to use their minds to play music in partnership with professional musicians. This is possible due to their bespoke AI software at the core of the BCMI system. This generates musical scores rather than synthesised music, and the composition created by Miranda enables patient-participants to choose a short ‘phrase’ of music with their eyes which is then relayed to the paired musician to play.
This research is one of the outcomes from the EPSRC-funded project “Brain-Computer Music Interface for Monitoring and Inducing Affective States” (£376,101.00), with Prof E.R. Miranda as Principal Investigator [3.1 – 3.3].
3. References to the research
J. Eaton and E. R. Miranda (2014). “On Mapping EEG Information into Music”. In E. R. Miranda and J. Castet (eds.), Guide to Brain-Computer Music Interfacing. London: Springer Verlag.
E. R. Miranda (2015). "Music Neurotechnology: From Music of the Spheres to Music of the Hemispheres", Symmetry: Culture and Science, 26(3):353-378.
Daly, D. Williams, F. Hwang, A. Kirke, A. Malik, J. Weaver, E. R. Miranda, S. Nasuto, (2016). "Affective Brain-Computer Music Interfacing", Journal of Neural Engineering, 13:4.
E.R. Miranda and J. Eaton (2018) “Music Neurotechnology: a natural progression”. In S. Emmerson (ed.), The Routledge Research Companion to Electronic Music: Reaching out with Technology. London: Routledge.
Daly, I., Williams, D., Malik, A., Weaver, J., Kirke, A., Hwang, F., Miranda, E. R. and Nasuto, S. J. (2018). “Personalised, Multi-modal, Affective State Detection for Hybrid Brain-Computer Music Interfacing”, IEEE Transactions on Affective Computing.
Daly, I., Nicolaou, N., Williams, D., Hwang, F., Kirke, A., Miranda, E. and Nasuto, S. J. (2020). “Neural and physiological data from participants listening to affective music”, Nature-Scientific Data 7(177).
Key Grant: E.R. Miranda, Brain-Computer Music Interface for Monitoring and Inducing Affective States, Engineering and Physical Sciences Research Council, 2012-2017, £367,101, EP/J002135/1
4. Details of the impact
Locked-in syndrome is a rare neurological disorder which affects around 1% of people who have a stroke - there are more than 100,000 strokes in the UK each year. There is no treatment or cure, and it is extremely rare for patients to recover any significant motor functions. It causes a complete paralysis of all voluntary muscles except for the ones that control the movements of the eyes. Individuals with locked-in syndrome are conscious and awake but have no ability to produce movements (outside of eye movement) or to speak.
Miranda’s BCMI system has challenged some of the myths and assumptions surrounding severe motor disability and has had a profoundly positive effect upon the mental health and wellbeing of patients, who were able to transcend their physical limitations to create art. It has led to changes in therapeutic practice at the Royal Hospital of Neuro-disability due to increased understanding of the benefits to patients of using this system, and more widely has contributed to public discourse around Locked-in Syndrome, of the opportunities created by the technology for those with it, and for imagining the possibilities of technology beyond that already achieved.
Increased mental health and wellbeing of patients
In 2019 Miranda created a proof of concept version of the BCMI for a care home in London. The custom-made system was tailored to the specific needs of the patient they worked with, and the flexibility of the system ensured that it could be adapted for individual requirements. The system had significantly beneficial impacts on the patient, increasing wellbeing and patient interaction, so Miranda built a personalized BCMI unit that was donated to the patient. The patient uses it on a regular basis, with a carer trained to support this activity. The patient emailed “I would like to thank you for the music system you specially made for me. Your research has been wonderful. It is lovely being able to choose and play music once again. This has improved my quality of life... I would love for other people, like myself, to have access to such a wonderful a piece of equipment” [5.1].
Miranda’s collaboration with the care home helped him to further understand the needs of patients. Although patients valued the technology, it only allowed them to choose from set pieces of music with little autonomy or creativity. They yearned for a system which allowed them to create original music and interact with others. This resulted in Miranda designing enhancements to the BCMI, which enabled four patients at the Royal Hospital of Neuro-disability (RHN) to collectively compose music. This pioneering system meant that the patient could direct a musician in performance, enabling the co-creation of music between patients and musicians, something that they had not previously been able to do. In trials, the RHN reported that the technology empowered the patients and boosted their confidence [5.2]. Crucially though, the unique and ground-breaking opportunity of this technology is that it enables this culturally marginalised and cognitively segregated group of patients to express themselves creatively through co-producing their own music. One music therapist explained: “ When you look at some of the patients we have here, communication internally is all intact, but they are cut out of the rest of the world by virtue of their disabilities. I think it is very important that there is technology that … offers patients that have locked-in syndrome, motor-neuron disease, multiple sclerosis, the ability to be creative. The BCMI bridges the gap that is there because of their inability to move or talk. This is a great means of transcending disability to offer individuals a unique experience of creating music with each other and interacting with skilled musicians to create original compositions.” [5.6]
Activating Memory: A pioneering performance pushing creative practice beyond its conventional boundaries**
In July 2015, the enhanced version of the BCMI was used in its first public performance when four musicians and four patients joined as the Paramusical Ensemble to co-create and perform Miranda’s compositon, Activating Memory, at the RHN. This pioneering performance was a significant moment for those participating. One of the patient-participants, Richard Bennet, who can only communicate via eye movement to select letters from a display, told a BBC News report: “ It was amazing to work with the others” [5.3]. Another patient, Steve Thomas said: “ This is a truly magical experience. It is a chance to play with other severely disabled musicians” [5.4] and: “ It’s more interactive with people actually getting my instructions. It was great to hear the musician play the phrase I selected” [5.5].
Change in public discourse around cultural access and marginalised groups
The preparation for and performance of Activating Memory was captured in a documentary, Paramusical Ensemble, which won Best Documentary at the BioFiction Science Art Film Festival 2019. Through this, the pioneering performance has been screened across the globe [5.6]. To December 2020, the film has been viewed 104,000 times on Vimeo alone.
The publicity around the performance and documentary also led to further performance opportunity, reaching a wide global audience; One of the patient-participants was Rosemary Johnson, a former violinist of the Welsh National Opera (WNO) who suffered a devastating injury in an accident in 1988, robbing her of speech and movement. Although her career was over, she never lost hope that one day she might make music and perform again with the WNO. In March 2017, this dream was realised when the media attention of Miranda’s research inspired a collaboration with Volvo and Grey advertising agency to produce a Sky Atlantic TV documentary about Johnson. Grey London approached Miranda following media attention to produce a film for the series ‘Defiant Pioneers’. The series focuses on ‘people who do things differently and whose relentless pursuit of craft and innovation would change the world’ [5.7]. Miranda worked with Johnson to adapt the system for a performance with an orchestra featuring her as a soloist at Kings Place, London. The first airing of the documentary [5.8] was shown during the advert break of the season finale of TV programme “Game of Thrones” on 27th August 2017, attracting 12.1 million viewers (global information & measurement company, Nielsen). This has led to greater awareness and understanding of the issues facing patients with locked-in syndrome. Johnson’s performance using Miranda’s BCMI technology contributed to her receiving an MBE for her services to music in December 2017 and was cited on the Royal Family Twitter feed [5.9].
Changes to medical practice regarding patient care
At a therapeutic level, Miranda’s collaboration with the RHN resulted in changes to the hospital’s work-based practice. Resident therapists, nurses and carers found BCMI technology to be a powerful tool for their practice, particularly music therapy, occupational therapy and palliative care. The system raised staff awareness that the patients led a full life before they came to RHN and provided a deeper understanding of the patients’ condition. It led to changes on how staff used the computer room with patients. Previously, they rarely linked eye movement tracking assistive technology with the music making software as it was time consuming and they did not appreciate the positive effects it had on patients. Dr Sophie Duport, Associate Director of Research, RHN said “ Since interacting with your system, they appreciate how much the patients enjoy it and how beneficial it is to their mental health and well-being. Therefore, they have changed their practise to facilitate a therapeutic intervention” [5.2].
5. Sources to corroborate the impact
E-mail from patient at Royal Hospital of Neuro-disability
Testimonial from Associate Director of Research, Dr Sophie Duport, Royal Hospital of Neuro-disability
BBC News, 12/02/2016, “Disabled musicians create music from brainwaves”:
Article in Neuroscience News, 20/07/2015, “Computer Interface Allows Disabled People to Control Musical Performance”:
Article in The Telegraph, 11/02/2016, “Brain-damaged violinist makes music for the first time in 27 years with mind-reading technology”:
Paramusical Ensemble documentary: https://vimeo.com/143363985
Volvo Defiant Pioneers https://www.volvocars.com/uk/about/humanmade/discover-volvo/defiant-pioneers
Music of the Mind, Sky Atlantic documentary: https://www.volvocars.com/uk/about/humanmade/discover-volvo/music-of-the-mind
The Royal Family Twitter @RoyalFamily:
Additional contextual information
Grant funding
Grant number | Value of grant |
---|---|
EP/J003077/1 | £367,101 |