Skip to main

Impact case study database

The impact case study database allows you to browse and search for impact case studies submitted to the REF 2021. Use the search and filters below to find the impact case studies you are looking for.

Search and filter

Filter by

  • Middlesex University
   None selected
  • 11 - Computer Science and Informatics
   None selected
   None selected
   None selected
   None selected
   None selected
   None selected
Waiting for server
Download currently selected sections for currently selected case studies (spreadsheet) (generating)
Download currently selected case study PDFs (zip) (generating)
Download tags for the currently selected case studies (spreadsheet) (generating)
Currently displaying text from case study section
Showing impact case studies 1 to 5 of 5
Submitting institution
Middlesex University
Unit of assessment
11 - Computer Science and Informatics
Summary impact type
Societal
Is this case study continued from a case study submitted in 2014?
No

1. Summary of the impact

Since 2008, Prof. Barn and colleagues have been researching the use of mobile apps in a variety of social care settings. This research has focused on marginalised young people in conflict with the law leading to the UK’s first mobile app to support Youth offending teams in the work with young people in the youth justice system. National impacts included: The mobile app was developed through a co-design process that generated impact of engagement of young people in the design, development and deployment of social technology. The MAYOT app is embedded in the strategy for West Mercia YoS. The app has been deployed and is in use in at Bromley Youth Offending Service by 25 case workers. A third Youth Offending Service, Essex YoS is in the process of finalising deployment before March 2021. Additionally, the co-design methodology is embedded in the design practice of an international software development house.

2. Underpinning research

Middlesex Computer Science undertakes leading research in technology for social good. The research undertaken for the impacts reported in this case study is an example of multi-disciplinary research with contributions from computer science, social policy and criminology. The research has been reported in leading international outlets over the research period of 2006 through to 2020.

Prof. Barn began work in the field of app development for social care settings in 2006 with app development for nurses. Applied research in social care education settings was further developed under another JISC funded project in support of social workers. The latter project was instrumental in exposing the importance of understanding users and how users interact and work with mobile devices in challenging settings. A second outcome was the importance of inter-disciplinary approaches to solution development.

Knowledge of the important role of mobile technology in their relationship to young people, was further enhanced through a research consultancy project funded by the Association of Chief Police Officers. Here, research elaborated a detailed understanding of the relationship between young people and mobile technology. Outputs from the research addressed the risk-taking behaviours of young people and were published in the Core-A-ranked Hawaii International Conference on System Sciences HICSS [R1]. This research was conducted in conjunction with Royal Holloway, University of London. An early insight from this research was the understanding that social care was critically neglected in terms of benefiting from advances in mobile technology.

This new understanding led to a proposal being submitted to the Nominet Trust in 2012 and awarded in 2013 as the MAYOT project (Mobile Apps for Youth Offending Teams) and conducted from 2013-2016. The research performed as part of this project developed a personalised mobile app for use by young people and their case workers in youth offending teams.

In the youth justice sector, Youth Offending teams were set up following the 1998 Crime & Disorder Act and comprise multi-agency teams coordinated by a local authority. YOTs work with young people on a range of community orders. At the initiation of this research (2013), there were over 20000 first time entrants into the youth system. Further, there 66,430 young people forming the case load of YOTs nationally (Data available from 2012). To ease this workload, we envisaged the use of a smartphone app to support interactions between case workers and young people.

Requirements for the app were elicited through the instigation of a value sensitive co-design approach where young people, their managers worked in close conjunction with the research team. The research was innovative in the use of such a methodology with a marginalised section of the community.

The sensitivity of the problem domain and the nature of the end-users, young people in the youth justice system, meant that value sensitive concerns were brought to the fore and in particular, issues such as privacy, security, autonomy and transparency were quickly established as critical. As a result, our research contributed a conceptual model for value sensitive concerns published in the Software Engineering in Society Track at the premier international software engineering conference, ICSE [R2]. This topic has subsequently become an important research area. An approach to early identification of value sensitive concerns using Floridi’s Informational Privacy theory was described in the ACM Symposium on Applied Computing (SAC) paper [R3]. More complex relationships between such concerns and desirable attributes such as resilience have also been illustrated [R4 – an extended paper of work first presented at the core ranked A conference, European Conference on Information Systems (ECIS). The value sensitive co-design approach was further adopted by a boutique software development company specializing in app development.

Reflecting on the interdisciplinary nature of the research, and the interplay between research and impact, research conducted during the deployment of the MAYOT app also addressed the impact of such technology on both case workers and young people. Techno-habitats of practitioners were explored in [R5].

The conceptual model of values is particularly important as it provides an important bridgehead between software engineering practice and non-functional requirements and has formed a substantive body of research reported elsewhere. Notably, we have also demonstrated how values can be more formally accounted for to support the design of resilient information systems. [R6].

3. References to the research

Balbir S. Barn, Ravinder Barn, and Jo-Pei Tan. Young people and smart phones: An empirical study on information security. In 47th Hawaii International Conference on System Sciences, HICSS 2014, Waikoloa, HI, USA, January 6-9, 2014, pages 4504–4514, 2014. DOI: 10.1109/HICSS.2014.554

Balbir S. Barn, Ravinder Barn, and Franco Raimondi. On the role of value sensitive concerns in software engineering practice. In 36th International Conference on Software Engineering, IEEE/ACM ICSE Companion , 2015. DOI: 10.1109/ICSE.2015.182

Balbir S. Barn, Ravinder Barn, and Giuseppe Primiero. An approach to early evaluation of informational privacy requirements. In Proceedings of the 30th Annual ACM Symposium on Applied Computing . ACM. DOI: https://doi.org/10.1145/2695664.2695788

Balbir S Barn and Ravinder Barn. An exploration of resilience and values in the co-design of sociotechnical systems. International Journal of Systems and Society (IJSS) , 3(1):1–17, 2016. DOI: 10.4018/IJSS.2016010101

Barn, Ravinder, and Balbir S. Barn. "Youth Justice in the Digital Age: A Case Study of Practitioners’ Perspectives on the Challenges and Opportunities of Social Technology in Their Techno-Habitat in the United Kingdom." Youth Justice 19, no. 3 (2019): 185-205. DOI: https://doi.org/10.1177/1473225419869568

Primiero, Giuseppe, Balbir Barn, and Ravinder Barn. "Value-sensitive co-design for resilient information systems." Studies in Logic, Grammar and Rhetoric (2020). DOI: https://doi.org/10.2478/slgr-2020-0032

4. Details of the impact

The impacts arising from this research are an example of close integration between research and practice. Impacts occurred both during the research process and from the outcomes of the research. The impact from the research has ranged from policy/strategy, public education and impact on the growth of an international SME working in the social enterprise sector.

Policy and Strategy

The reach of our work is potentially significant with a total of 153 youth offending teams in England and Wales. In March 2016, a total of 27,900 young people (aged 10-17) were convicted and sentenced by the courts (YJB, 2017). The vast majority of such young people are supervised in the community by the 12,000 personnel in Youth Offending Teams (YOTs). There are 153 multi-agency teams YOTs with representation from the police, probation, education, health and social services. The service also includes specialist workers, such as mental health practitioners, and substance misuse workers.

Our advocacy of the use of mobile technology (the MAYOT app) was embedded in the 2015-16 strategy of one large youth offending service, Worcestershire and Herefordshire YoS (now West Mercia YoS) [S1] as an example of innovative practice for engagement and dialogue between the case worker and the young person.

Before the app can be deployed in Youth Offending Service setting, staff undergo a rigorous training programme. Our conceptual and methodological training programme that focuses on the ethics of using social technology to promote engagement and empowerment in young offenders has been delivered to 20 YOTs in England and Wales in urban (e.g. London, Bradford, Leeds, and Newcastle), and rural (e.g. Worcestershire, Oxfordshire, and Cumbria) settings [S4].

The overarching and monitoring government body, the Youth Justice Board (YJB), has recognised the value and potential of our technology. In 2017, following a meeting between one of the co-investigators of the MAYOT project the then Parliamentary Under-secretary of State for Justice, Dr Phillip Lee invited the project team to contact the Youth Justice Board formally, to discuss our approach and update the YJB on the aims of the MAYOT app [S9]. The meeting took place on 29th November 2018 and included our deployment partners from the Youth Offending Services of West Mercia and Bromley. More recently, discussions have re-started and with the YJB and they are now considering an option to take ownership of the MAYOT technology to support wider deployment.

Further engagement with UK Government has also taken place through interactions with the Home Office. Following the rise in knife crime statistics in March 2019, the MAYOT team, along with the software development partner, GNB, met with the Home Office to elaborate how the MAYOT app could be used in the fight against knife crime. The team was invited to submit a proposal to evidence how the app could be applied in this setting by Tom McDonald from the Home Office.

The value of the app to the YoS was first noted by the Head of London Triborough YoS, who noted that the co-design process used by the team “helped develop confidence in our young people”. The senior leader was further was clear that: “The use of the MAYOT app encourages and supports positive engagement with young people and could reduce further offending by providing key information that young people may need such as what to do if you are arrested or drug information.” [S7].

The senior leader recognised the importance of such technology and when they moved to Bromley YOS, the team was again approached for engagement at Bromley.

In Autumn 2019, Bromley YOS agreed to adopt the MAYOT technology for use in the service. Dedicated training sessions to the entire Youth Offending Team and supporting services was provided (October 2019). (Letter of support/evidence: [S7]). Full deployment was planned for spring 2020 but has been impacted by the COVID-19 pandemic lockdown restrictions. A further deployment is now scheduled for February 2021.

Between July and October 2020, the Head of Western Essex YoS, responded to an Innovation call by Essex YoS and proposed the use of apps in a youth justice setting. Realising that such an app already existed, asked for MAYOT to be used at Western Region of Essex YoS. Training sessions for 8 case workers managing around 70 young people are scheduled for March 2021 with a deployment of the app planned for March 2021. The COVID-19 pandemic has resulted in further delays. The Essex YoS has produced a compelling video available as source [S8]. Confirmation of the Essex YoS deployment plan is noted in source [S3].

Stakeholder Engagement

We have worked with a range of key stakeholders through the entire design process. A key concern in the adoption of technology in challenging work environments, is the involvement of stakeholders. We instigated a co-design approach where case workers, managers (combined total, 17) and most importantly, the young people (10) were first class designers in the design processes used in the development of the MAYOT app. These were spread across three Youth Offending Services in England (London Triborough YoS, Hereford and Worcester YoS, and Oxfordshire YoS).

The voices of young people were central to the design of the app, their engagement in the co-design was an important characteristic and at times contradicted other stakeholders in the process such as magistrates. The co-design process ensured that all stakeholders were represented through the app. This tension of multiple views was a significant research outcome and has been documented in [R4, R5, R6].

Following availability of the app, a Train the Trainer workshop aimed at developing digital expertise in the use of the app, was delivered to all YoS in England, interested in attending. This workshop was attended by 21 case workers and their managers from 17 different YoS [S4]. In 2016, Further training was delivered to the Welsh Youth Offending Services: Monmouthshire & Torfaen, and Cwm Taf.

The professional body for Youth Offending team managers ( https://aym.org.uk) requested a contribution in the form of a short article for their regular magazine. The article is available in reference [S5]. Similarly, Magistrates form another critical set of stakeholders. In February 2015, we were invited to present to a meeting of 20 magistrates from Worcestershire. Following a detailed discussion, we were invited to contribute a short article for their newsletter (See Reference [S2]). Both articles are an example of knowledge transfer to communities outside the academic sector.

International Reach

There has been international component to the impact. In 2016, The novelty and value of the work was identified by researchers at the University of Copenhagen who invited members of the team to join a research team seeking funding from the Innovation Fund of Denmark to address similar concerns in that country.

Human, Economic and Technological Impact

The project has had two further impacts. Firstly, value-sensitive design and co-design is essential practice where end-users are from marginalised communities. This has manifested itself through human, personal development. The lead developer on the original MAYOT project, Mr Lalith Athiappan gained significant requirements gathering and development experience and user interface design skills such that he was instrumental to a successful KTP proposal (No. KTP010041) working with a social enterprise Global Notice Board ( http://www.gnb.com).

Athiappan was able to apply the good practice design guidelines and the ability to design for different types of end-users to develop a look and feel for user interface platform for use for a commercially available estate agents app that was derived from his experience on the MAYOT app. He was also able to use his development experience from the MAYOT project to contribute towards the establishment of a software development team in India that specifically serviced the social enterprise (GNB) in London. In February 2018, Prof. B. Barn delivered a workshop on value sensitive design for mobile apps to the software development in India, emphasising the importance of understanding users in the co-design process and how values underpin acceptance of technology when working with marginalised users [S6/S5].

5. Sources to corroborate the impact

“Innovative Practice: The YOS has been working with a multi-disciplinary academic team from Middlesex University and Royal Holloway University of London in piloting the Mobile Application for Youth Offending Teams (MAYOT). MAYOT is a smart phone application that provides a common platform for engagement and dialogue between the case worker and young person. The application allows communication around key activities, reminders for appointments, the provision of information and an activity meter/progress chart. Team members and young people from the South Worcestershire Team have been involved during 2014/15 in the iterative co- design and testing of the application. There are now twelve YOTs either using or planning to use the MAYOT application (Worcestershire Youth Justice Plan 2015-16: 8)”. (See page 8 of PDF).

March 2015, we were invited to contribute a short article to the Magistrates Association newsletter. In the words of one magistrate: ‘I was very interested to hear about the introduction of the MAYOT pilot study for young offenders. A mobile phone app that can remind these youngsters of important dates, key activities, even exclusion zone and progress on a court order can only help with compliance. It’s an exciting development in the communication process between young offender and caseworker.’ (Worcester magistrate). See: https://www.dropbox.com/s/76nnrmtbggbw3y5/Magistrate%20February%20March%202015-MAYOT.pdf?dl=0

Testimonial letter from Michael Kay, Essex Youth Offending Service.

Train the Trainer Event, Middlesex University, 25, June 2015. https://www.dropbox.com/s/s6wax0kgymsw6fe/Attendee%20Summary.pdf?dl=0 (Available)

In May 2015, the Association of Youth Offending Managers also invited us to contribute a short article to their newsletter. See: https://www.dropbox.com/s/gtqa0r0wouzg192/AYM%20Newsletter%20for%20May%2015v1.pdf?dl=0 (Available)

Training visit to India Software Development Lab for GNB – in letter.

Testimonial letter from the Betty McDonald, Head of Bromley Youth Offending Service.

Video produced by Essex YoS: https://www.dropbox.com/s/4u5ivkxpgm7vyxc/The%20Mayot%20Video.mp4?dl=0

Letter from Dr Phillip Lee, MP Under-Secretary of State for Justice inviting members of the team to present information about MAYOT to the Youth Justice Board.

Submitting institution
Middlesex University
Unit of assessment
11 - Computer Science and Informatics
Summary impact type
Societal
Is this case study continued from a case study submitted in 2014?
No

1. Summary of the impact

The Research Group on Development of Intelligent Environments creates and improves methods and tools from Software Engineering, Human-computer Interaction and Artificial Intelligence, through direct interaction with societal problems, notably those connected with health conditions. The Middlesex born innovation has advanced Computer Science generating measurable impacts in several directions and citizen groups often neglected by the technology giants. Here we present three of those areas: increasing inclusion for people with Down's syndrome, providing ambient assisted living support for older people in their homes, and encouraging citizens to be more physically active. Our research on context-aware systems guides our development and delivery of systems better tailored to the needs of individuals with specific needs.

2. Underpinning research

Intelligent Environments are closely related to areas such as Ubiquitous Systems and IoT systems, and refer to systems which exist in a physical environment enriched with sensing technology and Artificial Intelligence algorithms to provide context-sensitive help to humans. Specific challenges in these systems around the core concepts of contexts and context-awareness, which our work focus on. There has been work on contexts from an Artificial Intelligence perspective led by J. McCarthy in the 80’s and 90’s and then focused on how inferences in different contexts relate to each other in general. However our work is much more guided by the specific needs from practical contexts and what users expect from system services in those contexts.

Our research group ( http://ie.cs.mdx.ac.uk/) has been working to improve the development of Intelligent Environments (IE) since its creation in 2013. Amongst the challenges we faced there were those on the engineering side with a lack of methods and tools specifically helpful for developing these types of systems. Also, despite the interesting advances in AI, these powerful algorithms did not offer the right balance on expressiveness and efficiency to be run in systems with low resources and with fast reactions expected. Advice on engineering Intelligent Environments (IEs) systems has been patchy or not transferrable. Hence, we created our own refined versions of existing approaches to system development, including the “User-centred Intelligent Environments Development Process” [Augusto et al., 2017], an iterative process centred on stakeholder’s engagement. Part of that high level strategy also included our own method to gather “Requirements for Intelligent Environments” and an “Ethical Framework for Intelligent Environment Development which were used to influence requirements and from there the whole system [Jones et al., 2015]. These were then complemented with specific strategies for “Context-aware Systems Testing and Validation” [Augusto et al., 2020a], [Augusto et al., 2020b].

In applying AI we have specialized algorithms to make known AI techniques to work in real life IE scenarios. Our algorithms include real-time temporal reasoning to automate sensorized environments [Gimenez-Manuel et al., 2020], machine learning to learn user’s habits [Ali et al., 2019], and handling possibly conflicting user preferences [Oguego et al., 2018]. Those automated learning and reasoning algorithms combined with context-awareness resources and specialized interfaces provide a new system architecture [Augusto et al., 2020a] for intelligent environments. This consistent and integrated innovation allowed our team to win the BCS Machine Intelligence Competition RealAI (2019 edition): http://www.bcs-sgai.org/micomp/intro.php

These advances on user-centred engineering processes and tools facilitated the identification of relevant contexts and development of context-awareness required for the successful development and deployment of real-life services within various projects including:

  • Supporting independence for people with Down’s syndrome (Supported during 2013-2016 by the EU through a 4M Euros funded project: “PersOnalized Smart Environments to increase Inclusion of people with DOwn's syndrome (POSEIDON)”): to decide when users require advice or are in an emergency, or determine the system reaction in those detected contexts. Contexts of interest are usually organized at system level (e.g., battery level and connectivity), person level (e.g., physical or mental status), and environment level (e.g., weather or bus service).

  • Ambient Assisted Living (Partly supported during 2016-2019 by the EU through a 700K Euros funded project: “SecUre aCCESSibility for the internet of things (SUCCESS)”): the contexts our Smart Home system detects unhealthy sleeping patterns, unhealthy eating patterns, and ‘wandering’, all of which are well known to be meaningful to people experiencing dementia-like conditions. These contexts are detected through a specially designed rule-based temporal reasoning system fed by sensor data. House behaviour adapts to user preference through personalization interfaces and unsupervised learning algorithms. The Smart Home system was set up as part of the Smart Spaces lab in our campus: http://ie.cs.mdx.ac.uk/smart-spaces-lab/

  • Encouraging increase in physical activity within Barnet (a collaboration supported from 2018 onwards with £136,000 by Greenwich Leisure Limited, a charity company based in London working in the sports market with branches all over the UK, and by Barnet Council): this is a Gamification project to encourage citizens to be more physically active. The system achieves that by combining various techniques stemming from psychology (for example, behaviour change techniques or BCTs) which relate individual behavioural contexts to personalized habit formation and lifestyles changes. The stakeholders centred approaches and methods to identify and develop the context-awareness of the system have been used in this project. This project is currently ongoing and use in the community was delayed for a year due to Covid, so it is expected to start trialled in the community after spring 2021.

3. References to the research

This research was based on competitively funded projects, with robust peer review systems. The outcomes were published in leading peer review journals and conferences in the field:

  • S. M. Murad Ali, J. C. Augusto and D. Windridge (2019). Improving the Adaptation Process for a new Smart Home User. Proceedings of 39th SGAI International Conference on Artificial Intelligence (AI-2019). Cambridge, 17-19 December 2019. Available at: http://eprints.mdx.ac.uk/27908/

  • J. Augusto, D. Kramer, U. Alegre, A. Covaci and A. Santokhee (2017). The User-centred Intelligent Environments Development Process as a Guide to Co-create Smart Technology for People with Special Needs. Universal Access in the Information Society 17(1):115-130. Springer Verlag. Available at: http://eprints.mdx.ac.uk/21032/

  • J. C. Augusto, J. G. Gimenez-Manuel, M. Quinde, Ch. Oguego, M. Ali, C. James-Reynolds (2020a). A Smart Environments Architecture (SEArch). Applied Artificial Intelligence, Taylor and Francis. Available at: http://eprints.mdx.ac.uk/28682/

J. C. Augusto, M. J. Quinde, C. L. Oguego, J. G. Gimenez Manuel (2020b). Context-aware Systems Architecture (CaSA). Cybernetics and Systems, Taylor and Francis. Available at: https://eprints.mdx.ac.uk/31198/

  • S. Jones, S. Hara, J. C. Augusto (2015). eFRIEND: an ethical framework for intelligent environments development. In: Ethics and Information Technology, 17 (1):11-25. Available at: http://eprints.mdx.ac.uk/15705/

  • C. L. Oguego, J. C. Augusto, A. Munoz, M. Springett (2018). Using Argumentation to Manage Users' Preferences. Future Generation Computer Systems 81:235-243. Elsevier. Available at: http://eprints.mdx.ac.uk/22641/

  • J. G. Gimenez-Manuel, J. C. Augusto, J. Stewart (2020) Towards empowering people living with dementia in Ambient Assisted Living. Universal Access in the Information Society. Springer Verlag. Available at http://eprints.mdx.ac.uk/30290/

4. Details of the impact

The systems we developed based on our user-centred systems engineering improvements and in our context-awareness reasoning and learning algorithms improvements described in section 2 were used to create systems which had a diversity of positive effects:

Societal Impact: our systems contributed to the quality of life of different sectors of society with special needs. Some of them such as people with Down’s Syndrome are usually very much neglected by technology as they are not appealing to the larger dominant innovation companies dominating the digital markets. During the POSEIDON project a total of 200 EU citizens (PwDS, carers and representatives of national organizations supporting PwDS) participated from the workshops and pilots. We gathered evidence people with Down’s Syndrome were both keener and more able to use modern digital solutions than previously perceived (see supporting evidence [So1]). Our system to support people with early stages of dementia relate to an increasing section of our ageing population who is willing to stay independent and healthier for longer whilst the human resources required are not sufficient. There is also increasing awareness in society about the negative effects of sedentary life on humans’ health. Our system to encourage more active lifestyles is building on that awareness to encourage citizens in Barnet to do more physical activity and benefit from that. Some features of the system aims to increase self-esteem, others to provide reasons for habit building and others to increase social contact within the community (see supporting evidence [So2]).

Capacity-building Impact: as part of various projects we developed and perfected methods and tools. They are available in project repositories (see supporting evidence [Ca1]) which we also disseminated through Tutorials and Keynotes at International Conferences as well as eight research students which are now innovators working in other organizations, half of them in business/industry. Our most recent application of the principles and tools we have designed to assist citizens with context awareness has been used by a company, GLL, to transition from traditional gym based physical activity as their only business model into incorporating app supported individual and team based physical activity. (see supporting evidence [Ca2])

Cultural Impact: we informed relevant decision-makers of the findings of our research which have impact in their specific section of society:

  • We helped Down’s Syndrome associations to understand how people with Down’s Syndrome were able to increase independence and improve lifestyle choices through digital tools and how that can improve daily life experience for the family as a whole (see supporting evidence [So1]).

  • We also informed three teams of senior managers from different boroughs in London about the potential of Ambient Assisted Technologies to provide a digital safety net and point of advice to citizens experiencing early symptoms of dementia-like conditions (see [So2]).

Economic Impact: given our products were not marketed by ourselves and consisted of prototypes, methods and tools which help then design other marketable products, we rely on third parties reporting of economic impact. From our products the one which had best traceable benefits so far is our contribution to POSEIDON which has been used since by one of the Scandinavian companies involved in the project. The company used the concepts developed with our help during POSEIDON to improve their offer resulting in a number of different variants of specific products which have been delivered to the European market, especially in the North of Europe. These reportedly produced financial gains to the company which are described to the extent they are traceable by the company Karde (see supporting evidence [Eco1]).

International Reach: some of our activities involved partners from outside the UK and we are aware that at least at European level our work on supporting the developments of contexts and their linking with specific situations of interest for citizens with specific needs has been exploited by a Scandinavian company. Also as part of the legacy of the POSEIDON project we have raised awareness of the digital possibilities for this section of society and we are aware EDSA (the European Down’s Syndrome Association) still considers the project an important landmark (see supporting evidence [Int1]). An independent U.S.A. based science journalist who has written for several of the most important USA newspapers have made several interviews on our work within the Ambient Assisted Living area to include our views and work in an upcoming book addressing the impact of technology in modern indoor living [Int2]. POSEIDON is one of the 90 global projects pre-selected for the World Sumit on the information Society Prizes 2020 edition [Int3].

5. Sources to corroborate the impact

[So1] POSEIDON participant’s questionnaires, interviews, workshops and pilots attracted participation of hundreds of families across Europe. Their feedback and statements from the various non-academic stakeholders were positive on usefulness and usability and can be consulted in the deliverables available in the project webpage: http://www.poseidon-project.org/research-scientists/deliverables/ See also testimony of this in letter issued by DSA-UK and signed by the organization representative. The POSEIDON project then led to a smaller and more focused project with one of their branches: DSActive. They commissioned from us an exploratory research project funding a Master by Research student who produced a prototype of an app to educate children with DS on the concept of healthy food. See letter signed by DSACtive.

[So2] See letter from London Housing Association where they acknowledge their visit to our Hendon Campus Smart Home helped them to understand the benefits of work in this area and also the impact that caused on the visitors and the actions they will take in their respective boroughs: “ *…the visit to the lab changed for better their perception of what is feasible with Smart Homes technological augmentation …. Participants agreed on the importance of taking the knowledge gained from the workshop to their local authorities and other organisations.*”.

[Ca1] A number of guides, methods and tools for user-centred design and development of context-awareness features are offered for free through places such as:

These have been used to create the innovation in the three application clusters we have described, and externally at least by Karde (one of the Scandinavian partners of POSEIDON).

[Ca2] letter from GLL explaining the benefits of engagement with our Research Group on exploring Personalized Behaviour Change Techniques as an alternative to their business options.

[Eco1] See letter signed by the C.E.O. of Karde company, a previous partner of our POSEIDON project, where they acknowledge:

  • the positive influence of our participation in the group: “ *The collaboration with Middlesex University in the POSEIDON project had an impact on society through better understanding of the skills of people with learning disabilities and their abilities to become independent members of the society and actively take part in the working life. Also, our activities had a positive influence on public policy and services towards the target group because of increased attention from politicians and public sector. Finally, our collaboration increased the quality of life for the target group through more independence and better integration in the society.*”

  • and the financial benefits that taking part of POSEIDON brought to their organization “ *Since POSEIDON Karde has won several national and international contracts for projects which aims to help people with learning disabilities. You may say following the spirit of POSEIDON. The projects have lasted one-two years and have had budgets on around 50.000 - 200.000 euros. Karde has had about 20 projects at an average of 100.000 euros which adds up to 2.000.000 euros in the period 2015 – 2020. It will be wrong to say that all of them are results of POSEIDON, but it can be fair to say that 50% is in the POSEIDON spirit, making e-learning and support systems for people with learning disabilities.*”.

[Int1] See reference to the POSEIDON project in www.edsa.eu/poseidon-app/

[Int2] The Great Indoors, by Emily Anthes. Scientific American. 2020. Available from:

https://us.macmillan.com/books/9780374716684

[Int3] www.itu.int/net4/wsis/stocktaking/Prizes/2020/DetailsPopup/15428178241886969

Submitting institution
Middlesex University
Unit of assessment
11 - Computer Science and Informatics
Summary impact type
Technological
Is this case study continued from a case study submitted in 2014?
No

1. Summary of the impact

Middlesex has been advancing digital twin research in programming technologies and applications (Prof. Barn and Prof. Clark (2011-2016) with Tata Consultancy Services (TCS) and in digital twins for structural health monitoring of large-scale infrastructures (Prof. Nguyen).

Impacts have included industry-scale demonstrators for TCS clients, strengthening a robust Java implementation of Enterprise Simulation Language leading to a TCS product TwinX™, design simulations and non-pharmaceutical interventions for managing the COVID-19 Pandemic in Pune, India and software tools to produce a repair and maintenance plan for the Thăng Long bridge in Hanoi, Vietnam resulting in benefits of £9.1 million (£1.5m of savings on repair costs + £7.6m of estimated economy benefit).

2. Underpinning research

Middlesex Computer Science undertakes world-leading research to improve the quality of software systems development. Today, quality of software is dependent upon large-scale, networked, semi-autonomous interdependent systems. Such systems no longer have fixed behaviour and must invariably adapt to achieve their intended function and even change function over time. Consequently, software architects and designers require higher level abstract tools to understand system complexity and interdependencies. Simulation and modelling are core aspects of such work. Beyond software, similar methods of working are also applicable to engineering problems associated with large-scale infrastructures. During this REF research period, these approaches have coalesced around the notion of a Digital Twin (DT). Generally, a DT is a virtual representation of a real-world physical artifact or system that is linked to enable use of real-time data to enable learning, reasoning and dynamically adapting for improved decision making. DTs can be of many types depending upon the features in play and the purpose for which they are defined.

This section describes two broad directions of research activity associated with what is now widely known as a Digital Twin (DT): first, our work on software engineering and, second, our more recent work on structural health monitoring. The integration of model-based software engineering and the proving of conceptual advances in complex scenarios is a hallmark of our approach to DT research.

(a) Software Engineering: Barn and Clark’s work has developed language-based simulation and modelling techniques to design, analyse and adapt the quality-assured development of complex enterprise systems. After initial research on lightweight methods for enterprise modelling, the entirety of this work has been performed in collaboration with TCS and its research lab in Pune. Critically, it is not just an application of research, but much more: it is industry and academia working in collaboration to address emerging industry-scale problems. Barn and Clark’s contribution has led to new methods and technologies, the development of which is described in the rest of this section. The collaboration illustrates how the research has responded and adapted to the emerging sub-discipline of DT that first became significant in 2016.

Existing approaches to digital twin representation of enterprise modelling are not suitable as a basis for simulation and analysis. Barn and Clark’s collaborative research directly led to the construction of an executable modelling language called LEAP (Lightweight Precise Enterprise Architecture), together with a toolset for enterprise simulation [R1]. This research was reported in India’s premier Software Engineering Conference – ISEC, attended by representatives from most of India’s premier industrial research labs. ISEC has an acceptance rate of around 13-15% and is notable for the representation of industrial researchers. The LEAP work was identified by TCS as potential technology that could be exploited by TCS as they embarked on a research strategy focused on idea of the Model Driven Organization (MDO). LEAP thus formed the basis of a collaborative initiative between Middlesex and TCS Research leading to a conceptual research paper published in the Core-ranked A conference - HICSS [R2]. MDO was promoted by a series of dedicated workshops at international conferences. MDO subsequently became integral to the research strategy for TCS software engineering research area.

From 2013 to date, Barn has been hosted annually by TCS Research in Pune India for two weeks to co-develop research that has had a direct influence on the TCS Research strategy and their interactions with clients. These visits led to the co-creation of a technology-based method for organization modelling and decision-making, reported in the leading ACM/IEEE Models Conference series [R3]. This work was a development derived directly from the original work in LEAP. The key contribution of this work is a conceptual meta-model expressed in terms of goals, execution traces, simulation levers, and agents, for constructing enterprise models that can be used to aid decision-makers. Aspects of this work were also executed by TCS’s funding of a distance learning PhD student, a senior scientist at TCS, to undertake doctoral studies at Middlesex. Souvik Barat completed in 2018 and his work, supervised by Barn, led to the development of methodology for enterprise decision making. This annual collaboration, initially seeded by the groundwork of the research published in [R1] has led to new products being marketed by TCS [S7, S8].

To validate the conceptual model, a number of case studies were developed and reported in publications. A particular interest was focused on the use of the technology for modelling socio-technical contexts. Some of that research used ESL and its underlying conceptual model in an exploratory case study examining India’s demonetization strategy of 2016. This validation would provide confidence later in the work done on COVID-19 reported below.

Clark led on the development of a technology to support the modelling and decision-making approach. The agent-based technology takes the form of a novel programming language called ESL and an associated development platform called EDB ( http://www.esl-lang.org, [R4]) both of which are open source. The platform was then extended and used in a Digital Twin context with research being reported in [R5].

Following expansion and evolution of DT research, in 2019, Middlesex established the London Digital Twin Research Centre Centre ( https://dt.mdx.ac.uk/) . Further work now includes: Building Information Management, and Smart Factory (working closely with Festo and Siemens) as well as consolidating work on Structural Health Monitoring. The Centre’s ethos is that modelling gains from application, complexity spurs research innovation, and international collaboration enables both.

(b) Digital Twins for Structural Health Monitoring (SHM):

A well-documented use for Digital Twins is in the area of structural health monitoring of large infrastructure artefacts such as bridges and buildings. In 2018, Prof. Nguyen received funding for two international collaboration grants - from the Newton Fund ( https://dt.mdx.ac.uk/?page_id=36) and from the UKIERI ( https://dt.mdx.ac.uk/?page_id=37).

In collaboration with the University of Transport and Communications (UTC), Vietnam, Prof. Nguyen has been addressing the problem of how to detect damage and predict future maintenance requirements of large infrastructures such as bridges due to aging, environmental change, vehicular loads and other human-induced factors.

The research reported in [R6] addressed the issue of inherent high dimensionality of measured structure data in raw time series sensory signals for SHM using different deep learning techniques to assess the reliability as well as the trade-off between accuracy and of different deep learning models, assisting the relevant stakeholders to make informed decision in maintenance and operation of bridges.

The work in [R7] makes use of physical features embedded in raw data and an elaborated hybrid deep learning model, featuring two algorithms—convolutional neural network (CNN) and long-short term memory (LSTM). Building on advances in algorithms for health monitoring uniting machine learning, structural mechanics and signal processing Nguyen developed a novel hybrid approach that delivered highly accurate results in detecting damage and its severity even for multiple damage scenarios. The resulting method has been a practical end-to-end data-driven framework used for defining a Digital Twin for automatically monitoring the operational state of structures. This framework is reported in the IEEE Transactions on Automation Science and Engineering [R7]. These works were integrated in a Cloud based Digital Twin platform ( http://3.140.199.12/) that was critical to the repair plan development of the Thăng Long bridge in Hanoi, Vietnam by the Ministry of Transport resulting in savings of £1.5 million on repair costs and benefits of £7.6 million in economy [S12].

3. References to the research

Clark, T., Barn, B.S. and Oussena, S., 2011, February. LEAP: a precise lightweight framework for enterprise architecture. In Proceedings of the 4th India Software Engineering Conference (pp. 85-94). ACM. (doi: 10.1145/1953355.1953366)

Clark, T., Kulkarni, V., Barn, B., France, R., Frank, U. and Turk, D., 2014, January. Towards the model driven organization. In 2014 47th Hawaii International Conference on System Sciences (pp. 4817-4826). IEEE. (doi: 10.1109/HICSS.2014.591)

Kulkarni, V., Barat, S., Clark, T. and Barn, B., 2015, September. Toward overcoming accidental complexity in organisational decision-making. In 2015 ACM/IEEE 18th International Conference on Model Driven Engineering Languages and Systems (MODELS) (pp. 368-377). IEEE. (doi: 10.1109/MODELS.2015.7338268)

Clark, T., Kulkarni, V., Barat, S. and Barn, B., 2017, June. ESL: an actor-based platform for developing emergent behaviour organisation simulations. In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 311-315). Springer, Cham. (doi: https://doi.org/10.1007/978-3-319-59930-4_27 )

Barat, S., Kulkarni, V., Clark, T., Barn, B. (2019) An Actor Based Simulation Driven Digital Twin for Analyzing Complex Business Systems. Proceedings of the 2019 Winter Simulation Conference, 2019, Maryland, USA.(doi : 10.1109/WSC40007.2019.9004694)

H. V. Dang, M. Raza, V. T. Nguyen, T. T., Bui, and H. X. Nguyen, “Deep Learning-Based Detection of Structural Damage Using Time-Series Data,” Structure and Infrastructure Engineering, 2020. DOI: 10.1080/15732479.2020.1815225

H. V. Dang, H. Tran-Ngoc, N. V. Tung, B. T. Thanh, G. De Roeck, H. X. Nguyen, “Data-Driven Structural Health Monitoring using Feature Fusion and Hybrid Deep Learning,” IEEE Transactions on Automation Science and Engineering, Nov. 2020. DOI: 10.1109/TASE.2020.3034401

4. Details of the impact

International and industry collaborations have shaped Middlesex’s leadership in applied digital twin research. It has been arrived at from two different starting points but shares the underlying principle of model based design. This section outlines the impacts from both strands of work.

(a) Impacts arising from our Software Engineering research:

The work in software engineering, closely developed with TCS, is an example of a pioneering research collaboration in an emerging area of technology between academia and industry where knowledge transfer is continuous and facilitated by annual sabbatical residencies, presentations to TCS staff, joint publication, and supervision of TCS research staff.

TCS Research is a pioneer in core technology infrastructure to support Model Driven Engineering (MDE) and has delivered several large business critical software systems using this infrastructure for almost 20 years. They recognized that MDE could be applied to address all aspects of an enterprise such as vision, mission, goal, strategies and operational processes. These research questions were congruent with the LEAP research conducted by Barn and Clark [R1] and it is this that shaped the collaboration. The [S1] letter from the Executive Vice President and Chief Technology Officer of TCS confirms the basis of the collaboration, the ongoing work and the benefit of contributions of Barn and Clark to TCS activity.

After the initial influence of R1, the research collaboration between Middlesex and TCS was formally noted in a memorandum of cooperation. A co-designed research agenda was initiated, with the aim that it should lead directly to products for creating agent-based simulation models of enterprises. The research roadmap was conceptualized in the MDO vision paper [R2] and then formalized into a research roadmap report for TCS customers and shareholders [S3].

As a result of this research collaboration, TCS has been able to raise their profile in the area of enterprise modelling and simulation. This has had a positive impact on client satisfaction and loyalty, the generation of new revenue streams and their reputation for research-led innovation. Specific examples include:

  • Development of the TCS world-leading digital twin product – TwinX™ based on the meta model developed by Barn and Clark. The TwinX™ product is now being being provided as solution by TCS to companies such as VodafoneZiggo and has a product manager assigned – Kaustav Bhattacharya [S2].

  • Proof-of-concept demonstrators. The technology arising from the research has been showcased by TCS using proof-of-concept demonstrators [S1]. Case studies covering various business optimisation requirements based on ESL [research reported in R7] were developed. These included: Optimizing operation of a parcel sorting terminal; Optimization of order-to-activation process of a telecommunications company using digital twin based on ESL; and Optimized stock-replenishment for shops in a retail supply chain using digital twin based on ESL.

  • New Business. The extension of ESL with machine learning capabilities described in [R5] has been applied by TCS Research to a commercial problem provided by a European supermarket client. The company has, to date, been using a system-dynamics approach to simulation which takes several weeks to stabilize and produce a solution. Using the ESL-based machine learning simulation, TCS Research has been able to demonstrate the reduction of the stabilization time to a matter of hours. In this case, both the approach and the ESL technology have supported TCS in promoting TCS Research to existing and new clients [S1].

  • Conceptual scope in company R&D. TCS research labs incorporated MDO as one of their research themes as described in their promotional literature to clients and shareholders [S2].

  • Barn and Clark co-supervised a TCS Research employee on a doctoral programme at Middlesex from 2015-18. [S4]. The successful PhD has diffused ESL knowledge within the TCS through the recruitment of three new researchers specifically to develop local expertise in ESL. This was seen as critical for transferring research knowledge to the end user and building capacity in digital twin simulation at TCS.

To reach a broader audience beyond research, Barn is a co-editor of a book initiated by TCS Research ‘Advanced Digital Architectures for Model-Driven Adaptive Enterprises’ with IGI-Global. The book contains by-invitation contributions from industry leaders in the field and aims to establish TCS as a thought leader in the field of model driven enterprises and simulation-based methods [S5]. The book is aimed at practitioners seeking to understand digital twin technology. It was promoted via a post by Sankha Som, Chief Innovation Evangelist to his 500+ connections on LinkedIn. His post reported on the collaboration with Middlesex regarding Enterprise Modelling and stated how this “ area has already made its way into Enterprise Digital Twins that we are developing for various domains such as Telecom…” [S8].

Our earlier research had already indicated that the design approach and the ESL technology was viable in socio-technical systems [R6]. In early April 2020, colleagues at TCS developed a local Java implementation of ESL that originally developed at Middlesex. The TCS implementation has been productized as the Java TwinX™ Library. This software has been used to model scenarios for non-pharmaceutical interventions for the COVID-19 pandemic in the Pune region by working closely with a local Pune based NGO (PrayasPune.org). The fine-grained analysis of COVID-19 pandemic management was led by Dr Souvik Barat using the TwinX™ Java library based on ESL. Details of the simulation are described in [S7] where the role of Dr Barat as lead on the TCS team is documented. This (ongoing – March 2021) activity has been reported in a Pune-based newspaper and in national newspapers and is shown in [S6]. On 03/03/2021, the national TV channel, MirrorNow, ran a debate about Maharashtra’s recent COVID-19 pandemic spike in infections (available at: https://www.youtube.com/watch?v=x48G7-bOvPY). On this programme, Dr Abhay Shukla (national convenor, the People’s Health Movement-India), noted:

“The TCS-Prayas model that has been published, it has predicted the second wave back in mid-December, at a pretty accurate level.” [S9]

(b) Impacts from Digital Twins for Structural Health Monitoring:

The digital twin developments arising from the TCS collaboration and collaboration with Prof. Nguyen led to the Faculty of Science and Technology establishing the London Digital Twin Research Centre (LDTRC). Research outputs [R6, R7] from an international collaboration funded by the Newton Fund and UKIERI were brought to the attention of Ministry of Transport of the Vietnamese Government. The Ministry of Transport collaborated with a project team from Middlesex and the University of Transport and Communications, Hanoi. Using the conceptual and modelling developments reported in section 2. The collaboration has led to:

  • Preparation of a repair plan by the research team for the Thăng Long bridge in Hanoi, Vietnam by the Ministry of Transport resulting in savings of £1.5 million on repair costs.

  • Early re-opening of the Thăng Long bridge helped reduce traffic flow resulting in an estimated further benefit to the economy of Hanoi of around £7.6 million.

  • Policy development in digital transformation for infrastructures in Vietnam. The Ministry of Transport intends to use the digital twin technology developed by Middlesex and UTC on other bridges including Nhật Tân bridge, Cần Thơ bridge, and Mỹ Thuận bridge

The engagement and utilisation of the research [R6, R7] was essential in reducing the maintenance costs and was reported as critical in policy development in helping Vietnam towards digital transformation for infrastructures [S10], which was reported on the national TV programme “Digital Nation” in Vietnam on 7th March 2021 [S11].

5. Sources to corroborate the impact

Corroborating letter from the Chief Technology Officer of TCS Research. (PDF)

TCS Digital Transformation and TwinX™. (PDF)

Research strategy (Conceptual Scope Document) (PDF)

Staff development at TCS (PhD): http://eprints.mdx.ac.uk/26456/

Advanced Digital Architectures for Model-Driven Adaptive Enterprises – Edited Book for Practitioners. https://www.igi-global.com/book/advanced-digital-architectures-model-driven/226277

Newspaper articles from “The Week” and “Hindustan Times”. (See generated PDF)

COVD-19 Simulation report from PrayasPune Last accessed: https://prayaspune.org/health/images/galleries/Brief%20digital%20twin%20covid.pdf

LinkedIn Post by Sankha Som, Chief Innovation Evangelist at TCS. (see generated PDF).

Discussion of the TCS-Prayas Agent based Model on Indian National TV. https://www.youtube.com/watch?v=x48G7-bOvPY

Corroborating letter from Ministry of Transport, Vietnam. (PDF)

Digital Twin work reported on National TV programme ‘Digital Nation’: https://vtv.vn/video/quoc-gia-so-07-3-2021-488990.htm

Submitting institution
Middlesex University
Unit of assessment
11 - Computer Science and Informatics
Summary impact type
Technological
Is this case study continued from a case study submitted in 2014?
Yes

1. Summary of the impact

Middlesex University has pioneered the use of Electrical Impedance Tomography (EIT)

Key impacts:

  • Provision of imaging algorithms and clinical analysis impacting on clinical software (4.1).

  • Creation of the largest clinical data store for EIT clinical data in the world (> 50TBytes) for use by clinicians and industrial/academic researchers (4.2).

  • New wearable hardware for application on patents impacting on clinical usability of EIT (4.3).

  • Used successfully to monitor preterm neonates in the largest clinical study undertaken to date and identifying key parameters for the clinical management of neonates with respiratory conditions impacting on clinical practices (4.4).

2. Underpinning research

What is EIT? : EIT provides impedance changes arising from injection of small electrical currents into an electrode array placed on the subject and the measurement of the subsequent voltages. It can be used to image organ function in real time (100 images a second). Compared with existing technology it is highly portable, inexpensive and lends itself readily to remote imaging in order to save lives. The impact described here evolved from a series of specific developments employing Electrical Impedance Tomography. EIT requires the solution to an inverse problem to create the image of organ function in real time. This requires an accurate geometric finite element (FE) model, known as the forward model.

**First 2D images (1996 –2003)

Prior to REF2014 Prof Bayford pioneered software resulting in successful generation of the first 2D images of impedance change inside the human head using EIT. This led to a range of applications, including neuronal activity, stroke, visual evoked response and localising epileptic activity [1].

**Automatically generating subject-specific FE models (2003 –2007)

The 2D image work led to development of a method of automatically generating subject-specific FE models through elastic deformation from electrode position data for brain function [2]. In collaboration with Great Ormond Street Hospital, this work subsequently resulted in the application of EIT for monitoring lung development in pre-term neonates [3].

**Further algorithm development (2008 to present)

Prof Bayford developed a wavelet algebraic multigrid and estimated boundary form [4]. With an international team of colleagues, he instigated and developed a Graz consensus Reconstruction algorithm for EIT (GREIT). This could then be used as a benchmark for evaluating the effectiveness of future development of EIT algorithms for enhanced monitoring of lung function [5].

This was an award-winning article in Physiological Measurement with significant contribution by the research group at Middlesex comprising accurate forward models of adult male and female thorax, but more specifically of an infant’s thorax. These significant developments of EIT led to four significant grants that ultimately allowed this work to be applied in a clinical setting. The first from EPSRC in 2008, resulted in the contribution to the Electrical Impedance and Diffuse Optical Reconstruction Software project (EIDORS). This is a freely available website that provides software algorithms for forward and inverse modelling for EIT and Diffusion based Optical Tomography in medical and industrial settings. This site is also used worldwide to share data and promote collaboration between groups working in this area (over 2000 downloads and cited on over 100 published papers).

**Clinical translation and hardware development (2016 onwards)

The research described above allowed the team to develop algorithms and hardware for image reconstruction, parameter measurement and boundary form generation [6]. This culminated in the first large scale study monitoring the lung function of 200 neonates (preterm, high risk) for 72 hours each. As a result of this work the team at Middlesex University led a successful EU funding application (Horizon 2020) for €5M in 2016 for a project entitled “Continuous Regional Analysis Device for Neonate Lung (CRADL)” leading to a clinical system for use in neonatal intensive care units. Dr Bardill joined the project in 2016 to progress hardware development, create a new wearable device and continues to be part of the team with Prof Bayford and Dr Tizzard going forward. The work continues with follow-on funding (£1.8M) in early 2020 from EPSRC – “Preterm Neonate/neonatal Embedded Universal Microelectronic wearable, Acquisition for Cardiorespiratory Intensive Therapy” (PNEUMACRIT, EP/T001240). This project further develops clinical hardware for bedside monitoring of lung gestation of pre-term neonates. The research continues to flourish and diversify with the recent award of £700K from UKRI in 2020 to repurpose the hardware and techniques for monitoring Covid19 pneumonia in adult ITUs: COVID Regional Lung Electrical Impedance Tomography (CoRLEIT, EP/V044036).

3. References to the research

[1] Bayford RH, Gibson A, Tizzard A, Tidswell AT and Holder DS, (2001) Solving the forward problem for the human head using IDEAS (Integrated Design Engineering Analysis Software) a finite element modelling tool. Physiological Measurements (Institute of Physics), Vol. 22 No 1. pages 55-63. 0967-3334/01/010055.

[2] Tizzard A and Bayford RH. (2007) Improving the Finite Element Forward Model of the Human Head by Warping using Elastic Deformation. Physiol. Meas. Meas. 28 S163-S182 doi:10.1088/0967-3334/28/7/S13.

[3] Joo Moy Khor, Andrew Tizzard, Andreas Demosthenous and Richard Bayford. (2014) Wearable sensors for patient-specific boundary shape estimation to improve the forward model for electrical impedance tomography (EIT) of neonatal lung function doi:10.1088/0967-3334/35/6/1149.

[4] Bayford RH, Kantartzis P, Tizzard A, Yerworth R, Liatsis P and Demosthenous A. (2008) Development of a neonate lung reconstruction algorithm using a wavelet AMG and estimated boundary form Physiol. Meas. 29 S125-S138 doi: 10.1088/0967-3334/29/6/S11.

[5] A Adler, JH Arnold, R Bayford, A Borsic, B Brown, P Dixon, TJC
Faes, I Frerichs, H Gagnon, Y Gärber, B Grychtol, G Hahn, WRB
Lionheart, A Malik, RP Patterson, J Stocks, A Tizzard, N Weiler, GK
Wolf. (2009) "GREIT: a unified approach to 2D linear EIT reconstruction of
 lung images", Physiol Meas, 30:S35-S55,. ( Awarded IPEM’s Martin Black prices for best paper)

[6] Sven Nordebo, Mariana Dalarsson, Davood Khodadad, Beat Muller, Andreas Waldman, Tobias Becher, Inez Frerichs, Louiza Sophocleous, Daniel Sjoberg ,Nima Seifnaraghi , Richard Bayford. (2018) A parametric model for the changes in the complex valued conductivity of a lung during tidal breathing in Journal of Physics D Applied PhysicsDOI: 10.1088/1361-6463/aabc04.

4. Details of the impact

Software and data impact:

(4.1) The GREIT algorithm with new forward models is being adopted by a manufacturer of EIT systems (Swisstom/SenTec)1, which will represent a significant improvement for commercial medical EIT systems in product development for EIT-based monitoring of neonate patient respiration and regional air content within patient’s lungs at the bedside. (2018) (5.1) It has also been adopted by Emergex to extend EIT for other applications including cancer detection. We have also provided new models for these applications (See 4.2)

The models generated have also been used extensively by other groups internationally that focus on the development of imaging solutions. For example, Bayford and Tizzard, working with Dartmouth College and Florida State University in the USA, have been developing EIT and optical tomography imaging of both adult and neonate human heads and extensive use is being made of the public domain tool (eidors3d.sourceforge.net/) in this work.

Further work on the automatic generation of subject-specific forward models, namely the warping algorithm (2007), and with Prof. Janet Stocks, Great Ormond St. Hospital (2007 – 2009) which formed the basis of extending the process in the current REF period to imaging lung function specifically in neonates in collaboration with Prof. Andreas Demosthenous, Dept. Electronic and Electrical Engineering, UCL. This initial impact led to the CRADL project in 2016 (5.2).

(4.2) The creation of the largest data store for EIT clinical data in the world at Middlesex University (over 50TBytes). This resource is in use for ongoing clinical studies (5.4, 5.5 and 5.6). This resource is also being used by SenTec to improve their user interface and test their system.

The work improved breast tumour imaging in collaboration with Dr Andrea Borsic and Prof. Ryan Halter, Dartmouth College, NHR (2010), and is based on using elastic deformation to warp standard or idealised geometry – all of which provided extensions of the public domain toolset.

Hardware impact:

(4.3) In addition to its clinical use, the group obtained a patent that describes a flexible wearable device to extract boundary information for the warping algorithm. The system dynamically generates and modifies subject specific forward models in real time. This work addresses the urgent need for objective, non-invasive measures of lung maturity and development, oxygen requirements and lung function, suitable for use in small, unsedated infants, to define the nature and severity. This led to the following:

We have signed an NDA with Swisstom (now Sentec) to develop the wearable device [(H2020 cradlproject.org)(5.3,5.3)] and have a patent in place (Filed in 2015, WO2015025113A1, European patent number 3035846 2020). A new wearable device was developed with PEL during CRADL and are submitting a patent before arranging a licensing agreement.

The group is also working with Emergex to extend the application of EIT for the detection of cancer. A joint patent (WO/2010/052503, Detection of Cancer) is in place with this company. This work is also subject to an NDA, which limits the information we are allowed to disclose in this document. However, the system is presently being developed to locate new COVID vaccines in animal models in the USA.

Clinical Practice impact:

(4.4) Our models have been used successfully to monitor preterm neonates in two clinical studies using the CRADL system with the University of Oulu and Oulu University Hospital, Finland, and the Department of Neonatology, Emma Children’s Hospital, Amsterdam. The group’s electrical impedance tomography system has been used to detect ventilation distribution, end-expiratory lung impedance (EELZ) and tidal impedance variation during monitoring of preterm neonates requiring invasive ventilation and repeated surfactant treatment. This study demonstrated a significant effect of surfactant treatment on lung function (5.4, 5.6). This has enabled the use of EIT in new studies for adopting into larger clinical use.

Creation of the largest data store for EIT clinical data (> 50TBytes). This is an available resource for ongoing clinical studies (5.9) including Dartmouth and Florida use of EIDORS. It has

been used successfully to monitor preterm neonates in four large-scale clinical studies (5.3). The following organisations have benefited from these resources:

  • Consultant Paediatric Cardiologists and clinical researchers in the PEDEGO Research Unit, University of Oulu and Oulu University Hospital, Finland (2015 - on-going) (5.4)

  • The Department of Neonatology, Emma Children’s Hospital Amsterdam undertook the clinical study of CRADL (5.6)

  • Dr Karaoli Nicosia General Hospital (NGH), Cyprus also undertook the clinical study.

Consultant Neonatologists and clinical researchers at the Royal Hospital for Children, Glasgow, Scotland are working with the group on a new clinical study related to the PNEUMACRIT5 project which has been enabled through CRADL (5.5).

More clinical groups are using EIT as a result of the work at Middlesex and are identifying outcomes for imaging neonate lung function. The work has led to the recognition that EIT can address the urgent need to improve ventilation strategies in children. It is been clinically used to monitor lung function in neonates and adult patients (see link to Draeger/Sentec below) using some of the developments created for neonate imaging (5.8).

Industry impact:

(4.5) We are also working with PEL (Printed Electronics Limited)7 a UK based technology company providing advanced research and development, concept development and production capability for printed electronics and related functional material structures and systems, to develop print on flexible printed circuits for the EIT neonate system. They are members of the EU Graphene Flagship. PEL has worked with us on the CRADL, PNEUMACRIT and new CoRLEIT projects. PEL 3D printing are working with their commercial abilities to augment those in the Middlesex teams and concurrently we are enabling the company to get new markets and new business opportunities in the future (5.7)

Cost saving:

(4.6) EIT estimated cost saving of 928 to 10,705 euros per patient in the Dutch setting or 1,124 to 8,496 euros in the German setting. (5.10)

5. Sources to corroborate the impact

  1. https://www.sentec.com/products/eit/lms/lms-n/ and https://www.sentec.com/products/eit/lm-disposables/ (web links to SenTec showing the products produced from the Cradl project lead by Middlesex University.

  2. https://cordis.europa.eu/project/id/668259 EU project link for CRADL showing all the partners in the project showing that EIT is impacting on many organisation who had not previously been involved with it use. This includes Cyprus who had not previously been using EIT for clinical practice, hence increasing the user base of this technology.

  3. Web site (cradlproject.org) showing the device used in Hospitals (see video on website). Middlesex led the development of the CRADL project and coordinated it, along with key contributions in hardware (new belt designs) and software. This shows its impact on all areas, software, hardware and clinical practice. The industry partner was able to disseminate the use of EIT to a wider clinical group at conferences and major trade shows (Full list in evidence appendix)

  4. Consultants in the Pediatric Cardiologyand Intensive Carein the PEDEGO Research Unit, University of Oulu and Oulu University Hospital were invited as associate partners in the CRADL project which enabled them to adopt EIT in clinical practice. They are using EIT in Finland to develop new clinical management methods (See Oulu support letter)

  5. Consultant Neonatologists and clinical researchers at the Royal Hospital for Children, Glasgow are working with us on PNEUMACRIT to extend the technology and impact on clinical practices for a multi sensor system on neonates along with PEL (See Glasgow support Letter).

  6. Consultants from the Department of Neonatology, Emma Children’s Hospital, Amsterdam have used the CRADL system in clinical studies (See Amsterdam support letter) They are using EIT to identify apnoea in infants among other conditions.

  7. PEL (Printed Electronics Limited) (printedelectronics.com), helped in the development of the EIT belt and is involved in adapting it for COVID-19 use. (See PEL support letter) PEL is a UK based technology company providing advanced research and development, concept development and production capability for printed electronics and related functional material structures and systems. They are members of the EU Graphene Flagship. We developed with them flexible circuitry in the form of belts made from a soft fabric material that provide unintrusive patient interfaces for impedance spectroscopy in Covid patients and neonates. They contribute expertise and support on print on flexible printed circuits and access to their facilities (See support letter, and NDA would be needed with external partners).

  8. https://www.3sat.de/wissen/nano/201016-sendung-nano-102.html (Interview for German TV Nano Science program on CRADL (15mins into programme). Impacting on the visibility of EIT for clinicians, patients and general public. (web pages in support letter).

  9. The CRADL data is available on request subject to a number of ethical and GDPR requirements. It includes images of infants. The data is being used in studies to improve clinical management of patients. We can provide access to the data if the REF panel request it subject to the required conditions.

  10. CRADL implementation in NICU’s can lead to substantial medical cost savings especially in hospitalization and complication cost, while leading to improved health outcomes. The health economic analysis predicts the technology to be cost-effective in terms of ICER per deaths avoided and ICER per BPD cases avoided in the German and Dutch setting. (Cost benefit analyses undertaken by Panaxea, (isabelle.nefkens@panaxea.eu)(report available on request).

Submitting institution
Middlesex University
Unit of assessment
11 - Computer Science and Informatics
Summary impact type
Technological
Is this case study continued from a case study submitted in 2014?
No

1. Summary of the impact

.

Prof. B.L.W. Wong led the 17-organisation, €16.6 mil, EU-funded VALCRI project, Visual Analytics for Sense-making in Criminal Intelligence Analysis, from 2014-18. The output was a visual analysis system using tactile reasoning which enhances criminal and intelligence investigations.

Direct impacts claimed are:

  • Economic – through commercialisation. The VALCRI IP was acquired by Canadian global security systems company, Genetec Inc, with paying customers from Sep 2019

  • Improved performance, practices and policies for police investigations and subsequent societal benefits. VALCRI has already raised skills and technology awareness across a number of police and intelligence communities

  • Informed public debate - through active dissemination on best practice in intelligence-led policing.

2. Underpinning research

The VALCRI project, Visual Analytics for sense-making in Criminal Intelligence Analysis, (FP7-IP-608142, see valcri.org), is a €16.6 mil EU FP7 Integrating Project, that was driven by Wong’s research into the representation design of information and the interaction design of user interfaces to support human decision making in complex dynamic environments. He uses concepts and techniques from human-computer interaction, human factors, and cognitive engineering to understand and model the nature of expertise and cognitive work in these domains, and then develops appropriate technologies to support such work. In VALCRI, he continues research into the problems associated with making sense of big data using visual analytics, a recently established field that focuses on creating human-information discourse by coupling interactive visualisations with automated data analysis.

In prior research, Wong led other visual analytics research projects including the UKVAC (UK Visual Analytics Consortium) jointly funded by the US DHS and UK Home Office (2010-13; PNNL Contract No. 116077, DHS funded through NVAC at PNNL; Agreement No. 4112-46065, Sub-Agreement under the US DHS Cooperative Agreement No. 2009-ST-061-CI0001; Agreement with Home Office, dated 21 March 2012); and the EPSRC MakingSense project (2010-13; EP/H023135/1). In VALCRI, he combined this work with earlier research in which he invented the interaction design INVISQUE (2009; JISC RI Ref. No. IEDEVC19), the interactive visual search and query environment, that makes information grasp-able, enabling ‘tactile reasoning’, an epistemic action that facilitates sense-making and decision making. These ideas and principles drove the design of VALCRI.

Wong led the VALCRI consortium that comprised 9 universities and research organisations, 5 SMEs, and 3 Law Enforcement Agencies (LEAs) from across Europe. The Consortium’s 103 scientists and engineers, brought together a diverse set of expertise including ethics, privacy and law, human bias, sense-making and insight, argumentation and logic, knowledge management, ontology engineering, complex events processing, video analysis, data mining and semantic extraction, algorithm design, software security and access control, training design and development, and user interface design, to research and develop technology alongside an international team of police end-users. In this section we describe the research that underpinned the IP developed and owned by Middlesex University which was acquired by Genetec Inc.

Police intelligence analysts only ever have fragmented data from which to investigate cases and pre-empt terrorist attacks. They also operate in data-overload situations where they trawl through large volumes of forensic, operational, structured and unstructured data in multiple databases to discover the required data from which to construct the chains of evidence needed to investigate or prosecute cases. Police therefore need a combination of tools to discover relevant data across vast data sets e.g. other police reports that might be related to a case, while trying to make sense of fragmentary pieces of data.

We developed a model of sense-making for police intelligence analysis which we used to guide the design of the VALCRI system. The model was based on a number of different cognitive task analysis studies with intelligence analysts, some pre-dating the VALCRI project e.g. [Ref1] and a series of studies on ‘how analysts think’ conducted during the project e.g. [Ref2]. We discovered that analysts regularly use their expertise and intuition to make abductive inferences to create early, tentative explanations. Analysts also use inductive and deductive inference-making strategies, depending on what data is available, what rules are applied, and what goals are desired. Intuition plays an important role to create leaps of faith that produce insight and new lines of inquiry, laddering to elaborate, find or make associations. Critically, they also test the hypotheses.

Police face significant difficulties in discovering relevant data, partly due to the lack of adequate tools to discover relevant data in the large volumes of data in their systems. Criminals rarely provide enough data for police to prosecute them. When analysts discover relevant data, they assemble it into evidential chains and create narratives to explain what they concluded. These chains also need to be compliant with the laws of argument and logic so that such arguments can withstand interrogation, especially in a court of law. VALCRI technology aims to enable and support this model [Ref3].

The uniqueness and appeal of the VALCRI technology lies in the radically different user-interface based on the concept of tactile reasoning [Ref4]. Information is presented in ‘tiles’ which can be touched and moved around freely, as in the game of Scrabble. Initial ideas were researched as part of the INVISQUE project to develop a more tangible, epistemic action-based method for interacting with electronic library resources.

Each tile contains data or analysis functions such as a statistical process control chart to analyse crime patterns. These tiles can be assembled into arguments or explanatory sequences to re-construct crime events. Our studies suggest that user creativity and analytic reasoning performance can double when a user can rapidly re-arrange the tile sequences. We also found that users were better at maintaining provenance, or tracing their analytic reasoning pathways with the tactile reasoning approach.

Underlying the tile interaction are machine learning [Ref5] and database, e.g. ElasticSearch, algorithms that perform semi-automated semantic searches of both structured and unstructured text fields to discover similar reports in other databases and possibly associated information such as gang networks, using Formal Concept Analysis [Ref6].

We applied principles of representation design by mapping the analytic reasoning process on to key elements of the user interface. The VALCRI technology was designed around the concept of fluidity and rigour, where the interaction and transition between steps in a task is fluid, while ensuring that analytic processes retain the necessary analytic rigour.

The key outcome from this aspect of the VALCRI research has been a visual analytics technology where users can interact fluidly while ensuring analytic rigour, where hypotheses can be formulated and tested quickly, enabling investigators to discard or modify their hypotheses within minutes and hours, rather than days and weeks.

3. References to the research

[Ref1] Rooney, C., Attfield, S., Wong, B. L. W., & Choudhury, S. T. (2014). INVISQUE as a tool for intelligence analysis: the construction of explanatory narratives. International Journal of Human Computer Interaction, 30(9), 703-717.

[Ref2] Wong, B. L. W., & Kodagoda, N. (2015). How analysts think: Inference making strategies. In Proceedings of the Human Factors and Ergonomics Society 59th Annual Meeting, 26-30October 2015, Los Angeles, USA (pp. 269-273): SAGE Publications.

[Ref3] Wong, B. L. W., Seidler, P., Kodagoda, N., & Rooney, C. (2018). Supporting variability in criminal intelligence analysis: From expert intuition to critical and rigorous analysis. In G. Leventakis & M. R. Haberfeld (Eds.), Societal Implications of Community‐Oriented Policing Technology (pp. 1-11). Cham, Switzerland: Springer Open, Springer International Publishing AG.

[Ref4] Takken, S., & Wong, B. L. W. (2015). Tactile reasoning: Hands-on vs. Hands-off - what's the difference? Cognition, Technology & Work, 17(3), 381-390. doi:10.1007/s10111-015-0331-5.

[Ref5] Jentner, W., Sacha, D., Stoffel, F., Ellis, G. P., Zhang, L., & Keim:, D. A. (2018). Making machine intelligence less scary for criminal analysts: reflections on designing a visual comparative case analysis tool. The Visual Computer 34 (9), 1225-1241

[Ref6] Qazi, N., Wong, B. L. W., Kodagoda, N., & Adderley, R. (2016). Associative Search through Formal Concept Analysis in Criminal Intelligence Analysis. In Proceedings of 2016 IEEE International Conference on Systems, Man, and Cybernetics • SMC 2016 October 9-12, 2016, Budapest, Hungary: IEEE Press.

4. Details of the impact

We claim 3 forms of direct impact from our work:

1) Economic impact from commercialisation
  1. Twenty-four pieces of VALCRI IP produced by the Middlesex University team were acquired by Genetec, Inc., a Montreal-based, global security systems company, in April 2019 [Source A]. Genetec created a new business unit to market the new Valcri™ product, creating 14 new jobs to commercialise VALCRI. To enable the knowledge transfer, Wong and 4 key researchers from his lab were employed by Genetec in the new Valcri™ Product Group. The VALCRI acquisition catalysed Genetec to invest in a new Genetec UK head office in London in November 2019 to showcase Valcri™ and other Genetec products. Genetec is also combining VALCRI with its other security systems to create new products to open new markets. Genetec Valcri™ secured its first paying customer in September 2019 with expected annual sales thereafter. Potential Market Value: The US alone has 17,985 law enforcement agencies, although pricing information is competitor-sensitive, we estimate potential US market value at USD $3,579,000,000.

  2. i-Intelligence GmbH is a Zurich-based SME which specializes in teaching intelligence analysis globally. They translated VALCRI research in analytic reasoning and sense-making into curricula for training police analysts. Following trials during the VALCRI project with 214 law enforcement officers from 50 agencies in 16 countries, i-Intelligence completely overhauled their original training programs, and developed new executive level training that focused on managing complexity, ambiguity and uncertainty; issues central to VALCRI. [Source B]

  3. DSTL invests over £150,000 to productise VALCRI research. During the final year of VALCRI, further work was undertaken to address algorithmic opacity, i.e. the lack of transparency of machine-learning-based black-box algorithms. Our PhD student Sam Hepenstal, who works for DSTL (UK MoD), developed a conversational agent system for investigations based on our Algorithmic Transparency Framework. This enables a user to challenge the results, while also inspecting and verifying the system processes. This work has matured such that Dstl invested over £150,000 in 2020 to create a commercial product from this research prototype. [Source C]

2. Improved performance, practices and policies for police investigations and subsequent societal benefits

VALCRI has already raised skills and technology awareness across a number of police and intelligence communities, including:

  1. Partner West Midlands Police (WMP) have described the following impacts: (i) Through VALCRI’s research a wider range of analytic techniques have become central to the analyst training curriculum developed by WMP for the new Intelligence Academy. (ii) VALCRI’s research on ethical and privacy issues led the WMP to realize the importance of transparency for public services and the analysis of interconnected big data. New policies resulted for analysis of big data in their new Data Analytics Lab and the setting up of an Ethics Committee at WMP so that issues may be rationally debated while awaiting legislation. (iii) The WMP also re-developed one of VALCRI’s modules – the Statistical Process Control (SPC) Map, into a product compatible with their newly procured platforms. [Source D]

  2. Partner Lokale Politie Antwerpen (LPA) through their collaboration in VALCRI gained significant insights on how to design technology to transform traditional reactive-based policing methods to more pro-active Intelligence-Led Policing model. These insights led to an in-house developed software called ‘Focus’, comprising a suite of near-real-time crime analytics software for their newly formed department based on Intelligence-led policing. The VALCRI-inspired ‘Focus’ system is in use by all officers in the field and various analysis departments. Analysts can link data, discover associations, and present crime data in context. These are key VALCRI design concepts. [Source E]

  3. Partner Belgian Federal Police (BFP) adapted the associative search and visualization methods developed in VALCRI to build their own software for identifying large, organized crime groups. The software performed auto-discovery of crime organisations by searching for patterns of related activities, objects and flows, and associations of people and places. This has been used in the “Kali Beeldvorming” joint task force to speed up the discovery of information for identifying and taking down of large, organized crime groups and their networks. [Source F]

3. Informing public debate about intelligence-led policing through active dissemination
  1. VALCRI research sparked significant interests in the media after highlights of our research was published in the ‘New Scientist’ on 10 May 2017, “AI detective analyses police data to learn how to crack cases” [Source G]. A number of separate articles followed subsequently: Digital Trends, Government Technology, Legal Reader, Deccan Chronicle, ETech.com, and in the EU Research Magazine, 24 Jan 2018, “Plotting a path through crime data” [Source H].

  2. After the close of the project, VALCRI was chosen to be published in the European Commission’s “Success Stories” series in November 2018 as “Cyber-detective assists police with criminal investigations” [Source I].

  3. In 2020, VALCRI was selected as an example of best practice in UK on ‘ethics by design’. It was presented at the Roundtable on Crime Prevention, Justice and Artificial Intelligence in Latin America, 22 Oct 2020, organized by the United Nations Interregional Crime and Justice Research Institute (UNICRI), the British Embassy in Mexico, and C-Minds. Wong presented to Latin American government and police leaders, on how ethical safeguards were designed in VALCRI, and therefore how they can be incorporated into intelligence-led policing initiatives.

5. Sources to corroborate the impact

.

Letters of Support
  1. Letter of Support from Genetec, Inc.

  2. Letter of Support from i-Intelligence GmbH

  3. Letter of Support from DSTL

  4. Letter of Support from WMP

  5. Letter of Support from LPA

  6. Letter of Support from BFP

Key References relating to Section 4 Details of the Impact: Informing Public Debate about intelligence-led policing through active dissemination
  1. “AI detective analyses police data to learn how to crack cases”, New Scientist, 10 May 2017, see https://www.newscientist.com/article/mg23431254-000-ai-detective-analyses-police-data-to- learn-how-to-crack-cases/

  2. “Plotting a path through crime data”, EU Research Magazine, 24 Jan 2018, see https://issuu.com/eu_research/docs/valcri_eur14_low_res

  3. “Cyber-detective assists police with criminal investigations”, in European Commission’s “Success Stories” Series, 27 November 2018, see https://ec.europa.eu/research/infocentre/article_en.cfm?id=/research/headlines/news/article_18_11_27_en.html?infocentre&item=Infocentre&artid=49801

Showing impact case studies 1 to 5 of 5

Filter by higher education institution

UK regions
Select one or more of the following higher education institutions and then click Apply selected filters when you have finished.
No higher education institutions found.
Institutions

Filter by unit of assessment

Main panels
Select one or more of the following units of assessment and then click Apply selected filters when you have finished.
No unit of assessments found.
Units of assessment

Filter by continued case study

Select one or more of the following states and then click Apply selected filters when you have finished.

Filter by summary impact type

Select one or more of the following summary impact types and then click Apply selected filters when you have finished.

Filter by impact UK location

UK Countries
Select one or more of the following UK locations and then click Apply selected filters when you have finished.
No UK locations found.
Impact UK locations

Filter by impact global location

Continents
Select one or more of the following global locations and then click Apply selected filters when you have finished.
No global locations found.
Impact global locations

Filter by underpinning research subject

Subject areas
Select one or more of the following underpinning research subjects and then click Apply selected filters when you have finished.
No subjects found.
Underpinning research subjects