Impact case study database
Search and filter
Filter by
- The University of East Anglia
- 11 - Computer Science and Informatics
- Submitting institution
- The University of East Anglia
- Unit of assessment
- 11 - Computer Science and Informatics
- Summary impact type
- Environmental
- Is this case study continued from a case study submitted in 2014?
- No
1. Summary of the impact
Computer vision methods developed by Dr Mackiewicz and co-workers in the School of Computing Sciences (CMP), UEA are being used to monitor animal populations. Utilising computer vision and deep learning technology for environmental monitoring, CMP researchers are working with the Scottish Government, Marine Scotland, the British Antarctic Survey (BAS), and the Centre for Environment, Fisheries and Aquaculture Science (CEFAS) to monitor swarms of jellyfish, fish catches, and albatross populations. This has resulted in economic, commercial and environmental impacts including protection of nuclear power stations and desalination plants against jellyfish ingress (the closure of which can cost GBP1,000,000 per day), protection of threatened species, easier policy enforcement and new markets for industrial collaborators.
2. Underpinning research
JellyMonitor: The clogging of the seawater intake screens of coastal nuclear power stations and desalination plants by jellyfish is a concern for safety, security of supply and operational costs. JellyMonitor (2015-2018) was a research project aimed at developing a portable imaging platform that would detect jellyfish near to water intakes and provide an early warning of jellyfish ingress. Initial funding was from Innovate UK [Grant 1] for a collaborative project between the Centre for Environment, Fisheries and Aquaculture Science (CEFAS), CEFAS Technology Ltd (CTL), UEA, and EDF R&D UK Centre. The research continues with a NERC & EPSRC iCase PhD studentship (2018 to 2022) co-funded by the NEXUSS Centre for Doctoral Training (CDT) and CEFAS [Grant 2].
Within the collaborative Innovate UK project, the computer vision and machine learning algorithms were developed almost entirely by CMP researchers at UEA. The software component of JellyMonitor is designed to analyse sonar imagery in real-time whilst operating within the computational constraints of an embedded platform deployed on a seabed [3.1]. This novel system combines some conventional computer vision techniques to detect and track potential objects of interest, and a state-of-art deep neural network classifier, developed by researchers at UEA, to categorise and label them. This process was complicated by the large quantities of noise present in sonar imagery, the highly imbalanced dataset that was used to develop the prototype, the few occurrences of jellyfish in the dataset that were available during the development, and a very strict energy budget for the battery powered embedded platform. Consequently, the UEA expertise in semi-supervised image classification and domain adaptation [3.2] and the application of these was essential in achieving a working prototype of JellyMonitor.
CatchMonitor: Within the marine fishing industry, CCTV cameras improve data gathering on the observation of discards and support the catch quota management scheme. In 2014 the Scottish Government invited tenders to research and develop an automated video analysis system which could replace or enhance human observations of recorded videos. This work for Marine Scotland and the Scottish Government was carried out by CMP researchers as two small feasibility projects from 2014 to 2017 [Grants 3 & 4]. The developed algorithms comprising fish segmentation, classification and counting and the results confirming the feasibility of the application have been described in [3.3].
The results obtained during the feasibility projects allowed Marine Scotland and UEA to continue this collaboration as part of a much larger research consortium including commercial partners - H2020 Smartfish [Grant 5]. The result was the prototype of CatchMonitor [3.4] [3.5] - a computer vision system which can be installed on board fishing trawlers with the aim of monitoring and quantifying discarded fish catch. CatchMonitor software comprises four distinct parts: web user interface; isolation (segmentation) of each individual fish (based on Mask-RCNN – the state-of-the-art segmentation deep neural network); classification of segmented fish into a number of fish species (deep network based on ResNet – a modern classification deep neural network); and counting and classifying those fish instances in the video which end up as discards. Here, a novel algorithm was required as the view of the conveyor belt is often occluded by humans, which interrupts the fish tracking. The method developed at UEA is based on integration of the individual frame segmentation results, which are subsequently processed as fish density pixel maps and further aggregated to produce fish counts for a set of relevant species. Within the collaborative project, the algorithms constituting the software of CatchMonitor were developed solely at UEA. As with JellyMonitor, the expertise of the UEA team and their research in semi-supervised image classification and segmentation was key to the success of the project [3.2, 3.6].
Albatross Monitoring: Since 2017, UEA researchers have been working with the British Antarctic Survey on the development of a technology for albatross recognition and counting in satellite images, funded by NEXUSS CDT [Grant 6]. The developed method builds on the wider computer vision and deep learning expertise present within CMP at UEA. The algorithm is based on the U-NET deep learning architecture with the focal loss criterion that was required to tackle the extreme class imbalance that is one of the problems in this application [3.7].
3. References to the research
The underpinning research outputs have all been published in competitive, international, peer-reviewed journals/conferences and form part of a larger body of such published work.
(UEA authors highlighted in bold)
- JellyMonitor: Automated Detection of Jellyfish in sonar images using neural networks
French, G., Mackiewicz, M., Fisher, M., Challis, M., Knight, P., Robinson, B. and Bloomfield, A.
( 2018) IEEE International Conference on Signal Processing Proceedings, ICSP.
DOI: 10.1109/icsp.2018.8652268
- Self-ensembling for visual domain adaptation
French, G., Mackiewicz, M., Fisher, M.
( 2018) IEEE International Conference on Learning Representations ICLR.
- Convolutional Neural Networks for Counting Fish in Fisheries Surveillance Video, Machine Vision of Animals and their Behaviour
French, G., Fisher, M., Mackiewicz, M. and Needle, C.
( 2015) Workshop at the 26th British Machine Vision Conference. DOI: 10.5244/c.29.mvab.7
- Deep neural networks for analysis of fisheries surveillance video and automated monitoring of fish discards
French G., Mackiewicz M., Fisher M., Holah H., Kilburn R., Campbell N., Needle C. ( 2019 ) ICES Journal of Marine Science. DOI: 10.1093/icesjms/fsz149.
- H2020 Smartfish
Mackiewicz M., French G., Fisher, M.
( 2019) Public Deliverable 4.3, Prototype of CatchMonitor Dec 2019 (held on file at UEA)
- Semi-supervised semantic segmentation needs strong, high-dimensional perturbations
French, G., Aila, T., Laine S., Mackiewicz M., Finlayson G.
( 2020) In Proc. of the 31st British Machine Vision Conference (BMVA) arxiv.org/abs/1906.01916
- Using deep learning to count albatrosses from space: Assessing results in light of ground truth uncertainty
Bowler, E., Fretwell P., French, G., Mackiewicz M.
( 2020) Remote Sens. 2020, 12(12), 2026. DOI: 10.3390/rs12122026
Underpinning Funding
Grant 1: JellyMonitor: developing a jellyfish early warning system for coastal power stations
Collaborators: CEFAS Technology Ltd, Lowestoft; Centre for Environment, Fisheries and Aquaculture Science (CEFAS), Lowestoft, UEA, and EDF R&D UK Centre, London.
PI: Mark Fisher. Funder: Innovate UK. GBP126,415; 4/2015 - 3/2018
Grant 2: Four year iCase PhD studentship. Funded by CEFAS and NEXUSS CDT (NERC & EPSRC); 1/10/18 - 31/05/22
Grant 3: Automated Image Analysis for Fisheries CCTV
PI: Mark Fisher. Funder: Marine Scotland, Scottish Government Environment and Forestry Directorate. GBP74,400; 2014 - 2015
Grant 4: Enhanced Automated Image Analysis for Fisheries CCTV project (follow-up grant from Grant 3)
PI: M ark Fisher. Funder: Marine Scotland. GBP37,919; 2016 - 2017
Grant 5: Smart fisheries technologies for an efficient, compliant and environmentally friendly fishing sector: H2020 Smartfish
PI: Michal Mackiewicz. Funder: CEFAS Collaboration. GBP324,710; 1/2018 - 12/2021
Grant 6: Four year PhD research (2017 - 2021) funded by NEXUSS CDT (NERC & EPSRC)
4. Details of the impact
This case study demonstrates economic, commercial and environmental impact arising from research in CMP at UEA, including protection of nuclear power stations and desalination plants, protection of threatened species, easier policy enforcement and new markets for industrial collaborators.
Monitoring and Protecting Power Stations
In the right conditions, jellyfish can swarm and create serious problems for nuclear power stations, which rely on large volumes of cool sea water to control internal temperatures. If access to this supply is lost, the power stations have to shut down with consequent substantial financial losses. In 2011, the shutdown of the Torness Nuclear Power Station caused a loss of GBP1,000,000 per day for the parent energy company and a 20-day shut down in East Java of the Paiton Coal Power Plant cost USD21,700,000 (04/2016) [corroborating source A]. This is a global problem with further notable shutdowns in Oskarshamn Sweden in 2013 and Diablo Canyon, USA in 2008. The Director of CEFAS Technology Ltd (CTL) has stated that:
[textredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredacted]
[corroborating source B]
The project funded by the Innovate UK [Grant 1] resulted in the development of a fully functioning prototype of the jelly fish detection system – JellyMonitor – see Figure 1. The system was developed jointly by CTL and UEA with the support from EDF. The implementation uses a target embedded hardware platform specifically designed so that it can operate autonomously in the marine environment for the duration of the jelly-fish season. [textredactedtextredacted textredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredacted]
Figure 1
[textredactedtextredactedtextredacte dtextredactedtextredactedtextredacted textredactedtextredactedtextredacted textredactedtextredactedtextredacted textredactedtextredactedtextredacted textredactedtextredactedtextredacted textredactedtextredactedtextredacted textredactedtextredactedtextredacted textredactedtextredactedtextredacted textredactedtextredactedtextredacted textredactedtextredactedtextredacted]
Over the forthcoming three years there will be annual sea trials of the prototype over the entire jelly fish season so that more data can be captured. This will allow incremental changes to the system to be tested as CTL moves towards a commercial product.
The Director at CTL has stated:
[textredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredacted]
[corroborating source B]
Monitoring Fishing Discards
Marine Scotland is an agency of the Scottish Government responsible for management of Scotland’s seas. Marine Scotland first installed CCTV cameras on board fishing vessels in 2008 as part of a scheme to monitor fish discards, with the aim of facilitating the recovery of the North Sea cod stock. It quickly became clear that the volume of data would need a novel method of analysis when it took one person 3 full months to count and analyse the discarded fish from a single week-long trip of a Scottish trawler using recorded CCTV. A solution to this problem is especially pressing given that it is likely that CCTV systems on fishing vessels are likely to become the norm rather than the exception. [corroborating source C].
CatchMonitor was developed by UEA researchers in conjunction with Marine Scotland as an advanced onboard catch monitoring system allowing automatic image analysis of fish discards. The products developed are beginning to have significant impact on the fishing industry through the collaboration with Marine Scotland, Cefas and the wider H2020 consortium. The improvements in the accuracy and quantity of fish stock assessment data increases compliance with fishery regulations leading to reduction of fishing pressure and the environmental impact of fisheries. CatchMonitor is considered to be at Technology Readiness Level 5, with development work continuing with Marine Scotland on board their research vessel FRV Scotia. [corroborating source C]
Policy Development
The two articles on Marine Fishing monitoring [3.3, 3.4] were cited over fifty times including several times in policy making publications e.g., in Marine Policy [corroborating source D] and The European Landing Obligation. [corroborating source E]
Public Understanding
Our work has received considerable international publicity both in general popular science reporting and in the science policy press. [corroborating source F]
Albatross Monitoring
Despite all six species of Great Albatrosses are under threat according to the IUCN Red List, many colonies are not surveyed for decades at a time due to their remote and inaccessible nesting locations. Filling in the existing significant gaps in knowledge of distribution, population sizes and demography will significantly help in the conservation management of this endangered species. Using state of the art 31-cm resolution satellite imagery and computer vision methods, UEA researchers working alongside scientists from the British Antarctic Survey (BAS) have addressed this issue by automatically surveying populations directly from space. The developed protocol enables annual global censuses, which would not be possible with human observers, vastly improving our understanding of population trends and informing conservation action. Following the development of the algorithm at UEA, BAS has decided to upscale the albatross survey work to cover the whole Southern Ocean. Since December 2019, they have had an agreement in place with Maxar (DigitalGlobe) to task their satellites over Wandering Albatross breeding sites, allowing direct surveying of these remote populations for the first time. [corroborating source G]
5. Sources to corroborate the impact
- (i) Spineless attacks on nuclear power plants could increase (2015) – article from thebulletin.org (accessed on 26/01/2021)
(ii) How climate-related weather conditions disrupt power plants in Indonesia and affect people (2020) - article from theconversation.com (accessed on 26/01/2021)
Testimonial letter from the Director of Cefas Technology Limited (CTL), Lowestoft – dated 6th November 2020
Testimonial letter from the Chief Fisheries Advisor for Scotland, Marine Scotland, Aberdeen – dated 26th February 2020
Mortensen et al., Effectiveness of fully documented fisheries to estimate discards in a participatory research scheme. Fish. Res., 187 (2017), pp.150-157
Bergsson, Plet-Hansen, Jessen, Jensen, Bahlke. (2017). Final report on development and usage of REM systems along with electronic data transfer as a measure to monitor compliance with the Landing Obligation – 2016. Danish AgriFish Agency, Ministry of Food, Agriculture and Fisheries.
(i) Interview with Technical Writer Andrew Wooden, published on the Intel website - intel.co.uk (2018) (accessed on 02/12/2020)
(ii) Interview with Government Europa Quarterly 31 – governmenteuropa.eu (2019) (accessed on 02/12/2020)
- Testimonial letter from Principal Investigator at the British Antarctic Survey, Cambridge – dated 30th October 2020
- Submitting institution
- The University of East Anglia
- Unit of assessment
- 11 - Computer Science and Informatics
- Summary impact type
- Health
- Is this case study continued from a case study submitted in 2014?
- No
1. Summary of the impact
Statistical modelling of the effects of medical conditions on longevity and mortality risk, by Professor Kulinskaya and her research group within the School of Computing Sciences at UEA, has had a direct and demonstrable impact in three key areas:
Involvement with the UK and international actuarial profession has resulted in improved pricing and reserving within the insurance and pensions sector.
Uptake of improved mortality and life expectancy projections by the leading insurance provider Aviva has resulted in wider access to insurance and pension products for people with impaired health.
Influence on UK and international health and social care policy makers.
2. Underpinning research
Since 2014, we have developed a methodology for modelling the impacts of chronic medical conditions, medical advances and health interventions on longevity and mortality risks at both individual and population level. Our methodology is based on advanced methods of design and statistical analysis of the observational data from big health administrative databases, such as The Health Improvement Network (THIN) primary care database and the National Joint register (NJR). It involves the following steps:
design a longitudinal case-cohort study with the appropriate cohorts of cases and their controls selected from the big administrative dataset
build a sophisticated survival model enabling evaluation of survival benefits or harms of particular chronic health conditions, treatments and public health interventions
integrate these individual survival effects to evaluate population effects.
This approach was applied to three important examples of implemented or potential health interventions:
The survival benefits of statins at key retirement ages were established and published in 2016 [1].
The effects of a number of medical interventions on survival after heart attack were published in 2017 [2].
Optimal systolic blood pressure targets were evaluated and published in 2019 [3].
In [1], we demonstrated that the current internationally recommended thresholds for statin therapy for primary prevention of cardiovascular disease in routine practice may be too low and may lead to overtreatment of younger people and those at low risk of heart disease.
In [2], we quantified the hazards of death after myocardial infarction and found them to be lower than reported by previous studies. We also found that standard treatments using aspirin or ACE inhibitors may be of little benefit or even cause harm.
In [3], we compared intensive control of systolic blood pressure (SBP) at 120 mmHg (which is being implemented in the US) to standard control at 140 mmHg and quantified life expectancy implications of the two target SBP levels. We concluded that intensive treatment of SBP may be harmful in the general population in the UK.
Our novel methodology of integrating individual effects into population level longevity changes is presented in [4, 5]. This integration requires a combination of parametric assumptions about the underlying survival distribution, such as the Gompertz or Weibull distribution, with a survival model incorporating a number of modifiers. The latter can use a Cox’s regression when the proportional hazards assumption is satisfied, but may require more sophisticated modelling of shape and scale parameters. This “double-Cox” model was developed in [6].
Our research on methodology of incorporating results of survival modelling into evaluation of longevity and its applications to UK life expectancy changes was supported by a grant from IFoA. Development of novel survival models and their application to NJR data was supported by an ESRC BLG DRC grant.
3. References to the research
(UEA authors highlighted in bold)
- Survival Benefits of Statins for Primary Prevention: A Cohort Study
Gitsels, L.A., Kulinskaya, E. & Steel, N.
(2016) PLoS ONE, 11(11): e0166847. DOI: 10.1371/journal.pone.0166847
- Survival prospects after acute myocardial infarction in the United Kingdom: a matched cohort study 1987-2011
Gitsels, L.A., Kulinskaya, E. & Steel, N.
(2017) BMJ Open, 7(1) DOI: 10.1136/bmjopen-2016-013570
- Optimal systolic blood pressure targets in routine clinical care
Gitsels, L.A., Kulinskaya, E., Bakbergenuly, I. & Steel, N.
(2019) Journal of Hypertension, 37:837–843. DOI: 10.1097/HJH.0000000000001947
- How Medical Advances and Health Interventions Will Shape Future Longevity
Sessional paper and presentation, IFoA, Edinburgh, June 25 2018
Gitsels, L.A , Kulinskaya, E. & Wright, N.
(2019) British Actuarial Journal. DOI: 10.1017/S1357321719000059
- Calculation of changes in individual and period life expectancy based on proportional hazards model of an intervention
Kulinskaya, E., Gitsels, L.A., Bakbergenuly, I. and Wright, N.R
(2020) Insurance Mathematics and Economics, 93, 27-35.
DOI: 10.1016/j.insmatheco.2020.04.006
- Risk-adjusted CUSUM control charts for shared frailty survival models with application to hip replacement outcomes: a study using the NJR dataset
Begun, A., Kulinskaya, E. & MacGregor, A.J.
(2019) BMC Medical Research Methodology, 19, 217. DOI: 10.1186/s12874-019-0853-2
Key Underpinning Funding
Project: Smart Data Analytics for Business and Local Government. (Co-I) Kulinskaya, E
Funder: ESRC BLG DRC award
Value: GBP1,233,622. Dates: Jan 2014 – Dec 2020
Project: Use of Big Health and Actuarial Data for understanding longevity and morbidity risks. (PI) Kulinskaya, E
Funder: Actuarial Research Centre, Institute and Faculty of Actuaries
Value: GBP790,537. Dates: 2016 - 2020
4. Details of the impact
UEA research has had impact in the following three areas.
1. Improved pricing and reserving within the insurance and pensions sector
We ensure that our research findings and methodologies are utilised by actuaries through our involvement with the Institute and Faculty of Actuaries (IFoA). This is the UK’s only chartered professional body dedicated to the education, development and regulation of actuaries both in the UK and internationally. It represents and regulates over 30,000 members worldwide. Our impact generated with IFoA has primarily been through education and professional training. The Chair of the Research and Thought Leadership Board at the IFoA has stated:
“*The research undertaken by Professor Elena Kulinskaya and her team at the University of East Anglia into understanding longevity and morbidity is a key component of our work to ensure that all members maintain their competence through a programme of Continuing Professional Development (CPD).*”
[ corroborating source A].
Examples of our impact through interaction with the IFoA include:
Yearly webinars to actuaries in the UK and overseas where the key objective is to encourage a two-way dialogue and participation on the topics being researched. These events attract large actuarial audiences and are CPD accredited events for members of the IFoA [ corroborating source B]. Two webinars created for the IFoA (2018 and 2019) were watched by 1,585 people with 1110 hours of CPD being recorded.
On October 30, 2019, we provided a technical workshop at the IFoA, aimed at educating actuaries (83 attendees) in advanced statistical methods developed by the research team and their application [ corroborating source B].
We broaden access to our research findings through IFoA publications including the Actuary and the Longevity Bulletin [ corroborating source B].
To promote and embed our methods into actuarial practice, we were invited to participate in Working Parties by the IFoA, specifically the Working Party on Diabetes and the Working Party on Population Health Management. We also presented our results to the IFoA Health and Care Research Sub-Committee in 2018 and 2019 [ corroborating source B].
To engage individual insurance and finance companies, we made a presentation to the Longevity Science panel at Legal and General (May 10, 2017), to Aviva Life actuaries (18 October 2018), to PWC pensions team (October 24, 2019) and to Just actuaries (24 February 2020). [ corroborating source C]. Importantly, the Actuarial Research Council at the IFoA maintains a mirror of our UEA website to provide up-to-date and freely available information about our research to the actuarial community [ corroborating source D].
A key mechanism for ensuring wider uptake of our research findings within the actuarial community has been presenting our findings at professional actuarial and statistical conferences, including International Congress of Actuaries (Berlin, June 2018), International Biometric Society Conference (2018), International Society for Clinical Biostatistics Conferences (2017, 2019), Mortality and Longevity Symposiums (2016), Life Conference (2017), Royal Statistical Society conference (2017, 2019) [ corroborating source C].
2. Improved mortality and life expectancy projections resulting in wider access to insurance and pension products for people with impaired health
A key impact of our research is that it enhances the chances of people with certain health conditions, who had previously struggled to get insurance, of getting insured in the future. Our results on specific medical conditions and treatments have fed into and contributed to the underwriting of insurance longevity products in the following ways.
Aviva uses the results of our research in their underwriting, and our results help to quantify the longevity assumptions necessary for numerous longevity and population projections. The importance to Aviva is confirmed in a letter from the Life Analytics Director:
“*One of the risks that Aviva faces is the guaranteed income that Aviva promises to customers in annuity products. Our guarantee is based on an assessment of life expectancy and is underpinned by statistical models. Professor’s Kulinskaya pioneering work on survival models and their linkage to individual and population life expectancy has provided Aviva with confidence that these guarantees are sound and robust.*”
[ corroborating source E]
We have developed a free “My Longevity” App aimed at both Insurance professionals and the general public which models the impact of lifestyle and health choices on life expectancy. Since its launch on September 3, 2020 there have been more than 800,000 visitors to the site, and more than 200,000 life expectancy calculations made [ corroborating source F]. The accompanying optional survey produces feedback and allows users to communicate decisions the tool has influenced and therefore life expectancy gain for the user. Specific feedback from two users shows that the My Longevity App is providing long-anticipated answers to important societal questions:
“ As a kidney specialist writing medicolegal reports on Condition & Prognosis, I am often asked to give my opinion on life expectancy.… I have asked and asked statisticians if somehow I can manipulate risk ratios or hazard ratio data to give me amended life expectancy, and your paper is the first I have found to address this issue…. Your on-line calculator is amazing and just what everyone is looking for.”
“*Congratulations on what is a great development. Very easy to use and a clear result!.... We are looking to help our potential customers / community members understand the biological age, health span and lifespan expectations and obviously your app would be an interesting tool that people could use plus there are some interesting developments that could be looked at.*”
[ corroborating source F]
3. Influencing UK and international policy makers
To engage in dialogue with public bodies and policy makers, in 2018 we organised a one-day workshop entitled “The impact of medical advances and health interventions on longevity and population projections”, for over 40 attendees [ corroborating source G]. The workshop explored how various wide-scale medical advances or health interventions, for example via changes in National Institute for Health and Care Excellence (NICE) guidelines, may impact on longevity and therefore necessitate changes in population projections. The consequent impact on a variety of policies and business models, from public health to pensions and insurance products was also covered. Invited stakeholders included: Office for National Statistics, Department for Work and Pensions, Department of Health, Government Actuary's Department, NICE, Royal Statistical Society, World Health Organization, World Bank, BPI Pension Trust, and, from within the IFoA: Steering Group members; Practice Board and Research Committee Representatives; Mortality Research Steering Committee; Relevant Working Parties; Policy and Public Affairs. An article written by Jules Constantinou, the IFoA President highlighted the success of the impact workshop and sessional. [ corroborating source H]
In parallel, a high-level summary of our research findings was included in the IFoA’s response to the LSE-Lancet Commission call for evidence on the future of the NHS (2018) [ corroborating source I], and to the joint consultation between the Department for Health and Social Care and the Cabinet Office on Advancing our Health: prevention in the 2020s (2019) [ corroborating source J].
5. Sources to corroborate the impact
[A] Testimonial letter from the Chair of the Research and Thought Leadership Board, Institute and Faculty of Actuaries, 12/01/2021
[B] Interaction with the IFoA through:
Presentations:
1) Webinar “Use of Big Health and Actuarial Data for Understanding Longevity and Morbidity” (13/06/2017) (461 views by 10/01/2020) Downloaded from YouTube and stored on file at UEA.
2) Webinar “Use of Big Health and Actuarial Data for Understanding Longevity and Morbidity” (17/09/2018) (280 views by 10/01/2020) Downloaded from You Tube and stored on file at UEA.
IFoA technical workshop:
3) Technical Workshop “Beyond Proportional Hazards” (30/10/2019) (87 views by 04/02/2021) Downloaded from You Tube and stored on file at UEA
Dissemination via professional actuarial publications:
4) Kulinskaya E and Gitsels LA (2016) Use of big health and actuarial data for understanding longevity and morbidity risk. Longevity Bulletin by IFoA, Issue 9: Big Data in Health. ISSN 2397-7213. Page 15 ff.
5) Gitsels LA and Kulinskaya E. (2018) “Statins: figures on the pulse” The Actuary
Presentations to the IFoA Health and Care Research Sub-Committee:
6) At the IFoA, London on 12 September 2018 and 24 October 2019
[C] Details of conference presentations, and presentations to individual insurance and finance companies
[D] Mirror of UEA website from actuaries.org.uk - accessed on 04/02/2021
[E] Testimonial from the Life Analytics Director of Aviva, 23/11/2020
[F] MyLongevity App (pages 1 – 2), Google Analytics (pages 3 – 5) and feedback (pages 6 -15)
[G] One-day workshop “The impact of medical advances and health interventions on longevity and population projections ” (17/05/2018) (213 views by 4/02/2021) Downloaded from You Tube and stored on file at UEA
[H] Article by the IFoA President entitled ‘Big Health, Big Impact Big Data’ published in The Actuary magazine, July 2018 edition. Page 5 ff.
[I] IFoA’s response to the LSE-Lancet Commission: the future of the NHS call for evidence, 30 July 2018.
[J] IFoA’s presentation to the joint consultation between the Department for Health and Social Care and the Cabinet Office: Advancing our health: prevention in the 2020s, October 2019.
- Submitting institution
- The University of East Anglia
- Unit of assessment
- 11 - Computer Science and Informatics
- Summary impact type
- Technological
- Is this case study continued from a case study submitted in 2014?
- No
1. Summary of the impact
The Colour & Imaging Lab at UEA developed revolutionary algorithms for image fusion with applications in multiple domains including photography, security and accessibility (for colour deficient observers). This research was vested in Spectral Edge Ltd. which was spun out of UEA in 2011. Spectral Edge Ltd. attracted a total of GBP6,124,000 investment including, investment from the Iceni and Rainbow Seed corn funds and two rounds of venture capital funding.
In November 2019, Spectral Edge Ltd. was acquired by Apple Inc. and the Spectral Edge technology is now being incorporated into the Apple product pipeline.
2. Underpinning research
In the EPSRC grant [A] “Colour to Greyscale and Related Transforms” (with Professor Bloj, University of Bradford), Professor Finlayson’s research group at UEA explored image fusion in humans and, inspired by this work, developed new fusion algorithms for computer vision. At the same moment, the imaging industry started to manufacture devices (e.g. smart phones) equipped with multiple cameras which resulted in a sudden interest in, and need for, effective image fusion algorithms.
For intuitive reasons, most inquiry into image fusion operates in the edge or derivative domain, since edges are proxies for the detail we wish to fuse. In a standard methodology, for image fusion we are given N images from which we then calculate derivatives (or edges). The edges are fused together to create a composite edge-map which is reintegrated to solve for the final output image. Almost all the prior art treats image fusion as a N to 1 problem. Moreover, because the fused gradient field is, necessarily, non-integrable the fused images must both have artifacts and hallucinated detail. Spectral Edge theory teaches how to solve the more general N to M (where M is typically 3 for colour images) gradient fusion problem. In Spectral Edge image fusion [1], the available extra edge information is first be determined and is then used to accentuate the gradient field of a guide image (e.g. the original colour image). As before, the gradient field must then be re-integrated. To avoid the problems of artifacts and hallucinated detail, a new theory of integration was developed [2] which, provably, could not suffer from these problems. Spectral Edge theory was applied directly to many problems including, fusing the Near-Infra Red and visible images and bringing out more detail in diffusion tensor imaging [1].
Subsequently, the researchers at UEA, funded through an EPSRC follow-on award [B], re-imagined the Spectral Edge fusion theory as a tool for general image processing. In a first step a colour image is processed so that a specific property is enhanced. Then this property image is fused – using the spectral edge theory - with the original. As an example, the extra information that is available to observers with normal colour vision (the property image) is fused with the image available to colour deficient observers. The resulting fused image conveys dramatically more detail to colour deficient observers. In other research, a sharpened image was fused with the original to make a new and pleasing output image that had much greater local contrast (but no artefacts). In [3], colour and thermal images were fused together as part of a solution to pedestrian detection that was both simpler and faster to execute than the prior art.
Research was a key part of Spectral Edge’s mission, and UEA researchers continued to collaborate with the spin-out company up to the point of acquisition, at which time the focus was on both developing an extended theory of fusion based on a new definition of edges in images (inspired by operator theory from Physics), and [*redactedtext*redacted text*redacted text*]. Both of these research areas are part of the current “Future Colour Imaging” project [C], a 5-year EPSRC established career fellowship awarded to Professor Finlayson.
3. References to the research
(UEA authors in bold)
- Spectral Edge image Fusion: Theory and Applications
D.R. Connah, M.S. Drew and G.D. Finlayson,
( 2014) The European Conference on Computer Vision, 65-80.
DOI: 10.1007/978-3-319-10602-1_5
- Look-up Table based gradient field reconstruction
G.D. Finlayson, M.S. Drew and D. Connah,
( 2011) IEEE Transactions on Image Processing, 2827 – 2836.
DOI: 10.1109/TIP.2011.2134106
- Multi-spectral Pedestrian Detection via Image Fusion and Deep Neural Network
G. French, G.D. Finlayson and M. Mackiewicz,
( 2018) The Journal of Imaging Science and Technology, 176-181(6)
DOI: 10.2352/J.lmagingSci.Technol.2018.62.5.050406
Underpinning Funding
- C2GART Colour to Grey scale and Related transforms
PI: G. Finlayson
Funder: EPSRC
Value: GBP327,017.00 Dates: Oct 2006 - April 2010
- Spectral Edge Image Visualisation
PI: G. Finlayson
Funder: EPSRC follow-on-fund grant
Value: GBP63,000.00 Dates: Dec 2010 - Aug 2011
- Future Colour Imaging
PI: G. Finlayson
Funder: EPSRC
Value: GBP1,046,725.39 Dates: Sep 2019 - August 2024
4. Details of the impact
Company creation and growth
Following the initial research within Finlayson's Colour and Imaging Lab in the School of Computing Sciences at UEA Spectral Edge Ltd. was incorporated in 2011 [corroborating source A]. The Intellectual Property which formed the core of the patent portfolio underpinning Spectral Edge Ltd. included more than 10 patents covering image fusion, the integration of gradient fields, targeting images for different observers (e.g. to help colour deficient observers to see more) and in content-based image processing [corroborating source B].
Early funding for Spectral Edge Ltd. came from a Technology Strategy Board Proof of Concept grant (2012-2013 GBP35,798) and an Innovate UK grant (2014-2015, GBP110,000), for developing and validating using image fusion to help colour deficient observers see more [corroborating source C], together with Seedcorn funding from the ICENI and Rainbow funds (GBP538,000) [corroborating source D]. In 2016 and 2018 respectively, Spectral Edge attracted venture capital investment of GBP1,500,000 and GBP4,000,000 where IQ capital and Parkwalk were the lead investors [corroborating source E].
The company was initially based in Norwich before moving to Cambridge in 2014. Starting in the IdeaSpace (an incubation hub for early stage innovation), the company soon grew to a scale (6+ employees) where commercial offices were required. On acquisition at the end of 2019, Spectral Edge Ltd. was based in the Bradfield Centre on the Cambridge Science Park and employed 15 staff (headcount = 15; FTE = 15) of which 12 were software and hardware engineers.
Spectral Edge technology
Spectral Edge is a platform technology in the area of image fusion. In contradistinction to the prior art which focussed on fusing N images to 1, in the Spectral Edge approach all the detail is made available in a fused colour image (N to 3). Moreover, uniquely, the Spectral Edge algorithms produce images without artifacts or hallucinated detail, a major step forward.
Spectral Edge Ltd. first developed an algorithm called Eyeteq where the detail that a colour-blind observer can see is fused with the extra detail available to observers with normal colour vision. An empirical evaluation of Eyeteq processed TV and video content by the independent company i2 media research in 2015 showed that the resulting fused colour images convey much more information to colour deficient observers whilst simultaneously preserving colour fidelity for everyone else [corroborating source F].
Nighteq followed in 2016, providing blue light reduction technology for night-time viewing and in 2017 Vividteq was launched, in which contrast enhanced images are fused with a colour original to deliver a pleasing more detailed and artefact free image. Vividteq was licenced to NTT Data Italia in 2018 for inclusion in their broadcast infrastructure, which allowed their broadcast customers (including Sky Italia) to deliver high-dynamic range (HDR) like images irrespective of the limitations of the end-user hardware [corroborating source G]. A major focus of the company was the fusion of IR and visible images, which resulted in the Spectral Edge Fusion technology being launched in 2019. Spectral Edge Ltd. built its own cameras and processing pipeline which could fuse video streams in real time. This technology was being deployed commercially shortly before acquisition [corroborating source H].
Throughout development the Spectral Edge Technology proceeded with help and advice from industry experts, including the CEO of Agile Analog:
“I was very impressed with the core technology, and leapt at the chance to join the Spectral Edge Advisory Board, to enhance and assist in the future technology direction.
Given the quality of the underlying technology it is no surprise that Spectral Edge was acquired by such a global technology leader.”
[corroborating source I]
Acquisition of Spectral Edge Ltd. by Apple Inc.
In November 2019 Spectral Edge Ltd. was acquired by Apple Inc. with the Spectral Edge technology now being incorporated into Apple’s product pipeline. The Director, Cameras and Photos Software Engineering at Apple has confirmed that:
“Through Apple’s relationship with Professor Finlayson and University of East Anglia we were able to monitor the steady progress of Spectral Edge as they built their core technologies and talented team of engineers - several of which were a product of UEA’s Colour & Imaging Lab. The development of fundamental Spectral Edge algorithms into practical engineering deliverables gave us the confidence that these could quickly make an impact on Apple products and improve experiences for our customers. This team has also helped us attract new talent to grow our expertise at a new Apple facility in Cambridge.”
[corroborating source J]
An additional benefit of the acquisition is that the initial investments made in Spectral Edge Ltd. returned a significant profit to the investors, including the University of East Anglia, with the investment returns to both the ICENI Seedcorn Fund and IQ Capital now being invested in new ventures [corroborating sources D and E]. The acquisition was also significant for a number of Spectral Edge Ltd. employees, with 10 of the engineers being hired by Apple Inc. and the core Spectral Edge team being at the heart of Apple’s major expansion of its business in Cambridge.
5. Sources to corroborate the impact
Companies House certificate showing formation of Spectral Edge Ltd. in 2011
Underpinning IPR portfolio details
Technology Strategy Board (TSB) Grant 2012-2013 and from page 7, Innovate UK Grant 2014-2015 details
Letters from the Investment Director of the UK Innovation & Science Seed Fund and from the Fund Advisor for the Iceni Seedcorn Fund confirming funding amount, both November 2020.
Confirmation of venture capital investment from the Managing Partner at IQ Capital (Dec 2020) and the CEO of Parkwalk (Nov 2020)
i2 media research report (Oct 2015) on the evaluation of Eyeteq processed TV and video content
NTT Data and Spectral Edge bring HDR-like experience to any TV – article from advanced-television.com 05.04.2018 (Downloaded September 2020)
Spectral Edge Launches RGB and NIR Spectral Edge Solution for Surveillance Market - article from risk-uk.com, 29.03.2019 (Downloaded September 2020)
Testimonial from CEO, Agile Analog (March 2021)
Supporting statement from the Director, Cameras and Photos Software Engineering at Apple Inc. (Jan 2021)