Impact case study database
Computer Vision for Monitoring of Animal Populations
1. Summary of the impact
Computer vision methods developed by Dr Mackiewicz and co-workers in the School of Computing Sciences (CMP), UEA are being used to monitor animal populations. Utilising computer vision and deep learning technology for environmental monitoring, CMP researchers are working with the Scottish Government, Marine Scotland, the British Antarctic Survey (BAS), and the Centre for Environment, Fisheries and Aquaculture Science (CEFAS) to monitor swarms of jellyfish, fish catches, and albatross populations. This has resulted in economic, commercial and environmental impacts including protection of nuclear power stations and desalination plants against jellyfish ingress (the closure of which can cost GBP1,000,000 per day), protection of threatened species, easier policy enforcement and new markets for industrial collaborators.
2. Underpinning research
JellyMonitor: The clogging of the seawater intake screens of coastal nuclear power stations and desalination plants by jellyfish is a concern for safety, security of supply and operational costs. JellyMonitor (2015-2018) was a research project aimed at developing a portable imaging platform that would detect jellyfish near to water intakes and provide an early warning of jellyfish ingress. Initial funding was from Innovate UK [Grant 1] for a collaborative project between the Centre for Environment, Fisheries and Aquaculture Science (CEFAS), CEFAS Technology Ltd (CTL), UEA, and EDF R&D UK Centre. The research continues with a NERC & EPSRC iCase PhD studentship (2018 to 2022) co-funded by the NEXUSS Centre for Doctoral Training (CDT) and CEFAS [Grant 2].
Within the collaborative Innovate UK project, the computer vision and machine learning algorithms were developed almost entirely by CMP researchers at UEA. The software component of JellyMonitor is designed to analyse sonar imagery in real-time whilst operating within the computational constraints of an embedded platform deployed on a seabed [3.1]. This novel system combines some conventional computer vision techniques to detect and track potential objects of interest, and a state-of-art deep neural network classifier, developed by researchers at UEA, to categorise and label them. This process was complicated by the large quantities of noise present in sonar imagery, the highly imbalanced dataset that was used to develop the prototype, the few occurrences of jellyfish in the dataset that were available during the development, and a very strict energy budget for the battery powered embedded platform. Consequently, the UEA expertise in semi-supervised image classification and domain adaptation [3.2] and the application of these was essential in achieving a working prototype of JellyMonitor.
CatchMonitor: Within the marine fishing industry, CCTV cameras improve data gathering on the observation of discards and support the catch quota management scheme. In 2014 the Scottish Government invited tenders to research and develop an automated video analysis system which could replace or enhance human observations of recorded videos. This work for Marine Scotland and the Scottish Government was carried out by CMP researchers as two small feasibility projects from 2014 to 2017 [Grants 3 & 4]. The developed algorithms comprising fish segmentation, classification and counting and the results confirming the feasibility of the application have been described in [3.3].
The results obtained during the feasibility projects allowed Marine Scotland and UEA to continue this collaboration as part of a much larger research consortium including commercial partners - H2020 Smartfish [Grant 5]. The result was the prototype of CatchMonitor [3.4] [3.5] - a computer vision system which can be installed on board fishing trawlers with the aim of monitoring and quantifying discarded fish catch. CatchMonitor software comprises four distinct parts: web user interface; isolation (segmentation) of each individual fish (based on Mask-RCNN – the state-of-the-art segmentation deep neural network); classification of segmented fish into a number of fish species (deep network based on ResNet – a modern classification deep neural network); and counting and classifying those fish instances in the video which end up as discards. Here, a novel algorithm was required as the view of the conveyor belt is often occluded by humans, which interrupts the fish tracking. The method developed at UEA is based on integration of the individual frame segmentation results, which are subsequently processed as fish density pixel maps and further aggregated to produce fish counts for a set of relevant species. Within the collaborative project, the algorithms constituting the software of CatchMonitor were developed solely at UEA. As with JellyMonitor, the expertise of the UEA team and their research in semi-supervised image classification and segmentation was key to the success of the project [3.2, 3.6].
Albatross Monitoring: Since 2017, UEA researchers have been working with the British Antarctic Survey on the development of a technology for albatross recognition and counting in satellite images, funded by NEXUSS CDT [Grant 6]. The developed method builds on the wider computer vision and deep learning expertise present within CMP at UEA. The algorithm is based on the U-NET deep learning architecture with the focal loss criterion that was required to tackle the extreme class imbalance that is one of the problems in this application [3.7].
3. References to the research
The underpinning research outputs have all been published in competitive, international, peer-reviewed journals/conferences and form part of a larger body of such published work.
(UEA authors highlighted in bold)
- JellyMonitor: Automated Detection of Jellyfish in sonar images using neural networks
French, G., Mackiewicz, M., Fisher, M., Challis, M., Knight, P., Robinson, B. and Bloomfield, A.
( 2018) IEEE International Conference on Signal Processing Proceedings, ICSP.
DOI: 10.1109/icsp.2018.8652268
- Self-ensembling for visual domain adaptation
French, G., Mackiewicz, M., Fisher, M.
( 2018) IEEE International Conference on Learning Representations ICLR.
- Convolutional Neural Networks for Counting Fish in Fisheries Surveillance Video, Machine Vision of Animals and their Behaviour
French, G., Fisher, M., Mackiewicz, M. and Needle, C.
( 2015) Workshop at the 26th British Machine Vision Conference. DOI: 10.5244/c.29.mvab.7
- Deep neural networks for analysis of fisheries surveillance video and automated monitoring of fish discards
French G., Mackiewicz M., Fisher M., Holah H., Kilburn R., Campbell N., Needle C. ( 2019 ) ICES Journal of Marine Science. DOI: 10.1093/icesjms/fsz149.
- H2020 Smartfish
Mackiewicz M., French G., Fisher, M.
( 2019) Public Deliverable 4.3, Prototype of CatchMonitor Dec 2019 (held on file at UEA)
- Semi-supervised semantic segmentation needs strong, high-dimensional perturbations
French, G., Aila, T., Laine S., Mackiewicz M., Finlayson G.
( 2020) In Proc. of the 31st British Machine Vision Conference (BMVA) arxiv.org/abs/1906.01916
- Using deep learning to count albatrosses from space: Assessing results in light of ground truth uncertainty
Bowler, E., Fretwell P., French, G., Mackiewicz M.
( 2020) Remote Sens. 2020, 12(12), 2026. DOI: 10.3390/rs12122026
Underpinning Funding
Grant 1: JellyMonitor: developing a jellyfish early warning system for coastal power stations
Collaborators: CEFAS Technology Ltd, Lowestoft; Centre for Environment, Fisheries and Aquaculture Science (CEFAS), Lowestoft, UEA, and EDF R&D UK Centre, London.
PI: Mark Fisher. Funder: Innovate UK. GBP126,415; 4/2015 - 3/2018
Grant 2: Four year iCase PhD studentship. Funded by CEFAS and NEXUSS CDT (NERC & EPSRC); 1/10/18 - 31/05/22
Grant 3: Automated Image Analysis for Fisheries CCTV
PI: Mark Fisher. Funder: Marine Scotland, Scottish Government Environment and Forestry Directorate. GBP74,400; 2014 - 2015
Grant 4: Enhanced Automated Image Analysis for Fisheries CCTV project (follow-up grant from Grant 3)
PI: M ark Fisher. Funder: Marine Scotland. GBP37,919; 2016 - 2017
Grant 5: Smart fisheries technologies for an efficient, compliant and environmentally friendly fishing sector: H2020 Smartfish
PI: Michal Mackiewicz. Funder: CEFAS Collaboration. GBP324,710; 1/2018 - 12/2021
Grant 6: Four year PhD research (2017 - 2021) funded by NEXUSS CDT (NERC & EPSRC)
4. Details of the impact
This case study demonstrates economic, commercial and environmental impact arising from research in CMP at UEA, including protection of nuclear power stations and desalination plants, protection of threatened species, easier policy enforcement and new markets for industrial collaborators.
Monitoring and Protecting Power Stations
In the right conditions, jellyfish can swarm and create serious problems for nuclear power stations, which rely on large volumes of cool sea water to control internal temperatures. If access to this supply is lost, the power stations have to shut down with consequent substantial financial losses. In 2011, the shutdown of the Torness Nuclear Power Station caused a loss of GBP1,000,000 per day for the parent energy company and a 20-day shut down in East Java of the Paiton Coal Power Plant cost USD21,700,000 (04/2016) [corroborating source A]. This is a global problem with further notable shutdowns in Oskarshamn Sweden in 2013 and Diablo Canyon, USA in 2008. The Director of CEFAS Technology Ltd (CTL) has stated that:
[textredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredacted]
[corroborating source B]
The project funded by the Innovate UK [Grant 1] resulted in the development of a fully functioning prototype of the jelly fish detection system – JellyMonitor – see Figure 1. The system was developed jointly by CTL and UEA with the support from EDF. The implementation uses a target embedded hardware platform specifically designed so that it can operate autonomously in the marine environment for the duration of the jelly-fish season. [textredactedtextredacted textredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredacted]
Figure 1
[textredactedtextredactedtextredacte dtextredactedtextredactedtextredacted textredactedtextredactedtextredacted textredactedtextredactedtextredacted textredactedtextredactedtextredacted textredactedtextredactedtextredacted textredactedtextredactedtextredacted textredactedtextredactedtextredacted textredactedtextredactedtextredacted textredactedtextredactedtextredacted textredactedtextredactedtextredacted]
Over the forthcoming three years there will be annual sea trials of the prototype over the entire jelly fish season so that more data can be captured. This will allow incremental changes to the system to be tested as CTL moves towards a commercial product.
The Director at CTL has stated:
[textredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredactedtextredacted]
[corroborating source B]
Monitoring Fishing Discards
Marine Scotland is an agency of the Scottish Government responsible for management of Scotland’s seas. Marine Scotland first installed CCTV cameras on board fishing vessels in 2008 as part of a scheme to monitor fish discards, with the aim of facilitating the recovery of the North Sea cod stock. It quickly became clear that the volume of data would need a novel method of analysis when it took one person 3 full months to count and analyse the discarded fish from a single week-long trip of a Scottish trawler using recorded CCTV. A solution to this problem is especially pressing given that it is likely that CCTV systems on fishing vessels are likely to become the norm rather than the exception. [corroborating source C].
CatchMonitor was developed by UEA researchers in conjunction with Marine Scotland as an advanced onboard catch monitoring system allowing automatic image analysis of fish discards. The products developed are beginning to have significant impact on the fishing industry through the collaboration with Marine Scotland, Cefas and the wider H2020 consortium. The improvements in the accuracy and quantity of fish stock assessment data increases compliance with fishery regulations leading to reduction of fishing pressure and the environmental impact of fisheries. CatchMonitor is considered to be at Technology Readiness Level 5, with development work continuing with Marine Scotland on board their research vessel FRV Scotia. [corroborating source C]
Policy Development
The two articles on Marine Fishing monitoring [3.3, 3.4] were cited over fifty times including several times in policy making publications e.g., in Marine Policy [corroborating source D] and The European Landing Obligation. [corroborating source E]
Public Understanding
Our work has received considerable international publicity both in general popular science reporting and in the science policy press. [corroborating source F]
Albatross Monitoring
Despite all six species of Great Albatrosses are under threat according to the IUCN Red List, many colonies are not surveyed for decades at a time due to their remote and inaccessible nesting locations. Filling in the existing significant gaps in knowledge of distribution, population sizes and demography will significantly help in the conservation management of this endangered species. Using state of the art 31-cm resolution satellite imagery and computer vision methods, UEA researchers working alongside scientists from the British Antarctic Survey (BAS) have addressed this issue by automatically surveying populations directly from space. The developed protocol enables annual global censuses, which would not be possible with human observers, vastly improving our understanding of population trends and informing conservation action. Following the development of the algorithm at UEA, BAS has decided to upscale the albatross survey work to cover the whole Southern Ocean. Since December 2019, they have had an agreement in place with Maxar (DigitalGlobe) to task their satellites over Wandering Albatross breeding sites, allowing direct surveying of these remote populations for the first time. [corroborating source G]
5. Sources to corroborate the impact
- (i) Spineless attacks on nuclear power plants could increase (2015) – article from thebulletin.org (accessed on 26/01/2021)
(ii) How climate-related weather conditions disrupt power plants in Indonesia and affect people (2020) - article from theconversation.com (accessed on 26/01/2021)
Testimonial letter from the Director of Cefas Technology Limited (CTL), Lowestoft – dated 6th November 2020
Testimonial letter from the Chief Fisheries Advisor for Scotland, Marine Scotland, Aberdeen – dated 26th February 2020
Mortensen et al., Effectiveness of fully documented fisheries to estimate discards in a participatory research scheme. Fish. Res., 187 (2017), pp.150-157
Bergsson, Plet-Hansen, Jessen, Jensen, Bahlke. (2017). Final report on development and usage of REM systems along with electronic data transfer as a measure to monitor compliance with the Landing Obligation – 2016. Danish AgriFish Agency, Ministry of Food, Agriculture and Fisheries.
(i) Interview with Technical Writer Andrew Wooden, published on the Intel website - intel.co.uk (2018) (accessed on 02/12/2020)
(ii) Interview with Government Europa Quarterly 31 – governmenteuropa.eu (2019) (accessed on 02/12/2020)
- Testimonial letter from Principal Investigator at the British Antarctic Survey, Cambridge – dated 30th October 2020
Additional contextual information
Grant funding
| Grant number | Value of grant |
|---|---|
| 44757-320160 | £126,415 |
| SYS/011/14 | £74,400 |
| CR/2015/14 | £37,919 |
| H2020-SFS-2017-1 | £324,710 |