Impact case study database
Object Detection, Classification, Localisation and Tracking for Automated Wide-Area Surveillance
1. Summary of the impact
Our development of novel algorithms for combined real-time object detection, classification,
localisation and tracking within automated wide-area surveillance, carried out between 2013 and 2016 at Durham University, directly:
informed UK government science, technology and procurement policy for technical interface standardisation within wide-area, multi-sensor surveillance systems.
informed scientific policy-making activity by the governments of UK, USA, Canada, Australia, New Zealand and Netherlands on wide-area, multi-sensor surveillance systems.
informed the design, development and evaluation of wide-area surveillance sensors and associated technologies that were developed commercially by AptCore, Autonomous Devices, Blue Bear Systems Research, Createc, Cubica Technology and QinetiQ.
This directly contributed to GBP23.2 million investment in multi-sensor surveillance systems (UK/US government/industry), GBP11.9 million of additional commercial income and supported the creation of ~55 additional science and engineering jobs across six organisations (2013-2020).
2. Underpinning research
This research uses automated image understanding to provide long-term wide-area surveillance of dynamic scene objects (people, vehicles) addressing questions such as - “Is there anything there?” (detection), “What is it?” (classification), “Where is it?” (localisation) and “What is its behaviour?” (tracking). The key challenge is to be able to address these issues within real-time processing constraints (“as it happens”) and within the processing capability available within the size, weight and power (SWaP) constraints of a field-deployable, long-duration sensor unit. The Durham research team were able to introduce the idea that all-weather, long-term automated visual surveillance addressing all of these key questions could be achieved in real-time, within such a field-deployable unit, by the integrated co-design of both the sensor units (hardware, [R4]) and the associated algorithms (software [R1-R3], Fig. 1). Furthermore, we were able to develop a novel set of algorithms, suited specifically to our key design decision to perform sensing in the far-infrared (thermal) spectrum, enabling real-time processing performance within a field-deployable, all-weather sensor unit.
Figure 1: Field-deployable thermal imaging sensor units developed by Durham University (left/middle), and vehicle detection/tracking integrated within the SAPIENT user interface (right) – (Crown copyright).
This sensor unit design (hardware), and associated real-time algorithms (software), contributed to the design and validation of a novel wide-area surveillance concept that allowed sensors of varying capability and sensing modality to participate in a common sensor network by dynamically declaring availability, integrity and capability via a common interface [R4].
The key findings of this Durham-based research were that:
• robust all-weather detection and classification of pedestrians [R1] and vehicles [R3] can be achieved in real-time, within a limited field-deployable computational footprint, using far-infrared (thermal) sensing and a unique algorithmic combination of a multiple mixture of Gaussian background models to overcome inherent variations within the thermal imagery coupled with a feature-based foreground classification model [R1,R3]. This meant that objects could be detected and classified by type (pedestrian [R1], vehicle [R3]) during long-duration sensor deployment despite extreme changes in environmental conditions (illumination, temperature).
• photogrammetry can enable the passive recovery of 3D object position within the scene relative to the camera within far-infrared (thermal) video imagery and deliver positional accuracy to within the expected error bounds of GPS (for object localisation, [R1]). Furthermore, the expected impact of variations in human pose (for pedestrian targets) on this accuracy could be readily overcome via the use of the regressive posture estimation and subsequent algorithmic correction [R2]. This meant that objects detected and classified within the scene, could be tracked within real-world 3D co-ordinates via the use of photogrammetry based algorithms [R1, R2] without the need for active sensing as accurately as if explicit GPS trackers were placed upon the objects themselves. Our ability to process 3D object localisation in real-time further facilitated an extension to commonly-used Kalman filter based tracking, traditionally used for 2D pixel-wise object tracking, to full 3D object tracking in the scene [R1, R2].
• the combined use of detection, classification and (3D) tracking within far-infrared [R1-R3] further enables the robust secondary classification of objects to determine sub-type (for vehicles - car, van, 4x4, HGV [R3]) or behaviour (for pedestrians - running, walking, loitering, digging, crawling [R1, R2]) based on an accumulated feature representation [R1-R3].
• imperfections in the combined detection/classification/localisation/tracking capability of any deployed sensing modality and associated processing, such as far-infrared [R1-R3], can be mitigated via integration into a non-homogeneous sensing network, via a suitably defined common interface specification, such that these short-comings within current ambient conditions can be overcome via fusion with complementary sensing technologies [R4].
This Durham-based research contributed to the SAPIENT (Sensing for Asset Protection using Integrated Electronic Networked Technology) project (2014-2016) that developed a novel integrated, multi-modal wide-area surveillance network such that “ individual [deployed] sensors make low-level decisions autonomously … [and] are managed by a decision-making module which controls the overall system ... to reduce … the need to constantly monitor the output of the sensors.” [R4]. This joint research was carried out in collaboration with the Defence Science and Technology Laboratory (DSTL) (system requirements), QinetiQ (systems integration, visibleband camera sensors), Cubica Technology (decision-making module), AptCore (radar sensors), Createc (laser range sensors) and Durham (thermal imaging sensors) resulting in a joint publication [R4].
3. References to the research
[R1]. A Photogrammetric Approach for Real-time 3D Localisation and Tracking of Pedestrians in Monocular Infrared Imagery (M.E. Kundegorski, T.P. Breckon), in Proc. SPIE Optics and Photonics, Volume 9253, No. 01, pp. 1-16, 2014. [ http://dx.doi.org/10.1117/12.2065673]
[R2]. Posture Estimation for Improved Photogrammetric Localisation of Pedestrians In Monocular Infrared Imagery (M.E. Kundegorski, T.P. Breckon), in Proc. SPIE Optics and Photonics, Volume 9652, No. XI, pp. 1-12, 2015. [ https://doi.org/10.1117/12.2195050]
[R3]. Real-time Classification of Vehicle Types within Infra-red Imagery (M.E. Kundegorski, S. Akcay, G. Payen de La Garanderie, T.P. Breckon), in Proc. SPIE Optics and Photonics, Volume 9995, pp. 1-16, 2016. [ https://doi.org/10.1117/12.2241106]
The following is a joint publication on the overall SAPIENT concept with authors from all of the research partners:- DSTL (Thomas) QinetiQ (Marshall, Faulkner, Kent), Cubica Technology (Page, Islip), AptCore (Styles), Createc (Clarke), Durham (Breckon, Kundegorski):
[R4]. Towards Sensor Modular Autonomy for Persistent Land Intelligence Surveillance and Reconnaissance (P.A. Thomas, G.F. Marshall, D. Faulkner, P. Kent, S. Page, S. Islip, J. Oldfield, T.P. Breckon, M.E. Kundegorski, D. Clarke, T. Styles), in Proc. SPIE Ground/Air Multisensor Interoperability, Volume 9831, No. VII, pp. 1-18, 2016. [ https://doi.org/10.1117/12.2229720] [ Sapient Project YouTube Summary Video]
Papers have been peer reviewed as part of the publication process and show clear originality and rigour
4. Details of the impact
Effective wide-area surveillance is a key area of interest within both military operations and in civil infrastructure protection as it offers situational awareness – i.e. knowing what is happening and where it is happening within the environment (source: MoD Science & Technology Strategy, 2017).
Whilst the specification and evaluation of the operating requirements for these surveillance tasks is a policy matter for government, the technical development of sensing technologies to meet these requirements is carried out by commercial suppliers. In the UK, government science and technology policy in this area is informed by the UK Defence Science and Technology Laboratory (DSTL), an executive agency of the UK Ministry of Defence whose aim “is to maximise the impact of science and technology for UK defence and security” (DSTL). The research work at Durham [R1-R4] has had both policy and commercial impact within this area.
Impacts of Durham research on UK government policy are as follows:
• informed UK government science and technology policy for wide-area surveillance systems by enabling the UK DSTL to perform the “design, evaluation and validation of a new common interface specification for integrating varying types of intelligent networked sensors .. with direct reference to the performance characteristics of the [Durham] all-weather, all-condition camera-based thermal ... and the … requirements of the state-of-the-art real-time object detection, classification and tracking algorithms [provided by Durham -R2, R3]” [E1]. This was supported by “a total budget of GBP3 million (2013-2016), with additional technical and management support from ~10 DSTL staff scientists” - DSTL [E1]. The resulting UK government science and technology policy publication (SAPIENT Middleware Interface Control Document (ICD)) was publicly released under the UK Open Government Licence at ( http://www.gov.uk/sapient). Paul Thomas, Principal Scientist at DSTL, commented that this enabled “an important step forward in enabling sensors to ‘plug and play’ … [a] significant benefit both in the area of civil security such as the protection of infrastructure and in military systems such as for base protection.” [E2].
• informed UK government scientific policy-making activity by enabling the DSTL to test and evaluate SAPIENT “ for integrated wide-area surveillance within the inclusion of the thermal imaging solution provided by Durham [R1-R3], against a range of staged scenarios” including *“two large-scale technology demonstrations to inform ~60 senior personnel spanning UK military and civil infrastructure protection (from UK Ministry of Defence (MoD), Centre for Protection of National Infrastructure, Home Office, Dept. Transport) of the ‘art of the possible’ for integrated wide-area surveillance … (Malvern, Sept. 2015 / Throckmorton Airfield, June 2016)”* [E1] and technical evaluation “ against a range of unmanned aerial system [drone] targets flown in a variety of attack trajectories, using radar and electro-optic [sensors; R4].” [E3]. “The demonstrations provided an excellent showcase of sensor technology that was very useful in helping both military and civil [policy] stakeholders understand the progress, and more importantly the potential, for the technology.” - DSTL [E1].
• informed the UK government procurement policy for wide-area surveillance systems via the Defence and Security Accelerator (DASA). This led to the SAPIENT Middleware ICD, enabled by Durham research [in R1-R4], being adopted as the preferred interfacing option for commercial system suppliers to the UK MoD across GBP11.3 million of commercial research and development funding (2017-2020) [E4]. Furthermore, SAPIENT has been adopted for autonomous sensor management within the joint US Army / UK MoD procurement “Signal and Information Processing for Decentralized Intelligence, Surveillance, and Reconnaissance” (W911NF17S0003-US-UK, USD1.2 million, 2020). The UK Minister for Defence Procurement, Andrew Stuart MP, commented that by informing policy within UK defence procurement, SAPIENT “can act as autonomous eyes in the urban battlefield. Investing millions in advanced technology like this will give us the edge in future battles. It also puts us in a really strong position to benefit from similar projects run by our allies as we all strive for a more secure world.” [E2].
• enabled “collaborative experimentation internationally between the [governments of the] Five Eyes allied nations of Australia, Canada, New Zealand, the UK and the USA in the Contested Urban Environment experiment involving over 150 government and industry scientists and over 80 Canadian troops for three weeks in Montreal (2018)”* [E1]. This provided the five nations with a means to evaluate a range of differing sensors and operating methods for effective wide-area surveillance based on the work of [R1-R4] and informed the policymaking work of defence research scientists in the UK and partner nations. DSTL Chief Executive, Gary Aitkenhead, commented “[SAPIENT] is a fantastic example of our world-leading expertise at its best; our scientists working with our partner nations to develop the very best technology for our military personal now and in the future.” - [E2]. The collaboration has also informed the surveillance requirement policy of the military end user.
Figure 2: Briefing and site walk-around with demonstration attendees (upper), view from Durham sensor units (lower, left) and integrated multi-sensor visualisation via the common graphical display to operator (lower, right) – June 2016 (Crown Copyright).
“[SAPIENT] brings together our requirements as a user and DSTL as scientific advisers ... with our key allies in the five-eyes community.” said Lt Col Nat Haden, SO1 Intelligence, Surveillance, Target Acquisition and Reconnaissance Capability, British Army Headquarters [E2].
• enabled TNO (Netherlands, government research agency) to perform “rapid integration of raw and processed sensors by using open standards and the SAPIENT interface” [E3] designed and validated using [R1-R3] (2018). This allowed the Netherlands government to develop and evaluate additional novel sensing approaches to inform policy-making in wide-area surveillance and enabled international technical collaboration, via the SAPIENT ICD [E1], with UK-based commercial defence supplier, QinetiQ [E10].
• enabled the UK DSTL to perform “ongoing development of SAPIENT, via the DSTL-led SAPIENT Interface Management Panel, to the current SAPIENT Middleware Interface Control Document (Issue: 5.0, 11th May 2020) supported by an investment in excess of GBP5 million (2016- 2020) and ~20 research scientists” [E1] in order to revise and progress UK government science and technology policy in this area (2016-2020). “.. without the work of the Durham research team DSTL would not have been able to validate the performance and design of the SAPIENT interface or evaluate the SAPIENT concept of operation (con-ops) against the operating characteristics of this militarily important sensing modality. Moreover, the capability of the Durham [sensor] for detection, tracking and classification of behaviours … of pedestrians [R1, R2] provided a cogent demonstration of the military and security relevance of the wider SAPIENT system.” - DSTL [E1]
Commercial impact based on this research at Durham has enabled:
•AptCore (UK), [REDACTED]
•Autonomous Devices (UK), [REDACTED]
•Blue Bear Systems Research (UK), [REDACTED] This enabled Blue Bear “ to deliver the UK’s most complex autonomous air vehicle trial [to date]” – Williams-Wynn [E7], with a single human operator simultaneously controlling “20 fixed wing drones to form a collaborative heterogeneous swarm” and collaborative “payloads and payload support from Plextek, IQHQ, Airbus and Durham University …” at RAF Spadeadam, Cumbria in September 2020 (resulting company income: GBP2.5 million, [E2]).
•Createc (UK), [REDACTED]
•Cubica Technology (UK) , [REDACTED]
•QinetiQ (UK), lead technical authority on the SAPIENT interface definition, [REDACTED]
Summary: Research from the Durham team [R1-R4] has had policy impact and enabled collaborative policy-making activity with six national governments (Australia, Canada, Netherlands, New Zealand, UK, USA), contributed to a GBP23.2 million investment in multi-sensor surveillance (UK/US government/industry) and resulted in GBP11.9 million of additional commercial income to six companies supporting the creation of ~55 science and engineering jobs in the UK.
5. Sources to corroborate the impact
[E1]. Testimonial – Defence Science Technology Laboratory (DSTL), [REDACTED], August 2020.
[E2]. Press releases x 3 - DSTL / UK MoD, 18 December 2015, 24 Sept. 2018, 28 March 2019.
[E3]. Government white paper(s) x 2 – UK Government (DSTL): Thomas, P. et al. SPIE,
10802/108020D / Dutch Government (TNO): - Bouma, H. et al. SPIE , 10802/108020N, 2018.
[E4]. UK DASA + US Army – competition & contract award documents (x 5, 2017-2020).
[E5]. Testimonial – AptCore Ltd [REDACTED], August 2020.
[E6]. Testimonial – Autonomous Devices Ltd, [REDACTED], December 2020.
[E7]. Testimonial – Blue Bear Systems Research Ltd, [REDACTED], Feb. 2020 /
Press Article – ADS Advance, December, 2020.
[E8]. Testimonial – Createc Ltd, [REDACTED], August 2020.
[E9]. Testimonial – Cubica Technology Ltd, [REDACTED], August 2020.
[E10]. Testimonial – QinetiQ Ltd – [REDACTED], August 2020.
Additional contextual information
Grant funding
| Grant number | Value of grant |
|---|---|
| DSTL – 1000085877 | £457,000 |
| QinetiQ Ltd - E01 – 00417-CT | £0 |