Impact case study database

X-ray, and more recently CT-based, security screening is used in both transport and border security for the detection of explosives, weapons and contraband for the purposes of both law enforcement and counter terrorism. Whilst the regulatory mandate for this technology in aviation and border security is a policy matter for government, the technical development of security scanners to meet these regulatory requirements is carried out by commercial suppliers. The research work at Durham [R1-R6] has had both policy and commercial impact.

Impact on Government Policy: Based on this research at Durham, Professor Breckon has:

• advised the UK government in his role as a member of the Cabinet Office Cyber Experts Group, under the direction of the Civil Contingencies Secretariat reporting to the Chief Scientific Advisor for National Security, on aspects of computer technology within transport and border security (2015-present) [E1]. [REDACTED]

• advised the UK Government Office of Science (2016-2017), reporting directly to the Chief Scientific Adviser (CSA) to HM Government, [REDACTED]

• enabled the test and evaluation of reference detection algorithms from [R4] on classified X-ray imagery by scientists at the UK Home Office [REDACTED]

• informed US Department of Homeland Security (US DHS) – Transportation Security Administration (TSA) [REDACTED]

Fig2: Exemplar object detection in X-ray image for threat detection (single-view, 2D–left) [R4].

Commercial Impact: Based on this research at Durham:

1. Rapiscan Systems Ltd (UK) [REDACTED]

Conservative estimates put Rapiscan at a 40% market share of the global CT security scanner market (source: European Commission, DG Competition - Case M.8087). With the move to CT to meet the new security standards, this translates as a global reach of 1.64 billion passenger journeys per annum (based on: IATA statistics, 2018).

1. Smiths Detection GmBH (Germany) [REDACTED] Conservative estimates put Smiths Detection at a 30% market share of the global CT security scanner market (source: European Commission, DG Competition - Case M.8087) – with the move to CT to meet the new security standards this translates as a global reach of 1.23 billion passenger journeys per annum (based on: IATA statistics, 2018).

• Cosmonio Imaging BV (Netherlands) [REDACTED] “in collaboration with X-ray scanner manufacturer L3 Harris Technologies, [REDACTED] (resulting company income - GBP260k) [E7]. [REDACTED] (contributing to the acquisition of Cosmonio by Intel in 2020). Conservative estimates put L3 Harris as having at a 9% market share of the global X-ray security scanner market (source: X-Ray Screening Report, 360 Market Updates, 2019) - 369 million passenger journeys per annum (IATA statistics, 2018).

• Gilardoni S.p.A. (Italy) has integrated an algorithm from the research of [R4] within their X-ray scanner product line resulting in an industry showcase at the UK Security Expo (2017) [REDACTED] “Today, detection is done by measuring the atomic number [of an object]. Now we want to include shape recognition [to be commercialised by Gilardoni] and integrated into its products ... [This] will give us much more precise detection ... [and] increase our market share by having better equipment than our competitors ... The market wants to see these ... upgrades as quickly as possible. .. Whilst all new Gilardoni equipment should carry the latest algorithm [from Durham], there is also a strong possibility that it will be retrofitable on legacy systems.” Andrea Rotta, product specialist Gilardoni [E8].

• Kromek Ltd (UK) [REDACTED]

• Micro-X Ltd (Australia) [REDACTED]

• VisionMetric Ltd (UK) [REDACTED]

• Gatwick Airport (UK) [REDACTED]

• Battelle (USA) [REDACTED]

Summary: Research from the Durham team [R1-R6] has significantly impacted government regulatory policy in the US and UK (representing the 1^st and 4^th largest national passenger volumes - source: World Bank, 2018), directly helps secure 500+ million passenger journeys annually and enjoys global commercial impact across nine businesses that includes commercial reach to over 70% of the market – securing up to 2-3 billion passenger journeys annually.

Effective wide-area surveillance is a key area of interest within both military operations and in civil infrastructure protection as it offers situational awareness – i.e. knowing what is happening and where it is happening within the environment (source: MoD Science & Technology Strategy, 2017).

Whilst the specification and evaluation of the operating requirements for these surveillance tasks is a policy matter for government, the technical development of sensing technologies to meet these requirements is carried out by commercial suppliers. In the UK, government science and technology policy in this area is informed by the UK Defence Science and Technology Laboratory (DSTL), an executive agency of the UK Ministry of Defence whose aim “is to maximise the impact of science and technology for UK defence and security” (DSTL). The research work at Durham [R1-R4] has had both policy and commercial impact within this area.

Impacts of Durham research on UK government policy are as follows:

• informed UK government science and technology policy for wide-area surveillance systems by enabling the UK DSTL to perform the “design, evaluation and validation of a new common interface specification for integrating varying types of intelligent networked sensors .. with direct reference to the performance characteristics of the [Durham] all-weather, all-condition camera-based thermal ... and the … requirements of the state-of-the-art real-time object detection, classification and tracking algorithms [provided by Durham -R2, R3]” [E1]. This was supported by “a total budget of GBP3 million (2013-2016), with additional technical and management support from ~10 DSTL staff scientists” - DSTL [E1]. The resulting UK government science and technology policy publication (SAPIENT Middleware Interface Control Document (ICD)) was publicly released under the UK Open Government Licence at ( http://www.gov.uk/sapient). Paul Thomas, Principal Scientist at DSTL, commented that this enabled “an important step forward in enabling sensors to ‘plug and play’ … [a] significant benefit both in the area of civil security such as the protection of infrastructure and in military systems such as for base protection.” [E2].

• informed UK government scientific policy-making activity by enabling the DSTL to test and evaluate SAPIENT “ for integrated wide-area surveillance within the inclusion of the thermal imaging solution provided by Durham [R1-R3], against a range of staged scenarios” including *“two large-scale technology demonstrations to inform ~60 senior personnel spanning UK military and civil infrastructure protection (from UK Ministry of Defence (MoD), Centre for Protection of National Infrastructure, Home Office, Dept. Transport) of the ‘art of the possible’ for integrated wide-area surveillance … (Malvern, Sept. 2015 / Throckmorton Airfield, June 2016)”* [E1] and technical evaluation “ against a range of unmanned aerial system [drone] targets flown in a variety of attack trajectories, using radar and electro-optic [sensors; R4].” [E3]. “The demonstrations provided an excellent showcase of sensor technology that was very useful in helping both military and civil [policy] stakeholders understand the progress, and more importantly the potential, for the technology.” - DSTL [E1].

• informed the UK government procurement policy for wide-area surveillance systems via the Defence and Security Accelerator (DASA). This led to the SAPIENT Middleware ICD, enabled by Durham research [in R1-R4], being adopted as the preferred interfacing option for commercial system suppliers to the UK MoD across GBP11.3 million of commercial research and development funding (2017-2020) [E4]. Furthermore, SAPIENT has been adopted for autonomous sensor management within the joint US Army / UK MoD procurement “Signal and Information Processing for Decentralized Intelligence, Surveillance, and Reconnaissance” (W911NF17S0003-US-UK, USD1.2 million, 2020). The UK Minister for Defence Procurement, Andrew Stuart MP, commented that by informing policy within UK defence procurement, SAPIENT “can act as autonomous eyes in the urban battlefield. Investing millions in advanced technology like this will give us the edge in future battles. It also puts us in a really strong position to benefit from similar projects run by our allies as we all strive for a more secure world.” [E2].

• enabled “collaborative experimentation internationally between the [governments of the] Five Eyes allied nations of Australia, Canada, New Zealand, the UK and the USA in the Contested Urban Environment experiment involving over 150 government and industry scientists and over 80 Canadian troops for three weeks in Montreal (2018)”* [E1]. This provided the five nations with a means to evaluate a range of differing sensors and operating methods for effective wide-area surveillance based on the work of [R1-R4] and informed the policymaking work of defence research scientists in the UK and partner nations. DSTL Chief Executive, Gary Aitkenhead, commented “[SAPIENT] is a fantastic example of our world-leading expertise at its best; our scientists working with our partner nations to develop the very best technology for our military personal now and in the future.” - [E2]. The collaboration has also informed the surveillance requirement policy of the military end user.

Figure 2: Briefing and site walk-around with demonstration attendees (upper), view from Durham sensor units (lower, left) and integrated multi-sensor visualisation via the common graphical display to operator (lower, right) – June 2016 (Crown Copyright).

“[SAPIENT] brings together our requirements as a user and DSTL as scientific advisers ... with our key allies in the five-eyes community.” said Lt Col Nat Haden, SO1 Intelligence, Surveillance, Target Acquisition and Reconnaissance Capability, British Army Headquarters [E2].

• enabled TNO (Netherlands, government research agency) to perform “rapid integration of raw and processed sensors by using open standards and the SAPIENT interface” [E3] designed and validated using [R1-R3] (2018). This allowed the Netherlands government to develop and evaluate additional novel sensing approaches to inform policy-making in wide-area surveillance and enabled international technical collaboration, via the SAPIENT ICD [E1], with UK-based commercial defence supplier, QinetiQ [E10].

• enabled the UK DSTL to perform “ongoing development of SAPIENT, via the DSTL-led SAPIENT Interface Management Panel, to the current SAPIENT Middleware Interface Control Document (Issue: 5.0, 11th May 2020) supported by an investment in excess of GBP5 million (2016- 2020) and ~20 research scientists” [E1] in order to revise and progress UK government science and technology policy in this area (2016-2020). “.. without the work of the Durham research team DSTL would not have been able to validate the performance and design of the SAPIENT interface or evaluate the SAPIENT concept of operation (con-ops) against the operating characteristics of this militarily important sensing modality. Moreover, the capability of the Durham [sensor] for detection, tracking and classification of behaviours … of pedestrians [R1, R2] provided a cogent demonstration of the military and security relevance of the wider SAPIENT system.” - DSTL [E1]

Commercial impact based on this research at Durham has enabled:

•AptCore (UK), [REDACTED]

•Autonomous Devices (UK), [REDACTED]

•Blue Bear Systems Research (UK), [REDACTED] This enabled Blue Bear “ to deliver the UK’s most complex autonomous air vehicle trial [to date]” – Williams-Wynn [E7], with a single human operator simultaneously controlling “20 fixed wing drones to form a collaborative heterogeneous swarm” and collaborative “payloads and payload support from Plextek, IQHQ, Airbus and Durham University …” at RAF Spadeadam, Cumbria in September 2020 (resulting company income: GBP2.5 million, [E2]).

•Createc (UK), [REDACTED]

•Cubica Technology (UK) , [REDACTED]

•QinetiQ (UK), lead technical authority on the SAPIENT interface definition, [REDACTED]

Summary: Research from the Durham team [R1-R4] has had policy impact and enabled collaborative policy-making activity with six national governments (Australia, Canada, Netherlands, New Zealand, UK, USA), contributed to a GBP23.2 million investment in multi-sensor surveillance (UK/US government/industry) and resulted in GBP11.9 million of additional commercial income to six companies supporting the creation of ~55 science and engineering jobs in the UK.

Background: With the global car market comprising ~80+ million vehicle sales per annum, the global autonomous vehicle development market is estimated to grow from USD54.23 billion (2019) to USD556.67 billion in 2026 (source: Allied Market Research). The UK Industrial Strategy has seen over GBP200 million invested in UK Connected Autonomous Vehicle (CAV) research to date (UK Department for Transport), including ~GBP1 million on projects involving Durham University, and rising industrial research budgets annually from major automotive manufacturers and component suppliers (2019: Renault (~EUR4 billion), Jaguar Land Rover (~GBP2 billion), ZF (~EUR2.6 billion)).

Commercial Impact: Our underpinning research has enabled and informed the development, evaluation and safety testing of CAV/ADAS sensing in the UK and beyond:

• Renault (France) used [R1,R3] to support the development of their “Human Like Guidance” (HLG) ADAS / CAV concept whereby the driver and vehicle communicate verbally based on a shared visual understanding of the road environment (i.e. with the vehicle acting as if it were a human co-pilot / e.g. vehicle → driver: “ Turn left before the red car on the right, just after the railings”).

In collaboration with Durham, Renault used real-time object detection techniques from [R1] to “understand the performance characteristics of current state-of-the-art scene understanding techniques when applied to the extended requirements of HLG, including the ability to robustly detect a wide-range of road-side and urban environment objects and their attributes.” [E1] “This informed in the down-selection of both algorithms and on-vehicle sensing hardware for HLG evaluation within Renault” allowing Renault “to demonstrate the initial HLG concept via a simulation based prototype, that included a real-time scene understanding module for object detection supplied by Durham, culminating in a successful presentation / demonstration to Renault-Nissan executives (2017). This resulted in strong positive feedback on the value to the end-user and an executive directive to accelerate development of the HLG concept within Renault.” – [REDACTED], Renault [E1]

Subsequently, the Durham team co-developed a real-time software component to perform extended real-time object detection and attribute estimation [R1], coupled with low-cost dense stereo based object range estimation [R3] (Figure 1, top). This was supplied to Renault, under commercial contract with Durham, and installed onto on-vehicle GPU processing hardware with an integrated all-weather stereo camera rig, GPS and inertial measurement unit (Figure 2 – bottom middle/right). This enabled Renault to “construct and demonstrate an integrated on-vehicle HLG prototype, incorporating a scene understanding module for real-time object detection and range estimation supplied as a combined hardware and software solution by Durham (2018). This resulted in on-vehicle realisation of the HLG concept within Renault” [E1].

Figure 2: Exemplar on-vehicle sensor rigs used to support data collection and research under [R1-R6] in collaboration with Machines with Vision (left top/bottom), ZF Race Engineering (top middle), Jaguar Land Rover (top right) and Renault (bottom middle/right) – on test at Durham University.

This on-vehicle HLG prototype, constructed in collaboration with Durham, enabled Renault to “perform extended proof of concept evaluation of the HLG concept using an integrated on-vehicle HLG prototype with 60 different test drivers on open roads around a common pre-defined evaluation route in Versailles. This enabled extensive on-road experimentation whereby the test drivers where guided via HLG voice navigation commands that were automatically derived from a list of visible scene objects (type, position, attributes) provided in real-time from the Durham scene understanding module (2018/19).” [E1]

From this collaboration with Durham, “The resulting experimentation and analysis of the HLG concept, enabled by our collaborative research work, has directly informed the vehicle design and development process within Renault with HLG now being considered within the production design (post-research) phase of Renault vehicles. From the HLG prototyping exercise with 60 different test drivers on public roads, Renault has identified and ranked several uses cases with their associated customer value. In this way, Renault has built a road map for the integration of theses uses cases in accordance with this associated customer value. In addition, Renault has registered 2 patents on vocal ‘human like’ guidance sentences [patent: FR3078565A1, 2018]” in support of “autonomous driving technologies to be available in 15 Renault models by 2022 as part of our current ‘Drive the Future’ strategic plan” [E1].

• Jaguar Land Rover’s (UK) development and evaluation of CAV in “collaboration with ... Durham [based on R2, R4, R6], … resulted in research that has informed our ability to evaluate 3D depth mapping ..., scene segmentation and ability for accurately describing scene features. This enabled us to utilise this .. in .. investigating more feature-rich ADAS (Advanced Driver Assistant Systems), and to push forward vehicle autonomy.” - [REDACTED], Jaguar Land Rover [E2]

This has enabled JLR to “increase our understanding on ... the usefulness of vision systems on vehicles ... and the impact that they can have in autonomy.” [E2] “The impact of this research … for driver assistance systems and vehicle autonomy, in both the off-road [R2, R4] and on-road [R6] environment has notably informed our internal research and development process for the range of .. options that may feature in production vehicles [patent: WO2018007079A1, 2016] ... and will have a large, positive impact ... towards autonomy.” (Fig. 2, top/right) [E2]

• Machines with Vision’s (UK, Germany) development of a novel all-weather CAV localisation sensor in “collaboration … [with Durham] has enabled … on-road testing of early low-TRL versions of our localisation sensor [patent: US20190265038A1, 2016] ... benchmarking against visual stereo odometry” and facilitated development of “a refined higher-TRL [Technology Readiness Level] version of RoadLoc, our automotive-specific localisation sensor, in collaboration with both Durham and Jaguar Land Rover” [E3]. Based on the body of automotive research [R1-R6] underpinned by prior on-vehicle test and evaluation (see Figures 1 / 2 + [E3]), “the research and industrial experience of the Durham team within the automotive sector, has directly contributed to the successful award and … delivery of ... research contracts by Machines with Vision.” (additional income to company: GBP436,000 – [E3])

“Today, Machines with Vision… [has] a turnover of GBP547,789 (2020) and projected GBP1 million of revenue in 2021 ..., representing a significant growth from ... founding [in] 2016. … Without our research collaboration … [with] Durham we would never have had many of the insights and connections with the automotive sector, nor the ability to independently test and validate our sensor designs.” - [REDACTED] Machines with Vision [E3].

“Our ability to provide technical early-stage evidence of both on-vehicle testing and proof-of-concept benchmarking ..., in addition to the guidance on … sensor packaging / mounting / interfacing received from your team in the early days of our … localisation sensor, has been instrumental in contributing to ...” - approximately GBP600,000+ of additional company income [E3] and full commercialisation of a rail variant of the CAV localisation sensor . This will be fitted onto “all DB‎ (Deutsche Bahn, German Railways) measurement trains … by the end of 2019” [E4]. This task was completed and now provides localisation in areas of poor GPS coverage (e.g. tunnels, cuttings etc.) and improved 2cm localisation accuracy over the entire network at increased train operating speeds [E4]. “DB measurement trains” are specifically equipped track survey inspection trains that are used to routinely monitor track condition in order to ensure railway safety across the ~41,000km of track in Germany that carries ~2.6 billion passengers annually (2019, source: DB). This is in addition to ongoing work “with Network Rail (UK) on ... its measurement train (responsible for .. safety on all the UK’s high-speed lines)” that helps protect an additional ~1.7 billion passengers across ~16,000km of track in the UK.

• ZF Race Engineering - Conekt Engineering Services (UK) collaborated with Durham to use stereo-based 3D scene mapping from [R3] to “develop and demonstrate 3D content generation from multiple cameras” and to “design and develop the 3D scene reconstruction from key-frame based photogrammetry”- [REDACTED] (ZF), [E5] . “This enabled the realisation of high resolution real-time 360° 3D information using low-cost camera sensing … for use in ... platform autonomy, ... path sensing and object avoidance” [E5] using data obtained from the multi-camera rig constructed at Durham (Figure 2, top middle). *“This technical work, coupled with the 3D computer vision research expertise at Durham, contributed to the successful delivery of [2 projects to the] Defence Science and Technology Laboratory [UK Ministry of Defence]” [E5] resulting in GBP164,000 of additional income to the company (2015/2016) and contributing to growth in the company patent portfolio in this area [patent: US20190182467A1, 2017].

Summary: Research from Durham [R1-R6] has enabled research and development at leading automotive manufacturers including patentable technology (France/UK: Renault, Jaguar Land Rover, ZF), enabled commercial sensor systems to be supplied to UK MoD (ZF) and supported technology translation into rail which now helps to protect ~57,000 km of track and ~4.3 billion passenger journeys annually (UK/Germany: Machines with Vision).