Skip to main

Impact case study database

The impact case study database allows you to browse and search for impact case studies submitted to the REF 2021. Use the search and filters below to find the impact case studies you are looking for.
Waiting for server

Accelerating the deployment of automated driving technologies through advanced testing and international standards development, to ensure safety and enhance public confidence

1. Summary of the impact

The development of a novel methodology to create test scenarios and environments for testing has accelerated the safe roll-out of automated driving pods by two leading manufacturers – Aurrigo (UK) and Yamaha (Japan). Warwick research has directly fed into standards development, including the international standards for both low speed automated driving and for Operational Design Domain (ODD), with Warwick as the lead author on each. Defining the ODD, or the operating conditions of a driverless vehicle, is fundamental to ensuring its roadworthiness. The research has removed a key barrier to worldwide uptake of these greener and more inclusive transport systems, and regulators across the globe have been given the confidence that this emerging technology can be used safely. Beyond influencing expenditure for multiple government agencies [text removed for publication], the standards have enabled pilot schemes in Canada, Japan and the UK, with over 23,000 members of the public travelling a combined distance of more than 20,000km.

2. Underpinning research

While prototype Connected and Autonomous Vehicle (CAV) technologies have existed for some time, ensuring level of safety of these technologies has been a barrier to the commercialisation and adoption of CAVs. This is due to the challenges of testing automated driving systems in all of the complex scenarios they may face in public environments. There are five fundamental and intrinsically linked stages required for a new testing approach:

  1. Create – the choice of test scenarios

  2. Format – the articulation of test scenarios in appropriate form

  3. Store – the central repository of test scenarios enabling sharing and contribution

  4. Execute – the running of test scenarios in suitable environments

  5. Use – the usage of results in safety evaluation

The research outlined in this case study by the Warwick Manufacturing Group (WMG) covers the first four stages, and is key to the corresponding regulatory impact and standards development. CAV testing started at Warwick in 2014 with WMG’s 3xD simulator for Intelligent Vehicles [G1], a platform for developing and trialling alternate test methods and scenario generation concepts.

Stage: Create In order to demonstrate statistically that automated driving systems are safer than human drivers, it has been claimed that they will need to be driven for more than 11 billion miles – a concept known as the 11-billion-mile challenge (RAND, 2016 doi:10.7249/RR1478). However, Warwick research has shown analysing risk objectively is essential for a common understanding of safety targets across stakeholders [3.1]. Furthermore, Warwick conducted a global study with experts from research organisations and companies across the automotive supply chain in Japan, Sweden, Germany, India, UK and the USA which revealed that, instead of focusing on the number of miles travelled, testing should focus on understanding the different scenarios that the CAV experiences during its journey. In particular, test scenarios that probe the circumstances which cause the system to fail were the most valuable in advancing knowledge [3.2].

This led Warwick to propose the Hazard Based Testing approach . Furthermore, a novel hybrid methodology (knowledge-based [3.3] and data-based) for test scenario creation was developed in 2016. The methodology involves: 1) Using Systems Theoretic Process Analysis (STPA) as the basis, with a proposed extension (developed by Warwick as part of INTACT project **[G2]**); 2) comprehensive analysis of accident databases (e.g. STATS19 road safety data from the Department for Transport); and 3) analysis of 200,000 anonymized insurance claims records (latter two developed as part of OmniCAV project **[G3]**). This hybrid test scenario creation methodology provides an innovative means of evaluating and establishing the safety of systems.

Stage: Format Assessment studies of the sector demonstrated that regulators and users of simulation-based testing require different levels of detail in scenario description. This led to Warwick developing a Scenario Description Language (SDL), with two levels of abstraction to meet the differing needs of this diverse set of stakeholders [3.4]. SDL is enabling the exchange of scenarios across the CAV industry and research community. Warwick now leads various national and international standardisation activities on SDL and derived concepts [G4], such as Operational Design Domain (ODD) covered in BSI PAS 1883 (UK) [5.4] and ISO 34503 (international). This research has enabled Warwick to lead on the globally-applicable ASAM OpenODD project, which further defines language for ODD definition.

Stage: Store As part of the Midlands Future Mobility project [G5], Warwick researchers have developed the UK’s National Scenario Database (NSDB) as a central repository for exchange of scenarios. This work led to a collaboration with Deepen AI (Silicon Valley start-up) and the World Economic Forum who have adopted the Safety Pool database (re-branded from NSDB) as part of the Safe Drive Initiative policy framework for governments [5.8]. Scenarios in the Safety Pool database are stored in the SDL format developed by Warwick.

Stage: Execute The research has utilised a unique facility to learn about simulation based testing - the 3xD Simulator for Intelligent Vehicles, funded by the EPSRC [G1]. The team developed an experimental drive-in, driver-in-the-loop simulator – the first of its kind built in a Faraday Cage for uninhibited wireless communication – to provide a secure and safe environment for vehicle evaluation. To achieve this, emphasis was laid on a software architecture to enable modularity and re-configuration. The purpose was to bridge the gap between virtual and real-world testing and learn more quickly about the technology and how people will use, respond and react to it [3.5].

3. References to the research

Bold = Warwick

All research was peer-reviewed and supported by competitively awarded grants [G1-G5]

[3.1] Khastgir, S., Birrell, S., Dhadyalla, G., Sivencrona, H. and Jennings, P. (2017) Towards increased reliability by objectification of Hazard Analysis and Risk Assessment (HARA) of automated automotive systems. Safety Science, 99 (Part B). pp. 166-177. doi: 10.1016/j.ssci.2017.03.024

[3.2] Gangopadhyay, B., Khastgir, S., Dey, S., Dasgupta, P., Montana, G. and Jennings, P. (2019, October). Identification of test cases for automated driving systems using Bayesian optimization. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC) (pp. 1961-1967). doi: 10.1109/ITSC.2019.8917103

[3.3] Chen, S., Khastgir, S., Babaev, I. and Jennings, P. (2020) Identifying accident causes of driver-vehicle interactions using system theoretic process analysis (STPA). 2020 IEEE Conference on Systems Man and Cybernetics (SMC). doi: 10.1109/SMC42975.2020.9282848

[3.4] Zhang, X., Khastgir, S. and Jennings, P. (2020) Scenario Description Language for Automated Driving Systems: A Two Level Abstraction Approach. 2020 IEEE Conference on Systems Man and Cybernetics (SMC). doi: 10.1109/SMC42975.2020.9283417

[3.5] Khastgir, S., Birrell, S., Dhadyalla, G. and Jennings, P. (2018) Calibrating trust through knowledge: Introducing the concept of informed safety for automation in vehicles. Transportation Research Part C: Emerging Technologies, 96. pp. 290-303. doi: 10.1016/j.trc.2018.07.001

[G1] Bhattacharyya, K. (PI), Ceglarek, D., Chalmers, A., Harrison, R., Jennings, P., Ng, I. and Williams, M. A., Robotics and Autonomous Systems: The Smart and Connected Vehicle. EPSRC, Jul 2013 - Sep 2014, GBP3,148,000

[G2] Jennings, P. (PI), Dhadyalla, G., Birrell, S. and Maple, C. INnovative Testing of Autonomous Control Techniques (INTACT). Innovate UK, Apr 2016 - Sep 2018, GBP856,767

[G3] Jennings, P. (PI), Khastgir, S. and Dhadyalla, G., OmniCAV. Innovate UK, Dec 2018 - Mar 2021, GBP2,702,612

[G4] Khastgir, S. (PI). UKRI Future Leaders Fellowship – Evaluating a Novel Evaluation Continuum for CAVs. UK Research & Innovation. Jan 2020 – Jan 2024, GBP1,110,158

[G5] Jennings, P. (PI), Higgins, M. D., Birrell, S. and Dhadyalla, G., UK Central CAV Testbed (Midlands Future Mobility). Innovate UK, Mar 2018 - Aug 2021, GBP25,477,975 across 2 grants

4. Details of the impact

CAVs are a new unregulated technology that offer huge potential benefits, including improved safety, lowered emissions, inclusive mobility and more efficient journey times. However, in order to unlock this potential, it is essential that CAVs are introduced in a safe manner, and are also accepted and used by society. A recent RAND corporation study (doi: 10.7249/RR1478) suggested that CAVs would need to be driven 11 billion miles to prove they are even 20% safer than human drivers. Achieving some form of regulatory approval therefore posed a new challenge, one that could not be solved through distance driven alone.

Research at the University of Warwick addressed this problem by posing alternate methods and standards for autonomous vehicle safety and roadworthiness. The body of research described in this case study has been critical to the development of international standards and regulatory policy for Connected Autonomous Vehicles, and has accelerated the commercial development of driverless pods around the world in addition to increasing societal trust in CAVs.

Impact on Standards: In addition to contributions to 13 national and international standards in total [5.1], Warwick were the lead author on two key standards which allow different CAV stakeholders to speak a singular operational language. The establishment of a common understanding between stakeholders naturally forms a vital catalyst to regulatory uptake (see Impact on Regulation and Policy). These standards arose from the “format” stage of the testing process, where the universal Scenario Description Language (SDL) was first introduced [3.4].

The ‘first of a kind’ international standard for SAE Level 4 automated system – ISO 22737 – was initiated in 2017 through the outcomes of the INTACT project [G2]. A major focus of this standard was detailed test scenarios for low-speed automated driving systems which included tests for occluded targets, driveable area, minimal risk manoeuvres and false positives [5.2]. With support from Japan, USA, South Korea, Germany, China, Australia and Canada, the standard was publicly available (in DIS stage) by July 2020. The standard succeeded in filling a regulatory gap for low speed automated driving (LSAD) vehicles, an area which had been entirely unregulated previously [5.3]. In developing ISO 22737, Warwick has been praised not only for accelerating and creating a functional and practicable standard in under 2 years, but also for overcoming industry scepticism and building consensus between varied stakeholders [5.2].

The second standard details a common taxonomy and description of the Operational Design Domain (ODD). ODD defines the operating conditions of a CAV and its accurate definition is key for safety in order to define capabilities and limitations of the system, i.e., impart informed safety [3.5]. The standard, first published as the UK national standard BSI PAS 1883 in August 2020 [5.4], progressed to an international standard (ISO) ISO 34503 (in development). Publicly available specifications (PAS), which are designed to satisfy urgent business needs, have never progressed to ISO standards in CAV previously. However, this happened for the first time with BSI PAS 1883, which “is an exemplar in showcasing how the (UK) research … has led to the creation of a UK standard and now an international standard – ISO 34503” [5.5].

Impact on Regulation and Policy: Traditionally regulators and policy makers look to international standards to frame their own guidance and regulatory requirements. Over the assessment period, national and international organisations have used Warwick’s research to create a regulatory environment which enables commercial innovation, which has driven investment and created new job opportunities, while also preventing the introduction of unsafe or immature technologies.

Formed in 2015 as a joint policy unit between the Department for Transport and BEIS, the Centre for Connected and Autonomous Vehicles (CCAV) acts as a point of contact for industry and academia for CAV technologies, coordinating and enhancing UK Government activity in the sector [5.5]. Between 2015 and 2016, CCAV consulted with the research and automotive supply chain communities on requirements for CAV testing. Based on recommendations [text removed for publication] on how best to create and manage a CAV ecosystem in the UK to integrate virtual and real-world testing, CCAV invested GBP100,000,000 to create Zenzic (formerly Meridian) and Testbed UK [5.5]. These offer a range of facilities to safely take and test ideas from concept to deployment both virtually and physically. Following on from this initial collaboration and building on work from INTACT [G2], research on virtual validation of Automated Vehicles [3.2] in the 3xD simulation was pivotal in CCAV launching a GBP15,000,000 programme on CAV simulation in addition to further shaping investment opportunities [5.5]. The Deputy Head of CCAV outlined that “ the development of legal frameworks and regulations to enable safe and secure deployment for CAVs [is] critical for the future of transport in the UK. By making a significant contribution in this area, on standards and through input into regulatory discussions, Warwick “ have removed key barriers on the road to smarter mobility by helping [CCAV] increase public trust and confidence in the CAV technology. These advancements have not only led to strong growth in the sector, but have made the UK a leading authority on the development of new standards for CAVs[5.5].

Since 2018, Warwick have worked with Transport Canada on implementing CAV standards in real-world environments. Following the development of ISO 22737, Transport Canada have thoroughly piloted LSAD technology, performing 17 tests based on the standard in collaboration with driverless shuttle developer EasyMile. These tests transported 670 riders nearly 500km over 10 days in a mix of weather conditions, and “ generated vital insights for industry and Transport Canada to enhance safety, user experience and support continued connected and autonomous vehicle policy research” [5.3]. In terms of investment, in October 2020 the Canadian Federal Government invested [text removed for publication] in Area X.O., an Ottawa based test centre formed from the L5 Connected and Autonomous Vehicle (CAV) Facility, with [text removed for publication] being invested directly for equipment needed for the ISO 22737 test scenarios [5.3]. Transport Canada have further strategically allocated resources on advanced vehicle systems covering LSAD shuttles in 2020, including [text removed for publication] to build an intersection at their [text removed for publication] test centre, and a further [text removed for publication] to develop intersection test scenarios based on the success of the ISO 22737 testing [5.3].

On the benefits of the standard, Transport Canada’s Chief of Human Factors & Crash Avoidance explained, “ISO 22737 has accomplished exactly what international standards are designed for – providing a framework of guidance designed by experts which can be applied ubiquitously to its subject area. Simply put, these trials, which are at the forefront of autonomous driving in Canada, would have been fundamentally impossible without ISO 22737 being in place. ISO 22737 provided the means to collect the evidence to help determine if the shuttles were roadworthy” [5.3].

Warwick have further contributed to CAV policy by responding to joint Law Commission consultations, with these commissions being independent bodies who review the law across the UK. In August 2020, Jennings and Khastgir responded to the CCAV Call for Evidence on Automated Lane Keeping System (ALKS) [5.5]. Warwick’s responses to this directly fed into and shaped thinking in the Law Commission as a result, as noted in their third consultation paper: “Instead, we (The Law Commission) agree with WMG, University of Warwick. In their response to the Call for Evidence they stated: some ALKS systems may be unable to ‘respond’ to special vehicles and manufacturers may choose to define them out of their ODD. This is a perfectly legitimate thing to do. However, defining special vehicle as out of the system’s ODD, still put the onus on the manufacturer to detect the presence of that attribute, i.e. monitor the ODD” [5.6].

Commercial & Social Impact: With an ongoing race to secure market share, UK-based driverless pod manufacturer Aurrigo wanted their Autonomous Control System (ACS) to be cheaper to produce, with uncertainty around this safety critical technology being a key challenge. Based on a collaboration beginning in 2016 [G2], WMG were able to de-risk and resolve this issue: “ the novel virtual testing environment … Warwick has developed has enabled us to take our self-driving pods to market two years earlier than would otherwise have been possible. [text removed for publication]. [5.7]. Through collaboration on ISO 22737 and aligning their products to this prior to its publication, Aurrigo grew their revenues [text removed for publication] in 7 countries by the end of 2020 [5.7]. Additionally, joint work with Warwick has enabled the filing of 3 patents, as well as prestigious trials internationally and with British Airways to test autonomous baggage dollies at Heathrow Airport. On the market advantage gained, Aurrigo’s CTO stated: “ without the ISO 22737 LSAD standard, we would have been unable to unlock the [text removed for publication] market opportunity for [text removed for publication] our vehicles over the next five years”. Expanding further on the University’s ongoing role, their CTO added, “ WMG is critical in enabling the safe deployment of our technology in the real-world[5.7].

In Japan, Yamaha Motor Corporation – the world-leading motorcycle manufacturer – has collaborated with Warwick since 2017. The development of ISO 22737 removed the need for them to develop an equivalent in-house standard, saving approximately JPY200,000,000 (about GBP1,300,000) in a process which would have taken 2 years at a minimum [5.2]. [text removed for publication]. LSAD vehicles have been crucial for Yamaha’s ART for Human Possibilities initiative, which aims to achieve company growth while finding solutions to social issues. Driverless pods in particular assist people with accessibility needs including the elderly, people with impairments, pregnant women and families with small children [5.2]. Between 2019 and 2020, pilot schemes by Yamaha saw “22,000 members [transported], with a total distance travelled across all vehicles of more than 20,000km”. The company confirmed that by March 2020, LSAD vehicles had been socially implemented in Eiheiji Town in Fukui Prefecture, Chatan Town in Okinawa Prefecture, and Kamikoani Village in Akita Prefecture, where they are being used as a local mobility option for the community [5.2]. Aurrigo have similarly piloted their vehicles in social initiatives, working with the charity Blind Veterans UK to provide mobility for 426 blind veterans, with 21% of individuals having significant mobility issues [5.7]. [text removed for publication].

At the interface of industry and regulation, the World Economic Forum (WEF) – an agenda-setting international NGO – launched the Safe Drive Initiative (SafeDI) for AVs, with community membership from Aurora, Cruise and Microsoft. In November 2020 WEF released a key AV policy framework white paper for SafeDI, linking industrial AV safety expertise with regulators who wish to promote deployment while understanding what can be considered safe for AVs. Downloaded 3,000 times within its first week and adopted by the Dubai Roads and Transport Authority, the guidance was founded on Warwick’s research on the ODD based scenario repository, and enables “ seamless incorporation by regulators, such as a national, state, regional or municipal authorities, removing the essential barrier of conceptualising a framework inside the organisation[5.8].

5. Sources to corroborate the impact

[5.1] List of CAV standards developed by the University of Warwick

[5.2] Statement from the Top Manager of LSAD development and commercialisation, Yamaha

[5.3] Statement from the Head of Crash Testing and Human Factor, Transport Canada

[5.4] PAS1883: 2020, Operational Design Domain (ODD) taxonomy for an automated driving system (ADS) – Specification.

[5.5] Statement from the Deputy Head of CCAV, Department for Transport / BEIS

[5.6] AV: Consultation Paper 3 (Law Commission, Dec 2020) https://tinyurl.com/32wuvu2s

[5.7] Statement from the CTO of the RDM Group and Aurrigo

[5.8] Statement from Lead for Automotive & Autonomous Mobility, WEF, with SafeDI white paper

Additional contextual information

Grant funding

Grant number Value of grant
EP/K011618/1 £3,148,000
102587 £856,767
103287 £1,568,255
104035 £17,556,098
104529 £2,702,612
105094 £7,911,897
MR/S035176/1 £1,110,158