Impact case study database

Our work on reasoning infrastructure for ontologies and knowledge graphs has led to a range of impacts in diverse settings. These include:

• impact on ontology developers, through the development of technical standards and open source tools that support these standards;

• impact on non-profit organisations and businesses that develop and use ontology-based systems;

• direct impacts on commerce and production through the spinout company OST and the application of its patented technology in industry.

Impact on ontology development through open source reasoners for OWL 2 ontologies.

Horrocks, Motik, and Cuenca Grau have been influential in the development of the OWL ontology language, editing key specification documents for the refined and extended OWL 2 standard (second edition published by W3C in 2012) [ E1]. The standard allows ontologies to be shared across applications, and has stimulated increasing use of ontologies in a range of sectors. Reasoning systems that support OWL, in turn, are an important enabling factor for the wider adoption of the standard and the increased uptake of ontology-based systems. Effective reasoners are critical for ontology development and use, for example to verify the consistency of the knowledge they express, to make implicit knowledge explicit, or to access data via queries. The OWL reasoners HermiT [ R1– 2] and ELK [ R3] have achieved wide reach as open source projects and through distribution with the open source Protégé OWL ontology editor, in which HermiT – the only reasoner that fully and correctly supports OWL [ R2] – is the standard reasoner. Protégé is the most popular tool for ontology design, with over 360,000 registered users, and is widely relied on for building and maintaining ontologies in industry and in large government projects such as the National Cancer Institute Thesaurus and the WHO’s International Classification of Diseases [ E2].

Large clinical and biomedical nomenclatures like these are a prominent application area for ontologies; they are gradually superseding existing medical classifications, and will provide the future platforms for gathering and sharing medical knowledge. One such ontology is SNOMED CT, the world’s most comprehensive clinical terminology – “a codified vocabulary that is now accepted as a common global language for health terms” [ E3]. It is owned and administered by the non-profit organisation SNOMED International, and is officially used in over 80 countries (including the UK, a founding charter member) to support the direct management of individual health and care and to improve the flow of data across health and care systems. SNOMED CT is an NHS Digital information standard, applying to all organisations providing NHS, Public Health, and/or Adult Social Care services. Since 2018 it has been implemented across Primary Care settings within the NHS, and is a core component of the NHS patient record service; all secondary, dentistry, and optometry services are required to establish a detailed implementation schedule by 31 December 2020 [ E3, E4].

SNOMED International has used ELK since 2016 to perform complex reasoning tasks in its new authoring platform: “for example, to check the logical consistency of (the definitions of) SNOMED concepts, and to automatically organise concepts into a hierarchical structure, which is essential when dealing with such a large terminology” [ E3]. SNOMED CT contains more than 400,000 terms and is “under constant revision and extension”. “ELK was a game changer for us as it reduced the time needed to reason with SNOMED CT from hours/days to only a few seconds. Not only that but, unlike some earlier systems, ELK was developed by researchers who were also able to provide formal and peer-reviewed guarantees as to the correctness of the algorithms embodied in ELK [ R3]. This is clearly of great value given the critical role played by SNOMED CT in the NHS and many other national healthcare systems” [CIO, SNOMED International: E3].

B2i Healthcare is a software engineering firm that specialises in SNOMED CT and healthcare information standards and exchange. B2i has used ELK across the whole assessment period to provide reasoning services in its authoring platform, Snow Owl. Snow Owl is deployed in over 3,000 locations in over 80 countries, and B2i is able to state that “ELK is the de facto global standard for classifying SNOMED CT ontologies by national eHealth programs and organizations”. These include national health ministries and agencies in the UK, Belgium, Denmark, Estonia, Ireland, Norway, Sweden, Switzerland, Australia, Singapore, New Zealand, and the USA [Non-Executive Chairman, B2i Healthcare: E5]. B2i uses ELK because of “its outstanding performance and the correctness guarantees provided by its basis in world leading research [ R3]. ELK performs description logic classification in parallel on modern multi-core computers, which allows the full international SNOMED CT plus the Australian extensions (830,926 relationships) to be classified and checked for equivalencies in about 10 seconds on a modern desktop computer. This is in stark contrast to earlier systems which took hours to perform this task, if they were capable of performing it at all. The orders of magnitude performance improvement provided by ELK means that reasoning can be used ‘on the fly’ during the authoring process, resulting in both improved quality and productivity” [ E5].

Besides medical and life sciences, OWL ontologies are increasingly used in industry to address information integration and access problems. Aibel is a multinational company headquartered in Norway that provides engineering and construction services in the oil, gas, and offshore wind industries. Aibel has used HermiT across the whole assessment period to (i) support the development of a new master ontology system for design requirements and specifications; and (ii) perform reasoning and query answering over this custom-built ontology to configure design and product selection. These tasks are now “performed with greater precision and less effort than with Aibel’s legacy system, ultimately resulting in a design of higher quality, which again reduces the total time and cost of construction” [Senior Manager: E6]. Aibel uses HermiT because of “its full support for the OWL ontology language, and its superior performance, reliability and scalability” in ontology reasoning [ E6; see R1– R2]. Using HermiT, Aibel has been able to document significant reductions in data errors and duplications. In the context of the very large projects delivered by the company, such deficiencies in data quality can have significant cost implications: for example, the presence of duplicate design artefacts leads to erroneous bulk orders of materials. The knowledge-based approach enabled by HermiT supports “a more efficient and precise description of design artefacts…removing practically all duplicate design artefacts recorded in the system. (It is estimated that more than 30% of the legacy system’s data was duplicate data.) The lack of duplicates and added detail in design descriptions make it easier to manage material storage and order a better selection of materials…The effect is an estimated cost reduction of approximately 5% for bulk material orders, which in large projects amounts to more than EUR100,000,000” [ E6]. Aibel states that: “in summary, HermiT plays an important role at Aibel, and has enabled us to deploy knowledge-based solutions that significantly reduce cost” [ E6].

Impact in industry through commercialisation of high-performance RDFox system.

Pathway to impact. The highly scalable RDFox tool was initially released for research purposes in 2014 under an open source licence. A patent was also filed on the underpinning research: this would later be significant to the commercialisation of the technology through the spinout company OST [ E7]. The Oxford team subsequently worked with industrial partners to develop the technology into a mature platform with performance bottlenecks identified and removed [ R5, R6]. RDFox was applied in several pilot industry projects, including with Siemens, for ontology-based data access enhancement (2015); Statoil (now Equinor), for integration and analysis of oil production and geological survey data (2016); Kaiser Permanente, in patient data analysis for healthcare compliance (2016); EDF Energy, to manage and analyse information about their electricity distribution network (2016); and other partners including Skyscanner, for recommender engines, and Armasuisse for text analysis [ E8]. During these pilot projects the functionality of RDFox was extended to match business requirements.

In 2017, OST was formed to bring cutting-edge research in semantic technologies to industry. OST has raised GBP4,100,000 in investment, including GBP3,000,000 in Series A investment led by Samsung Ventures, announced in June 2019 [ E9]. By September 2019 OST employed 6 full-time staff; 9 FTEs were employed by the end of the assessment period. OST’s patented technology is sold under licence to customers for a fee of approximately GBP50,000 per licence. Since April 2018, the company has secured licence sales worth over GBP570,000 [ E10]. Customers include Festo, a German multinational production line equipment company, and electronics giant Samsung.

Festo produce and sell pneumatic and electrical control and drive technology for factory or process automation. They have used RDFox since 2018 to implement a new semantic approach to data processing through the Festo Semantic Platform [ E11, E12.2]. In turn, the semantic approach has allowed them to optimise their sales process. Festo’s “choice of RDFox was based on its outstanding performance, reliability and scalability” compared to other knowledge graph systems, and its basis “in published algorithms whose correctness has been formally verified”, giving “high confidence that RDFox will continue to provide correct answers regardless of the knowledge and the queries that we use” [Head of Smart Data Services: E11]. RDFox’s performance advantages have a direct impact on production at Festo. The company’s product catalogue includes thousands of components that can be combined in millions of different configurations, with configurations being subject to many complex constraints. RDFox is used to compute possible configurations and to ensure their validity. The previous approach, using multiple relational databases, “took several hours to compute valid configurations, and also required complex data maintenance and deployment management. In contrast, RDFox can compute valid configurations in a few seconds…This saves us a huge amount of time and also allows for much greater flexibility in updating our product catalogue” [ E11]. The company has found that “the semantic approach is also much easier to maintain and extend” [ E11]. Queries are now “much easier to write”, and “extensions or changes in the information model due to technical or business demands are much faster accomplished and validated. That is a significant increase in flexibility as well as saving time and computing resources” [ E12.1]. As a result, Festo are extending the Semantic Platform to cover further technical product domains. As “a key component of the Festo Semantic Platform”, RDFox is central to this new approach across the company, “and as such is already having a significant impact on our data management business” [ E11; see also E12.2].

Prior to their investment in OST, Samsung had already funded various projects, worth approximately GBP500,000, with a view to using RDFox in several different customer-facing applications. Samsung are now using the RDFox knowledge graph system “to drive applications running on a variety of platforms including embedded systems and mobile devices (phones and tablets)”; these include, for example, lifestyle recommender systems linking very large and heterogeneous data sets covering areas such as food, health, and exercise. “When fully deployed, [Samsung] expect such applications to be running on millions of devices, all of which will access relevant knowledge via RDFox” [ E13]. Samsung’s choice of RDFox “was based on its outstanding performance, reliability and scalability.” Other knowledge graph systems tested by Samsung “were all found to be much slower than RDFox, often by orders of magnitude, and could also produce incorrect answers…In summary, RDFox is a key component of our knowledge-intensive applications, and we expect such applications to be used by hundreds of millions of Samsung customers worldwide” [ E13].

Ongoing aviation modernisation efforts are focussing on improving efficiency and safety whilst lowering costs. Fundamental to this change is having more detailed data streams: this includes protocols used in ATC (Air Traffic Control)-related operations such as aircraft surveillance and collision avoidance, but also protocols used for general data-link communication between aircraft and ground stations, and carrying information such as the medical status of passengers and payment details for in-flight transactions.

Most of these systems were designed decades ago, and lack resilience against malicious actors under well-accepted modern threat models. The main reason for this is that ATC has long focussed on safety, i.e., making systems resilient to faults that occur naturally and unintentionally. This ‘historical mindset’ ignores intentional malicious behaviour, which is at the centre of security research. Our impact stems from our demonstration of the scope for malicious activity – providing the evidence base for policy changes throughout the industry. Concretely, our work has led to the following:

• Impact on regulatory policy and commercial operations

• Impact on national-level procurement

• Open provision of aircraft data

Impact on Regulatory Policy and Commercial Operations.

At the international level, the International Civil Aviation Organization (ICAO) has recognised for the first time the need to validate ADS-B-derived information [ E1]. Our work is cited as evidence that ADS-B data cannot be solely relied upon for ATC, and so that a proposed full transition away from secondary transponder radar or multilateration systems cannot take place without new security measures.

At the level of national governance, our work is cited in policy documents or directly confirmed by government agencies in multiple countries. In 2016 the United States Department of Defense (DOD) issued a Call for Proposals for research towards securing ADS-B [ E2], referencing our work [ R3] as principal motivation. The call’s objective was to “develop a modular, secure, and affordable solution for Automatic Dependent Surveillance Broadcast (ADS-B) for Air Force platforms”. It uses our research as the primary evidence for the need to mitigate the lack of security in ADS-B: “while ADS-B will play an essential role in the future of air traffic control, the inherent lack of security measures in the ADS-B protocol is a reason for concern. The problem has recently been widely reported in the press and at hacker conventions. Academic researchers, too, proved the ease of compromising the security of ADS-B with current off-the-shelf hard- and software (Ref 2). It has also been estimated that it will cost billions of dollars to retrofit all DoD aircraft with ADS-B technology. Given these numbers and the looming ADS-B Out FY 2020 mandate, it is readily apparent that there is a need for a practical, secure and affordable solution to transitioning NextGen technology into the DoD/civilian fleet of aircraft.”

“(Ref 2)” refers to the preprint version of our paper [ R3] and in turn elements of [ R1] where the experimental analysis of active attacks on ADS-B was first conducted.

More recently, the United States Government Accountability Office issued a report to congressional committees in January 2018 on the “Urgent Need for DOD and FAA to Address Risks and Improve Planning for Technology That Tracks Military Aircraft” [ E3], again citing our work in [ R3]. In Switzerland, meanwhile, “based on the recent research insights regarding the insecurity of many civil and military aircraft communication systems” articulated in [ R1- R6], the government “has invested significant funds to build a new avionics laboratory in its headquarters in Thun” as “a priority undertaking”, and the Swiss Air Force “has decided to make a distinction between traditional electronic warfare threats and new cyber security threats to aircraft” [ E4]. Our work has further been incorporated in assessments of cyber-security for critical national infrastructure by the governments of Singapore [ E5] and Sweden [ E6], citing our work on ADS-B, and the Netherlands [ E7] (citing [ R1- R3]).

Indeed, our work has promoted awareness of cyber-security issues across the entire aviation community, from individual pilots [ E8] to airlines [ E9] – underpinning pressure on air traffic regulators to address the insecurity issues in aircraft data links. Swiss Air undertook system improvements to mitigate a vulnerability we discovered [ R5] and responsibly disclosed to them [ E9], in which passenger credit card details were broadcast without encryption whenever an onboard payment was made. Our work thereby benefitted not only Swiss Air from a data-protection perspective, but also any of the more than 16,000,000 passengers carried by the airline annually who have made onboard purchases since the system improvement.

Impact on National-level Procurement.

Armasuisse is the Federal Office for Defence Procurement within the Swiss Confederation. It is the sole procurement organisation for defence and civil protection purposes in Switzerland. They confirm in their letter [ E4] that our aviation security research has changed their procurement process in two major ways. Firstly, introducing the requirement for systems to provide physical-layer data to enable detection and mitigation of attacks on ATC communication links (based on our work [ R1- R6]). Secondly, requiring the inclusion of cyber-security testing in the procurement of national ATC surveillance systems. This testing particularly considers the fake message injection attacks identified in our work [ R3]. This second activity is already occurring in practice, with Armasuisse writing in [ E4]: “we have used approaches, knowledge, and software developed in collaboration with Oxford to conduct extensive in-the-field penetration testing of the new Multilateration system currently procured by the Swiss air traffic control company Skyguide”.

Open Provision of Aircraft Data.

The OpenSky Network operates over 2,000 sensors worldwide and serves approximately 3,000 members, including national air traffic regulators and EUROCONTROL (the Europe-wide air traffic body). The value of this open data is keenly recognised in the air-traffic sector. EUROCONTROL’s Performance Review Unit is incorporating feeds from the OpenSky Network into their activities in “monitoring and reviewing the performance of the pan-European air navigation services (ANS) system”, as part of a broader mission to coordinate and harmonise air-traffic management across Europe [ E10]. The unit notes that with the incorporation of the OpenSky Network as a source “[t]he additional data will enhance the tracking of aircraft movement, particularly in terms of better accuracy, higher reporting rate and faster access to data” [ E10].

The importance of open data has come rapidly into focus with the emergence of the COVID-19 pandemic. In an April 2020 publication [ E11], the International Monetary Fund (IMF) recommended the data provided by the OpenSky Network as a source for use in measuring an economy’s external sector (that country’s economic relationships with the rest of the world) when faced with emergency circumstances. Further, in May 2020, the OpenSky Network provided flight data to the Bank of England for the preparation of its quarterly Monetary Policy Report [ E12], which studied the economic impact of COVID-19 upon the UK. The data were used to highlight the sharp decline in the number of aircraft departures, both in the UK and worldwide. Data from the OpenSky Network is also referred to as a high-frequency indicator for gross domestic product (GDP) forecasting. The report states: “as business output surveys are providing a less useful steer than usual, Bank staff are using a wider range of indicators to gauge how GDP is likely to evolve”. Four data sources were highlighted as such high-frequency indicators, with the OpenSky Network the sole resource for flight traffic levels. This was contributory to the bank’s estimate that “monthly GDP will fall by enough in March to pull GDP down by around three percent” [ E12.1]. Use of OpenSky data continued in the following quarterly Monetary Policy (August 2020) [ E12.2].

In addition to the impacts on regulatory policy, commercial flight operations, and defence procurement arising from our work on the security of aircraft communications systems, the open provision of aircraft data through OpenSky has given rise to further impacts on national-level data policy and on systems development in industry. The development of OpenSky in collaboration with Armasuisse was itself a technical case study in big-data management for security and defence purposes [ E4]. Beyond uses in government and policymaking, OpenSky Network data is also integrated into more than 20 open projects including the LiveTraffic module for the commercial-grade and FAA-certifiable flight simulator X-Plane. (It is also included as core functionality in the free GeoFS flight simulator.) [ E13]. In industry, companies manufacturing ATC monitoring systems now provide interoperability with the OpenSky Network: for example, Günter Köllner Embedded Development GmbH (trading as jetvision), which produces and markets receivers specifically for the OpenSky Network [ E14]; and SeRo Systems GmbH, which also provides high-end sensors that are interoperable with OpenSky, and used for collection and processing of Mode S and ADS-B data for aviation traffic management [ E15].

Pathway to impact. Semmle was founded in 2006 to create the novel technology that realises the potential of Datalog for software analysis demonstrated in the Oxford team’s research, widening the scope of application to business intelligence rather than just program analysis. Six US patents were filed by Semmle to protect these further advances after the creation of the company [ E1]. In September 2014 Semmle raised USD8,000,000 Series A investment from the leading venture capital firm Accel Partners. This allowed the team to expand from its Oxford office to two new sites in Copenhagen and New York, and to make the product enterprise-ready. The company grew from 18 employees in 2014 (FTEs: 18), to 35 following Series A investment (FTEs: 35), to 48 by 2018 (FTEs: 48). In August 2018 USD21,000,000 Series B funding was raised from a consortium led by Accel Partners. By the time of its 2019 acquisition by GitHub, a Microsoft subsidiary since its USD7,500,000,000 acquisition in 2018, Semmle employed 81 staff (FTEs: 81) across six offices in Oxford, San Francisco, New York, Seattle, Valencia, and Copenhagen [ E2, E14.3]. It is not possible to give confidential information about company revenue, or the sale price. However, business insights company Owler estimates 2019 annual revenue at USD10,000,000, and financial data company PitchBook estimates the sale price at USD410,000,000 [ E14.1, E14.2].

Semmle’s technology is built on a novel approach to program analysis that combines two disparate disciplines, object-oriented programming and database logic. By treating source code as a relational database, and analysis problems as queries against a database, deep semantic analyses can be expressed as concise queries in an object-oriented query language. The creation of such queries is an order of magnitude quicker than previous methods of creating code analyses. The query language is an object-oriented form of Datalog, whose evolution can be traced in [ R1– R6]. Cutting-edge techniques are used to make the queries perform well. Semmle’s analysis engine thus makes software easily and accurately searchable, allowing complex questions to be asked at a previously unachievable pace and scale [ E3, and below]. This REF period coincided with the application of Semmle’s technology to its ‘killer’ commercial use case, software security. Semmle’s unique solution for variant analysis – when a new vulnerability is identified in a software system, to find all occurrences of the same logical mistake in the same code base, or indeed in a portfolio of codebases – has delivered business-critical benefit to numerous major global clients since August 2013. These include Behavox, Credit Suisse, Dell, Google, Microsoft, Mozilla, Murex, NASA, Nordea, Uber, and other clients who cannot be named publicly. Three examples are described below. The period also saw the growth of Semmle’s wider contribution to software development and integrity through its open and shared approach to security. It made its CodeQL analysis engine freely available through an open analysis platform, LGTM.com, used by over 700,000 developers [ E7], and worked closely with top commercial security teams and the open source community to find and report vulnerabilities in widely used software [ E5.1–3, E6].

Software security and development in industry. Microsoft’s products and services are used by billions of individuals and millions of companies every day. When a software vulnerability is identified or reported, failure to find and patch all variants at the same time increases the risk of code being exploited in the wild. Since 2017 Microsoft has incorporated Semmle automation into its code review processes, using it to analyse bugs reported in a given component (for example the JavaScript engine of the Edge browser) and define queries that can find similar patterns in other components. A single such query has in many cases “provided a number of actionable results across multiple codebases”. Microsoft then store these queries in a central repository, to be re-run periodically by security teams across the organisation. In addition to variant analysis, Microsoft’s software researchers use Semmle’s technology proactively to review source code and identify vulnerabilities. Its ability to identify even the most complex semantic patterns means that an “explorative approach” can be maintained while automating what would otherwise be a “tedious and error-prone” process of manual audit. “Using Semmle to scale up our code review capabilities was an easy choice” (Security Software Engineer, Microsoft Security Response Center) [ E3, E4].

Nasdaq Corporate Solutions provides business and market intelligence through software and consultative services to thousands of organisations globally. After benchmarking static analysis tools and software analysis products, Global Corporate Solutions Technology (GCST) at Nasdaq chose Semmle’s LGTM product in 2018 as the only solution capable of running the continuous and accurate analytics needed to monitor development standards across a growing and highly complex application portfolio. Nasdaq have used Semmle analysis to provide actionable insights for strategic decision-making and budgeting across the organisation, enabling improved tracking of development projects, increasing the efficiency of engineering teams, and benefitting the individual developers who use Semmle’s tools. As a result, Nasdaq “completely changed the way people are writing code” (SVP of Global Corporate Solutions Technology), and brought down “technical debt” across the application portfolio (i.e., the implied future time and development cost accrued by using limited or imperfect solutions in the short term) by more than 75% within one year. This reduced software risk, and freed developer time for value-creating tasks [ E3].

BlackLine is a leading global provider of cloud financial software, serving over 2,300 organisations in more than 150 countries. After its initial public offering in late 2016, BlackLine turned to Semmle to meet the challenge of increasing the complexity of its code architecture by the addition of new software features and developments, while maintaining the security and integrity of its clients’ critical financial data. Blackline used the technology to identify complex data integrity issues, automate code review, and implement robust coding standards across the organisation. Blackline’s Director of Software Development was “blown away by [Semmle’s] power”; “no automated solution we had tried was able to find problems of this complexity, but using Semmle we became able to quickly find these types of issues across our portfolio. Within weeks this eliminated manual efforts that were consuming significant cycles of our SDLC” (Software Development Life Cycle). Using Semmle to enforce the organisation’s coding standards, moreover, enabled BlackLine “to reduce our maintenance spend drastically”. “On top of that, our developers are relieved of routine tasks related to maintenance work and can now focus on the creative side of their work” [ E3].

Securing open source software. Almost every software product today relies on open source code at some point in its supply chain. Through its commitment to security as a shared responsibility, Semmle has made a significant contribution to reliably securing such code, benefitting both developers and the consumers of software, and helping to ensure the continued growth and sustainability of open source. Semmle’s technology was made freely available to open source developers through its LGTM.com analysis platform, allowing them to run shared queries against their code or create their own. As of January 2020, the service analysed every commit on over 135,000 projects worked on by a user community of over 700,000 developers [ E7]. Security research findings contributed by Semmle’s security research team and by its customers were shared through an open Git repository, amplifying the expertise of top security researchers across the world [ E5.1]. Semmle’s security research team also directly discovered and reported many vulnerabilities in open source projects [ E5.2]. By September 2019, when Semmle was acquired by GitHub, the team had responsibly disclosed over 100 CVEs (Common Vulnerabilities and Exposures) in high-profile projects like U-Boot, Apache Struts, the Linux Kernel, Memcached, VLC, and Apple’s XNU, allowing project developers to patch vulnerabilities and protect users [ E5.2]. As GitHub’s SVP of Product noted, Semmle’s unique approach to code analysis “makes Semmle far more effective, finding dramatically more issues and with far fewer false positives”: “just as relational databases make it simple to ask very sophisticated questions about data, Semmle makes it much easier for researchers to identify security vulnerabilities in large code bases quickly… Because QL [Semmle’s analysis engine] is declarative and object-oriented, creating a new analysis is much easier than with traditional code analyzers. Customers frequently find vulnerabilities they couldn’t find with other tools and accomplish tasks that used to take weeks or more in hours” [ E10.3]. Some of the vulnerabilities discovered using the technology have had drastic data security implications. In 2017 and 2018, for example, Semmle security researchers discovered critical remote code execution (RCE) vulnerabilities in Apache Struts, a widely used framework for Java applications [ E8]. Equifax’s failure to patch a similar RCE vulnerability in Apache Struts earlier in 2017 led to a serious data breach that had directly cost the company USD440,200,000 by the end of 2018 [ E9]. According to some expert commentators, the 2018 Semmle-discovered vulnerability was potentially even more concerning than the previous RCE vulnerabilities, including the one exploited in the Equifax breach, because it “operates at a far deeper level within the code” [ E6.2, at pp. 15–17: synopsys article on CVE-2018-11776, August 2018].

The success of its open, community-driven model allowed Semmle to double its customer base and to increase open source usage tenfold in the year 2018/19, resulting in its acquisition in September 2019 by GitHub, the leading global platform for open source resources and collaboration. GitHub announced their excitement in “bring[ing] the world’s most powerful semantic code engine to the world’s largest developer community” [ E10.1], describing the move as “a big step in securing the open source supply chain” [ E10.2]. GitHub’s CEO called Semmle’s engine “revolutionary”, and the SVP of Product stated that “no other code analysis tool has a similar success rate” in finding vulnerabilities [ E10.2 & 3]. The CEO stated: “Semmle’s community-driven approach to identifying and preventing security vulnerabilities is the very best way forward… As a community of developers, maintainers, and researchers, we can all work together toward more secure software for everyone” [ E10.2]. Following the acquisition, the Semmle security research team formed the core of a new GitHub Security Lab, which was launched in November 2019 with the mission to find vulnerabilities in open source projects and to make tools available “to make it easier for others to find those vulnerabilities within their own codebases” [ E11.1 & 2]. Semmle’s “industry-leading semantic code analysis engine”, CodeQL, was accordingly released as a GitHub product in November 2019 [ E11.2, E12.1]. It is free for research and open source, and is available to all public repositories and enterprise customers via GitHub’s continuous integration and deployment service [ E12.2]. As of July 2020 GitHub reported having over 50,000,000 developer accounts, over 2,900,000 enterprise accounts, and over 100,000,000 repositories worldwide [ E13].

Route to impact. The research contributions were protected through two patent applications: one on map matching and lifelong learning [ R2– R4] has been granted in the US and Australia and is under consideration in Europe and China; one on pedestrian dead reckoning [ R1] is under consideration in the US, Europe, China, and Australia [ E1]. These patent applications were licensed to Navenio Ltd in December 2015. The algorithms have since been further refined by Navenio’s engineers under the guidance of Trigoni [ E2], making them robust enough to be deployed in complex multi-building hospital sites.

In 2018, Navenio received GBP427,714 in funding from the GBP17,000,000 Digital Health Catalyst competition run by UKRI, to develop their tracking technology – “an Uber for porters” that enables hospital staff to be in the right place at the right time [ E3]. Navenio has developed two location-based services that use the novel indoor positioning technology developed by Trigoni’s research group in [ R1– R4]: 1) an intelligent workforce solution (IWS) for porters and other hospital staff; and 2) a cleaning compliance solution for hospitals.

Economic impact of Navenio. Navenio initially deployed their technology into two Manchester hospitals in 2018, and in the 2017–2018 financial year reported income of GBP175,782 and provided employment to 24 people. In 2018–2019 the company reported income of GBP489,326 and employed 58 people. In May 2020, Navenio raised GBP8,850,000 in a Series A funding round led by QBN Capital [ E2]. As of December 2020, Navenio employed 67 staff (headcount: 67; FTEs: 65.2) and 3 consultants, with contracted annual recurring revenue of approximately GBP700,000 [ E4]. The company has deployed its IWS portering technology to 9 UK NHS hospitals across 7 NHS trusts: Tameside and Glossop Integrated Care; Manchester University; North Tees & Hartlepool; Buckinghamshire Healthcare; East Kent Hospitals; Epsom & St Helier; and Royal Cornwall. Over 400 portering staff and over 700 task requesters in these hospitals are using the technology daily [text removed for publication] [ E4].

At the 2020 annual Women in IT Awards, Trigoni was named CTO of the Year, recognising not only the impact of Navenio’s technology services in healthcare to date, as documented below, but also Trigoni’s important impact on diversity through her leadership role in a sector where female representation still stands at only 19% [ E5].

Impacts on organisational efficiency in the NHS. Navenio uses intelligence about a hospital, its workflows, and the tasks it needs to perform in order to determine the optimal way of completing portering or cleaning tasks, and provides real-time tasking and tracking to streamline staff workflows. As evidenced below, the technology allows hospitals to identify and address non-optimised staff workflows, rota management, and resource utilisation. This leads to greater task completion at higher speed, reduced loss of ward staff time, and reduced spend on agency and helpdesk staff. The technology also enables the provision of data on employee activity, task completion, and adherence to the Service-Level Agreements (SLAs) with which hospital trusts are required to comply. These operational insights, provided via an analytics dashboard, allow trusts to make evidence-based improvements in resource use and allocation, and to avoid fines for non-compliance with SLAs.

Workflow efficiency and cost savings. A major inefficiency currently experienced in NHS hospitals is the significant amount of nurses’ time spent on portering tasks, due to difficulties in finding and tasking porters. Ward staff allocate tasks to porters via a helpdesk, with tasking affected by uncertain availability, and often reliant on individuals’ site or staff knowledge. High call volumes can lead to delays and even service failure, and nurses’ time is frequently spent chasing porters or completing portering tasks themselves. This reduces the time nurses spend on core patient care tasks, while hospitals recruit costly agency staff to make up for the shortfall.

Navenio’s IWS has allowed hospital management to correct such inefficiencies and identify others, leading to substantial cost savings. The deployment of the technology across three hospitals in the East Kent trust, for example, led between November 2019 and September 2020 to a 15% reduction in portering calls to the helpdesk [text removed for publication] [ E6.1]. Since its deployment at Tameside and Glossop in December 2017, the technology has removed the need for a helpdesk altogether, releasing capacity of 40 hours per week [ E7.2]. Service delivery for portering has improved dramatically: Tameside and Glossop have recorded a 94% increase in portering tasks completed, while task response times and task completion times are 40% and 12% quicker respectively [ E7]. At the East Kent trust, 18% more tasks are handled, despite reducing helpdesk calls; speed of task assignment has increased by 39%; task response times are 29% faster; 26% more tasks are completed; and 10% fewer tasks are cancelled. As a result, it is estimated that East Kent Trust hospitals have saved “up to 200 hours per week” of ward staff time previously spent “carrying out porter duties or chasing porters”, with this time now freed for patient care [ E6.2, E6.1]. Tameside and Glossop have also recorded 175 hours per week of released ward capacity, “as a function of reducing the need for ward staff to carry out portering tasks” [ E7].

These gains are achieved through the technology’s intelligent workflow optimisation, not by placing additional pressure on personnel: portering staff now walk fewer miles (per hour per resource), and “positive feedback has been received from all users”, including porters [ E6.1, E8]. At Tameside and Glossop, the Logistics Manager reports that “ward staff are able to book tasks quickly, and then be kept up to date on task progress – allowing them to plan their days better, in turn freeing up more of their time for patient care”, while “porters find the solution easy to use – it allows them to focus on the most important tasks and ensures daily tasks are handled with the minimum amount of effort” [ E7].

Evidence-based resource allocation. The detailed analytics enabled by Navenio’s location-tracking technology have informed changes in resource alignment at NHS hospital trusts. At East Kent, the data “informs decision making and has allowed the Facilities Management team to review their staffing demand and understand where improvements need to be made. This allows for optimisation of staff productivity and allows managers to put forward a business case for an increased number of porters if needed” [ E6.1]. The Facilities Manager states: “we have found the data from Navenio to be invaluable, it has allowed us to see a true reflection of what is actually going on ‘on the floor’. Previously our data was just a data dump … but now we get to see where the operational focus should be” [ E6.1]. Fine-grained resource demand analysis, “with the changing hospital flows, down to 15-minute intervals”, has allowed departments across the hospital “to change their way of working to avoid unnecessary delays”, and “overflow one team to another team when there is unexpected capacity or an increase in a specific service demand, i.e. COVID-19” [ E6.1].

Insights based on location tracking can also highlight significant problems that were previously invisible. At one hospital, Navenio showed that a much higher volume of blood samples was being collected than at comparably sized sites [text removed for publication] [ E6.1].

Data-enabled management. The actionable insights provided by the Navenio technology have resulted in changing management practices at hospital trusts, where Navenio data has been incorporated into regular decision-making processes. The East Kent trust uses Navenio’s insights to give “all managers a complete overview of what has happened and where workflow can be improved”, and “monthly review calls are carried out to help managers understand and interpret the data so it can be used to improve processes and be disseminated for presentation to the board” [ E6.1]. The Facilities Manager reports that “Navenio and our teams have…spent time building up our portfolio of data to feed into the SLT and Board papers to ensure that they see the highlights and trends they need to inform decision making” [ E6.1].

Other trusts similarly report wider benefits flowing from data-enabled decision-making. At Tameside and Glossop, “management now have access to highly usable information, providing fascinating insights on operational practices and compliance levels, and that helps us address inefficiencies across the organisation as a whole” [ E7]. In April 2019, the Care Quality Commission drew special attention in a Use of Resources report to these ‘Outstanding Practices’ for improving care at NHS Tameside: “the trust was able to demonstrate a clear theme of using technology and innovation to improve productivity throughout the trust through examples such as… Navenio – a portering app which monitors all portering activity in real time, with a live dashboard to inform team leaders of any demands on the service. The app has allowed the trust to review activity and response, has removed the need for clinical staff to pick up portering duties due to unavailability, has resulted in a reduction in the time spend logging calls and a reduction in overtime by increasing shift efficiency” [ E9]. The impact of Navenio’s technology across the breadth of hospital operations is recognised by new clients, who are investing in order to become “truly data-led organisation[s]” [Managing Director, NTH Solutions (facilities management for North Tees and Hartlepool Hospitals NHS Foundation Trust): E10].

Impacts on cleaning standards and compliance outcomes. The Navenio system configuration takes into account hospital compliance requirements, and analytics are generated that allow hospital management to monitor and improve compliance outcomes. These insights can mitigate significant financial risks for hospital trusts, whose SLAs include provisions for substantial financial penalties correlated with performance targets. Hospitals have reported measurable improvements in compliance outcomes due to the use of Navenio: in August 2019, for example, Queen Elizabeth The Queen Mother Hospital achieved task handling compliance levels for portering tasks of 100% for emergency moves, 97.3% for urgent moves, and 96.2% for routine tasks. Overall, facilities management for the three East Kent hospitals recorded a 12% increase in task compliance in the 10 months following November 2019 [ E6.1]. Tameside recorded SLA compliance of 95% using Navenio [ E7]. At [text removed for publication], potential fines were identified by feeding Navenio’s data insights into the hospital’s PAYMECH compliance system. The hospital was then able to avoid penalties by adapting staffing capacity in response.

In 2020, Navenio won GBP441,616 in UK government funding to develop their applications to support the NHS in dealing with COVID-19 [ E12]. This accelerated the development and launch of Navenio’s cleaning compliance solution, which has already been purchased by 5 hospital customers [ E4]. The tracking and tasking solution improves workflow and resource efficiency to achieve greater task completion and better cleaning standards and compliance. The technology can also be used directly to identify sources of hospital-acquired infections by determining which staff and assets have been in contact with infected patients.

Impacts on staff and patient experience and healthcare delivery. The efficiencies enabled by Navenio’s IWS have benefitted staff user groups across NHS hospitals, with concomitant benefits for healthcare delivery. As part of the deployment process, Navenio gathers requirements from all relevant staff and departments, such that staff feel they are fully “able to input our needs into the system” [Radiology Lead, Epsom & St Helier: E8], and provides training to all users of the technology. Necessary adaptations are made, e.g. for partially sighted users. For portering and cleaning teams, adoption can mean a wholesale shift to a digital working culture. The technology has been embraced because it has shortened journey times, eliminated unnecessary journeys, and made it easier for porters and cleaners to manage their workload (for example, a porter stated: “it knows what stage of the task you are on…it will make things so much easier”) [see staff feedback in E6.1, E8].

The experience of clinical staff, and in turn the quality of patient care, has improved as a direct result. At Tameside “ward staff are able to book tasks quickly, and then be kept up to date on task progress…freeing up more of their time for patient care” [ E7]. At Epsom & St Helier, “we now have visibility of when porters have accepted tasks, an estimate of when they will arrive and their progress. It has really improved the situation for staff and patients alike” [ E8]. Patients experience these benefits in the form of faster response times (at the East Kent trust “response time to patients has been 29% faster, reducing from an average of 14 minutes to 9.9 minutes”), and more time for clinical staff to care for patients, “as they are spending less time doing porter tasks or chasing completion of porter tasks” [ E6.1]. This translates both into better patient care – for example, faster turnaround times for X-rays – and improved patient safety, as in the blood sample case described above. An April 2019 CQC inspection report noted that NHS Tameside had been rated ‘Good’ for medical care service responsiveness in part because “technology had been used to improve productivity and understand service demand and capacity” [ E11].

Our research exposed critical failures in the security of Bluetooth that nullified the confidentiality and authentication properties of the technology. Bringing these facts to light carries direct impacts: in making unknown security risks known, in allowing informed choices to be made by manufacturers and users of Bluetooth, and in directing the development of commercial communication systems. However, the most profound impact of this programme has come from the research team’s coordinated disclosure process, which has repeatedly allowed the most serious risks to be mitigated before they can be abused – thereby preventing their potential harm. As Bluetooth is in use at enormous scale (almost 3,000,000,000 Bluetooth BR/EDR devices were shipped in 2019 alone [ E4.1]), and the exposed vulnerabilities affected all standard-compliant devices, the steps taken to remedy the deficiencies found by the research team have helped protect the security and privacy of a substantial proportion of the world’s population.

The coordinated disclosure of security vulnerabilities is a pillar of beneficial security research. Once discovered, a vulnerability is reported to the affected standards bodies and manufacturers so that they can remedy it before it becomes publicly known. The research team provides details of their discovery and analysis, interacting with industry security specialists as they devise an appropriate countermeasure. One researcher typically acts as the point-of-contact between the research team and the industry parties. Papers describing vulnerabilities are embargoed from public release before a given publication deadline to allow the remediation to take place in secret. During this process, vulnerabilities are assigned a Common Vulnerabilities and Exposures (CVE) Identifier and evaluated for severity using the Common Vulnerability Scoring System (CVSS). This provides a clear and common format for controlled distribution to affected manufacturers, along with a measure of criticality for prioritising fixes. Typically, researchers will have proposed potential mitigations in their papers, which can form the starting point for practical fixes. When successfully executed, the coordinated disclosure process allows critical vulnerabilities to be fixed for the majority of users before the vulnerabilities can be maliciously exploited. This provides an obvious benefit to the users themselves, be they individuals or businesses, and also to the manufacturers who avoid the costs and liability of security breaches resulting from the vulnerability. As such, firms often operate Bug Bounty programmes to reward researchers for their efforts.

The coordinated disclosure process was conducted twice by Kasper Rasmussen’s research team, with contact first made for the KNOB vulnerability described in [ R2 & R4] and the established relationships then used for the BIAS work in [ R3]. The result of [ R2] was initially reported to the Bluetooth Special Interest Group (SIG) and CERT Coordination Center (CERT/CC) in October 2018. This established communication with Intel, who were the responsible partner for the cross-industry Unified Security Incidence Response Process (USIRP) as defined by the International Consortium for Advancement of Cybersecurity on the Internet (ICASI) [ E1]. The research team was in dialogue with Intel from January 2019 onwards regarding action to coordinate mitigation efforts, with Antonioli acting as the point-of-contact via email and telephone. The research paper was embargoed until August 2019 to aid these efforts. The later work in [ R3] was reported in December 2019 through the same channels, with the associated paper embargoed until May 2020. In both instances, the research team provided detailed technical information, analysis, and potential countermeasures as derived from their work.

The main point of contact for the research team in both cases was with the Intel Product Security Incident Response Team (iPSIRT). In addition to chairing the USIRP, Intel also conducted their own vulnerability remediation, as a major manufacturer of Bluetooth devices. As the collaboration progressed, the research team was in direct contact with ICASI as well as through Intel, further working with CERT Coordination Center (CERT/CC) and the Bluetooth Special Interest Group (SIG) directly. CERT/CC is a non-profit body hosted by Carnegie Mellon University and funded by the US Government that coordinates security response events with industry. The Bluetooth SIG is the specification body for Bluetooth industry, representing over 36,000 member companies [ E4.1]. These bodies in turn coordinated a response among their members.

The KNOB attack in [ R2] was evaluated by Intel as “Critical”, with a score of 9.3 (out of 10) [ E1], which resulted in Intel awarding a Bug Bounty Firmware Payout of USD30,000 [ E1], their maximum firmware payout [ E7]. CERT/CC issued CVE Identifier, CVE-2019-9506, and a Vulnerability Note, VU#918987 [ E2]. The attack was also evaluated at an industry-wide level as “High”, with a score of 8.1 under CVSS v3.0 [ E2]. For context, this score is higher than that for the well-known Spectre vulnerability (5.6, “Medium”), Heartbleed bug (7.5, “High”) or WPA2 KRACK attack (6.8, “Medium”). This score therefore clearly indicated that the attack required a priority response. As documented below, industry acted very quickly to provide fixes, interrupting normal development schedules to ensure that timely mitigations were made available.

The subsequent BIAS attack in [ R3] was issued CVE-2020-10135 by CERT/CC, who released Vulnerability Note VU#647177, and evaluated as 5.4 “Medium” under CVSS v3.0 [ E9]. While this score is lower than for the earlier work, it only addresses the direct effects of the attack and does consider that it further enhances the power of the KNOB attack – a fact explicitly stated in the vulnerability note [ E8] and discussed in the Bluetooth SIG response [ E4.4].

Working together with the researchers and CERT/CC, ICASI subsequently led disclosures to the whole ICASI membership [ E1]. They also coordinated with Apple and Lenovo through ICASI Collaborator NDAs. As ICASI noted of the handling of [ R2]: “the goal of this coordination was for CERT/CC and the Bluetooth SIG to notify as many potentially impacted vendors as possible so that they could develop the appropriate fixes, while minimizing the risk that the vulnerability would be disclosed prior to a fix being available” [ E3].

Industry acted quickly to provide fixes during the coordinated response process. Most major platform vendors released patches immediately following the public disclosure: Microsoft for Windows; Apple for macOS, iOS, and watchOS; Google for Android; Cisco for IP phones and Webex; Huawei for Android phones; and BlackBerry for Android-powered devices [ E5]. Patches were similarly made available for popular Linux distributions such as debian, Red Hat and Ubuntu [ E5]. Intel published a white paper and a statement in response to the disclosure of the research findings, recommending that “in all cases, components participating in a secure Bluetooth connection employ the highest level of encryption possible” [ E6].

On 13 August 2019, the Bluetooth SIG announced that it had “updated the Bluetooth Core Specification to recommend a minimum encryption key length of 7 octets for BR/EDR connections” [ E4.2] as recommended in [ R2]. The SIG “strongly recommends that product developers update existing solutions to enforce” this minimum, and “is also broadly communicating details on this vulnerability and its remedy to our member companies and is encouraging them to rapidly integrate any necessary patches” [ E4.3]. The SIG represents over 36,000 member companies [ E4.1]. CERT/CC also issued an advisory referring Bluetooth host and controller suppliers to the updated specification and instructing downstream vendors to refer to their suppliers for updates [ E2]. Both the Bluetooth SIG’s technical update and CERT/CC’s advisory describe the technical contribution of the paper [ R2].

As described in their Security Notice [ E4.4 & 5], the Bluetooth SIG made further fixes to the standard in direct response to the BIAS attack [ R3]: “to remedy this vulnerability, the Bluetooth SIG is updating the Bluetooth Core Specification to clarify when role switches are permitted, to ensure a role switch in the middle of a secure authentication procedure does not affect the procedure, to require authenticating the peer device in legacy authentication, and to recommend checks for encryption-type to avoid a downgrade of secure connections to legacy encryption. These changes will be introduced into a future specification revision.”

Due to the well-executed disclosure process, the final experience for most users was the seamless deployment of software patches to protect them [ E5]. In the case of the KNOB attack [ R2, R4], by the time the vulnerability information became public – and thus at greater risk of malicious exploitation – mitigations were already widespread. Further, there had been sufficient opportunity for industry to analyse the intricate technical details of the research papers and produce accessible guidance that users could readily act upon [ E3, E4.2–5, E6].

As a fundamentally preventative activity, and while parts of the industry response are still ongoing, it is difficult to precisely quantify the investment in remedying the KNOB and BIAS vulnerabilities – or indeed the economic savings that pre-empting any malicious exploitation has yielded. However, for context, the costs associated with the WPA2 KRACK attack (CVSS v.3.0 score of only 6.8, lower than the KNOB attack’s 8.1) have been placed in the tens of millions of dollars [ E9], while the cost of the Heartbleed bug (CVSS v.3.0 score of 7.5) has been estimated in the hundreds of millions of dollars [ E10]. What is certain is that the researchers’ coordinated disclosure allowed remediation to begin before malicious exploitation could take place, and that the corrections to the Bluetooth Core Specification guarantee that this work [ R2– R4] will continue to have an industry-wide impact in the future, protecting many millions of Bluetooth users from malicious attacks and associated harm.