Impact case study database

Model-based software engineering is the practice of promoting domain-specific models to first-class citizens of the software engineering process, using such models to analyse, simulate and reason about properties of the system under development, and eventually automatically generate from them (parts of) its implementation. Epsilon is an open-source platform [5.1] that offers an integrated set of state-of-the-art languages and tools for automating many of these tasks.

Epsilon has been adopted across the systems and software engineering industry by a wide range of organisations such as Rolls-Royce, BAE Systems, Codebots, IBM, Blackbelt, DevBoost, NASA, Thales, Siemens, Raytheon, ATOS, Ergon, Talend, JC Chapman, Corpus Solutions and the Develop Group [5.8]. Epsilon is also used as a major component in several open-source projects, more than 40 of which are listed (with supporting evidence) under [5.9]. Components of Epsilon have also been used in the context of software modelling and model-based software engineering teaching in at least 15 higher education institutions worldwide [5.5].

As of December 2020, Epsilon’s user forum contains more than 9,700 questions and answers posted by users and developers of the project (more than 4,700 of which have been posted to the forum since August 2013), and is the 21^st largest forum (top 5%) among the 395 forums hosted by the Eclipse Foundation. The high-quality support that Epsilon’s development team provides to users through the project’s forum is recognised as an important contributor to Epsilon’s continued adoption and impact [5.6] (quotes #59 - #64).

Below are some examples of industrial uses of Epsilon. Quotes for the first three industrial uses of Epsilon (Codebots, Blackbelt and Leonardo) have been extracted from a report [5.6] commissioned by the University of York. The report was compiled by the independent consulting company Fresh Perspectiv after interviewing several industrial users of Epsilon. The report also contains supporting quotes from organisations such as IBM and Altran and from senior management at the Eclipse Foundation.

Examples of Industrial Value

“Epsilon has a positive impact on our revenue. The core of Codebots projects use Epsilon software. Through our products it also allows our clients to get their products quicker to market.” [5.6]

Epsilon is a core underpinning technology [5.6] (quote #22) of the commercial Codebots low-code software development platform ( http://codebots.com/), developed by the Codebots company in Brisbane, Australia (Codebots also has offices in London and Singapore). According to the CEO of Codebots, Epsilon reduces the technical debt and costs for the company’s customers [5.6] (quote #35) and allows them to ship their products to the market faster. The comprehensive support that Epsilon provides for domain-specific modelling, model validation and code generation also reduces the training effort for software developers in the company: “Epsilon pulls all the tools into one family. It simplifies the number of software tools I need to train my team on. Epsilon has the full gamut for us.” [5.6] (quote #40). Compared to competing offerings, Epsilon is seen to provide better support for type checking and functional decomposition, leading to “improved quality, reduced errors, and improved maintainability.” [5.6] (quote #54).

“Without Epsilon, this tool maybe wouldn’t exist” [5.6]

Epsilon is also a core technology of the commercial Judo enterprise software development platform ( https://www.judo.codes), developed by Blackbelt Technology in Hungary. According to Blackbelt’s lead architect, Epsilon is such a fundamental technology of Judo that without it the platform might not exist [5.6] (quote #19). Epsilon’s integrated nature has delivered substantial maintainability benefits for the company [5.6] (quote #18), and its openness, ease of use and versatility have also been highlighted [5.6] (quotes #43 and #65) as significant reasons for the adoption of Epsilon in the context of the Judo platform.

“[With Epsilon] we can produce more and better products in a shorter time.” [5.6]

At Leonardo UK ( https://uk.leonardocompany.com), Epsilon has been used since 2016 to produce software for aircraft-mounted systems which process positioning and sensor data [5.6] (quote #7). The solution developed using Epsilon “has proven much more robust and scalable than previous ones, has made it easier to check and compare intermediate models, and has allowed the correctness of the specification transcript to be maintained even as source and destination formats evolve” [5.10]. Epsilon increases productivity and reduces the cost of producing avionics products [5.6] (quote #25), by reducing the amount of code that developers need to write manually [5.6] (quote #52) thus increasing profit margin [5.6] (quote #17); the company estimates savings of GBP200,000 per annum as a result of this application of Epsilon [5.10]. The use of Epsilon is also seen as an improvement to the software engineers’ quality of life as it eliminates repetitive and dull work and allows engineers to spend more time on solving interesting problems [5.6] (quote #68).

At JC Chapman ( https://www.jcchapman.com), multiple components of Epsilon were used in 2015 to develop a platform for rigorous specification and automated translation of constrained natural language rules in the financial regulation domain (predominantly rules published by the Prudential Regulatory Authority of the Bank of England). This work was carried out in the context of a Knowledge Transfer Partnership (KTP) co-funded by the company and Innovate UK. The Epsilon-based solution developed in the KTP was able to capture 97% of the regulatory rules it was evaluated against, with ~50% less code on average compared to SQL [5.7], leading to significant productivity and maintainability benefits for customers of the company.

At BAE Systems ( https://www.baesystems.com), Epsilon has been used since August 2013 (started from 2010) in the context of a model-based product for automated consistency checking of Tactical Data Link (TDL) specifications, that are used to specify the communication capabilities of defence platforms. The process of migrating from a document-based to a model-based development process using Epsilon, and the benefits realised from this transition were presented in a joint paper between academics at York and engineers at BAE Systems [5.2]. According to the engineers at BAE Systems [5.2] “the Epsilon language components serve to provide a parsimonious implementation of a complex domain” with the company estimating productivity benefits “of an order of magnitude, given similar skilled levels of practitioners”, which also extend into the maintenance phase of the product.

At Rolls-Royce ( https://www.rolls-royce.com), Epsilon has been used extensively since 2017 to support model-based engineering tasks such as model validation and model-to-text transformation in the context of an in-house domain-specific model-based engineering solution, CaMCOA Studio [5.3]. CaMCOA Studio will be used to architect and integrate the software for all future Rolls Royce engine control and monitoring systems [5.4]. In the absence of Epsilon, Rolls-Royce would have needed to use a mix of independently-developed and syntactically inconsistent languages, which would hinder reuse and increase development and maintenance costs. Epsilon is seen as an integral part of Rolls-Royce’s modelling toolkit and currently around 20 engineers in the company are using the tools and languages it provides. Professor Kolovos is the PI in an ongoing 3-year KTP with Rolls-Royce, co-funded by InnovateUK, to support further embedding of Epsilon technologies in CaMCOA Studio. According to the company, “… during the development of CaMCOA Studio, several process improvement events have shown a 96% reduction in the number of man-hours compared to the process CaMCOA Studio replaces. This could result in an initial cost avoidance figure of approximately GBP2,500,000 per annum when deployed” [5.4].

In the first part of this section, we provide necessary details of the Route to Impact, describing how the underpinning research was exploited in the development of an innovative Worst-Case Execution time (WCET) analysis technology now called “RapiTime”, and transferred to industry via the formation of a successful spin-out company, Rapita Systems Ltd. (https://www.rapitasystems.com/\), hence providing the evidential link between the underpinning research and the impact in the relevant REF period (i.e. since Aug 2013). The Impact during the REF period is then detailed in the second part of this section.

Route to impact:

During the EU FP5 NextTTA project (1st Jan 2002 to 31st Jan 2004) members of the RTSRG group, Guillem Bernat, Antoine Colin, Stefan Petters, and Alan Burns, introduced the underpinning research on hybrid measurement-based WCET analysis. This approach combined both measurement and static analysis techniques to accurately estimate the WCET of complex software components running on advanced microprocessors. As part of the project, they also developed a prototype WCET analysis tool called pWCET [3.5]. This tool was evaluated on an Audi drive-by-wire system. Audi was an industrial partner in the NextTTA project. Audi’s expression of interest in pWCET and its capabilities led directly to the formation of a spin-out company to transfer this technology into industry.

In 2004, members of the RTSRG; Guillem Bernat, Ian Broster, Antoine Colin, and Robert Davis, and the University of York founded a spin-out company called Rapita Systems Ltd. (www.rapitasystems.com\) to commercialise the technology and bring it to market. All rights to the technology and prototype tools were transferred to the company by the University of York which became a shareholder in the company. In 2005, Rapita Systems received GBP200,000 of funding from Viking Investments Ltd. and an associated group of Business Angels. Following the initial technology transfer, the pWCET prototype was re-implemented as a commercial quality tool and re-branded as “RapiTime”. RapiTime was then extended to support analysis of systems written in C++ as well as the C, and Ada programming languages.

The low-overhead tracing, source code instrumentation, and parsing technology developed as part of RapiTime were used as the basis for two complementary products: (i) a code coverage tool (RapiCover) and (ii) an on-target test solution (RapiTest) that automates the creation and execution of unit, integration and system tests. Together, RapiTime, RapiCover, and RapiTest form part of the Rapita Verification Suite (RVS).

From 2008 to 2014, Rapita’s revenues increased from [text removed for publication], and as of Aug 2013, Rapita employed [text removed for publication] people in its offices in York and Cambridge.

Impact during the REF period:

Since 2014, Rapita has focused on sales of its RVS product, including RapiTime, RapiCover, and RapiTest, to customers ( https://www.rapitasystems.com/about/customers) in the aerospace and automotive markets.

RapiTime ( https://www.rapitasystems.com/products/rapitime) enables companies in the aerospace and automotive electronics industries to reduce the time and cost required to obtain confidence in the timing correctness of the systems they develop. It provides a cost-effective means of targeting software optimisation, such that new functionality can be added to existing systems without the need for expensive hardware upgrades. Further, RapiTime is portable across a wide range of different microprocessors, meaning that companies can use the same technology across multiple projects without the need for re-training or adoption of multiple solutions.

RapiCover ( https://www.rapitasystems.com/products/rapicover) reduces the time and effort required for companies to obtain structural code coverage data for their critical embedded software. RapiCover achieves this by integrating with existing software build systems, and utilising extremely low overhead, on-target tracing technology. This reduces the number of builds needed to collect coverage data, eliminating unnecessary testing time and effort.

RapiTest ( https://www.rapitasystems.com/products/rapitest) drives the inefficiencies out of functional testing in critical software verification projects, by automating the creation and execution of unit, integration and system tests. It reduces the cost of software verification, particularly in the avionics industry.

RapiTime technology has been deployed on, and in continuous use (within the REF period) on a number of major long-term aerospace projects world-wide, examples include:

• [text removed for publication]: Flight Control Computer (FCC) and the Cockpit Displays for the [text removed for publication] (RapiTime in continuous use since Aug 2013, started in 2006).

• [text removed for publication]: FADEC (Full Authority Digital Engine Control) for the [text removed for publication] (RapiTime in continuous use since Aug 2013, started in 2009).

• [text removed for publication]: ARBS (Aerial Refueling Boom System) for the [text removed for publication] (RapiTime in continuous use since Aug 2013, started in 2011).

• [text removed for publication]: Flight Control System for the [text removed for publication] (RapiTime in continuous use since Aug 2013, started in 2010).

• [text removed for publication] a European Space Agency experimental [text removed for publication] (RapiTime in continuous use since Aug 2013, started in 2012).

• [text removed for publication]: Used in a proof-of-concept relating to new processes for the development of Flight Control Systems. (RapiTime in continuous use since Aug 2013, started in 2010).

• [text removed for publication]: Evaluation and tool qualification for use on the [text removed for publication]. (RapiTime in continuous use since Aug 2013, started in 2008).

• [text removed for publication]: Development of AUTOSAR software modules. (RapiTime in continuous use since Aug 2013, started in 2009).

• [text removed for publication]: Flight management system (since 2015), [text removed for publication] Systems (since 2015), [text removed for publication] (since 2015), [text removed for publication] (since 2014), more than 10 projects, certifications and new developments during 2014-2020.

• [text removed for publication]: all new [text removed for publication], using the [text removed for publication] processor, analysed by RapiTime. (Since 2014).

• [text removed for publication]: started use in 2015, developed onto new multicore platform since 2019.

• [text removed for publication]: started use in 2017, broader adoption in 2019, new multicore RapiTime use since 2019.

• [text removed for publication], started RapiTime use in 2019 for new project.

• [text removed for publication], started RapiTime use in 2018.

• [text removed for publication], RapiTime and multicore analysis since 2019.

• Multiple companies in [text removed for publication] adoption from 2016 to 2020, various projects. [text removed for publication].

• Projects have also started with [text removed for publication] in 2017, [text removed for publication] in 2019, and [text removed for publication] in 2019. [text removed for publication].

The majority of Rapita’s revenues come from the Rapita Verification Suite (RVS) products and services based on the RapiTime technology.

As an exemplar, [text removed for publication] has been a major user of the Rapita RVS products, based on the RapiTime technology since 2014. Today, they are a key customer for Rapita, using several RVS tools including both RapiCover and RapiTest.

The [text removed for publication] work started in 2013, to see if the low overheads of RapiTime could be used to monitor an unusual and esoteric system. The trial was successful, and the company adopted a few licences for the verification of an update to a key [text removed for publication] project. As the technology proved itself and confidence in the tools increased, additional [text removed for publication] development projects, primarily cockpit and controls systems, started to use RVS. From 2017 onwards, these projects have been through multiple certifications with the FAA and other bodies, using the certification kits also provided by Rapita. Today, the [text removed for publication] units tested by RVS are flying in at least four commercial [text removed for publication] aircraft.

Without RVS, the timing measurement, code coverage analysis and testing of the safety critical software would be done either manually or with other less powerful tools. The low-overhead of the original RapiTime technology has been successfully applied to enable efficient measurement of structural code coverage of systems at [text removed for publication]. The benefit is that more of the software can be tested at once, so that the time and cost of testing is lower. In a key avionics control systems upgrade, before RVS 15 test runs were required, which was reduced to only 2 using RVS, meaning that the testing costs 13% of the original. Rapita received the following feedback from a senior engineer at [text removed for publication] in 2019: “This is an excellent tool and everyone on the project likes what it can do”.

In 2016, Rapita was acquired by Danlaw Inc to support its growth and expansion into the automotive and aerospace industries. In the 2018-19 financial year, Rapita’s annual revenues exceeded [text removed for publication] (up from [text removed for publication] for the financial year prior to the start of the REF period). Further the number of employees has increased from a total of [text removed for publication] as of August 2013, to over [text removed for publication] in York, and [text removed for publication] in the USA (Rapita Systems Inc.) as of April 2020. The success and indeed the existence of the company is a consequence of the underpinning research as described in the narrative.The Department of Computer Science at the University of York continues to have strong links with Rapita, through joint work on UK Research and Innovation projects such as SECT-AIR and HICLASS.

All of the facts presented above about the customers, projects, revenues, and headcount of Rapita Systems Ltd., are confirmed and corroborated in [5.1].

Sectoral Impact

1) Automotive

York’s research has had an impact on automotive standards [5.1] and on major automotive suppliers JLR (an Original Equipment Manufacturer (OEM)), Bosch (a tier 1 supplier) and HORIBA-MIRA (an independent assessment house) [5.2]. York (Kelly) was invited onto the British Standards Institution (BSI) committee developing the automotive standard ISO 26262 specifically to draft the Part 10 material on SC development [5.1B]. ISO 26262 is the international standard for programmable electronics and software used by all automotive manufacturers. York and JLR were the founding members of the Motor Industry Software Reliability Association (MISRA) SC Working Group. York’s work, with the other working group partners [3.3], has led to our recognition as a significant contributor to the resulting MISRA SC Guidance [5.1A, 5.2D]. York has also contributed to the work on “Safety of the Intended Function” which addresses some of the issues arising from autonomy [5.1C, 5.2D]. The global automotive market is circa USD2,000,000,000,000 with the UK alone having a turnover of GBP82,000,000,000 and employing 168,000 people directly. ISO 26262 and the SC guidance apply mainly to the electronic elements of vehicles; these represent more than one third of the vehicle value. As an example, the SC work now used by Bosch “established release conditions for such systems … spread across worldwide locations” [5.2A]. Ongoing work is also influencing German standards [5.2A], British and International standards and guidelines [5.2B]. The work has had impact beyond York’s direct collaborators [5.3]; for example, the German Pegasus project has adopted York’s SC approach for Safety Argumentation [5.3A] and Uber uses GSN for their autonomous vehicle SC [5.3B].

2) Defence and Aerospace

York’s work has influenced UK Defence Standards [5.4] and the UK Defence and Aerospace sector [5.5]. York’s 4+1 principles [3.1] are the basis for Defence Standards 00-056 and 00-055 [5.4A, 5.4B]; York (McDermid) was the lead author on 00-056 [5.4A] and a contributor to 00-055 [5.4B]. York’s research on risk, confidence, and compliance arguments directly shaped the UK Military Aviation Authority’s (MAA) Manual of Air System Safety Cases [5.4C] to support MAA Regulatory Article RA1205. The Defence Science and Technology Laboratory (Dstl) confirm the scale of impact of this work on projects of value circa GBP20,000,000,000 per annum [5.5C] and attests to its wider impact: “I doubt that there is a safety case worldwide that has not been influenced by this work” [5.5C]. Also, referring to the AAIP, the international impact is clear: “this initiative provided me support at a NATO research meeting on managing and assuring autonomy” [5.5C]. The industrial impact is reflected in [5.5A, 5.5B] (including influences on software development beyond the work referenced above [3.1-3.6]). For example, [5.5B] identifies influences on a range of certification problems including data-intensive systems (see also [5.6A]); it also identifies two programmes “over which York’s research has provided influence and direction” with a value of over GBP2,000,000,000.

3) Other/Non-Domain-Specific

York has played a leading role in running the Safety Critical Systems Club (SCSC) a UK-based, but international, professional community. The SCSC has produced a range of guidelines which are used widely in the UK and internationally [5.6]. For example, the Data Safety Guidance [5.6A] addresses the problems of assuring safety of data intensive systems, adopting the ‘4+1’ principles (albeit citing another York paper, rather than [3.1]); there is a similar influence on the Service Assurance Guidance [5.6B]. York (Alexander) led the development of work on autonomous systems [5.6C] which builds on the work of the AAIP and is already beginning to be used.

The Global Mining Guidelines Group (GMG) is a worldwide consortium of all the major players in the mining and quarrying industry, including manufacturers, e.g., Caterpillar, operators, e.g., Rio Tinto, and government agencies, e.g., the Government of Alberta. The sector increasingly uses autonomy and is developing guidelines on the design, assurance and use of autonomous systems in mines and quarries (there are significant safety risks, with individual machines carrying circa 500t of material). New guidelines have been developed with input from York (McDermid) on safety of autonomous systems [5.7], building on the principles developed by the AAIP and “not only helped GMG complete and publish the guideline, but it also helped the mining industry align on the topic” [5.8B] since the presence of operators, manufacturers and regulators in the GMG inevitably brings tensions. Further, although recent, the “guideline is being used in a college curriculum and in an application for regulatory approval” [5.8B].

Impact on Industry 4.0 by York (through the AAIP) is further illustrated through work in support of the Global Manufacturing and Industrialisation Summit in the area of autonomy, where “the University of York will continue to bring a unique and impactful understanding and approach to the challenges we face in assuring the safety of complex manufacturing systems in the future” [5.8C].

In healthcare the CONSORT-AI (Consolidated Standards of Reporting Trials–Artificial Intelligence) Guidelines [5.9] provide “a new reporting guideline for clinical trials evaluating interventions with an AI component” which directly references our work on accountability and safety [3.6].

The impact of the HISE group’s work internationally (through one of the Fraunhofer Institutes) can also be seen [5.8A]; for example, this shows “GSN … implemented into Fraunhofer’s safety engineering tool (safeTbox) … successfully applied assurance cases” and cites influences on other standards in Germany and internationally, e.g. the Structured Assurance Case Meta-Model (SACM), and highlights the influence of the AAIP on projects in Germany.

Pathways to Impact

A) Direct Engagement with solution developers and regulators

HISE has worked with a wide range of developers of safety-critical systems, e.g., BAE Systems including funding for PhD students [5.5B], Bosch (through the AAIP) [5.2A], and JLR including EngD projects [5.2C]. Work with regulators includes the MAA [5.4C] and ongoing work through the AAIP embraces the Health and Safety Executive (HSE), and the Medicines and Healthcare products Regulatory Agency (MHRA) and the Care Quality Commission (CQC) in healthcare.

B) Standards and Guidelines

The references (section 5) contain 12 standards or guidelines which build on York’s work; the letters cite a further 5; York’s research is (and has been) influential in a number of standards activities, e.g. ISO 21448 and ISO/TR 4804 [5.2A], VDE-AR-E 2842-61, ISO/IEC AWI TR 5469, and the Structured Assurance Case Metamodel (SACM) [5.8A].

C) Collaborative Research

In the period HISE has been involved in 11 major projects of value circa GBP24,200,000 (spend in the period of circa GBP17,200,000) with 27 industrial partners. This has led to direct impact in 8 countries. Ongoing research projects such as the ICON project with two Fraunhofers, Fraunhofer IESE (Institute for Experimental Software Engineering) and Fraunhofer IKS (Fraunhofer Institute for Cognitive Systems), [5.8A] is building on the work on dynamic risk [3.5].

D) Education and Training

York has taught an MSc programme in Safety Critical Systems Engineering (SCSE) since 1995 and has also delivered continuing professional development (CPD) courses direct to industry; the teaching is research-led. The York SCSE CPD team has taught 666 individuals from 111 organisations in 12 countries over the REF period; in addition, there has been in-house training for specific organisations, e.g. to 300 staff at NHS Digital [5.10A] and 100 at JLR [5.2C]. The SCSE MSc is increasingly introducing material on autonomy, based on the work of the AAIP, and specialist CPD courses are also being presented on the challenges of safety engineering for autonomy. Whilst only recently started, this material has already been presented to more than 70 engineers, from 45 organisations and 3 countries, and to 80 within the NHS [5.10A].