Impact case study database
Improving the quality assurance of large software models
1. Summary of the impact
Collaborative research between Edge Hill University (EHU) and British software consultancy company SimCon Ltd has led to the development of an automated tool for checking and improving the quality of large codes, removing and correcting errors and inconsistencies. The lead developers of the Weather Forecasting Model (WRF), a major atmospheric research weather forecasting model, have deployed the tool in the repository where the model code is made available to the community of WRF developers and users. The tool checks that every change into the code is free from a wide range of errors and inconsistencies before every release. The improvement in the quality of code has also allowed SimCon Ltd, to improve the quality of their existing software products, used by a global client base, and grow their market position. Since 2014 the existing knowledge was integrated into the procedures of development and testing of the WRF code and this work has been extended to other large codes, some of which use the latest Fortran Standards.
2. Underpinning research
The research findings underpinning the impacts described, are the product of research undertaken at Edge Hill University (EHU) with initial work completed by Prof Mark Anderson and continuation led by Prof Ella Pereira. Prof Anderson’s work on the behaviour and structure of the source code using the Weather Research and Forecasting (WRF) model as a case study was submitted to REF 2014. Outputs 1 and 2 comprise the underpinning research in the 2014 case study. The work originally commenced in June 2011 and continues to date.
The WRF model (a next-generation mesoscale numerical weather prediction system) serves the atmospheric research and operational weather forecasting needs of its users. The National Center for Atmospheric Research (NCAR, US), the National Oceanic and Atmospheric Administration (NOAA, US) and the US Air Force Weather Agency, supported by a wider community of developers, have led the WRF development. The model has been designed to support implementation on a broad set of hardware (single processor laptops to high-performance computing (HPC)). Its modelling system is complex and exploits many Fortran capabilities that permit compile-time checking, including modules and interfaces. Anderson’s research was undertaken in collaboration with the University Corporation for Atmospheric Research (UCAR), a consortium of 100+ universities and academic affiliates, and funded by the US Air Force. Anderson designed a suite of tests and used the FPT toolkit developed by SimCon Ltd to explore the extent and nature of inconsistencies within the WRF model. The outcome of this work demonstrated that inconsistencies in output generated by the model could be caused by the absence of a single and uniformly applied set of Fortran code standards, compiler inconsistencies or by building the code on different systems/ hardware (Output 1).
Further tests have been developed to configure the software environment, perform runs on test cases, determine differences between the outputs and develop a software tool to visualise the differences between outputs in the model. Almost 80% of the WRF model was put under scrutiny, including large modules such as chemistry and physics, for possible errors and their effect on the outcome of the model. This research activity (Output 2) presents the classification of the issues that existed across the suite of modules within the WRF model. This work led to the integration of a new automated test procedure which supports testing by UCAR developers into the online repository that is used by all WRF developer and user communities.
Since 2014, further research has been carried out by the EHU team in partnership with SimCon to analyse large code bases including the WRF developed with the latest Fortran standard (2018) and to further enhance error classification. This research has revealed a need for FPT to be updated in order to capture errors in code written in the Fortran 2018 standard. This work has been led by Pereira at EHU and forms the basis of a forthcoming publication, and along with outputs 1 and 2 underpins the impacts described in section 4.
3. References to the research
Work was scrutinised by target audiences in both the engineering and academic communities, published in peer-reviewed international journals. The research was also included in UCAR’s successful funding application to the US Air Force.
[1] Conference Contribution, Journal Article: Collins, J., Farrimond, B., Anderson, M., Owens, D., Bayliss, D. and Gill, D. “Automated Quality Assurance Analysis: WRF – a case study”. In Proceedings of 5th International Conference on Computer Engineering and Technology (ICCET), Vancouver, Canada, 13-14 April 2013 (also Journal of Software, JSW, ISSN 1796-217X). doi:10.4304/jsw.8.9.2177-2184
[2] Journal Article: Collins, J., Farrimond, B., Flower, D., Anderson, M. and Gill, D. “Removal of Numerical Drift from Scientific Models: A case study using WRF”. International Journal of Software Engineering and Applications (IJSEA), 4(2), pp 51-61, March 2013. doi: 10.5121/ijsea.2013.4204
4. Details of the impact
The Edge Hill team formed the initial partnership with SimCon and the University Corporation for Atmospheric Research (UCAR) in 2011 and worked collaboratively to ensure engagement and increase the potential for adoption of the tool. This involved presenting ongoing research work at the workshops and other gatherings organised by UCAR, attended by both developer and user communities as well as providing direct support to SimCon with the tool updates. The initial impact of this collaborative project occurred in 2013 and was reported as a case study in REF2014. Work on this project re-commenced in 2018 and led to the further impacts within the current REF period.
Further work was carried out by the EHU team and SimCon in 2018, using previously designed techniques (output 2), enabling the tool to be extended to the Fortran 2018 standard. Before this work, the tool was only able to detect and fix errors and anomalies in legacy Fortran code. Additionally since 2018 the tool has been updated to be able to detect a wider range of errors. This work has allowed SimCon to further widen their customer base and increase their competitive position within the market. At the same time, EHU and SimCon re-commenced work with UCAR and supported FPT development for WRF regression tests. While in the previous REF period UCAR was able to use FPT in the development of new versions of WRF, work carried out since 2018 led to integration of FPT into the bug-fix branch that is the more frequently generated release. FPT is now integrated in both the development and bug-fixing branches ensuring further stability of the WRF model leading to improved forecast reliability.
Impact 1: The FPT toolkit has been improved/updated resulting in its integration into the WRF build procedure reducing inconsistency and improving code accuracy
The WRF model has more than 48,000 users from 160 different countries and is the most widely used model in research and development for weather forecasting and climate change. It is managed at UCAR (University Corporation for Atmospheric Research) as a collaborative partnership between federally funded US agencies, principally NCAR, the NOAA, the US Air Force Weather Agency, Naval Research Laboratory, the Federal Aviation Authority and universities worldwide. The model is the backbone for a large number of real-time forecasting systems including several national centres, TV stations and is also embedded into a number of proprietary systems supporting the agriculture, transportation and alternative energy sectors. Users range from organisations engaged in climate and weather research to governmental and military organisations who all use the model to inform operational decisions. The accuracy of the model and reduction in errors is therefore critical across a number of industries and applications.
EHU’s collaboration with SimCon Ltd has shaped the way in which the WRF model is developed and built, improving its quality and reliability. This knowledge has also been deployed across other areas, such as those described in impact 2.
The underpinning research that took place at EHU demonstrated that issues exist in the source code within WRF. Moreover, the research work led by Anderson at EHU has demonstrated that programs often behave differently under different compilers or in different computing environments. Those issues can be removed or corrected through an automated approach to code analysis.
EHU, SimCon Ltd and UCAR have collaborated on the integration of FPT with the existing regression test suite of WRF. The WRF branch flow consists of the develop branch, bug-fix release branch and master branch. The FPT position within this flow lies in the bug-fix release branch. For that purpose, SimCon Ltd has developed a new version of FPT designed specifically for the needs of this task. The integration of FPT into the build procedure requires that FPT runs in a variety of different environments. FPT has now been migrated to Mac OS and to ARM processors.
As a result of the collaboration between EHU, SimCon and UCAR the new version of the FPT toolkit has become available to the team inside UCAR who are responsible for the supervision of the code quality and maintenance. This is also available to be used by the wider research community which contributes to WRF. This means that regardless of the platform or compiler used by the developer community they are now able to use the FPT tool to identify and fix inconsistencies in their version of WRF model.
‘The benefits to the WRF community include the reduction/elimination of inconsistences in WRF and thus a reduced exposure to modelling problems. The resultant revision of WRF has already and will continue to provide more robust performance across different architectures and compilers.’ Lead software engineer, UCAR [S1]
The EHU team has demonstrated the value of integrating automated testing utilising relative debugging techniques to assist in the identification of issues in incremental releases of the WRF model. This has resulted in a more consistent output being generated by WRF irrespective of the underlying hardware architecture or compilers used by the developers to implement the model. Moreover, having FPT integrated into the bug-fix release branch of the WRF repository ensures that every update submitted to the WRF repository is checked for inconsistencies and quality, while in the previous usage of FPT by UCAR and developers this was not possible. This saves considerable time and manual effort during regression testing.
This integration ensures that any alteration to the original code will cause no damage and verifies the removal of existing problems, resulting in an accurate output of the model. The latest updates to FPT also resulted in 100% of WRF developer and user communities being able to use the tool for checking and removal of anomalies from the code. These anomalies are related to ever developing source code and not to the underlying climatological model. Hence, removing code related errors ensures accurate implementation of the model. ‘So, not only have errors been removed from the source code, but there is the confidence that we know that a very large array of potential problems is absolutely not a concern’ [S1]. The software development team at UCAR have benefitted from the work with ‘improved run time stability’ of the WRF model and code that is easier to identify errors reducing the release times for new versions. The wider user base of the model benefit through a more consistent model with fewer code errors, and greater accuracy. Given the high importance of the WRF model to the weather forecasting userbase and range of decisions it is used to inform, this significant step change in the ability to update and monitor errors in the model’s code is critical.
Impact 2: Improving the market position of British software company SimCon Ltd, and enhancing software quality
SimCon Ltd is a British software products and consultancy company, specialising in real-time simulation and scientific computing. Its key service areas are Fortran Analysis, Engineering and Migration; key customers include Airbus, Boeing, DENEL OTR, QinetiQ and the US Navy. SimCon has been collaborating with Edge Hill on the WRF software quality assurance project since 2011 as well as being one of the beneficiaries of the research outcomes. The company has deployed its FPT toolkit in their work on the WRF model and in projects for other clients (particularly migration and quality assurance). During the software quality assurance project, the research undertaken by Edge Hill enabled SimCon to make several improvements to this toolkit.
One of the major improvements was the addition of pre-processor support enabling WRF and other large Fortran programmes to be re-engineered, therefore, not only finding errors in code but to fix them, allowing permanent automatic changes in the code base.
The tool is now able to assess inconsistencies in code developed using different underlying software and hardware platforms, and compilers as well as the latest features of the Fortran language. Based on the testing facilities developed by the EHU team, SimCon has also extended the range of errors handled by the tool. Before the work carried out with Edge Hill team the tool was capable of analysing and detecting errors in legacy Fortran and in Fortran up to the standard Fortran 95. Since the partnership with EHU team the tool has been extended to handle most of the Fortran 2018 standard and embedded cpp macros’, Technical director, SimCon Ltd [S2].
Having the tool extended to the latest Fortran standards covering a wider range of errors has enabled SimCon Ltd to acquire a new customer base. The cpp macro handling EHU and SimCon developed for WRF can quickly be adapted for use on other large codes. This means that SimCon are now able to use the enhanced toolkit in both ongoing and new migration and quality assurance projects for clients in the scientific, engineering and other communities where later versions of the Fortran coding standard are in use. This expands not only the range of potential customers but also the potential scale and scope of the analysis meaning SimCon have been able to perform quality assurance and analysis on large scale national infrastructure projects. Most notably these improvements enabled SimCon to move into new industries to complete the re-engineering of a code which controls energy distribution in a large part of central Europe and to support a large-scale software migration project.
As noted by SimCon’s technical director, the upgrades result in cpp macro handling only taking ‘a few days’ to adapt to new large codes [S2]. This considerable time saving has widened the opportunities to expand into further markets.
On top of the direct business benefits, SimCon have been able to pass on the benefits to their client base with improvements in software quality. ‘In addition to SimCon widening its customer base, both SimCon and the FPT user community have gained significant benefit from this collaboration in improvements in the software quality of a wider variety of customer Fortran codes’ [S2].
5. Sources to corroborate the impact
Corroboration is provided by factual statements from SimCon and UCAR, alongside data demonstrating changes made in WRF software releases since 2014.
[S1] Factual Statement Lead Software Engineer, UCAR
[S2] Factual Statement Technical Director, SimCon