Impact case study database
Novel Multigrid Solvers for Significantly Improving the Met Office's Weather and Climate Forecast Capabilities
1. Summary of the impact
Fast numerical weather and climate models are crucial for the success of the Met Office as a leading meteorological organisation. Novel multigrid solvers developed by University of Bath allow more accurate forecasts in a shorter time. This generated impact through improving the Met Office forecast products by implementing multigrid solvers in the Unified Model. Starting from Technological Readiness Level (TRL 1), those solvers have been used operationally (TRL 9) since 9 December 2020. In operational configurations multigrid results in an average reduction of the model runtime by 13%, which translates into significant cost savings of GBP300,000 per annum for the Met Office.
2. Underpinning research
Context
At the heart of the Met Office numerical forecast models a partial differential equation (PDE) for the pressure correction has to be solved in every time step. The PDE solver is one of the computationally most expensive components of the code (typically accounting for 25% - 50% of the runtime), and hence any improvements will translate into time- and cost- savings. Solvers have to be algorithmically optimal, computationally efficient and scale on massively parallel computers. As the model resolution increases, the iterative solver previously used by the Met Office became prohibitively expensive due to the large number of iterations, as confirmed by [Maynard and Walters (2019) Comp. Phys. Comm., 244, pp.69-75]. Furthermore, this solver lacked robustness, which can manifest itself in model crashes and unpredictable runtime.
Figure 1: multigrid hierarchy on the latitude-longitude grid used by the Met Office
Multigrid for the Met Office forecast models. To overcome this issue, a new geometric multigrid solver tailored to thin spherical domains and latitude-longitude grids used by the Met Office was developed by Professor Scheichl and his research team [Buckeridge and Scheichl (2010)] at University of Bath. Subsequently it was adapted for the full model by Dr Mueller at University of Bath. By using a hierarchy of coarser grids (see Figure 1), the multigrid solver can take into account long range correlations and is able to deal with the convergence of gridlines at the poles more effectively. [Buckeridge and Scheichl (2010)] and [Mueller and Scheichl (2014)] confirmed that tensor-product multigrid algorithms [Börm and Hiptmair (2001) Num. Alg., 26(3), pp.219-234] based on horizontal coarsening combined with vertical line relaxation are efficient when solving the atmospheric pressure correction in flat domains: multigrid substantially reduces the number of solver iterations. In addition to the simplified setups explored in [Buckeridge and Scheichl (2010)], the efficiency of the algorithm under real-life conditions was demonstrated by implementing it in the current Unified Model ‘ENDGame’ code. The code is routinely run on thousands of cores of the Met Office supercomputer to produce numerical weather forecasts under tight time constraints and research in [Mueller and Scheichl (2014)], [Mueller, Scheichl and Vainikko (2015)] confirmed the scalability of tensor-product multigrid solvers on massively parallel machines.
In addition to improving the current operational 'ENDGame' dynamical core (which has been using multigrid since December 2020), the Met Office is implementing a next-generation forecasting code. This new model (codenamed ‘LFRic’) is based on advanced discretisations, which require a complete redesign of the solver technology. Work carried out by Professor Scheichl and Dr Mueller during the NERC funded GungHo project (2011 – 2015) has been crucial to implement massively parallel multigrid solvers in this context. After carrying out more realistic simulations on triangular grids in [Dedner, Mueller and Scheichl (2016)], a prototype mixed finite element multigrid solver based on the tensor-product idea and velocity mass-lumping was developed [Mitchell and Mueller (2016)] and integrated into the LFRic code base [Maynard, Melvin and Mueller (2020)]. As shown there, the new multigrid solver in LFRic is around 2 times faster than alternative methods and improves parallel scalability of the model.
3. References to the research
Buckeridge, S & Scheichl, R 2010, 'Parallel geometric multigrid for global weather prediction', Numerical Linear Algebra with Applications, vol. 17, no. 2-3, pp. 325-342. https://doi.org/10.1002/nla.699
Mueller, EH & Scheichl, R 2014, 'Massively parallel solvers for elliptic partial differential equations in numerical weather and climate prediction: scalability of elliptic solvers in NWP', Quarterly Journal of the Royal Meteorological Society, vol. 140, no. 685 , pp. 2608-2624. https://doi.org/10.1002/qj.2327
Mueller, E, Scheichl, R & Vainikko, E 2015, 'Petascale solvers for anisotropic PDEs in atmospheric modelling on GPU clusters', Parallel Computing, vol. 50, pp. 53-69. https://doi.org/10.1016/j.parco.2015.10.007
Dedner, A, Mueller, E & Scheichl, R 2016, 'Efficient multigrid preconditioners for atmospheric flow simulations at high aspect ratio', International Journal for Numerical Methods in Fluids, vol. 80, no. 1, pp. 76-102. https://doi.org/10.1002/fld.4072
Mitchell, L & Mueller, E 2016, 'High level implementation of geometric multigrid solvers for finite element problems: applications in atmospheric modelling', Journal of Computational Physics, vol. 327, pp. 1-18. https://doi.org/10.1016/j.jcp.2016.09.037
Maynard, C, Melvin, T & Müller, E 2020, 'Multigrid preconditioners for the mixed finite element dynamical core of the LFRic atmospheric model', Quarterly Journal of the Royal Meteorological Society, vol. 146, no. 733, pp. 3917-3936. https://doi.org/10.1002/qj.3880
4. Details of the impact
Nature and extent of the impact
The key impact consists in improvements to the Met Office numerical forecast products resulting in code which has been used operationally since 9 December 2020. A recent Met Office news release states [E]: “The collaborative efforts of the Met Office and […] the University of Bath have resulted in the implementation of a dramatically faster pressure correction solver in the Met Office Unified Model, which has a discernible impact on the total model runtime. The new multigrid solver will allow higher resolution forecasts to be run in the future and is already allowing better utilization of supercomputer resources”. This translates into improved weather and climate forecasts; multigrid solvers also increase the robustness of the model.
The Head of Dynamics Research at the Met Office states [A] “By convincing the Met Office to adopt [multigrid] both for its current and next-generation models, the research carried out by the University of Bath has directly impacted on important decisions [at the Met Office]”. Research at University of Bath led to significant changes to computer codes which are crucial for the Met Office. According to a Principal Fellow in Supercomputing and Manager of the Met Office HPC Optimisation team [B], “The overall investment of around 0.8 person-years (approximately £100k) of senior staff into this project demonstrates the importance the Met Office is placing on the new multigrid solver technology”.
Maturity of Met Office code is measured in terms of Technological Readiness Levels (TRLs) (on a scale from 1 [lowest] to 9 [highest]). Prior to the involvement of Bath the maturity of multigrid could be classed as TRL 1. As explained in [B] “ [multigrid] has achieved TRL 9 (“Actual system "mission proven" through successful mission operations”) when it started to be used in operations from 9 December 2020 onwards”.
Details of the beneficiaries
The direct beneficiary of the research is the Met Office. As an internationally leading meteorological organisation the Met Office provides weather- and climate- forecast for the general public, commercial customers and policy makers. The Unified Model code is used by a range of international partners and research organisations, such as the National Institute of Water & Atmospheric Research (NIWA) in New Zealand [F].
How the research underpinned the impact
The Met Office has limited resources to introduce more radical changes to its numerical algorithms and had therefore not explored alternative solver technologies. The research carried out in Bath assured the Met Office that a significant reduction of the total model runtime can be achieved with multigrid.
Process through which the research led to impact
Research of Dr Mueller and Professor Scheichl was carried out as part of the “GungHo” project and convinced the Met Office to invest in multigrid solver technology. Dr Mueller established a close working relationship with senior atmospheric scientists (Prof Nigel Wood, Dr Ben Shipway, Dr Thomas Melvin) and computational specialists (Dr Christopher Maynard, Dr Paul Selwood, Dr Andy Malcolm) through frequent mutual visits (between 2016 and 2020 Mueller spent 3 months at the Met Office as a visiting scientist) and joint meetings [C]. The multigrid solver was documented [D] and integrated into the ENDGame model by Mueller and Dr Andy Malcolm (between 2012 and 2019). Following rigorous internal code review and evaluation in operational configurations (between autumn 2019 and summer 2020), the Met Office fully incorporated multigrid into the operational forecast model on 9 December 2020 [B].
Evidence of the extent of the impact
Impact is evidenced by the reduction in overall model runtime achieved by the multigrid solver as quantified in [B & E]. The currently operational configuration of the model corresponds to a computational grid with 2560 x 1920 x 70 = 3.4·108 points and a spatial resolution of around 10km in mid-latitudes. For a 75 hour deterministic global forecast the model runtime is reduced by almost 10%. As further explained in [B], “Similar performance gains have been shown for other configurations; in particular for ensemble runs on smaller core counts the savings are higher, with around 15%”. The news release [E] clarifies that savings in overall runtime can be clearly attributed to improved solver performance: “[multigrid] requires substantially less iterations and is more efficient overall: […] one linear solve with the new multigrid solver takes approximately half the time that was required with the previous […] method”.
Figure 2: Reduction of time per solve for different processor counts (see [source E])
To demonstrate the significance of the achieved savings in model runtime, consider the resulting monetary savings through better supercomputer utilisation. The current yearly cost of all global operational runs is GBP3,700,000. As discussed in the letter [A] “Operational tests demonstrate an average reduction of 13% in total runtime (corresponding to significant cost savings of £300k per annum) [...] performance improvements of this order of magnitude are rare and typically only achieved once every 5 years”. The wide reach of the impact is demonstrated by the fact that [A & F], “the new multigrid code is currently used by one of [the Met Office’s] international partners, the NIWA research institute in New Zealand”.
Dates when impact occurred
Tests of the ENDGame multigrid solver in operational configurations were carried out between autumn 2019 and spring 2020; inclusion in pre-approval package trials from March 2020 onwards corresponds to TRL 7. By inclusion in the recent “Parallel Suite”, the solver reached TRL 8 in June 2020. The multigrid code has been used operationally, TRL 9, since 9 December 2020.
5. Sources to corroborate the impact
Letter from the Head of Dynamics Research group at the Met Office, 30 November 2020.
Letter from a Principal Fellow in Supercomputing and Head of High Performance Computing at the Met Office, 9 December 2020. Contains details on recent operational performance tests of the multigrid solver in ENDGame.
List of joint meetings and visiting scientist appointments of Dr. Eike Mueller at the Met Office (including details of external investment), 2016 – 2020.
Technical documentation of multigrid solver in Met Office ENDGame model (Met Office-internal user documentation document, 2019).
Link to Met Office Research News article, August 2020. https://www.metoffice.gov.uk/research/news/2020/multigrid-solver
Email from a Staff Scientist at National Institute of Water & Atmospheric Research Ltd (NIWA), New Zealand, 23 February 2020.
Additional contextual information
Grant funding
| Grant number | Value of grant |
|---|---|
| NE/J005576/1 | £180,063 |
| NE/K006754/1 | £180,532 |