Impact case study database

Environmental science is heavily dependent on big data and has always been dependent on bleeding edge data handling technologies. The research described here has directly enabled science outcomes which would have otherwise been difficult, or even impossible, to achieve given data volumes and disparate communities. These outcomes exploited three separate activities: (i) the use of the JASMIN data commons, (ii) the use of the Earth System Grid Federation, and (iii) the re-use of metadata techniques and tools by third parties; all of which arose from the research summarised here.

JASMIN data commons usage: JASMIN is used by a range of environmental scientists who wish to either share data, or directly exploit the massive volume of data held in the archives of CEDA. CEDA itself is a key tenant of the JASMIN system, alongside a number of other organisations and individuals. The architecture and GBP20m implementation of the JASMIN was a primary research outcome – the operational implementation is now supported by annual budget in excess of GBP 1m via contributions from the National Centre for Atmospheric Science (NCAS), National Centre for Earth Observation (NCEO), the Natural Environment Research Council (NERC), and the UK Space Agency.

The JASMIN data commons depend on the data gravity asserted by the petascale CEDA archive: users are incentivised to bring their data alongside the archive and bring other users (and their data) for collaborations. In excess of 1,500 users exploit JASMIN directly, with approximately 25,000 additional users exploiting CEDA services hosted on JASMIN (Statistics, [S5.1]). The 1,500 direct users are organised into more than 250 distinct groups (tenancies) sharing data and or compute resources for data analysis – users can, and do, belong to multiple groups. JASMIN is now an integral part of UK and global science, yielding a wide variety of outcomes. Representative key JASMIN usage examples are listed below (but see also [S5.2]):

1. The ESA climate change portal ( http://cci.esa.int/data) is deployed in the JASMIN cloud, making use of metadata and data distribution technologies which have evolved from the original DataGrid work [R3.7].

2. JASMIN is providing the underpinning services for UKSA contingency planning for retaining access to EC Copernicus data post-Brexit and for the commercial activities of the Science and Technology Facilities Council’s RALSpace and others [S5.3].

3. The production of unique rainfall estimates and insurance products for over three million African farmers are now predicated on climate services deployed on JASMIN [S5.4].

4. The dominant mode of analysis for UK CMIP6 work feeding into the sixth assessment report of the IPCC is the use of JASMIN, [S5.3] and [S5.5].

5. The most recent UK State of Nature Report, a key government indicator for guiding environmental policy, covering how biodiversity has changed from 1970 to 2015 analysed 12,000 species using over 34 million records on JASMIN [S5.6].

All these outcomes were predicated on the co-location of curated data, user data, and specialised computing which arose from the University of Reading research into ‘disparate communities with common data’.

Earth System Grid Federation (ESGF) and policy change: The ESGF provides the mechanisms for global management and distribution of data products produced by environmental groups worldwide – the ‘common community distributed data’ problem. In particular, it was designed to support the fifth and sixth global coupled model intercomparison projects – CMIP5 and CMIP6 – which themselves were timed to support the fifth and sixth assessment reports of the IPCC. The IPCC is a UN entity created to provide policymakers with regular scientific assessments on climate change, its implications and potential future risks, as well as to put forward adaptation and mitigation options. Those responsible for the fourth assessment report of the IPCC shared in the 2007 Nobel Peace Prize.

As of July 2020, the ESGF hosts 28 different projects with 21 petabytes of data. These are from 7.3 million datasets, distributed across 15 data nodes, which provide data downloads to 140 countries [S5.7]. The UK ESGF data nodes are deployed by CEDA and hosted on JASMIN, one cataloguing UK data for third-party download, and one hosting third-party data which has been replicated to JASMIN to avoid multiple petascale downloads to the UK. The research described here was integral to the initial architecture and deployment in CMIP5 [S5.3].

Much of the ESGF data is simulation data, used directly by academics, climate change consultants, and indirectly by those making commercial and policy decisions. All can find it difficult to understand the difference between different simulations. The Reading research on simulation provenance [R3.4] provided technologies to support the creation, extraction, and comparison of simulation documentation directly addressing this issue. As a consequence, the Reading researchers developed new systems and all modelling groups have been mandated [R3.2] to use them to construct simulation documentation for the sixth global model intercomparison project (CMIP6). (The mandate is described in [R3.3] and further evidenced in [S5.5]).

Reuse: Two examples are presented: (1) how the climate forecast (CF) metadata conventions [R3.1] are integral to weather forecasting and earth observation [S5.8]; and (2) how the JASMIN concept was copied and implemented by the Met Office to provide them with internal capability (SPICE) which matched the JASMIN provision to the wider UK community [S5.9].

The importance of the CF conventions to applications in weather and climate is well known and accepted, but prior to the advent of Reading’s data model, they were not very interoperable with other systems. With this data model in place, the World Meteorological Office is now exploring the development of new regulations to encourage such interoperability. Whether or not these are put in place, the use of CF and the data model is now changing information transfer in global weather forecasting (on top of its integral role in climate model intercomparison). CF is now also becoming integral to earth observation, allowing major satellite data providers to lower costs and make data available to more users. The Reading team’s data model has been a key part in making data products accessible to more of the users of Europe’s largest provider of satellite based meteorological data, EUMETSAT [S5.8].

JASMIN was the first environmental ‘‘super data computer’’, showing the benefit of centralising high-volume data with dedicated high-performance analysis compute. The existence of JASMIN has enabled international research collaborations like PRIMAVERA (an EU Horizon 2020 research project involving the University of Reading and 19 European partners with the aim of developing a new generation of advanced global climate models for the benefit of governments, business and society) [S5.10]. What is more, JASMIN has sped up research workflows to the extent that some tasks which previously would have taken years, now take days. This is allowing some people to have, and test, as many ideas in a year, as they could have done in their entire career with traditional systems. The impact on research science was significant enough that the Met Office designed and procured their own ‘‘mini-JASMIN’’ called SPICE (Scientific Processing and Intensive Compute Environment) – contracting Lawrence to provide advice and quality assurance [G3.5]. SPICE now underpins Met Office research across both weather and climate [S5.9].

Summary: Large-scale environmental science depends on vast volumes (petabytes) of data, typically in millions of files. The research carried out at Reading on techniques for managing and manipulating data at this scale (on vocabularies and tools, the earth system grid federation, and the design and implementation of JASMIN - a computational facility that delivers a ground-breaking petascale analytical environment) has been essential to the work of multi-national agencies (for example, EUMETSAT). It has underpinned the research that led to delivery of the UK climate impact projections, the Paris Agreement of the UN Framework for Climate Change [S5.5], and the research that will underpin the next assessment of the IPCC. The next decade will see new challenges in moving from petabytes to exabytes, necessitating further developments in tools and information systems.