Skip to main

Impact case study database

The impact case study database allows you to browse and search for impact case studies submitted to the REF 2021. Use the search and filters below to find the impact case studies you are looking for.
Waiting for server

Nanosecond timing for a particle accelerator network

1. Summary of the impact

The GSI-Helmholtz Centre for Heavy Ion Research (GSI) in Germany runs a particle accelerator facility used by some 3,000 researchers from 50 countries. Glyndwr University research into timing in data transmission in networks has contributed to GSI’s ability to deal with the problem of achieving deterministic control of large real-time systems under the assumption of a common, highly accurate notion of absolute time. In turn, this has supported GSI’s work in establishing the Facility for Antiproton and Ion Research in Europe (FAIR), a more sophisticated and versatile accelerator facility capable of parallel execution of multiple experiments with different particle beams.

2. Underpinning research

In 2000, Grout and Davies began working on various aspects of Internet traffic analysis, particularly the statistical analysis of ‘connectionless’ packet distributions and timings compared to the older ‘connection-based’ telecommunications networks. Although the general ‘inappropriateness of the Poisson model’ was by then widely accepted, Grout argued that such behaviour might still be evident in high-speed networks (where packets would appear as non-continuous entities with inter-packet gaps) or (for example) control networks (in which data comprised short, independent packets with no self-similarity). The apparent non-appearance of such patterns was a puzzle at first but eventually shown to be a widely-implemented oversight in the way timings were recorded over discrete (as opposed to continuous) intervals [ reference 1]. Grout and colleagues continued to develop this contribution to the field that became loosely known as ‘network calculus’ [ reference 2] and applied it to the analysis and control of ‘self-similar’, ‘heavy-tailed’, ‘scale-free’ and ‘bursty’ network traffic [ reference 4]. A particular focus that emerged by necessity was the development of tools (both theoretical and practical) to deal with ‘guarantees’ of worst-case network behaviour rather than simple averages [ reference 3]. This need for ‘determinism’ in network performance analysis further extended capabilities of the network calculus.

Between 2012 and 2014, Grout and Davies continued their work in collaboration with personnel (Kreider and others) at GSI-Helmholtz. The first report of the ‘White Rabbit’ timing system [ reference 5] described what was initially an experiment implemented at GSI/FAIR and CERN to extend the network calculus of ‘system-on-chip’ timings to the larger ‘network-on-chip’ model. This is achieved via a new ‘EtherBone’ protocol over a gigabit Ethernet network to remote ‘field-programmable’ gate arrays/processors (FPGAs). In simple terms, iIt demonstrates that the modelling techniques developed by Kreider and Grout work in the real world and on a large enough network. The precise performance requirements of the timing systems to be implemented at GSI and CERN demonstrated that they were viable in a production environment [ reference 6].

3. References to the research

  1. Grout, V. (2001), ‘Measuring network traffic: the hidden distribution’, Mathematics Today, December 2001, pp.179-183 [available on request]

  2. Grout, V., Cunningham, S., Oram, D. and Hebblewhite, R. (2004), ‘A note on the distribution of packet arrivals in high-speed data networks’, in Proceedings of the IADIS International Conference, 2004, ISBN 972-99353-0-0 [available on request]

  3. Rezaul, K.M. & Grout, V.(2007), ‘Managing the Bursty Nature of Packet Traffic using the BPTraSha Algorithm’. [Paper presented to the Third Collaborative Research Symposium on Security, E-Learning, Internet and Networking 14th-15th June 2007]. Plymouth: Plymouth University [available via https://glyndwr.collections.crest.ac.uk/274/1/fulltext.pdf]

  4. Rezaul, K.M. & Grout, V. (2007) ‘A Survey of Performance Evaluation and Control for Self-Similar Network Traffic’. [Paper presented to the Second International Conference on Internet Technologies and Applications (ITA 07), held at Wrexham, Wales 4-7 September 2007]. Wrexham: Glyndwr University [available via https://glyndwr.collections.crest.ac.uk/339/1/fulltext.pdf]

  5. Kreider, M., Bär, R., Beck, D., Terpstra, W., Davies, J., Grout, V., Lewis, J., Serrano, J. & Wlostowski, T., “Open Borders for System-on-a-Chip Buses: A wire format for connecting large physics controls”, American Physical Society Physical Review Special Topics – Accelerators and Beams, Volume 15, Issue 8, August 2012. https://glyndwr.repository.guildhe.ac.uk/id/eprint/8324/

  6. Kreider, M., Bär, R., Beck, Terpstra, W., Davies, J. & Grout, V., “New Developments for the FAIR Data Master”, Proceedings of the 10th International Workshop on Personal Computers and Particle Accelerator Controls (PCaPAC 2014), Karlsruhe, Germany, 14-17 October, 2014 (Winner of the ‘Isamu Abe Price’ award for ‘Best Paper’). https://accelconf.web.cern.ch/PCaPAC2014/papers/fpo022.pdf

4. Details of the impact

The GSI-Helmholtz Centre for Heavy Ion Research (GSI) in Germany runs a particle accelerator facility used by some 3,000 researchers from 50 countries. The Facility for Antiproton and Ion Research in Europe (FAIR) is a heavy ion particle accelerator being built at the GSI campus, as an international collaboration with associations to the United Kingdom. FAIR will offer a superconducting heavy ion synchrotron ring with a circumference of 1100m, several storage rings and various and diverse experiment sites. The existing accelerators at GSI will be used as a pre-acceleration stage and injector for FAIR. Full operation is planned in 2025, the current investment (in 2020) being 1.262 billion Euro (based on 2005 price levels). While FAIR will not match the energy of the Large Hadron Collider for protons, it is much more versatile. FAIR will be capable of accelerating heavy elements and anti-protons to up to 99 percent of light speed while running four experiments in parallel.

As part of the FAIR extension, research has been undertaken supporting the development of a control system capable of handling the highly complex interplay between various machines and systems. Each action must be executed at a predefined time, as part of an overall master plan. To obtain nanosecond accuracy, the need for new levels of absolute deterministic performance guarantee was recognised quickly and expert advice sought. GSI’s contacts at Darmstadt University of Applied Sciences identified Grout as a consultant for both deterministic network calculus and the unconventional behaviour of short control packets in an ultra-high-speed data network.

From 2010, Grout and Davies worked with the GSI team, particularly a GSI lead developer, Mathias Kreider, who then enrolled as a Glyndwr PhD student under Grout’s supervision (thesis title ‘On Time, in Style: Nanosecond Accuracy in Network Control Systems’’, completed 2017). Together they began to develop the FAIR timing system that would allow GSI’s new facility to operate within deterministic performance guarantees. The research led to the development of a new time synchronisation system and a new master unit architecture known as the Data Master (DM) The biggest challenge was the extension of the simple performance guarantees obtained from its ‘network-on-chip’ core to the broader, inherently unpredictable nature of the wider accelerator and synchrotron facility.

The DM is responsible for direct real time control of timed devices across the facility. The new DM is an FPGA based controller, using White Rabbit (WR) timing. WR is a hardware boosted variant of the IEEE1458 Precision Timing Protocol (PTP), capable of reaching 1 nanosecond accuracy with a jitter of < 20 ps. DM control granularity is at 1 nanosecond, which is an improvement by three factors of magnitude over the old system. The DM system is alarm-, not event-based, meaning messages are sent slightly in advance and are executed in the synchronised receiver precisely at the included deadline, removing the necessity for cable delay compensation. Use of the DM allows direct control of kicker magnets with real-time feedback. This is essential for coordination of beam transfers between FAIR accelerator rings. The DM and new timing system also allow synchronisation of the PHELIX peta-watt laser down to nanoseconds.

As opposed to the legacy system, the DM can generate commands programmatically from its own graph based control language, carped. carpeDM can handle complex interactions between accelerator systems as well as user requests, the only limit being FPGA resources and bandwidth. Another advantage is the introspection capabilities in real time and the built-in visualisation, providing insights for engineers and physicists developing machine control models. This transparency drastically reduced the necessity for on-site measurements during commissioning, which were time consuming and hard to correlate.

February 2019 trials of the new control system showed a very successful beam time, earning a lot of positive feedback from experimenters involving linear stage UNILAC, SIS18 synchrotron and ESR storage ring. Experiments ranged from materials research, astrophysics, biophysics, plasma physics, nuclear research all the way to solar wind simulation for the European Space Agency. The DM has proven to be able to command multiple accelerators in parallel. The DM’s accurate command distribution was also successfully employed to allow for time-of-flight measurements on particles without specialised pure optical time synchronisation systems. This is seen as proof of the new control system being able to utilise the superior stability of the White Rabbit time distribution system. The new system was thus delivering better clock synchronisation than the former single purpose hardware.

The adoption of the DM by GSI for the FAIR development has not only contributed to increased efficiency within the particle accelerator network by improving timed devices across the facility, it also indicates strengthened industry collaborations between Glyndwr and GSI. The DM was critical to the design and development of establishing FAIR, where trials showcased the flexibility and usefulness of the DM for various scientific fields and possible commercial ventures.

5. Sources to corroborate the impact

1.

GSI Helmholtzzentrum fur Schwerionenforschung GmbH, Design Engineer Timing Systems, corroboration of collaborative research activities, and impact

2.

Letter from GSI Helmholtzzentrum fur Schwerionenforschung GmbH, Design Engineer Timing Systems, corroboration of collaborative research activities, and impact

Additional contextual information