Results and submissions : REF 2021

Commercial impact in the rapidly growing market for automatic voice recognition ...

Submitting institution: The University of Sheffield
Unit of assessment: 11 - Computer Science and Informatics
Summary impact type: Technological
Is this case study continued from a case study submitted in 2014?: No

Download case study PDF

1. Summary of the impact

Work by Professor Thomas Hain and colleagues at the University of Sheffield underpins the automatic speech recognition (ASR) systems of VoiceBase, a major US provider of voice data analytics services to companies operating large call centres. Thanks to state-of-the-art ASR tools provided by Sheffield, VoiceBase is able to accurately transcribe and analyse over a billion minutes of calls per year, offering their customers the ability to mine their entire volume of calls for a wealth of nuanced information (e.g. relating to fraud prevention and performance management). The competitive edge provided by Hain’s technology has allowed VoiceBase to grow their business significantly by adding at least $100m to the value of the company and attracting clients such as Uber, Home Depot, NASDAQ, Delta Dental, and GrubHub.

2. Underpinning research

ASR for conversational speech is a challenging research problem, particularly in the context of adverse acoustic conditions such as over the telephone or in multi-party meetings. Commercial ASR can be achieved only by systems that combine numerous state-of-the art components, many of which are machine learning models that require massive volumes of speech data to be processed with computational efficiency. Professor Thomas Hain’s team at the University of Sheffield has addressed these issues since 2000, developing research in three key areas:

Robust methods for conversational speech. Hain’s work on ASR of telephone conversations was first developed in the context of international evaluation campaigns organised by the US National Institute for Standards and Technology (NIST) (2000-2004). This research yielded highly adaptive system architectures for ASR that encompassed both lexical and acoustic variations. Building on this, research was conducted for the EU projects Augmented Multi-party Interaction (AMI) (2004-2006) and Augmented Multi-party Interaction with Distance Access (AMIDA) (2006-2009), which focused on the automatic understanding of multimodal data generated in meetings [R1]. Sheffield addressed several key areas of machine learning in ASR: improvements in front-end feature extraction, automatic system optimisation using sampling techniques, automatic language model data collection and adaptation, and methods for improving ‘far-field' performance (i.e. using distant microphones). Sheffield-led systems won NIST competitions for rich speech transcription in 2007 and 2009.

The work on the recognition of conversations in meetings (2004-2009) led to advances in the robustness of model estimation and system structure design. Model robustness, especially for novel deep learning architectures, was achieved by enhanced confidence-based methods to filter and select training data. The research community adopted so-called sequence training of deep neural networks for improved robustness using a state-level supervision method developed by Sheffield. A combination of such methods was shown to allow efficient and highly accurate filtering of training data [R2].

Adaptation to background conditions. Starting in 2011 with the EPSRC-funded Natural Speech Technology (NST) programme grant (2011-2016), Hain’s team developed novel methods for adaptation to complex and non-stationary background conditions [R3]. Unsupervised switching of background models was developed in the context of classical Gaussian mixture models and expanded to neural networks. A novel technique was developed – acoustic latent Dirichlet allocation (aLDA) – which allowed acoustic models to be informed by subtle acoustic variations. Adaptation of language content was shown to be highly effective in ASR of multi-party meetings using acoustic and text LDA [R4].

Adaptation was further facilitated by metadata information pertaining to the speaker and domain. The team developed a novel method for diarisation of multi-party recordings that achieved outstanding performance even with overlapping speakers. A novel extension involved so-called h-vectors, which improved the characterisation of speakers for adaptation purposes [R5]. Novel models for sentiment and emotion recognition were introduced to enable ASR sensitivity to unusual conditions (e.g. a speaker becoming angry during a phone call) [R6].

Scalable tools. The complexity of ASR systems in both training and recognition is high. To alleviate this problem, the Sheffield team developed a resource optimisation toolkit (ROTK), which brings a novel graph/metadata approach to system construction, allowing optimisation of system structures and their deployment. The group has also developed a scalable and fully automatic framework for integrated training of acoustic and language models on a very large scale (>10,000 hours of acoustic data).

3. References to the research

Sheffield staff and students in bold.

Hain, T., Burget, L., Dines, J., Garner, P. N., Grezl, F., El Hannani, A., Huijbregts, M., Karafiat, M., Lincoln, M., & Wan, V. (2012). Transcribing Meetings With the AMIDA Systems. IEEE Transactions on Audio, Speech, and Language Processing, 20(2), 486–498. https://doi.org/10.1109/tasl.2011.2163395. Cited by 124.

Saz, O., Deena, S., Doulaty, M., Hasan, M., Khaliq, B., Milner, R., Ng, R. W. M., Olcoz, J., & Hain, T. (2018). Lightly supervised alignment of subtitles on multi-genre broadcasts. Multimedia Tools and Applications, 77(23), 30533–30550. https://doi.org/10.1007/s11042-018-6050-1. Cited by 2.

Saz, O., & Hain, T. (2017). Acoustic adaptation to dynamic background conditions with asynchronous transformations. Computer Speech & Language, 41, 180–194. https://doi.org/10.1016/j.csl.2016.06.008. Cited by 1.

Deena, S., Hasan, M., Doulaty, M., Saz, O., & Hain, T. (2019). Recurrent Neural Network Language Model Adaptation for Multi-Genre Broadcast Speech Recognition and Alignment. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(3), 572–582. https://doi.org/10.1109/taslp.2018.2888814. Cited by 10.

Shi, Y., Huang, Q., & Hain, T. (2020). H-Vectors: Utterance-Level Speaker Embedding Using a Hierarchical Attention Model. Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Barcelona, Spain, 7579-7583. https://doi.org/10.1109/icassp40776.2020.9054448. Cited by 8.

Jalal, M. A., Milner, R., Hain, T., & Moore, R. K. (2020). Removing Bias with Residual Mixture of Multi-View Attention for Speech Emotion Recognition. Proceedings of the 21st Annual Conference of the International Speech Communication Association (Interspeech), Shanghai, China, 4084-4088. https://doi.org/10.21437/interspeech.2020-3005. Cited by 0.

4. Details of the impact

In May 2018, Sheffield launched the Centre for Speech & Language Technology, a joint research centre with US voice data analysis company VoiceBase (with Hain as Director). VoiceBase provides speech recognition and speech analytics services through web-based software to companies whose business involves large call centres. Sheffield technology underpins the ability of VoiceBase to transcribe 1 billion minutes of calls every year, processing up to 60,000 live audio streams simultaneously [S1].

When VoiceBase was launched in 2010, it relied on temporary solutions and licensed third party software to support its services. However, with rapid progress in artificial intelligence (AI) and speech recognition technology, the company needed to develop a world-class speech engine of its own, supported by an internationally recognised research base, in order to compete in a rapidly growing market. VoiceBase’s collaboration with Sheffield has impacted the company in three key areas:

Company-wide infrastructure for safe speech system updates. The collaboration between Sheffield and VoiceBase began with Sheffield’s creation of a framework for the development of new speech models, which was immediately implemented across the whole of the company (60-70 employees across the US, Russia, India and the UK). Drawing on his expertise in this field and the ROTK developed at Sheffield, Hain developed a system that allowed staff in any location to incorporate novel research and technology from a multitude of different platforms and apply them to diverse aspects of the company's speech engine consistently and safely, without jeopardising the system’s operation [S1].

Upgraded speech engine. A speech engine comprises some 30 different strands of fast-moving technology, each one of which is the subject of multiple research projects worldwide. Keeping pace with the leading edge across all of these developments and identifying the key breakthroughs and advances requires an exceptional understanding of the field. Hain's research makes him one of only a small handful of people to have such understanding, allowing him to direct the development of VoiceBase's speech engine, deciding which elements to replace, and identifying the most advanced solution to use [R1-R6] [S1].

New speech models. The Sheffield VoiceBase Centre for Speech & Language Technology has developed a set of new (probabilistic) speech models that were rolled out to all VoiceBase clients in 2020. These models are based on hundreds of thousands of hours of archived recordings and their transcriptions, together with processes that optimise how audio source data are cleaned, processed, aligned, and separated to improve the accuracy of future transcriptions. Sheffield's expertise in deep learning architectures, emotion detection, and sentiment analysis were key to the development of these new models [R4-R6] [S1].

The technology developed allows VoiceBase to create highly accurate transcriptions of all their customers’ calls, including non-verbal elements (such as pauses, pace, and voice pitch of speakers) and emotion (such as empathy, anger, and excitement). AI is then applied to allow tailored, bespoke analysis of these transcripts to generate the performance and predictive data required by clients for a range of purposes, including the following:

Fraud prevention: keywords, pauses, and pitch/tone analysis allow clients to monitor all of their call centre operatives automatically for fraudulent or illegal activity, and take swift action to avoid millions of dollars in fines from the US Federal Communications Commission and its equivalents [S2].
Performance monitoring and management: enabling the ability to identify when staff talk over callers, respond empathically to anger or distress, and provide the information requested [R6].
Maximising sales and allocating resources: giving a reliable percentage score in the opening minute of a call to indicate the likelihood that it will lead to a sale or appointment, allowing the company to decide whether and how to follow up.
Protection of sensitive information: automatically identifying and deleting sensitive information (e.g. credit card details) from call recordings and transcripts.
Market analysis: automatically identifying common demands and complaints to improve offerings.
Meeting analysis: minutes, action points, keywords, identification of participants, individuals' input, concerns, and sentiment, etc. [R1].

Since the collaboration began, the company has recorded strong growth (100% in 2018 and 100% in 2019, with the same expected in 2020) and is expanding into Europe and Australia. Voicebase's CEO attributes a large part of this success to Sheffield research. “ Everything we do as a company relies on Thomas [Hain] . Since the end of Q2 2020, 95% of our speech recognition technology has been based on Sheffield research. Based on acquisitions by Apple, Google, Facebook over the last ten years, the new engine and research capability provided by our collaboration with Sheffield is worth at least USD100M+ if looking at company valuations and merger and acquisition historical comparables.” [S1].

VoiceBase attributes their ability to compete with the likes of Microsoft, Google, Amazon, and IBM to attract multi-billion-dollar companies such as HomeDepot, NASDAQ, Uber, Dish Network, GrubHub, Delta Dental, Centurylink, and Twilio as clients to the increased transcription accuracy enabled by Sheffield and the global reputation of the research team [S1]. The company is also partnering with call centre hardware providers such as Poly (70% share of the global market, 90% of the US market) and GN Netcom/Jabra to launch products directly connected to VoiceBase Speech analytics [S1]. The Senior VP for Strategic Alliances and Partnerships of Poly said, “ I was blown away by the analytic capabilities of your [VoiceBase] platform and how you're disrupting the economics of what it means to do sentiment analysis.” [S3].

The COO of Proactive Dealer Solutions (PDS), which assists car dealerships by maximising call centre sales, gives an example of the impact of Sheffield technology on VoiceBase clients: “ Within a matter of minutes using the predictive indicators that we built with VoiceBase […] we are able to score the call on 30 different metrics […] The average dealership loses 20% of their inbound calls and only appoints about 18%. Thanks to the tool we built, our clients across the board are seeing a fail rate of less than 10% and we are now appointing at 55%. The average dealer that works with us and really embraces our management tool sees an increase in sales of over USD100,000 a month.” [S4].

Enhancing quality and productivity and reducing costs of machine translations

Submitting institution: The University of Sheffield
Unit of assessment: 11 - Computer Science and Informatics
Summary impact type: Technological
Is this case study continued from a case study submitted in 2014?: No

Download case study PDF

1. Summary of the impact

Machine translation (MT) is inexpensive, fast, and accessible, but it lacks the reliability of human translators. Sheffield research on quality estimation (QE) in MT has enabled the identification of the likelihood of error, allowing MT to be used with greater confidence and underpinning impacts for multiple organisations. Microsoft introduced MT for text strings in program code in 37 languages, allowing more than 20% of translated software to be published without human intervention. Facebook improved its offering of approximately 6bn automated translations per day in >300 language pairs. In addition, the research shaped the latest development of the European Commission Directorate-General for Translation’s (DGT’s) MT system.

2. Underpinning research

In the context of the global spread of social media and a global population that is mostly monolingual, MT allows individuals to communicate across language barriers in a way that was impossible a generation ago. Professor Specia’s research established the field of QE for MT , with the developed approaches summarised in [R1], the only book on the topic, written by invitation for the main book series on natural language processing (NLP). Specia developed algorithms to score machine-translated documents for their predicted translation quality at the word, sentence, or document level. These methods can successfully identify text likely to be ungrammatical and poorly written and text that reads fluently but is likely to contain meaning errors.

In 2012, Specia and her collaborators established the Quality Estimation Shared Task (QuEst) [R2], which provides automatic methods for estimating the quality of MT output at run time without reliance on reference translations. The task has run yearly since. In 2013, Specia published QuEst, the first sizeable research output in the area [R3] and the first framework released as a tool for MT QE. This framework provided the first language-independent feature set, which made QE applicable to any language pair and offered sentence-level quality prediction. In 2015, Specia and Dr Scarton expanded the framework to include word-level prediction and document-level prediction by adding novel, appropriate features for these two levels, leading to the development of QuEst++ [R4].

In 2018, Specia and Dr Blain developed and released deepQuest, a version of QuEst relying on deep learning, in response to new developments in MT software [R5]. Unlike previous approaches, deepQuest avoids the need for “feature engineering”, thereby providing increased language independence as well as improved performance. Standard NLP pre-processing is extremely computationally intensive, but the lightweight deep learning approach in deepQuest, called the bidirectional recurrent neural network (biRNN) approach, does not require pre-processing or pre-training. In 2017, deepQuest was shown to perform on par with the state-of-the-art approach, Predictor-Estimator, with the advantage that it can be trained in a fraction of the time.

Finally, in 2020, Specia and Blain developed and released a new version of deepQuest [R6]; combining the biRNN approach with state-of-the-art, pre-trained representations. Their investigation of the new version showed that it allowed QE with unsupervised learning and provided state-of-the-art performance based on current benchmarks.

QuEst, deepQuest and QuEst++ are freely available on GitHub.

4. Details of the impact

The Sheffield research contributed to rapid improvements in the quality of MT services. QuEst++ gives a reliable indication of probable quality, enabling users to focus efforts more effectively (reducing time and costs) and increasing confidence and awareness of quality risks when using raw MT texts. These improvements have led many organisations to use MT as their primary translation method, with human translators performing editing.

Microsoft E&D Global

In May 2018, Microsoft E&D Global introduced MT with automated QE based on QuEst into software localisation (i.e., the process of adapting software for a specific region or language, by translating text and adding locale-specific information). Although Microsoft was previously using MT for ~75% of their content, its use for software proved more complex, as the risks were greater since functional bugs in a product negatively impact customer experience (and result in a loss of customers). By 2019, MT was enabled for 37 languages on 90% of the Office product line. QuEst has allowed Microsoft to publish >20% of machine-translated word count without review, with no measurable negative impact on customer satisfaction [S1].

According to Microsoft E&D’s Senior Program Manager, “The key benefit of this process for Microsoft has been savings of 20% in the cost of human translation and post-editing, but it has also delivered time savings by removing a stage from the translation and publishing process. With over a million words now published using raw MT across 37 languages, these savings are significant. […] Without QuEst we would not have been able to develop and deploy a quality estimation solution fit for purpose, therefore QuEst was absolutely fundamental in achieving the volumes we have managed to reach to date. [...] QE provided us with confidence without which I don't think we would have had buy-in from product owners to start publishing raw MT. If they did approve it, we would have been severely restricted in the volumes we would have been able to publish”. Summarising, the Senior Program Manager stated, “We can safely say that QE gave us a 20-30% relative gain, in terms of volume of raw MT that we could publish, compared with using other mechanisms” [S2].

Facebook

Specia collaborated with Facebook within the REF assessment period on research to inform Facebook’s development of prediction models of translation quality. The Research Scientist Manager at Facebook AI explained, “ Facebook offers its users machine translation of posts into their chosen language. Some translations are shown automatically instead of the original foreign language post – and the decision on which translations are shown automatically is based on confidence level. If our system is confident that the translation is likely to be of good quality then it will be shown automatically – if not, then it will not. Professor Specia’s original research and the software she has been pushing forward (QuEst++ and deepQuest) largely set the standard for the area and provided inspiration for the prediction model used to make these decisions” [S3].

“Facebook has set up a research agenda around quality estimation based on her work and in collaboration with Professor Specia, creating new corpora and data sets to increase the coverage of the prediction model from higher research languages (e.g. German, Chinese) to lower research languages (e.g. Sinhalese, Nepali)” [S3].

“Facebook’s goal is to bring down language barriers and allow people to connect regardless of language. This is an important part of our mission, and we take it seriously. Professor Specia’s research has helped us to achieve our mission by enabling the technology that ensures only the highest quality and most accurate translations are shown to our members. Facebook shows more than 6 billion translations daily in more than 300 direct language pairs, and each translation is accompanied by a quality estimation prediction” [S3].

European Commission DGT

The research on biRNNs played a key role in guiding the introduction of automatic QE into eTranslation, the MT system of the European Commission DGT. The DGT aimed to improve the usefulness of their services through the QE technology and methods developed in Specia’s research, as the previous quality control system was overly simplistic. eTranslation is both a supporting tool for DGT translators (who translate over 2 million pages of official documents into 26 languages every year) and a main translation route for other EU institutions and institutions of its member states (courts, parliaments, etc.). eTranslation has been used to translate up to 1 million pages of text in a single day, and usage continues to grow [S4].

The components of the QE module were successfully integrated, and the module became fully operational and interoperable with the eTranslation system. The Sheffield technology estimates the translation quality from two methods: MT and “translation memory” (the default method used by the DGT, which employs a database of previous translations). The likely superior version is then recommended. For other users, each MT includes a quality/accuracy “health warning” based on a range of factors. These warnings allow greater confidence in the translated texts and highlight the areas of greatest risk.

The Project Manager for Machine Translation at the DGT stated, “Without the work of the Sheffield Research Group we would not have been able to contemplate integrating quality estimation functionality into eTranslation at this point in time. Specia worked closely to advise and guide our internal IT and translation experts, including delivering a workshop to our staff entirely devoted to quality estimation. Her influence on our direction of travel and our approach to the development of our software has been fundamental” [S5].

A pilot of the module is currently underway.

Localisation and translation technology specialists

Specia’s work is the cornerstone for QE in MT across the industry. A host of localisation and translation technology specialists worldwide, such as Belgian consulting and systems integration company CrossLang [S6] and ModelFront (formerly SignalN) [S7], have been guided by Specia’s research or used her products to develop and benchmark their own product offering. The CEO of ModelFront stated, “ Specia’s work has defined the category of Quality Estimation of machine translation, and all companies working in this area are indebted to her for her early vision and leading the research community for the last decade. As a dedicated provider of Quality Estimation technology, ModelFront is grateful to her for proving this technology both scientifically and in the eyes of the industry” [S7].

Shaping international policy and public debate on lethal autonomous weapons systems ...

Submitting institution: The University of Sheffield
Unit of assessment: 11 - Computer Science and Informatics
Summary impact type: Societal
Is this case study continued from a case study submitted in 2014?: Yes

Download case study PDF

1. Summary of the impact

Research into the capabilities and ethics of artificial intelligence (AI) conducted at the University of Sheffield has fuelled debate at national and international levels on the development of lethal autonomous weapons systems (LAWS). Evidence from the research prioritised the issue on the UN agenda and ensured that technological issues were fully understood and considered by member states. In addition, it underpinned a global civil society campaign against LAWS, accomplished in large part through a coalition of NGOs co-founded by Professor Noel Sharkey, whose ranks swelled from 7 to 172 NGOs in the submission period. The arguments of the campaign persuaded national governments and political unions worldwide to support a ban on the development of LAWS.

2. Underpinning research

Two strands of Professor Noel Sharkey's research underpin the impact: one on robotics and another on the ethics and legality of the use of robots in LAWS.

Throughout the 2000s, Sharkey extended upon his earlier novel research on the close relationship between the physical embodiment of robots and their control systems to understand the potential and limitations of future robots. In a 2001 study comparing human and robot embodiment, he argued that the two are fundamentally different and that strong embodiment, either mechanistic or phenomenal, is not possible for present day robots [R1]. This conclusion both provided an argument against strong AI and underpinned Sharkey’s developing views on robot ethics.

Sharkey investigated robot programming, learning, and construction, particularly with regards to the use of neural networks for learning robot behaviours. Examples include (1) work on robot localisation – the problem of determining a robot’s location quickly, reliably and accurately – to which he and A.J.C. Sharkey proposed a novel solution exploiting self-organising maps and ensemble techniques [R2]; and (2) work on robotic arm control – the problem of how to rapidly adapt a robotic arm controller given the new geometric space that arises when sensor position is changed or sensors are replaced – to which he proposed a new solution combining genetic algorithms and neural networks [R3]. This body of work yielded in-depth insights into the capabilities and limitations of autonomous robots and established Sharkey’s technical credibility as a robotics expert.

In 2005, Sharkey began to investigate the ethical issues surrounding the use of robots in various applications, particularly military applications, leading him to explore whether the use of robots in LAWS could be deemed either morally defensible or legal according to international laws and conventions governing weapons and warfare. With reference to specific robot technologies and military robots, Sharkey analysed the perceptual and cognitive capabilities required for robots to distinguish between combatants and non-combatants, showing that current robot technologies fall far short of possessing these capabilities, despite the claims made by arms developers and military organisations. He therefore argued that autonomous robot weapons fail to meet two key principles of international humanitarian laws governing warfare (such as the Geneva and Hague Conventions): discrimination (it must be possible for an attacker to distinguish combatants from non-combatants) and proportionality (the anticipated loss of life and damage to property incidental to attacks must not be excessive in relation to the concrete and direct military advantage expected to be gained).

Sharkey also explored the concept of autonomy in robots, determining the degrees of autonomy that military robots can exhibit and finding that the failures of current non-autonomous, "human-in-the-loop" remote military weapons, such as drones, are even more likely to occur in LAWS. Supported by a Leverhulme Trust Senior Research Fellowship, this research has led to 11 publications in military ethics, law and technology journals (e.g. [R4] on the use of automated robots in wars and the new type of battle stress they introduce), as well as high-quality science and engineering journals (e.g., [R5] on the application of AI to discriminate between innocents and combatants in modern warfare), with over 185 total academic citations.

Drawing on his background in psychology, Professor Sharkey conducted further research to explore what “human control” means in relation to LAWS. By referencing theories of automatic and deliberative human behaviour and examining 60 years of data on human “supervisory control” of machines, he introduced a new framework that reframes autonomy in terms of supervisory control and allows for greater transparency and allocation of responsibility [R6].

4. Details of the impact

Impact on a global campaign against the development of LAWS

From 2007 to 2013, as part of an engagement campaign, Sharkey presented his research-based technological and ethical case against the development of LAWS, which select targets for lethal force without human intervention [R2-R5]. He became the leading voice of expertise on this subject in the media, giving evidence to national and international military and government bodies and calling urgently for formal international discussion. This formed the basis of a REF2014 impact case study.

With formal international discussions on LAWS beginning in November 2013, Sharkey continued to draw on his research [R1, R6] to drive international debate through both the International Committee for Robot Arms Control (ICRAC - an NGO that he co-founded in 2009 to bring together interdisciplinary academic experts who shared his concern about the use of robots in LAWS) and the Campaign to Stop Killer Robots (CSKR - a coalition of 7 humanitarian NGOs, including ICRAC, that he co-founded in 2012 whose mission is “ to ban fully autonomous weapons and thereby retain meaningful human control over the use of force”) [S1a].

Sharkey supported the mission of the CSKR by providing expert knowledge and research-based evidence to CSKR stakeholders in the debate throughout the submission period. The CSKR campaign leader, a Nobel Peace Laureate, confirmed, “ Professor Noel Sharkey's research into the ethics and capabilities of AI has provided the inspiration and the evidence base to support the Campaign's goals since its inception and continues to be crucial to our work in driving this debate forward today” [S2]. Sharkey presented the evidence base [R1, R4-R6] in support of a ban to many technologically savvy potential donors, which was instrumental in winning their support. The CSKR campaign leader noted, “ These activities have generated a significant amount of funding for CSKR, which we have used to enable NGOs and members of the campaign from less wealthy, often underrepresented countries to work with their own governments and attend UN events so their voices could be heard in the debate” [S2]. CSKR reports virtually all campaign funding has been provided by a Geneva-based group of anonymous private donors ($1.3m in financial year 2019-2020) [S1b].

Since 2013, the CSKR grew from 7 NGOs to a global movement of 172 national and international organisations (including Nobel Peace Prize-winning Human Rights Watch (HRW), Amnesty International and the Nobel Women’s Initiative) in 65 countries [S1a], with new regional subgroups continuously being established – the latest being CSKR South East Asia created in 2019 [S1c].

Impact on discussions of the UN Convention on Certain Conventional Weapons (UNCCW)

In November 2013, the UNCCW decided to convene a meeting of experts to formally discuss LAWS. The decision was in large part a response to two reports (in 2010 and 2013) by the Special Rapporteur on Extrajudicial Executions for which Sharkey provided key evidence [S3], as well as Sharkey’s engagement campaign conducted since 2007. Formal discussions of LAWS were held at annual international meetings of experts (2014-2016) and, as the issue increased in importance, at international Group of Governmental Experts (GGE, 2017-2019) meetings. Sharkey represented the ICRAC at all of these events [S4], contributing as an invited expert in 2014 and producing a report on meaningful human control in 2018 [S5] [R1, R6]. The report demonstrated to the delegates the crucial difference between automatic and directive control and led to the incorporation of meaningful human control into the draft guiding principles for LAWS. One meeting chair described the value of Sharkey’s input as follows: “ His in-depth understanding and research-based knowledge of potential capabilities of Lethal Autonomous Weapons and Artificial Intelligence in the military domain provided a factual and very much appreciated contribution that informed the group’s discussions, in particular on the subject of meaningful human control” [S6].

In addition to his direct contributions, Sharkey has indirectly influenced debate surrounding LAWS through the contributions of the CSKR at these expert meetings, organising side events and producing briefings and statements [S1c, S4]. At the vast majority of the expert meetings, CSKR members accounted for all (or all but one) of the NGO members present [S4].

Impact on national governments and political unions

Sharkey and the CSKR have engaged directly with national governments (Helsinki, Brussels, Berlin, Paris, Buenos Aires, and Rio de Janeiro) and groups of nations (the European Parliament, African Union, Non-aligned Movement, and Nordic Group) to generate support for a ban on LAWS by the UN [S1c, S4]. Some of the funding Sharkey helped raise for the CSKR has been given to members to hold local events [S1b, S2], and the group has actively lobbied and co-hosted side events at the UN General Assembly [S1c, S4].

A total of 99 countries have now raised LAWS in their remarks at the UN General Assembly, with dozens more aligning themselves with statements by political unions [S1d]. The number of countries mentioning LAWS during their statements rose from only 16 states in 2013 to 37 in 2020. Previously, 42 did so in 2019, 49 in 2018, 34 in 2017, 36 in 2016, 32 in 2015, and 23 in 2014 [S1d]. During 2020, Brazil, Japan, and Germany have hosted their own international meetings on LAWS, with Austria scheduling its meeting for early 2021 [S1d].

The UN Secretary General (2018), as well as the Pope (2020), the Dalai Lama (2014) and other faith leaders (2014), have joined 30 nations from Europe, Africa, South America, Asia and the Middle East in calling for a ban on LAWS [S1c, S1d, S1e]. However, since progress towards a UN accord has reached a stalemate, with Russia, Israel, the USA, South Korea, and Australia opposing any kind of regulation, the CSKR has refocused its efforts on finding a national sponsor for an international treaty [S2], which would not need unanimous UN backing to become law. The CSKR proposed the key elements of an international treaty [S7] with an emphasis on human control [R1, R4-6] and HRW has identified this as a key interest among nation states [S8, pages 1-7].

Guided by Sharkey’s research, the CSKR targeted Germany and France as potential sponsors through public events and media campaigns, resulting in French and German foreign ministers identifying the threat from LAWS as one of six issues requiring urgent and priority multilateral action at the 2019 UN General Assembly. They also led their counterparts from 16 other countries to co-sign a political declaration endorsing the objective of “developing a normative framework” that would address autonomous weapons [S8, page 1 footnote 2].

Impact on UK political debate

In 2017, Sharkey was invited to give evidence on LAWS [R1, R6] to the House of Lords Committee on Artificial Intelligence as part of their inquiry ‘AI in the UK: Ready, Willing and Able?’ The former chair of the committee commented, “ His expert evidence made our members aware of the issues caused by the lack of clear definitions for ‘autonomous’ when applied to weapons, and also of the fact that the United Kingdom’s definition differs significantly from the global consensus” [S9]. At the committee’s request, Professor Sharkey provided a report [S10] on the various definitions of autonomous weapons, which directly underpinned their recommendation in a wide-reaching 2018 report that the Ministry of Defence change their language on autonomous weapons and meaningful human control to align more closely with the international consensus [S9].

Sharkey also provided evidence to the All-Party Parliamentary Group (APPG) on AI; the co-chair of the APPG attested that his contributions, “have greatly contributed to our discussions on this and other areas of ethical AI development and deployment as well” [S9].

5. Sources to corroborate the impact

Combined: CSKR website information (All accessed 20th Jan 2021).

About CSKR. https://www.stopkillerrobots.org/about/
CSKR annual report 2019 reports income and expenditure activities ( https://bit.ly/3tP8491) pp.7-8 & 20-22.
History and achievements. https://www.stopkillerrobots.org/action-and-achievements/
75th UN Assembly Meeting. https://www.stopkillerrobots.org/2020/10/un-diplomacy/
Positions of countries on the call to ban fully autonomous weapons (July 2020). https://www.stopkillerrobots.org/wp-content/uploads/2020/05/KRC_CountryViews_7July2020.pdf

Confidential testimonial statement from the 1997 Nobel Peace Prize Laureate and CKSR campaign leader (2020). Corroborates the importance of Professor Sharkey’s research in the CSKR and ICRAC efforts.

Confidential testimonial and report from the UN Special Rapporteur on extrajudicial, summary or arbitrary executions (2013). https://www.ohchr.org/en/issues/executions/pages/srexecutionsindex.aspx. Corroborates Professor Sharkey’s contribution. (Accessed 20th Jan 2021).

List of UN events where Professor Sharkey has contributed (2014 - 2019).

ICRAC report authored by Professor Sharkey “Guidelines for the human control of weapons systems” (2018). (Accessed 28th Jan 2021). https://bit.ly/3tQHVXv

Confidential statements from a Chair of the Informal Meeting of Experts of the UNCCW (2015 & 2016) and a Chair of the Group of Governmental Experts of the UNCCW (2019) confirming Professor Sharkey’s contribution to the debate.

Elements for a treaty on fully autonomous weapons proposed by the CSKR (2019). (Accessed 20th Jan 2021). https://bit.ly/3f4Heps

Human Rights Watch report “Stopping Killer Robots - Country Positions on Banning Fully Autonomous Weapons and Retaining Human Control” (August 2020). (Accessed 15th Dec 2020). Corroborates signing of political declaration (page 1 footnote 2) and calls for an International Treaty (pages 1-7) https://www.hrw.org/report/2020/08/10/stopping-killer-robots/country-positions-banning-fully-autonomous-weapons-and#_ftn7

Confidential testimonial letter from the former Chair of the House of Lords Select Committee on Artificial Intelligence and Co‐Chair of the All Party Parliamentary Group (APPG) (2020). Corroborates Professor Sharkey’s role and contribution to the committee discussions.

Professor Sharkey’s report submitted, on request, to the House of Lords Select Committee on Artificial Intelligence (2018). (Accessed 15th Dec 2020). http://bit.ly/3tiIZDr

Societal and commercial benefits of large-scale data collection and analysis

Submitting institution: The University of Sheffield
Unit of assessment: 11 - Computer Science and Informatics
Summary impact type: Technological
Is this case study continued from a case study submitted in 2014?: No

Download case study PDF

1. Summary of the impact

The Organisations, Information and Knowledge (OAK) group at the University of Sheffield have developed novel technologies to capture and analyse massive volumes of data from mobile phones, sensors, and social media. These technologies have enabled the creation of a successful sports media company with a turnover of £1.8m; underpinned the development of an app to support a national Public Health England (PHE) campaign (800,000 downloads); informed the creation of cycle paths and a Clean Air Zone in Birmingham; underpinned disaster response strategies in Italy (involving 27,000 citizens); and facilitated the development of novel traffic management products that have transformed the field and improved lived experience in busy urban areas of the UK.

2. Underpinning research

The OAK group, led by Professor Fabio Ciravegna, focuses on the development of novel algorithms and infrastructures to address challenges underpinning the acquisition, fusion and modelling of data at very large scales, and their application to generate impact. The underpinning research was developed by the OAK group through leadership and participation in a number of European and EPSRC projects. Ciravegna led WeSenseIt (2012-2016), SETA (2016-2019), LODIE (2012-2015), and RAnDMS (2012-13), and the OAK group was a partner in WeKnowIt (2008-2011) and ReDites (2013-14). The research spans five themes:

Theoretical foundation for digital health interventions. In 2013, interdisciplinary work conducted by Ciravegna in collaboration with psychologists at Sheffield and Sussex developed a clear theoretical foundation for digital interventions in the health domain. This was achieved via some of the earliest work for assessing behaviour change techniques (self-affirmation, theory-based messages, and implementation intentions) to target health behaviours in a smartphone intervention [R1]. Subsequent studies extended this methodology to a large-scale PHE digital intervention scheme (the behaviour change mobile application Active 10 with hundreds of thousands of users), in which physical activity was directly monitored using smartphone sensors [R2]. The same research also underpins the SETA technology which allows detailed and granular information on user mobility to be collected.

Robust analysis of mobility at a massive scale. To enable participant monitoring over long periods of time and at very large scales, fundamental research was performed to address problems related to the power efficiency of 24/7 mobile tracking algorithms and their robustness across a diverse range of contexts, situations and mobile device types [R2]. Algorithms able to continuously and precisely model mobility with limited impact on a phone’s battery life were developed for multifarious applications spanning from health and wellbeing to emergency response [R3]. These innovations allowed for the efficient and reliable collection of over one billion data points between March 2017 and January 2019.

Computing context. The mobility of citizens cannot be fully understood without also considering the wider context of their sensor data. For example, it is necessary to combine information from current and historical motion data, GPS, and transport network maps with environmental (such as temperature, weather, and road morphology) and external (e.g. the use of walking aids) factors to determine whether a citizen is running, driving, or using public transport. The OAK group developed novel algorithms for determining the context of motion data using sensor fusion and graph-based approaches [R3].

Methodologies for citizen engagement. Working with social scientists at IHE Delft in the Netherlands, the OAK group developed a framework to identify drivers for increased citizen participation in environmental monitoring, and carried out empirical research in the UK, the Netherlands, and Italy between May and November 2013. This led to important insights into the interactions between individual motivations for participation, the objectives and needs of authorities/institutions, and governance structures [R4].

Novel methods for social media analysis. Methods for largely unsupervised, large-scale mining of data from the web were developed, including approaches for populating knowledge bases that addressed well-known problems such as data sparsity and noise (which can be caused by web publishers using inconsistent vocabularies or making errors). These approaches were shown to increase the precision of relation extraction in a number of internationally available corpora and real-world data sets (F1 score increased between 7% and 40% over baseline models) [R5]. This research supported the development of novel methods for social media analysis via natural language processing by allowing the context of a micropost (e.g. a Twitter message) to be inferred from a knowledge base. A particular focus was the early detection of rumours on social media, which led to the development of graph-based techniques for rumour detection [R6].

3. References to the research

University of Sheffield staff and students in bold

Epton, T., Norman, P., Sheeran, P., Harris, P. R., Webb, T. L., Ciravegna, F., Brennan, A., Meier, P., Julious, S. A., Naughton, D., Petroczi, A., Dadzie, A.-S., & Kruger, J. (2013). A theory-based online health behavior intervention for new university students: study protocol. BMC Public Health, 13(1), 107. https://doi.org/10.1186/1471-2458-13-107. Cited by 45.

Ciravegna, F., Gao, J., Ireson, N., Copeland, R., Walsh, J. & Lanfranchi, V. (2019). Active 10: Brisk walking to support regular physical activity. Proceedings of 13th EAI International Conference on Pervasive Computing Technologies for Healthcare (Pervasive Health2019), Trento, Italy, 20th-23rd May 2019, 11-20. http://eprints.whiterose.ac.uk/145438/. Cited by 2.

Ciravegna, F., Gao, J., Ingram, C., Ireson, N., Lanfranchi, V. & Simanjuntak, H. (2018). Mapping mobility to support crisis management. Proceedings of the 15th Annual Conference for Information Systems for Crisis Response and Management (ISCRAM), Rochester NY, USA, 305-316. http://eprints.whiterose.ac.uk/130788/. Cited by 2.

Wehn, U., Rusca, M., Evers, J., & Lanfranchi, V. (2015). Participation in flood risk management and the potential of citizen observatories: A governance analysis. Environmental Science & Policy, 48, 225–236. https://doi.org/10.1016/j.envsci.2014.12.017. Cited by 155.

Zhang, Z., Gentile, A. L., Blomqvist, E., Augenstein, I., & Ciravegna, F. (2016). An unsupervised data-driven method to discover equivalent relations in large Linked Datasets. Semantic Web, 8(2), 197–223. https://doi.org/10.3233/sw-150193. Cited by 11.

Varga, A., Basave, A. E. C., Rowe, M., Ciravegna, F., & He, Y. (2014). Linked knowledge sources for topic classification of microposts: A semantic graph-based approach. Journal of Web Semantics, 26, 36–57. https://doi.org/10.1016/j.websem.2014.04.001. Cited by 37.

4. Details of the impact

The collection and analysis of large-scale data has a wealth of applications. This is illustrated by the diverse fields in which the OAK group’s research has delivered impact: health, transport, disaster response, and sports media.

National societal impact

In the UK, Sheffield’s research on tracking technology, application efficiency, and server infrastructure [R1-R2] were utilised in the “Move More” initiative of PHE’s £3m “One You” campaign. Between 2017 and 2019, the Active 10 app achieved approximately 800,000 downloads. PHE confirms, “ With the support of Sheffield's research, PHE were able to develop and launch the first free-to-use mobile app that provided the user with information on time, intensity and periodicity [of physical activity]. The app played a significant role [...] and made a major contribution to the overall success of the One You campaign” [S1].

In Italy, the OAK group used technologies developed during the WeSenseIt project [R3-R6] to analyse data from mobile phones, sensors, and social media, managing the successful evacuation of 27,000 citizens from Vicenza in April 2014 following the discovery of a WWII bomb [S2, S3]. This led the Italian government to adopt citizen observatories as support for water and flood risk management (European Flood Directive 2000/60) [S2]. The Special Projects Manager at the Eastern Alps District River Authority states: “ The contribution of the OAK group in this process was key. The WeSenseIt project made the concept real and applicable; the technology developed by OAK provided concrete proof of the power of the citizen observatories as well as a powerful benchmark for requirement analysis and for the development of the final production technology” [S2].

Impact on mobility and local authorities

As part of the “Big Birmingham Bikes” scheme (ongoing from 2015), Birmingham City Council (BCC) gave a large number of bikes (2,000 initially) to communities historically unlikely to take up cycling. The project initially used a dedicated GPS tracker, but as the number of bikes on offer rose, the cost of such trackers to gather evidence in support of the scheme became prohibitive. By offering free tracking on recipients’ mobile phones from 2017-2019, the Sheffield-designed SETA tracking technology [R1-R2] allowed for significant savings (£130k per year) while collecting even more detailed and granular information on overall mobility. Data showing how cycle use correlated with the routes chosen by riders, combined with vehicle tracking information provided by a SETA partner, caused BCC to revise the proposed locations of new cycle routes to better meet the needs of cyclists and reduce safety risks [S4]. After the Wellbeing Service at BCC became the Active Wellbeing Society (an independent charity), it bid successfully to become one of Sport England’s 12 local delivery pilots, being awarded a total of £11m between 2018 and 2019 (of which £420k was devoted to free bicycles). “ The experience, evidence and skills we gained by using the SETA app made a significant contribution to the success of our proposal, allowing us to put ourselves forward confidently as a cutting-edge organisation with a key strength in the use of data gathering and evidence. [...] Our core objective has always been to increase cycling and increase funding for cycling – and Fabio’s SETA app has helped us achieve this” [S5]. To supplement the prior cycling and vehicle data, Sheffield provided tracking apps for thousands of cyclists, walkers, and runners within Birmingham, giving BCC a detailed understanding of mobility within their city, upon which they based the design and operation of their Clean Air Zone [R3].

The Floow Ltd, a company that collects second-by-second data on vehicles in motion to provide risk analysis services to the motor insurance industry, used their participation in the SETA project to develop six new data products for traffic management within the REF period. According to their Chief Innovation Officer: “ The University of Sheffield was the driving force behind the SETA project [...] Sheffield's expertise in active travel and complex visualisation systems [...] helped The Floow understand the needs of traffic managers and the potential of products that might help support them” [S4]. These products have enabled local authorities to more effectively manage transport for the benefit of their residents. The Floow’s contract with Greenwich has transformed the council's priorities for traffic management by overturning previously false assumptions with precision data, e.g., it was revealed that the proportion of traffic from outside the region was 75% and not 20% [S4]. This approach has been expanded and repeated, providing new insights to make streets safer and less polluting across London and the UK [S4].

Commercial impact

As a result of these new data products, The Floow has been able to win commercial contracts in new sectors. Working with Sheffield and the other SETA partners “ ensured that emerging new to market products better met wider needs of end users including a broader overview of 'mobility'. […] These gains were fundamental to our success in these areas and are thanks to SETA”. The new products have generated agreements in excess of £400k, over the financial year 2019-2020, and led to contracts in new markets. These markets include governmental, with supply to the Department for Transport, highway agencies and local authorities across the UK; autonomous, with new aspects of support helping to set routes for robo-taxi deployments; and traffic management, with supply of traffic data to large consultancy organisations and monitoring teams [S4].

Football Whispers, a start-up aiming to create an online platform for football fans and sports journalists to analyse and discuss the latest football news and transfer rumours, approached Ciravegna in October 2015 for help with developing the technology needed for their launch in January 2016. The OAK group’s social media-based technology [R6] was developed for the prediction of football transfers by analysing messages in social media across multiple languages and 36 international leagues [S6]. According to the Founder and former CEO of Football Whispers: “ Thanks to the work of the OAK group at the University, we were able to launch on time in January 2016 and with our full service offering – something that we would not have been able to accomplish without their input. [...] In that time our business grew from 0 to 2,500,000 unique monthly users. [...] It was adopted as a tool by Sky Sports, ESPN (the world’s largest sports television company) and football magazine FourFourTwo, and was also used by the sports pages of The Sun and The Daily Mail. During the period of my Chairmanship (2015-2017) Football Whispers generated substantial income from advertising, and from the separate professional services we offered to media organisations, football team press officers and sports journalists, alongside their service to fans” [S7, S8].

The company expanded into other sports and now operates as All Sports Whispers to reflect its involvement in the NFL, NBA, and other organisations. All Sports Whispers employs 35 people and has generated over £1.8m in revenue since its founding in 2017 [S9].

5. Sources to corroborate the impact

Confidential email from the Product Lead – Marketing at PHE (2020). Corroborates user download data for the Active 10 app and how it has helped the PHE campaign.

Confidential testimonial from the Special Projects Manager at the Eastern Alps District River Authority (2020). Corroborates the role of Sheffield’s technology in the evacuation of the city of Vicenza and its adoption by the national and regional governments of Italy to meet the requirements of the European Directive 2007/60 and the Water Framework Directive (2000/60) of the Eastern Alps District.

The Times article “Italian city is evacuated for British bomb”, 26 April 2014 (subscription required). Reports number of citizens evacuated in Vicenza whilst a recently discovered Second World War bomb is defused (paragraph 4). (Accessed 24 Feb 2021). https://www.thetimes.co.uk/article/italian\-city\-is\-evacuated\-for\-british\-bomb\-q2wgxtpkrg9

Confidential testimonial letter from the Chief Innovation Officer, Director and Co-founder at The FLOOW Ltd (2020). Corroborates how Sheffield’s research was used to a) influence the BCC’s cycle routes, b) provide a number of UK councils with precise data to allow for the development of improved local traffic management plans and c) develop new to market products for The Floow. The end users of The Floow’s products and the economic impact of these products are presented.

Confidential testimonial letter from the Director of Insight and Knowledge at the Active Wellbeing Society (2020). Corroborates the critical role that Sheffield’s research played in achieving the charity’s objective and allowing them to become one of Sport England’s 12 local delivery pilots.

Football Whispers brochure (scan of hard copy only) confirming the importance of the University of Sheffield’s involvement in the Football Whispers product (2016).

Confidential testimonial letter from the Founder and former CEO of Football Whispers (2020). Corroborates the critical role Sheffield’s research in Football Whispers product launch.

Various media pages reporting on Football Whispers:

Sky Sports (2016) (Accessed 7th Aug 2020). http://bit.ly/3exKI3C

Football Whispers award nomination (2017) (Accessed 2nd Oct 2020). http://bit.ly/3qFGlWo

ESPN (2018) (Accessed 2nd Oct 2020). https://es.pn/3bH898s

The Sun (2018) (Accessed 2nd Oct 2020). http://bit.ly/3tiPu9j

Company accounts filed at Companies House (2017-2019). Corroborates employment and financial figures for All Sports Whispers. (Accessed 29th July 2020). http://bit.ly/2Na3oLh

Truth and trust: combating disinformation and abuse in social media

Submitting institution: The University of Sheffield
Unit of assessment: 11 - Computer Science and Informatics
Summary impact type: Societal
Is this case study continued from a case study submitted in 2014?: No

Download case study PDF

1. Summary of the impact

Sheffield’s big data analytics has probed the veracity, sentiment, and sharing patterns of social media posts and exposed the ways social media can be used and abused to shape opinions about significant political events, such as elections or the Brexit referendum. The methods and findings have been used to promote truth in public discourse, underpinning UK and international policy responses to misinformation and the misuse of social media in relation to various issues, including COVID-19 and online abuse directed at politicians around elections and major national events. The work has also fuelled extensive media coverage on the misuse of social media that has raised public awareness of its risks.

2. Underpinning research

By the beginning of this REF assessment period, work on information extraction (IE) involving Sheffield had established a new orthodoxy for automated data capture from unstructured and semi-structured text. Shallow analysis methods that combined finite state and statistical pattern matching made it possible to identify entities, relations, and events in sources such as news articles to a much higher level of accuracy than previously. At this time, the Sheffield research programme centred around GATE (a General Architecture for Text Engineering, our open-source software architecture for IE **[R1, R2]**) and published results that have attracted more than 10,000 citations.

In the 2000s and 2010s, Professor Kalina Bontcheva and her team adapted and extended these shallow analysis methods to the specific case of social media, showing how the structured and contextual elements of this new textual form can boost the accuracy and richness of data capture with IE. In the same way that hypertext allowed new applications to exploit the web's link structure (e.g. Google's PageRank algorithm), we showed how social media hashtags, @mentions, likes, association networks and location data opened up a wide range of possible analytics. Bontcheva's research on micropost indexing, semantic annotation, search and visualisation demonstrated real-time analytics of web-scale Twitter streams [R3]. As before, the published results were complemented by open-source data and processing infrastructure that enabled experimental repeatability and boosted external take-up.

The team applied the same approach in work on sentiment analysis and rumour and veracity measurement that coincided with the rise of fake news, disinformation, and online abuse [R4, R5, R6]. The new work built on the team’s previously developed taxonomies and application programming interfaces for flexibly and efficiently modelling text processing components, achieving the best compromise between expressive power and efficiency for pattern matching over textual annotations, thereby effectively allowing the combination of statistical counts with derivations based on linguistic intuition. Exploiting the ability to mine vast quantities of consumer-generated media, learn from datasets and recognise groups of words with similar mood/meaning rather than individual terms, Bontcheva's team created datasets of hundreds of millions of tweets and commissioned journalists to annotate the tweets regarding their veracity in relation to a set of rumours prevalent at the time. The team then used crowdsourcing to classify the stance of each tweet towards these rumours (agree, disagree, comment, question, etc.). The datasets the team created were among the very first of their kind available, offering fine detail. Their usefulness is illustrated by their subsequent uptake in other research and applications, with 58 citations for the original dataset paper, 173 citations for the 2017 dataset, and already 48 citations for the dataset published in 2019.

4. Details of the impact

The ability to analyse large volumes of social media streams for sentiment and subject in real time has underpinned national and international policy and informed public debate.

National policy

Bontcheva has contributed to parliamentary inquiries and policy fora on abuse of MPs and disinformation in the submission period.

In December 2017, the House of Commons Digital, Culture, Media, and Sport Committee (DCMS) launched an inquiry into disinformation and fake news. Bontcheva was one of two academics invited to introduce the committee to social media analytics to provide context for them to better understand the subsequent evidence. In addition, she submitted supplementary written evidence. Both were cited in the final report [S1].

In 2019, Bontcheva provided an analysis of abusive tweets directed at MPs from the first six months of that year to the Joint Human Rights Committee inquiry on democracy, free speech and freedom of association, who used the evidence in their report to show the scale of the problem [S2].

In November 2020, Bontcheva was invited to a virtual expert roundtable with social media companies and experts hosted jointly by the Digital and Health Secretaries, focussing on the threat that mis- and disinformation pose to the acceptance and uptake of a COVID-19 vaccine [S3a]. The group explored ways to build long-term processes for working together to tackle disinformation related to COVID-19 and beyond, and understand the impact of existing interventions. This resulted in an agreement with Facebook, Twitter, and Google on new measures to limit the spread of vaccine misinformation and disinformation, and to help people find information about any COVID-19 vaccine [S3b]. Bontcheva was then invited to attend the first Counter-Disinformation Policy Forum in December 2020, which was set up by the Minister for Digital and Culture and brought together industry, civil society, and academia to develop a collective response to the evolving threats to the information environment [S4].

Bontcheva developed a close collaborative relationship with policymakers at the DCMS. She conducted a longitudinal study on the abuse of MPs and candidates leading up to the 2019 general election to inform the government response to this increasingly prevalent issue. The DCMS stated, “ Sheffield University’s research in this area, and particularly their abuse work spanning across 2015, 2017, and 2019 elections has been extremely valuable and has contributed towards our initial policy development” [S5]. The study also provided an intuitive web-based visualisation to enable policy makers to track online abuse directed at UK politicians in real time, “DCMS will be the primary beneficiary, but other Government departments – including the Defending Democracy team in the Cabinet Office, who protect candidates during elections, and Home Office colleagues, who look after the physical security and protection of MPs and monitored MP abuse during the campaign – will also benefit” [S5].

Bontcheva continued her collaboration with the DCMS by using Twitter to compare the COVID-19 pandemic with an election in terms of the abuse of MPs, specifically examining the responses elicited by ministers’ and MPs’ communications. A November 2020 report outlined the topics attracting the most engagement/abuse during the early stages of national lockdown (up to 25 May), with research covering June 2020 scheduled to report in Q1 2021 [S5].

International policy

In March 2020, due to the potentially fatal results of the spread of disinformation about the COVID-19 pandemic – what the WHO described as a “ massive infodemic” – UNESCO commissioned Bontcheva to co-author two policy briefs. She drew on her previous research to: provide a detailed analysis of the types of viral disinformation helping to drive the pandemic; investigate how individuals, the news media, internet communications companies, and governments are responding to contamination of the information ecosystem; offer rich food for thought about actions undertaken to combat the disinfodemic; and assess the potential risks associated with restrictive measures and provide recommendations to align crisis responses to international human rights standards on access to information, freedom of expression, and privacy. On the 24 April 2020, these policy briefs were published in three languages on the UNESCO website, receiving 15,487 visits by the end of December 2020 [S6a, S6b].

Bontcheva expanded on these briefs by co-editing and contributing to ‘ Balancing Act: Countering Digital Disinformation While Respecting Freedom of Expression’ for UNESCO in September 2020 [S6c]. The report features global-scale, comprehensive analyses as well as sector-specific actionable recommendations and a 23-point framework to test disinformation responses. It is available on the International Telecommunications Union and UNESCO websites and has received a combined 8,408 unique visits by the end of December 2020 [S6b, S6d].

In June 2020, the team began working with First Draft, an influential international network of journalists, news organisations, policymakers and social media platforms (including the BBC, Facebook, Twitter, and Google) who promote integrity in the world’s information ecosystem. Sheffield contributed technical knowledge and data visualisation expertise to create a novel conceptual framework centred around identifying vulnerabilities and areas requiring intervention, rather than looking for viral misinformation. Sheffield solved problems such as how to calculate a ‘data deficit’ number, and make that readable for humans in numbers and visualisations, resulting in the First Draft COVID-19 Debunk Dashboard [S7a] and accompanying in-depth report [S7b], which launched worldwide in July 2020. Sheffield also created a uniquely comprehensive database of fact checks that First Draft used for further editorial work [S7c]. First Draft’s Impact and Policy Manager stated, “Throughout, they fed into our thinking around what data deficits are, how they can be measured, and how those measurements can be represented in an actionable way for users”. He explains that the concept of ‘data deficits’ has been adopted in most major disinformation initiatives and says that the research has contributed to “an important, broader shift in the problem-definition of the misinformation field” [S7d].

Increased public awareness of the use and misuse of social media

Bontcheva’s research has made it possible to interrogate millions of tweets about different topics and extract more nuanced, granular insights and quantitative data than ever before . An ongoing collaboration with journalists at Buzzfeed News led to three key articles (below) being reported widely by national and international print, broadcast, and online media [S8].

Attitudes towards key issues in the Brexit referendum (December 2016), an analysis of 3 million tweets about the Brexit referendum, gave valuable insights into the concerns and online behaviours of both “remain” and “leave” supporters, and showed that immigration was by far the principal concern of leave voters despite attempts by mainstream politicians to play down its importance.

Abuse directed at politicians on Twitter (July 2017) was picked up by BBC2 in a special report for its flagship programme Newsnight, watched by some 200,000 people. The article was reported widely in local, national, and international print and online media as well as national radio and television news, reaching an estimated 149.3 million people [S8]. In July 2017, the Committee for Standards in Public Life (tasked with investigating abuse of public office-holders through social media) included this Buzzfeed report in the evidence it examined [S9].

Suspect activity amongst Brexit Party Twitter followers (May 2019) revealed interesting connections between suspicious Twitter accounts. In particular, it identified an extensive fake network whose component accounts worked together to “amplify” pro-Brexit messages, providing clear evidence of the existence and modus operandi of such social media networks. The article’s author, the senior political correspondent for Buzzfeed News, stated that the underpinning research provided “ the case study I needed to illustrate the problem. It was the first time such a network had been mapped, and the discovery made a significant contribution to the final article. […] My article aimed to raise public awareness of such misinformation, and to expose the fact that, around a very important democratic event, certain anonymous actors were creating a lot of noise on social media that was not genuine” [S10]. The story was picked up by outlets with a combined estimated reach of 17.7 million [S8].

5. Sources to corroborate the impact

Final report from House of Commons Digital, Culture, Media and Sport Committee inquiry on Disinformation and ‘fake news’ (2019). Corroborates the use of Sheffield’s oral (page 101 Q1-51, 2017) and written (page 59 para 206) evidence. (Accessed 16th Jun 2020). https://publications.parliament.uk/pa/cm201719/cmselect/cmcumeds/1791/1791.pdf

Report from Joint Committee on Human Rights inquiry on Democracy, freedom of expression and freedom of association: Threats to MPs (2019). Corroborates the use of Sheffield’s written evidence (page 16 para 35, page 39 para 100 and page 62 item 24). (Accessed 1st Feb 2021). https://bit.ly/2OSPQ7I

Combined: DCMS & DHSC vaccine disinformation roundtable:

Confidential personal invitation to the round table of experts (2020)
Government statement following the round table outlining the agreement reached (2020). (Accessed 29th Jan 2021). http://bit.ly/3bF6jF8

Confidential personal invitation to the first Counter-Disinformation Policy Forum (2020).

Confidential statement from Policy Advisor at DCMS (2020). Corroborates the value of Sheffield’s research to DCMS.

Combined: UNESCO information:

Two UNESCO policy briefs co-authored by Professor Bontcheva on COVID-19 disinformation (24th April 2020). (Accessed 7th Oct 2020). http://bit.ly/3vk88iE
Report for UNESCO website viewing figures for S6a & S6c up to 31st December 2020.
UNESCO report co-edited and contributed to by Professor Bontcheva “Balancing Act: Countering Digital Disinformation While Respecting Freedom of Expression” (18th Sept 2020). (Accessed 15th Dec 2020). https://en.unesco.org/publications/balanceact
Email confirming the International Telecommunications Union website viewing figures for S6c up to 31st December 2020.

Combined First Draft information:

COVID-19 Debunk Dashboard (Accessed 29th Jan 2021). https://datadeficits.firstdraftnews.org/
In-depth report citing the use of Sheffield’s research. (Accessed 29th Jan 2021).https://firstdraftnews.org/long\-form\-article/data\-deficits/
First Draft article based on the fact checks database produced by Sheffield. (Accessed 18th Mar 2021). https://firstdraftnews.org/latest/the\-first\-six\-months\-of\-the\-pandemic\-as\-told\-by\-the\-fact\-checks/

d. Confidential testimonial from the Head of Impact and Policy at First Draft (2021). Confirms Sheffield’s contribution to the collaboration.

Summary of media reach of all three Buzzfeed stories from

Committee of Standards in Public Life 17th report, “Intimidation in Public Life” (2017). Cites the Buzzfeed article (p43). (Accessed 16th June 2020). https://bit.ly/3tdSCDm

Confidential testimonial statement from the senior political correspondent for Buzzfeed News (2020). Corroborates that Sheffield’s research provided the fundamental data this article was based on.

Impact case study database

Search and filter

Filter by

Commercial impact in the rapidly growing market for automatic voice recognition ...

1. Summary of the impact

2. Underpinning research

3. References to the research

4. Details of the impact

5. Sources to corroborate the impact

Enhancing quality and productivity and reducing costs of machine translations

1. Summary of the impact

2. Underpinning research

3. References to the research

4. Details of the impact

5. Sources to corroborate the impact

Shaping international policy and public debate on lethal autonomous weapons systems ...

1. Summary of the impact

2. Underpinning research

3. References to the research

4. Details of the impact

5. Sources to corroborate the impact

Societal and commercial benefits of large-scale data collection and analysis

1. Summary of the impact

2. Underpinning research

3. References to the research

4. Details of the impact

5. Sources to corroborate the impact

Truth and trust: combating disinformation and abuse in social media

1. Summary of the impact

2. Underpinning research

3. References to the research

4. Details of the impact

5. Sources to corroborate the impact

Filter by higher education institution

Filter by unit of assessment

Filter by continued case study

Filter by summary impact type

Filter by impact UK location

Filter by impact global location

Filter by underpinning research subject