Impact case study database

The impacts described here are the result of the EPO's sustained engagement with diverse groups of electoral stakeholders throughout the REF assessment period. This includes electoral commissions, governments, polling agencies, electoral observers, and the High Court. The research has been used to make concrete changes to the running of elections (e.g. to polling station organisation and ballot paper design), developed the skills and capacities of those working on elections, informed changes to polling methodologies, and shaped and encouraged the recognition of best practice among electoral practitioners. These changes have contributed to an improved experience of elections, notably among disabled, first-time, and minority voters, and to greater accuracy in election polling. Details of these impacts are described below.

Optimising electoral procedures for voters in the Palestinian Territories

In May 2015, Bruter and Harrison led a two-day training session for senior staff of the Central Electoral Commission (CEC) of the Palestinian Territories [A]. This session was funded by the United Nations Development Programme (UNDP) to strengthen democracy in the Palestinian Territories and built upon earlier engagement with the CEC that also included a smaller capacity-building event in London in 2015. This particular session addressed a variety of electoral psychology issues identified as important to the Palestinian context - including social divisions, gender inequalities, and civic education - and considered how these interact with, and might precipitate a change in approach to, electoral management [1] [3] [4]. The CEO of the CEC credits this session - allied to other insights from EPO research on electoral ergonomics - with having prompted a shift in the CEC's role " from mere implementer of elections processes to more open and proactive roles promoting [the] democratic process in full partnership with the community it serves", particularly evident in a changed " approach to address first-time voters and [introduction of] a new design of the polling stations to respond to the needs of people with disabilities" [A].

Subsequently, electoral reforms enacted in 2016 have introduced new laws on candidate eligibility, allowing greater gender representation and lowering the minimum age requirement, both initiatives intended to promote greater feelings of inclusion and identification among women and younger voters. Technical changes have led to the redesign of ballot papers and polling stations to better accommodate the needs of disabled voters. And the establishment of a new Electoral Awareness Fund and outreach initiatives led by civil society organisations form part of a drive to increase participation in the forthcoming 2021 municipal elections.

Addressing these various initiatives, the CEC CEO has said: " We at the CEC firmly believe that using insights from the Bruter and Harrison research [has] contributed to better understanding [of] the way we serve voters in elections regardless of age, gender or disabilities, and will lead to a better experience and participation of the Palestinian citizens in the coming years". These benefits extend to 2.2 million voters in the municipal elections [A].

• Ensuring equitable elections and effective access to the vote for all [text removed for publication] citizens

At the request of [text removed for publication], Bruter and Harrison provided expert advice to [text removed for publication] on proposed reforms of the country’s electoral system. Their submitted report [B] convinced the government to limit its planned reforms on the basis that they would increase inequality in access to the vote for some minorities and be psychologically unacceptable to parts of the population, resulting in a compromise reform in December 2014.

The government’s proposal put forward two alternative plans based on a mixture of plurality and best-loser system, with a top up proportional allocation based on an “unreturned votes elect” system. Bruter and Harrison designed six institutional tests and six psychological tests that should be met by any viable electoral system reform and simulated the impact of the two electoral system reforms under consideration. Following the collection of electoral psychology data, Bruter and Harrison’s analysis, firmly grounded in the underpinning research [1] [2], showed that the proposed reforms could increase unfairness, negatively affect the representation of some of the country’s minorities, entrench gender inequality, and be viewed negatively by - and lack the acceptance of - [text removed for publication] [B]. The original reform plan was consequently shelved by the Prime Minister and replaced by cross-party discussions.

• Providing expert witness advice in an Irish constitutional referendum case

Bruter served as Expert Witness in cases tried by the Irish High Court and the Irish Supreme Court on the impact of the Irish Government’s framing of the Referendum on Children’s Rights on voters’ behaviour. (This referendum was held to approve an amendment to the Constitution of Ireland which inserted clauses relating to children's rights and the right and duty of the state to take child protection measures.) The government’s campaign was accused of being biased in favour of a “Yes” vote (i.e. approval of the amendment) and the Courts were asked to judge whether the campaign was, in fact, neutral and thereby constitutional (first case), and also whether it could have materially affected the outcome (second case). The government lost the first case in a landmark judgment but won the second (i.e. the referendum result was not overturned) in a final Supreme Court judgment of 24 April 2015. Bruter’s evidence [2] [3] figured prominently in proceedings, as noted in the judgment delivered by Mr. Justice John MacMenamin: “ the appellant sought to base a substantial part of her case on Dr Bruter’s conclusions, reached following his analysis of the raw data” [C, para. 48 ]. Bruter’s expert evidence is referenced extensively throughout the judgment [C, paras. 40-61 ].

Improving political polling methodologies to more accurately forecast election results

The EPO has worked regularly and extensively with Opinium, the political polling agency, throughout the current REF assessment period. They have worked together on numerous international electoral survey studies, with the EPO providing research input on the design of sample frames, the scripts for questionnaires, and on the deployment of surveys to nationally representative samples of citizens of various countries, at various points in the election cycle [D]. Meetings with the Opinium electoral forecasting team were held to discuss weighting variations based on EPO findings on electoral psychology, including developing new measures identifying which voters are most likely to change their minds in the run up to an election, and controlling for registration before applying weightings to avoid over-estimating youth abstention [1] [3].

The Chief Executive of Opinium considers the learnings from this collaboration to have been key to the Opinium political polling team having been the most accurate research agency at the 2019 General Election and one of just two agencies to have accurately predicted the outcome of the EU referendum in 2016: " Without a doubt, credit must be given to the inspiration and learnings we have gained from the EPO which has undoubtedly influenced how we as an agency look to understand the voting psychology of the British voters. For example, insights from EPO’s research have been critical in taking into account psychological variables in panel and sample design. Conversely, their work on question phrasing has helped us optimise variables that we use in political polling and barometer studies. Most importantly the importance of personality and emotion, both of which now have a place in our own methodology" [D]. Opinium carry out frequent election surveys in the UK and the US and provide regular political polling for national and international media, notably The Observer.

Influencing best practice in electoral management

EPO research has also been used by electoral stakeholders who work internationally to improve the practices and processes of elections [1] [3]. For example, the Delian Project is a non-governmental organisation dedicated to helping jurisdictions implement positive change in the democratic voting process. Managing Director, Dr Allan Best describes how engagement with the EPO has prompted the Delian Project to place greater emphasis on electoral psychology in the course of its work: “ one of the biggest ways that [Bruter] and [Harrison]’s research has really opened the eyes of senior individuals in election management bodies is the fact that in EMBs we tend to look more at the logistics of an election as opposed to the science and emotion of an election. This is critical. The fact that they have been able to bring analytical and critical-based research, atmospherics that actually let us understand what is the target audience, what are they looking to glean from an election, how do we increase voter turnout, and more importantly how do we make a successful election that actually is relevant to the voter in a way that is human, that is emotive, that is actually something that caters to them as opposed to something that is an exercise in logistics” [E].

Influence on electoral observation practices in the Falkland Islands

This work has also had an effect on electoral observation missions. It was put into practice when the Delian Project served as an election observer for the independence referendum in the Falkland Islands. Using questions developed by the EPO [1] [2], the Delian Project was able to gain insights into the unique emotion of an independence referendum, even among a very small population of 1,600. Best has since said that the answers to those questions on voters’ electoral experience were “ very useful to us to understand that we have to be very cognisant of emotions” [E].

Establishment of new awards fostering and rewarding best practice

Bruter and Harrison have shared best practice in election processes [1] [2] with election management bodies (EMBs) through the International Centre for Parliamentary Studies (ICPS), which promotes effective policymaking and good governance through capacity-building and specialises in electoral affairs [E]. Arvind Venkataramana, former CEO of the ICPS, has attested to the usefulness of the underpinning research in helping EMBs to improve voter satisfaction specifically. According to Venkataramana, “ where [Bruter and Harrison] make a significant impact is to improve the overall voter experience, which, in turn, results in greater voter turnout, which is a success for election management bodies.” EMBs have found the research “ very useful to implement strategies that they’ve been wanting to implement for a while, especially to attract first-time voters, to improve access for disabled voters, adapt new technologies in their regions, and also they’ve used their insights to design ballot papers more innovatively” [E].

After Bruter presented findings on electoral psychology and electoral ergonomics at their annual international conferences, the ICPS created four new categories in its Electoral Stakeholder Awards, to better recognise outstanding work by EMBs. The new award categories were in Electoral Ergonomics, Citizens Engagement, First-Time Voters, and Accessibility. These have been bestowed every year since December 2013, encouraging greater innovation among EMBs in trying to improve the overall voter experience. Venkataramana has identified the Electoral Ergonomics category, in particular, as being among those to receive the most - and most creative - nominations [E]. Bruter served on the awards committee until 2019.

Influence on the production of a new television series

Findings from the EPO research [2] [4] have also informed the writing of a new Apple Television spy series. Life Undercover, starring Brie Larson, dramatises a fictional French election, set against the background of an extremist threat and electoral disruption. A production team research assistant discovered the Bruter and Harrison research and invited them to a meeting in Los Angeles. Briefing the team of nine writers and producers, Bruter and Harrison offered expert input on the dramatic scenario, including insights on voter psychology. Megan Martin, the lead writer, says: "[ Bruter] and [Harrison]'s insights into the transmission of extremist ideology, and the psychology of voters were invaluable...As the writers continued to shape story lines, [Bruter] and [Harrison]'s time with us grounded the narrative in ways that continued to surprise and impress me" [F]. Production has been halted due to the Covid-19 pandemic.

Helping to give first-time voters and cognitively disabled voters a voice in the UK, South Africa, and Australia

EPO research has also been used in the course of initiatives intended to give first-time voters and those with learning and cognitive disabilities a greater voice in electoral processes in the UK, South Africa, and Australia [1] [3]. Since 2019, Bruter and Harrison have worked on the All-Party Parliamentary Group on Votes at 16 to help it build a case for the extension of the voting franchise to 16 and 17-year-olds, with a first report published in 2019. Similarly, since 2016, Bruter and Harrison have been working with the South African Independent Electoral Commission (IEC) to test pilot measures intended to make the electoral experience of first-time voters feel rewarding and more likely to motivate sustained electoral engagement [G]. The work involved collaboration with South African youth organisations and one of the IEC Commissioners. The IEC subsequently began running pilot information campaigns in a selection of electoral wards in August 2016.

In parallel, EPO organised two policy labs (in 2018 and 2019) in collaboration with the Australian Electoral Commission (AEC) and the state-level commissions of South Australia and the Australian Capital Territory. Here, research findings focusing on the experience of voters with mental health issues were shared with commissioners and a team of 20+ divisional heads at the AEC. EPO and AEC organised a joint observation of the 2019 Federal Election, focused on the experience of disabled and special-needs voters, in particular. Bruter and Harrison worked with staff in polling stations dedicated to accommodating disabled voters [H].

The underpinning research and its associated software tools have had many diverse impacts. The development of quanteda has led directly to the establishment of a new community interest company, been used by commercial organisations, regulators, and data science professionals to improve their products and services, informed media reporting and analysis, and been widely used in training and development contexts, benefitting learning and understanding internationally.

Establishment of new community interest company and subsequent provision of training internationally

The innovation of the quanteda R package [1] lay in its provision of new tools allowing the implementation of both new text analysis methods developed in QUANTESS, and of dozens of existing methods for which there was substantial demand but only limited tools. To facilitate more efficient dissemination of and engagement with the quanteda suite of text-analysis software tools [1] [2], and to ensure their continued development and long-term sustainability, the Quanteda Initiative - a non-profit, community interest company - was established by Benoit and Watanabe in January 2018 [A]. Since then, a series of QI-branded training workshops have been held internationally. Up to 31 December 2020, nine training events were delivered for clients in six countries (Germany, Norway, Switzerland, Australia, Japan, and Ireland) [A]. Tutorial materials have been prepared in five languages, including Chinese, Japanese, and Hindi. Workshops have typically accommodated groups of around 20 per session. Revenues have subsequently been reinvested into the company to cover the costs of web servers, cloud computing, software licenses, and the services of the three private contractors who have been employed to deliver these expert tutorials.

The quanteda suite of text-analysis software tools itself remains completely free to use. And because it is open-source, and therefore completely open to expert scrutiny, the software can be trusted by users. quanteda follows a software-as-service model which means that all computation is done on a cloud server; as such, it requires only access to a web browser to use it [B]. quanteda is designed to be accessible to non-expert user.

Uptake has been significant: by 31 December 2020, it had been downloaded more than 580,000 times [B]. This includes by stakeholders in the private sector, by political analysts and commentators at international media organisations, and by data science professionals. User testimonials are available on the quanteda website [C]. Details of its various impacts and initiatives are included below.

Benefits for commercial users and a communications regulator

In March 2019, Swiss start-up company Grünenfelder Zumbach began using quanteda. Grünenfelder Zumbach provide data analytics, policy evaluation, and consulting services for a diverse client base in Switzerland. The speed and stability of the quanteda tools have contributed to the more efficient, improved service Grünenfelder Zumbach now provides to its customers, as attested to by a member of its team:

“We are a young company based in Zurich and specialise in policy evaluation. Wherever possible, we try to leverage the potential of data science methods in our projects. quanteda is our first choice when dealing with large amounts of text, because it is fast and stable, it offers many possibilities to quickly find patterns and it provides excellent interfaces for further analysis. Our customers are always impressed [by] how much insight can be gained from large amounts of unstructured text data in short time. Without quanteda our job would be less easy and definitely less fun.” [C]

Facebook has also used quanteda. Its Core Data Science team conducts large-scale, global, quantitative research to gain deeper insights into how people interact with each other and the world around them. The team’s findings directly inform improvements to the user experience for Facebook’s 2.7 billion monthly active users. A Research Scientist has confirmed the team’s use of quanteda to conduct its analysis:

"Data scientists at Facebook have used quanteda for a variety of analytic purposes, both as a tool for exploratory analysis and for measurement in relation to production implementation. For example, it has been used to conduct text analysis to measure the linguistic diversity of entity recommendations." [D]

Ofcom is the UK’s government-approved communications regulator. Ofcom has duties and powers in topics such as media literacy and media plurality, amongst others. Its Economics Group used quanteda to conduct analysis during one of its research projects, as confirmed by an Ofcom economist:

“Economists at Ofcom used quanteda to produce summary statistics, explore possible applications of natural language processing tools, and conduct text analysis – for example it has been used to uncover the topics covered in various documents.” [E]

Informing media analysis and reporting

quanteda’s text analysis tools have been used to inform media commentary and analysis. In February 2020, i24 news (an Israeli national news and current affairs platform) used quanteda in an experimental analysis which compared the texts of two key peace initiatives [F]. The first of these was the Peace to Prosperity plan, unveiled by the Trump administration in January 2020 and embraced by Israel; while the second was the Arab Peace Initiative of 2002, put forward by Saudi Arabia and subsequently garnering full Palestinian support. i24’s quanteda-enabled analysis was able to generate new insights, such as the Trump plan’s use of language focused on economic initiatives and its glaring absence of the word “peace”, for example.

In July 2020, The Washington Post used the quanteda Lexicoder Sentiment Dictionary in analysis of whether Russian disinformation campaigns were targeting African Americans [F]. This analysis built on a 2018 Senate Select Committee on Intelligence report which concluded that Russian information operatives seeking to interfere with US elections were mainly targeting African Americans. The Washington Post used quanteda to analyse almost 40,000 purportedly divisive tweets sent from Ghana and Nigeria between June 2017 and March 2020 and believed to be associated with Kremlin-backed sources. Again, the newspaper’s quanteda-run analysis was able to generate new insights, showing how these malicious accounts tweeted a mixture of sentiments to cultivate followers and manipulate US narratives about race, racial tensions, and police conduct.

Benoit’s analysis of the use of Twitter in the run-up to the Brexit referendum [6] was used to inform reporting by the UK’s Daily Express. In January 2017, some six months after the vote, the Express quoted comments made by Benoit at an LSE public event and drew upon the EUENGAGE research findings specifically to describe the key impact of more “positive” Brexit campaigners, who were said to have been “ more positive, more certain and more forward-thinking than those supporting Remain, a scientific study of 26 million tweets has revealed” [F]. Similarly, in April 2017 Benoit himself published an article in The Washington Post on the question of whether US political discourse is becoming “dumber”, in which findings from the State of the Union address analysis [4] were discussed extensively.

Impacts on curricula, teaching, and learning

quanteda has provided new tools for teaching in universities around the world, having been added to the curricula of higher education courses in the US and Germany. The University of North Carolina at Charlotte uses quanteda in the “Visual Analytics” course of its Data Science and Business Analytics Program [G]. In Germany, the University of Frankfurt uses quanteda in its Quantitative Text Analysis course [G]. Additionally, libraries at higher education institutions have incorporated quanteda into their own training and support resources. Notable examples include Princeton University, University of Virginia, WZB Berlin Social Science Centre, and University of Mannheim [G].

Testimonials from quanteda users also attest to the software’s successful applications to teaching. A user at the University of Bremen describes quanteda as: “ an excellent resource for both research and teaching that complements R in a way that is invaluable to me…It is far superior to related packages and so well documented that I use it centrally when teaching text mining” [C]. Another user, from Johns Hopkins University (Baltimore, USA), explains: “ I recommend quanteda to students in the Johns Hopkins University Data Science Specialization Capstone project. […] quanteda provides a rich set of text analysis features coupled with excellent performance relative to Java-based T packages for text analysis” [C].

quanteda’s utility for teaching led to its inclusion in SAGE Publications’ Research Methods series, the company’s flagship programme of research support resources. Benoit recorded a video explainer for use in “An Introduction to the Quanteda R Package” module, while the software tools have been used in other modules, including "Data Management: Preprocessing with Quanteda" and "Researching Brexit Tweets Using Quanteda Software". quanteda also forms part of the syllabus of the SAGE Campus online course, “Fundamentals of Quantitative Text Analysis” [H].

The citation accompanying the 2020 Best Statistical Software Award quanteda received from the Society for Political Methodology also expressly mentioned its facility for use in teaching and training. The committee wrote: “[quanteda’s] extraordinary documentation not only makes it accessible for researchers from a variety of backgrounds, it also facilitates the further creation of packages and utilities, and supports its usage in teaching and training… quanteda's innovation, accessible documentation, and functionality are testaments to the collaborative efforts of both junior and senior scholars that can serve as a model for future software development” [I].

Impacts on data science professionals

In September 2020, Analytics India Magazine named quanteda as one of its top ten R packages for natural language processing, noting its “ fast, flexible, and comprehensive framework for quantitative text analysis in R” [J]. In October 2019, Towards Data Science included the “ all-encompassing” quanteda in its list of the “5 Packages You Should Know for Text Analysis with R”, describing it as “ the go-to package for quantitative analysis” [J]. Towards Data Science is an online tech journal based in Canada, with 500,000 subscribers.

quanteda has featured on the R-bloggers site, affiliated to the Foundation for Open Access Statistics. Research carried out in 2019 used quanteda to analyse 18,000 music reviews posted on the Pitchfork website, with the objective of understanding the linguistic signals of album quality, identifying those adjectives most associated with positive and negative reviews [J].

This research has been instrumental in promoting civil society-led reconciliation and the importance of regional, restorative approaches to post-conflict justice as a complement to the national-level, retributive approaches. Stakeholders to have benefitted from engagement with this research include UK foreign policymakers, the EU, Balkan civil society organisations (most notably RECOM), and the United Nations.

Influence on UK foreign policy related to the Western Balkans

Through a series of interventions advocating for the inclusion of civil society actors as part of a regional approach to reconciliation in the Western Balkans, Kostovicova's research has influenced the development of UK policy relating to the legacy of conflict in the region. This was achieved by first ensuring the question of reconciliation was included on UK policymakers’ agenda [1] [2], before subsequent sustained, committee-level engagement culminated in the Balkan states' unprecedented joint commitment “ to overcome the legacy of the past through promoting reconciliation” [D].

The UK-sponsored Summit was part of the so-called “Berlin Process”, which has supported the EU integration process of the Balkan countries since 2014. In July 2017, Kostovicova was part of a small group of experts who met with the House of Lords International Relations Committee to discuss the scope of its new Western Balkans inquiry. This meeting, and Kostovicova's contributions on reconciliation, unresolved legacy issues of conflict and violence, and their impact on rising nationalism [3] [4] [5], were key to the question of conflict-resolution and inter-ethnic relations being included among the inquiry's questions of interest [A, p. 61 ], which also included geopolitical and governance challenges.

In September 2017, Kostovicova submitted written evidence to this committee on impunity for war crimes (especially wartime sexual violence) [4] [5] and the role of civil society in post-conflict justice and peacebuilding [2], outlining the perils of an inherently unstable region with “peace without reconciliation”, where a lack of “consensus on the causes and nature of violence committed in the conflicts” feeds into post-conflict nationalism. This evidence was included in the committee’s final report [A, p. 12 ], which recognised the importance of reconciliation in its recommendations: "[ t]he UK must continue to support political capacity-building, strengthening the rule of law, gender equality, women’s participation and post-conflict reconciliation" [A, p. 36 ].

Kostovicova made further contributions to UK policy approaches to the Balkans by submitting evidence on post-conflict justice and reconciliation to the House of Commons Foreign Relations Committee Balkans Inquiry, and its “Global Britain and the Western Balkans” report [B]. Findings from Kostovicova’s research on civil society and its role in peacebuilding (including reconciliation) in the region [3] were cited in the report, which quoted her evidence on the “ weakness of an already fragile civil society [that] is under increasing pressure from increasingly authoritarian leaderships”, which she discussed in the context of the peacebuilding, stabilisation, and rule of law [B, p. 6 ]. In its response to the report, the government outlined its policy on the Western Balkans [C], particularly on the recommendation contained in the committee’s report [B] that it should outline how it wants to achieve its policy goals in the Western Balkans - with “specific milestones and metrics for success” - including in the area of civil society: “ Another key metric will be the degree to which the region has resilient institutions that underpin the rule of law, and enable inclusive and democratic societies with a free press and an engaged civil society” [C].

The Western Balkans Summit was held in London on 10 July 2018. In spite of the Balkan states elites’ general resistance to coming to terms with the criminal past, the UK’s approach in the run-up to and at the Summit itself, informed by the political complexities and sensitivities of dealing with accountability for mass atrocity perpetrated in the region, resulted in the unprecedented joint commitment of all Balkan states “ to overcome the legacy of the past, through promoting reconciliation and good neighbourly relations” [D], with specific joint commitments on missing persons and establishing facts of war crimes against all victims, including victims of conflict-related sexual violence [D]. This declaration is important as it establishes an inter-state-level commitment to continue to work on the legacy of war and work towards reconciliation.

Before and after the London Summit, the UK distinguished itself among EU member states by becoming a firm champion of the region’s so-called “legacy issues”, marked by limited progress in addressing war crimes, which had undermined efforts towards peacebuilding and stability in the region. The government outlined how it would subsequently be “working closely” with the British Group International Parliamentary Union (BGIPU), which “ will empower parliamentarians from the region to scrutinise their respective governments on the commitments they made at the Summit to help ensure that all such pledges are delivered” [C]. Again, Kostovicova was invited to present her research [1] [2] and engage on issues related to legacy and reconciliation with representatives of the UK parliament at an event with parliamentarians from all Western Balkan countries, organised by the BGIPU, in September 2018. This seminar represents the operationalisation of the UK’s policy towards the Western Balkans, both in general and during the post-Brexit period.

Enhancing the advocacy capacity and effectiveness of civil society actors, especially RECOM, benefitting its interactions with the EU

RECOM is a regional network of civil society organisations which seeks to establish a regional fact-finding commission about war crimes and other serious violations of human rights committed in the former Yugoslavia. One of its principal means of achieving this is through diplomatic representations to the EU. The Kostovicova research has proved vital to the robustness of these representations, and to the effectiveness of interactions with the EU [1] [2] [3].

Kostovicova has engaged regularly with RECOM throughout the REF assessment period, sharing findings from her analysis of regional-level post-conflict justice debates, including the representation of women in the process [2] [6]. The RECOM Process Coordinator has noted how the Kostovicova research “ has increased the advocacy capacity of the RECOM Network, provided evidence of the effectiveness of a regional approach to post-conflict justice, and enhanced the evidence base to support our efforts to establish a regional fact-finding commission, particularly in the context of RECOM's interactions with the EU” [F].

Accordingly, at the Poznań Summit in July 2019, the successor summit to the 2018 London event, the Chair’s conclusions explicitly took up the issue of reconciliation and RECOM’s role in it: “ The legacy of the past creates a particularly acute sense of unfairness to victims still looking for justice and deep divisions between neighbours and communities. Action is needed at all levels across the region to reduce these impediments and to develop a process of confidence-building and reconciliation to fully unleash its potential. […] Participants underlined their commitment to reconciliation, including through the RECOM initiative” [I]. This declaration indicates the embedding of legacy issues within the Berlin Process initiated at the London Summit.

Further impacts from Kostovicova’s research emerged from direct engagement with RECOM [6]. At a public event involving some 200 civil society representatives in Sarajevo, Bosnia and Herzegovina in January 2018, attendees heard the first scientific evaluation of the regional versus national debates in which they had participated, as well as evidence of a need for more opportunities for women to present their views and concerns [1] [2] [6]. This was significant as it was later said to have reinvigorated both the commitment of victims’ family members and of human rights activists to continue to press for the regional route to justice and reconciliation (which still faces opposition from some quarters). The RECOM Process Coordinator has confirmed that the research " provided important evidence of [the] reconciliatory potential of the RECOM process…[This] finding was motivating for their engagement in the advocacy and [for] keeping this issue on the public agenda despite opposition from authorities who prefer to avoid confronting the past" [F]. The research findings on gender inequalities reaffirmed RECOM’s endorsement of a more inclusive approach thereafter, as noted by its then Outreach and Partnerships Director: “ It was Dr Kostovicova’s research on gender issues within the broad consultative process on truth-seeking in post-Yugoslav states that focused our attention more on the importance of a fully inclusive and participative approach in dealing with the past” [J].

Influence on the United Nations' approach to its reconciliation projects - promoting a regional approach and foregrounding women's voices

In March 2017, the United Nations Department of Political and Peacebuilding Affairs (DPPA) and the Conflict Prevention and Peace Forum invited Kostovicova to contribute to a strategic brainstorming session on conflict prevention and peacebuilding in the Western Balkans, hosted by the Diplomatic Academy of Vienna. Here, Kostovicova again highlighted the value of a regional, bottom-up approach to reconciliation led by civil society actors, based on insights from her novel application of quantitative text analysis methods [1] [2] [3]. Following this session, the UN produced and distributed a summary of the discussion, in which it recognised that the “ UN could promote a regional approach to reconciliation as a way to unblock national efforts which remain stymied by the elites” [K].

The Vienna meeting prompted further DPPA interest in a scientific evaluation of UN-funded reconciliation activities which would feed into the UN’s policy process. Further interactions with the Senior Political Affairs Officer at the DPPA took place at events in London and at UN Headquarters in May 2019, held as part of the joint research project with King's and UAL, at which the Kostovicova research was again part of evidence presented and discussed [1] [2] [6]. The Senior Political Affairs Officer has subsequently attested to how Kostovicova’s research insights on gender bias in reconciliation activities have informed his working practices in a way that benefits women and helps to redress gender imbalances in peacebuilding discussions facilitated by the UN. Citing the clear " narrowing [of] women’s deliberative space and opportunities to develop and sustain arguments that reflect their concerns" identified by the research, he has described a changed approach to meetings and events facilitated by the UN: " Your research established conclusively that the sequences of men talking after men are longer than those of women talking after women lead to a gendered outcome...I have benefitted from your insight and evidence-based approach in my work to ensure that women get equal, if not more space, in the workshops" [L]. This new approach has since been implemented in the course of 2019 workshops in Eswatini, organised with the Government of Eswatini, and Tanzania, and with the Southern African Development Community [L].

Sidel's research has made a significant contribution to the Coalitions for Change programme, firstly by shaping its understanding of the political context of the Philippines and how best to work within it, but also by making decisive interventions in specific programme initiatives. These initiatives have led to reforms in disaster risk reduction, education, electoral processes, healthcare, land governance, public transportation and infrastructure, and tax policy. Sidel's analysis of the CfC programme has also influenced the approach of the Australian Department of Foreign Affairs (DFAT) and Trade and The Asia Foundation (TAF) to development assistance and provided evidence and analysis in support of an innovative approach to development - "thinking and working politically" - of interest to a broader audience of practitioners and policymakers in overseas development agencies and multilateral institutions.

Contributions to successful reform initiatives in the Philippines: CfC, funded by DFAT and undertaken in partnership with TAF, is a AUD68 million programme designed in line with experimental and innovative ideas about “doing development differently” and “thinking and working politically”. Instead of providing development assistance to the Philippine government and/or NGOs in pursuit of policy goals, CfC has sponsored a diverse set of advocacy campaigns for reforms across a range of policy areas. These advocacy campaigns have involved the exploration and conceptualisation of the policies themselves, followed by their promotion through the lobbying of government officials and legislators.

Sidel’s work with CfC began at the programme’s inception in 2012. He was invited by the Australian Embassy and the TAF office in Manila on the basis of his research and writings on Philippine politics [1] [2] and expertise on the context of policy reform in the country. His research and writings on Philippine politics have helped to shape the CfC programme’s overall understanding of the political opportunities and obstacles enabling and constraining reform advocacy work at the national and local levels [A] [B] [C] [D].

In 2014, and on the strength of his research and engagement with CfC during those first two years, Sidel was appointed to the Programme Strategic Panel (PSP), set up to robustly interrogate CfC’s analysis and progress and advise the executive-level Joint Steering Committee. As a PSP member, he has played a direct role in internal programme decision-making, reviewing concept notes for new initiatives and work/progress reports on ongoing initiatives, and providing extensive comments and recommendations. His accumulated expertise on Philippine politics and his empirical research and analysis was integral to shaping the programme as its work unfolded, including substantive input to tactics, strategies, and programming decisions. The importance of these contributions is confirmed by the Senior Programme Officer of the CfC programme: " Given Sidel’s knowledge of politics, the Philippine context, and Southeast Asian politics, his contributions to the strategic analysis and tactical approaches of CfC were critical" [A]. Examples of his impact on individual streams of the programme include:

• He conducted an in-depth investigation - through interviews and analysis of documentation - of CRI+D (Coordinating Roads and Infrastructure Investments for Development), an initiative for improving local government decision-making on roads and other infrastructure projects [4]. This investigation identified limitations and loopholes in the design and implementation of CRI+D, most notably the varying input and effectiveness of local chambers of commerce in the provinces and the weak footing of the programme in terms of the national bureaucracy. His analysis helped to push CfC to clarify and strengthen its procedures for local business participation in CRI+D and to institutionalise its reforms within the national government’s budgetary processes for funding of infrastructure projects [E].

• He assisted and advised the CfC-supported electoral reform advocacy group LENTE (Legal Network for Truthful Elections) in designing successive research projects as it drove forward the passage of the Electoral Service Reform Act in 2016, drafted implementing rules and regulations for the new law in 2017, and documented its impact over 2018-2019 [4]. The electoral reform legislation improved conditions for the service of public schoolteachers on local boards of election inspectors, reducing their vulnerability to political influence and interference, thus enhancing the quality of the electoral process [F].

• He provided careful analysis of the role of the CfC-backed reform advocacy group Action for Economic Reforms (AER) in the passage of the 2012 landmark “sin tax” reform [3] [4], repeatedly warned against AER lapsing into mere “backstopping” of the Department of Finance (DoF) in promotion of comprehensive tax reform since 2016, and strongly recommended a renewed focus on promotion of higher cigarette taxes to fund the Universal Health Care (UHC) programme [A] [C]. He thus helped to nudge AER to lobby DoF on this front, leading to government pressure on legislators and passage of a new “sin tax” linked to UHC funding by congress in June 2019, with new revenues estimated at USD800 million as of early 2020 [G].

• He provided strategic advice and analysis to CfC’s Mobility team on urban transport reform initiatives, produced a systemic analysis of the political economy of traffic congestion in Metro Manila [6], and prodded CfC-supported transport reform advocates to rally behind a clear blueprint for the national capital region’s transportation system to meet the challenges of the COVID-19 global pandemic in the spring of 2020. He thus helped CfC in its successful promotion of unprecedented transport reforms - the establishment of a network of bicycle lanes and government service-contracting of buses and jeepneys - by summer 2020, thus providing a basis for safe, socially-distanced public transportation and promoting sustainable solutions to longstanding traffic congestion in the national capital region [A] [D] [H] [I].

• His investigation of CfC’s work in Mindanao and the Sulu Archipelago through interviews, participant observation, and analysis of available documentation produced a critique of CfC’s - ill-conceived and ineffectual - activities in support of the peace process in the southern Philippines [4]. This critique helped to convince the programme to cease support for localised conflict resolution efforts and “security sector reform”, and eventually to terminate its broader involvement in this area of policy work, thus allowing the programme to focus on its core areas of strength and success in policy reform.

In addition to these specific and more targeted interventions, Sidel’s research has also underpinned wider CfC efforts which have assisted the passage of new laws, promulgation of new regulations, and introduction and incorporation of new practices, procedures, and processes into the working of the Philippine government [A] [C]. To date, CfC’s achievements in terms of reforms include the following:

• Introducing and institutionalising new bureaucratic procedures and budgetary provisions for addressing overcrowding in public schools [J].

• Introducing and institutionalising new practices and procedures enhancing accessibility of polling places for people with disabilities in elections [F].

• Extension of the Residential Free Patent Act of 2010 to cover public schools and other government building sites, thus enabling titling of these lands and enhancing their security and eligibility for improvement and/or extension [K].

• Passage of the Agricultural Free Patent Reform Law, streamlining and simplifying the bureaucratic procedures for farmers to obtain titles to agricultural land and to engage in agricultural land transactions [K].

• Securing government commitment to Clark International Airport through funding for a new terminal and rail links, thus alleviating air traffic congestion in Metro Manila [L].

• Introduction of new guidelines for Local Road Network Development Plans (LRNDPs) to include more transparent, participatory, and evidence-based procedures for decision-making on investments in local roads and infrastructure, thus enhancing the implementation of local road projects across the Philippines [E].

The Senior Advisor for Program Strategy at The Asia Foundation has attested to the positive impact of the programme: " CfC has achieved dramatic success in a wide range of sectors in the Philippines on reforms previously thought to be politically impossible, and it has done so in each case faster, cheaper, and with greater sustainability than would have been possible using conventional project modalities" [M].

Evidencing the value to programme funders of this mode of development assistance: Sidel’s research and analysis of the CfC programme has also been crucial to the continuation of its funding and to demonstrating its success to DFAT and TAF. He has particularly promoted the preservation and strengthening of those aspects of CfC which represent the most significant departure from traditional development assistance programming. In the 2014 interim review of CfC and the 2017 extension of the programme under “Phase 2”, the argument in favour of continued autonomy and adaptability for CfC won the day. DFAT has continued to support a programme which does not conform to standard bureaucratic procedures for budgeting, programming, monitoring, and evaluation. Sidel’s independent assessments, analysis, and advice have helped to ensure CfC’s extension and enhanced autonomy of operations [A] [B] [M].

Shaping best practice and influencing the design of other international aid programming: Finally, Sidel’s papers and his book [4] have provided the basis for independent documentation and analysis of one of the most experimental and innovative development programmes in the world. Over the past decade, a growing number of practitioners and policymakers in the development sector have become interested in ideas about “doing development differently” and “thinking and working politically” in development. One example is the Thinking and Working Politically (TWP) Community of Practice, whose regular meetings and steady stream of communications and publications have included senior representatives from leading development agencies. Another example is the influential From Poverty to Power blog of Oxfam Senior Strategic Adviser Duncan Green, which has showcased Sidel’s research [N]. The former Assistant Secretary for Governance and Fragility at DFAT describes how the CfC programme " is now cited internationally as one of the best examples of adaptive governance programming within global development practice and is cited regularly by bilateral development donors and multilateral agencies as an example of both innovation and best-practice" [B]. Sidel’s book is now being used as a standard text at the Overseas Development Institute and in the United States Agency for International Development [A] [O], while his work on CfC is now informing other international aid programming within multilateral organisations, including the World Bank and agencies of the United Nations [B]. Sidel’s research on CfC has thus provided much-needed evidence that support for reform advocacy campaigns can be effective and impactful, and much-needed analysis of the kinds of institutional arrangements, operating procedures, and political circumstances which determine developmental outcomes for reform initiatives such as those sponsored by CfC [B] [M] [O].

Hertog's research has had impacts throughout the GCC countries, most notably in Qatar, which has introduced major changes to its labour market, ending the requirement for migrant workers to obtain their employer’s permission to change jobs. These reforms will deliver benefits to workers, employers, and the national economy. The research has also made a critical contribution to the debate and formulation of labour policy in the UAE, leading to reforms which have resulted in a considerable relaxation of worker mobility laws; and in Saudi Arabia, where an employer-borne levy on foreign workers was introduced to narrow the cost gap to citizen employees in the private sector. Lastly, Hertog's research has also benefitted the World Bank, which is very active in the GCC, advancing its understanding of the segmentation of GCC labour markets and the implications for both local and foreign workers.

Changes to labour and mobility legislation for foreign workers in Qatar

The most significant policy impact of Hertog’s research on Gulf labour markets was delivered through a project for which he functioned as academic lead from 2018 to 2019 on behalf of the International Labour Organization. Since 2018, the ILO has been working with the Qatari Ministry of Administrative Development Labour and Social Affairs to reform migrant worker sponsorship rules in the country. In this context, Hertog prepared a 30,000-word research and policy paper – submitted in May 2019 – detailing the rationale and options for further reforms of the migrant labour sponsorship system in Qatar, with the Qatari government as the main recipient [A]. The project has culminated in extensive liberalisation of sponsorship rules in October 2020, details of which are below.

This policy paper [B] directly draws on and cites Hertog’s research [3]-[6] about the sponsorship system in the wider region and how it can be reformed; it is the key research and policy document for the bilateral ILO-Qatar project. It develops the economic and policy rationale for sponsorship by investigating the costs and unintended consequences of the current system in Qatar in detail, with new empirical data. It also provides a detailed policy package – consistent with the recommendations outlined in [3] and [4] – for migrant labour mobility reform. This paper proposes to narrow the labour rights gap with Qatar nationals both to: a) improve wages, productivity, and working conditions for migrant workers; and b) make it relatively more attractive to employ nationals in the private sector once foreigners and locals enjoy the same rights. It also investigates informal labour markets in Qatar and makes the case that sponsorship reform would reduce these. Finally, it develops a strategy for winning over reform allies in the private sector, namely firms which already provide better working conditions and will therefore benefit from a more fluid labour market in which they can recruit experienced workers locally at lower cost.

The paper [B] is based on field interviews with policymakers, frontline bureaucrats, employer representatives, employers, domestic and private sector workers, local experts, and academics, as well as econometric analysis of individual-level wage data from the Qatari government. More importantly, it draws on a decade of academic work [1-6] on the segmentation of GCC labour markets and its unintended consequences.

The ILO currently enjoys an unusual embedded position in the Qatari government, which it has been able to negotiate as a result of the heavy international pressure that Qatar has been under through both the recent UAE-Saudi blockade and the unfavourable international reporting about labour conditions in Qatar in the context of the 2022 FIFA World Cup. This has created a unique window of opportunity to shape policy in Doha.

As a result of the ILO project, in October 2019 the Qatari government announced that the current labour sponsorship system would be abolished. This was followed in August 2020 by the adoption of a law that allows foreign workers mobility between employers and reforms the termination of employment [C]. Many details of the reform directly correspond to the report’s policy proposals [B], such as the inclusion of domestic workers under the new mobility rules. The reform also incorporates the report’s recommendation to create an initial probation period that ties workers who have newly entered the country to their initial employer for six months – while also creating a “buy-out” option, suggested in the report, that allows other employers to hire a worker before the end of this probation period by financially compensating the previous employer, thereby balancing the interests of all parties while maintaining mobility rights. These key recommendations were first proposed in the underpinning research described above [3] [4].

The new law has also followed the report’s proposal to create a one-month notice period that gives workers sufficient flexibility in moving to new jobs, and the recommendation to put strict limits on the use of “no-compete clauses”, which otherwise could be used to re-establish mobility restrictions. The Qatari government introduced a general minimum wage for all foreign workers in October 2020, which the report (as well as Hertog’s previous work **[3]**) analysed as complementary to the abolition of sponsorship and is expected to naturally improve wage levels.

The ILO has confirmed that Hertog’s paper has been central in providing the policy input to this process [A], highlighting the strong economic and social rationale the paper [B] provides for the reforms and the importance of his previous research in setting this agenda. Qatar has now moved from having the most restrictive migrant labour regime in the region to the least restrictive one, and the intellectual case for sponsorship reform in his paper has been central in facilitating these reforms, affecting more than two million foreign workers [A].

Impacts in Qatar are likely to include higher productivity and wages for foreign workers, with the mean wage gain estimated at 14% in the report (with stronger impacts for lower earners), better skills accumulation, and reduced informal markets, all of which will benefit both workers and legitimate Qatari businesses. Perhaps most importantly, sponsorship reform will mean better working rights and fewer abuses for migrant workers. All stakeholders involved have welcomed the reforms, and underscored their overall significance:

“By introducing these significant changes, Qatar has delivered on a commitment. One that will give workers more freedom and protection, and employers more choice. We are witnessing what can be achieved when governments, workers and employers work together with the ILO to promote decent work for all." Guy Ryder, ILO Director-General [D].

“The State of Qatar is committed to creating a modern and dynamic labour market. In line with Qatar Vision 2030, these new laws mark a major milestone in this journey and will benefit workers, employers and the nation alike.” Yousuf Mohamed Al Othman Fakhroo, Minister of Administrative Development, Labour & Social Affairs [D].

"*This is very good news for migrant workers in Qatar. The leadership shown by Qatar in dismantling the kafala system and introducing a minimum wage is long-awaited news for all workers. The ITUC stands ready to support the Government of Qatar in the implementation of this historic move, to ensure all workers are aware of the new rules and benefit from them. Other countries in the region should follow Qatar’s example.*” Sharan Burrow, General Secretary of the International Trade Union Confederation [D].

"*These reforms will make a major contribution to the efficiency and productivity of the Qatar labour market. IOE stands ready to support the Qatar Chamber of Commerce and Industry and the Government in supporting employers during this transition. Our congratulations to Qatar and its Chamber of Commerce!*” Roberto Suárez Santos, Secretary-General of the International Organisation of Employers [D].

“*If these reforms are implemented in line with the announcements, and migrant workers are genuinely able to change employers, it will be significant. It would remove a central element of the kafala system, giving workers more ability to determine their own fate if their employers deceive or mistreat them.*” James Lynch, director at migrant rights group, FairSquare [E].

Improving the labour rights of migrant workers in the United Arab Emirates

Hertog’s research has also impacted mobility reforms for foreign workers in the UAE. In 2009, he was invited to work as a technical expert as part of a World Bank team drawing up proposals for labour policy reforms in the GCC region, work that has continued into the current REF period. This invitation was extended based on the strength of Hertog’s academic work in this area [F].

In the course of his work on the UAE, Hertog was commissioned by the Ministry of Labour to produce a substantial policy paper on the state of internal labour mobility in the GCC and policy options for sponsorship reform [G]. The full report, running to 15,000 words and firmly grounded in the underpinning research [2]-[5], was presented at a ministerial meeting in Abu Dhabi in 2015. This report has subsequently informed the regional policy debate on the question of labour mobility reform. Dr Omar Alnuaimi, Assistant Undersecretary for Policies and Strategy at the UAE Ministry of Labour from 2012-2017, has commended Hertog’s important contributions through his work with the World Bank team and, addressing the policy report specifically, noted its “ critical contribution” to internal debate and its importance in “ making the economic and policy case for such mobility reforms” [H].

Again, these reforms will significantly improve the labour rights of migrant workers in the UAE. Addressing the benefits of the reforms and contribution of Hertog’s policy report, Dr Alnuaimi has said: “ it informed our own internal debate about reforms to the labour sponsorship system, which resulted in a considerable relaxation of mobility rules in the UAE in 2016, allowing lower-skilled workers to switch jobs with agreement of their employer after six months of employment and allowing higher-skilled workers to switch jobs without employer agreement. The reforms significantly enhance mobility rights of foreign workers" [H].

An indication of the practical impact of these reforms is provided by analysis of similar – in fact, more limited – reforms in 2011, which have been shown to have substantially improved wages and worker mobility (see, for example, the econometric work by Suresh Naidu, Yaw Nyarko, and Shing-Yi Wang, Journal of Political Economy, 124(6), 2016), precipitating a doubling of employer transitions and an improvement in real earnings of 10% for foreigners switching jobs.

Additionally, in 2014-15 Hertog conducted further research on the potential introduction of a minimum wage for foreign workers in the UAE. The results of the research were presented directly to the UAE Minister of Labour and, although the recommended wage increase is yet to be implemented, the issue continues to be debated internally.

Contributions to labour mobility reform in Saudi Arabia

Hertog’s work in his capacity as a technical expert with the World Bank’s GCC team has also contributed to progressive policy advancements in Saudi Arabia too. He was the lead author of a 120-page report prepared for the Saudi Ministry of Labour, which outlined a suite of fiscal tools and policy options intended to reduce the cost gap of employing migrant versus local workers and made the case for labour mobility reform. In 2017, Saudi Arabia introduced an employer-borne levy on foreign workers which narrowed the cost gap to citizen employees in the private sector, boosting national employment. Dr Andras Bodor, Senior Economist at the World Bank and previously GCC region task team leader on labour policy, noted how " Dr Hertog's work has been critical in triggering and informing the internal debate on this measure” [F].

Advancing the understanding of GCC labour markets at the World Bank and the International Monetary Fund

As outlined above, Hertog has worked with the World Bank since 2009 and throughout the current REF period. Hertog’s input has benefitted the World Bank team by improving its understanding of GCC-specific labour issues and enhancing its capacity to formulate effective policy responses. Indeed, Bodor has attested to the importance of this contribution: “ Dr Hertog is probably the leading academic expert on GCC labour markets and labour policy. His work has substantially advanced our understanding of how the segmentation of markets between foreigners and citizens compromises the quality of employment of foreign workers and keeps GCC citizens out of private employment. It is important to note that labour market policy advice in the GCC cannot rely on the adaptation of labour policies employed elsewhere in the world; the nature of the GCC labour markets is so different because of the vast majority of foreign workers in the workforce. Dr Hertog understands this well and developed out-of-the-box, unique policy solutions for the particular GCC-specific challenges. He has developed several policy tools in his publications for narrowing the labour mobility and labour cost gap between foreigners and citizens, notably through reforms to the sponsorship system and fiscal tools such as fees and subsidies" [F].

Hertog’s research has been widely read in the policy community in the Gulf monarchies as well as within international organisations and consultancies working with regional governments on labour market reform. His work has also informed research and policy reports produced centrally by the World Economic Forum and the International Monetary Fund [I].