Impact case study database

Wachter’s research on the legal and ethical underpinning of AI systems—particularly her work on a right to explanation, counterfactual explanations, data protection, and algorithmic fairness—has substantially informed policy guidance on AI systems in the UK and Europe, and has also informed significant changes to business practice.

Informing policy guidance on a right to explanation:

The automated decisions enabled by AI systems, while extremely attractive to government and industry because of their time and cost efficiency and accuracy, present a new and significant challenge to regulation, with important implications for equality, fairness and human rights. Wachter’s paper [R1] on the (lack of a) “right to explanation” of automated decision-making and significant loopholes in the GDPR generated significant public debate, including press coverage, public hearings, and discussion in academic and policy circles, on ways to increase algorithmic accountability. It also informed the House of Commons Science and Technology Committee’s public inquiry on ‘Algorithms in decision-making’ in 2017. Wachter was invited to give evidence to the inquiry on 14 November [C1a and C1b], and an entire section of the Committee’s enquiry and report was devoted to exploring the right to explanation, in which Wachter’s evidence is cited [C1c, pp.29–30]. Wachter also testified as an expert witness before the House of Lords Committee on Artificial Intelligence on October 31, 2017. The House of Lords subsequently submitted amendments to the draft Data Protection Bill, demanding a right to explanation for automated decisions. Four amendments directly speak to Wachter’s evidence [R1]: amendments 74 (automated processing), 119 (explanation), 134 (automated decision-making), and 183 (the inclusion of a new clause: “Right to information about individual decisions by public bodies based on algorithmic profiling”) [C2]. These amendments were intended to close loopholes identified [R1] and thereby increase algorithmic transparency and accountability.

While the amendments were later dropped, Wachter’s research on the right to explanation has put this issue firmly on the policy agenda. It has been cited in over 30 reports on data governance published by influential bodies, including the Royal Society and British Academy, the Nuffield Foundation, the Parliamentary Office of Science and Technology, the European Parliament, and the Council of Europe. Concerning European protections, the Article 29 Working Party, which issues guidance and interprets matters of EU data protection law, fixed the loopholes she raised [R1] in their “Guidelines on Automated individual decision-making and Profiling” in 2018 [C3]. [R1] points out that the GDPR only gives protection for fully automated decisions (i.e. with no “human in the loop”)—although this very rarely happens in practice, and “token humans” (i.e. nominal human involvement) can be added to a system in order to avoid such protections. Article 29 fixes this loophole by clarifying that “ To qualify as human involvement, the controller must ensure that any oversight of the decision is meaningful, rather than just a token gesture”, that is, these tokens can’t be used as a way to avoid protection under the GDPR. They confirm that these guidelines take account of Wachter’s research and list her publications [R1, R3] in their recommended reading [C3, p. 37]. The guidelines recommend solutions [C3, pp. 20–22, 24–27] and offer legal clarifications inspired by Wachter’s research to close the extensive legal loopholes on algorithmic accountability she identified.

Changes in business regulation and practice: counterfactual tools to explain AI decisions

Having put the issue AI accountability on the policy agenda, Wachter’s subsequent paper [R3] – showing that it is possible to provide meaningful explanation of so-called black-box decisions by providing “counterfactual explanations” – has indicated a practical way forward for both policy makers and industry. The guidelines for good explanations issued by the Information Commissioner’s Office [C4], the UK’s data protection regulator, are heavily influenced by this work; as is the guidance on automated decision-making issued by the Article 29 Working Party citing her work on counterfactuals as good examples of explainable AI [C3].

Following extensive engagement with industry, Wachter’s work has also contributed to significant changes to business practice. Companies have come under increasing public and political pressure to increase transparency of automated decision-making, and Wachter’s work on counterfactual explanations [R3] currently offers the only practical means to do so in a human-understandable way. It has thus been seized on by industry as a solution to the problem, and is becoming increasingly used by developers and businesses around the world.

In their 2019 White Paper on "Issues in AI Governance" [C5], Google cites Wachter's work [C5, p. 23] on counterfactuals [R3] as a means by which AI system operators can better understand a system’s outcomes. Google had already implemented counterfactual explanations in an interpretability interface in September 2018, releasing the “What-If Tool” for use by consumers, businesses, and developers. Designed for Google’s TensorFlow, one of the leading machine-learning frameworks, the tool is intended to make “it easier for a broad set of people to examine, evaluate, and compare machine learning models - whether you're a developer, a product manager, a researcher or a student” [C6]. Google acknowledges that the tool is directly underpinned by Wachter’s research [R3] by providing a means for ordinary consumers to understand complex “black-box” decisions [C6].

Sky, for example, has said that “ *Understanding how models arrive at their decisions is critical for the use of AI in our industry. […] With tools like What-If Tool […] our data scientists can build models with confidence, and provide human-understandable explanations.*” [C7]. Google has also implemented counterfactual explanations into their cloud services, making this available to tens of thousands of businesses [C7]. The Head of Google Cloud AI’s Division, has summed up the benefits of implementing counterfactual explanations at Google, saying that: “ it’s really important for societal reasons and fairness reasons and safety reasons” [C8] – that is, when it comes to trusting the technology to make important decisions that affect people’s lives, in an accountable way.

Other businesses that have implemented counterfactual explanations in their products include IBM, Accenture, Vodafone South Africa (in its “Just 4U” payment plans) [ Corroborator 1], and the drone insurance company Flock. Far from being “impossible to understand”, black-box decisions about people’s finances, medical diagnosis, hiring decisions, university admissions, criminal justice (etc.) can now be made human-understandable [R2] as an industry standard, by using counterfactuals as a means of understanding what has / has not been important in coming to a particular automated decision or prediction.

Changing business practice: accountability tools for detection of AI bias

AI systems rely on training data to create their models: we know that (gender, ethnicity etc.) biases in these data can significantly skew models, leading to discrimination against protected groups. While aligning judicial understandings of fairness with AI models has been difficult [R5], Wachter’s paper [R6] presented a practical means of reconciling these two approaches. In December 2020, an AI accountability toolkit on “fairness detection” was released in SageMaker, Amazon's machine learning service [C9]. The tool implements ‘conditional demographic disparity’ (a concept coined and described in R6) as a baseline statistical measurement that allows testing of fairness in automated systems by detecting bias in training data. As Amazon explains, “ This metric is useful for exploring the concepts of direct and indirect discrimination and of objective justification in EU and UK non-discrimination law and jurisprudence” [R5, R6] [C9]. That is, by aligning with the European Court of Justices’ ‘gold standard’, the tool enables the contextual approach to judicial interpretation of bias practiced under EU non-discrimination law.

Through significant engagement with both policy makers and industry – across the three areas of a right of explanation for automated decisions [R1, R2], counterfactual explanations as a means to accomplish this [R3, R4], and detection of bias in AI systems [R5, R6] – Wachter’s work on transparency and accountability of AI systems is now affecting millions of people in all sectors. Important loopholes in European protections on automated decision making have been closed, and industry and consumers now have, for the first time, a means to intuitively understand the reasons for automated decisions, including ways to assess whether and why they have been discriminated against.