Skip to main

Impact case study database

The impact case study database allows you to browse and search for impact case studies submitted to the REF 2021. Use the search and filters below to find the impact case studies you are looking for.
Waiting for server

Protecting Civilians in Conflict Areas and Raising Standards in Survey Research

1. Summary of the impact

Professor Spagat’s research and methodological innovations on the measurement of war casualties has influenced the conclusions and recommendations of The Report of the Iraq Inquiry (E1, also known as “The Chilcot Report”). This stimulated a policy debate that directly led to improved record keeping on civilian casualties in UK military operations. The research has also informed advocacy groups and learned societies, leading to changes in their practices for recording war casualties and improving survey standards.

Spagat’s research played a formative role in The Chilcot Report’s recommendation that governments, including the UK, should record civilian casualties incurred in their military operations and use that information to minimize them. Subsequently, the UK Ministry of Defence adopted the practice of civilian casualty recording at its field hospitals. This research is also used by a network of NGOs, the International Network on Explosive Weapons, in their advocacy campaign to reduce the harm from explosive weapons to civilians, particularly in populated areas.

Professor Spagat’s research also has introduced methodological contributions that influenced the polling industry, setting standards for transparency and the prevention of data fabrication in survey research. This has had an impact on a large number of surveys and organizations worldwide, including the United States’ State Department.

2. Underpinning research

Professor Spagat’s body of research in conflict economics includes the development of a methodology for casualty recording in war, a key input in the quantitative analysis of conflict, its effects and its causes.

Spagat applied these methods to measure and account for death in war (e.g. the human cost of the Iraq war). He identified the main threats facing Iraqi civilians in the war, exposed false claims in the scientific literature concerning deaths in Iraq both prior to, and during, the 2003 war. This research produced a milestone case study of data fabrication and violations of established transparency standards in survey research.

In Reference 1, Spagat developed methods to identify and code lethal events in war. The analysis of the data established strong regularities in violent event patterns in the nine modern wars for which it was possible to compile or obtain event data conforming to the methodological standards of the programme. Importantly, for this case study, this work validated methods for compiling event data on war, including those of Iraq Body Count, a programme to record and maintain a large database of lethal events occurring within the Iraq War from 2003 to present.

References 2 and 3 used the large Iraq Body Count database to classify perpetrators and weapons used in the Iraq war, according to their violent impact on civilians. The Chilcot Report accurately summarised the findings of Reference 2 as “in events with at least one Iraqi non-combatant civilian casualty, the methods that killed the most non-combatant civilians per event were aerial bombing (17 per event), combined use of aerial and ground weapons (17 per event) and suicide bombers on foot (16 per event). Aerial bombs killed on average nine more non-combatant civilians per event (17) than aerial missiles (8).” Reference 3 added a finding that deaths caused by Coalition forces of Iraqi civilians, women, and children peaked during the invasion period, with relatively indiscriminate effects from aerial weapons.

References 4 and 5 debunked a widely cited survey, published in The Lancet, which ostensibly measured the number of deaths in the Iraq war. Reference 4 demonstrated a fundamental flaw in the sampling scheme that came to be known as “main street bias.” Reference 5 found evidence of data fabrication and ethical violations which included breeches of transparency standards for survey research that were established and are maintained by the American Association for Public Opinion Research, the largest professional organization of public opinion and survey research professionals in the U.S.

Reference 6 debunked a widely cited survey-based estimate which claimed that economic sanctions levelled against Iraq during the 1990s caused the deaths of hundreds of thousands of Iraqi children.

3. References to the research

  1. Common Ecology Quantifies Human Insurgency.  Bohorquez, J.C.; Gourley, S.; Dixon, A.; Spagat, M.; Johnson, N.F. In: Nature, Vol. 462, No. 7275, 17.12.2009, p. 911-914. QI: Scopus citations, 133; Altmetric Score, 100; cover article for that issue of Nature; Journal Impact Factor, 41.6. DOI: 10.1038/nature08631

  2. The Weapons that Kill Civilians: Deaths of Children and Noncombatants in Iraq, 2003-2008. Hicks, M. H.; Dardagan, H.; Serdán, G. G.; Bagnall, P. M.; Sloboda, J. A.; Spagat, M. In: New England Journal of Medicine, Vol. 360, No. 16, 16.04.2009, p. 1585-1588. QI: Scopus citations, 39; Altmetric Score, 67; Journal Impact Factor, 79.3. DOI: 10.1056/NEJMp0807240

  3. Violent Deaths of Iraqi Civilians, 2003–2008: Analysis by Perpetrator, Weapon, Time, and Location. Hicks, M. H.; Dardagan, H.; Serdán, G. G.; Bagnall, P. M.; Sloboda, J. A.; Spagat, M. In: PLOS Medicine, Vol. 8, No. 2, 02.2011, e1000415. QI: Scopus citations, 40; Altmetric Score, 81; Journal Impact Factor, 11.7. DOI: 10.1371/journal.pmed.1000415

  4. Bias in Epidemiological Studies of Conflict Mortality. Johnson, N. F.; Spagat, M.; Gourley, S.; Onnela, J. P.; Reinert, G. In: Journal of Peace Research, Vol. 45, No. 5, 09.2008, p. 653-663. QI: Scopus citations, 26; Altmetric Score, 17, paper of the year award for the journal; Journal Impact Factor, 2.4. DOI: 10.1177/0022343308094325

  5. Ethical and Data-integrity Problems in the Second Lancet Survey of Mortality in IraqSpagat, M. In: Defence and Peace Economics, Vol. 21, No. 1, 02.2010, p. 1-41. QI: Scopus citations, 11; Altmetric Score, 29, Journal Impact Factor, 1.1. DOI: 10.1080/10242690802496898

  6. Truth and death in Iraq under sanctions. Spagat, M. In: Significance, Vol. 7, No. 3, 09.2010, p. 116-120. QI: Scopus citations, 4; Altmetric Score, 24. DOI: 10.1111/j.1740-9713.2010.00437.x

4. Details of the impact

Professor Spagat’s research on the recording of war civilian casualties have influenced the policy and public debate, and his new methodologies improved its statistical accuracy in three ways. First, his research output has informed policy reports and recommendations, most notably in The Chilcot Report’s inquiry into the Iraq War. Second, the findings surrounding civilian casualties are providing scientific support to advocacy groups’ objective of banning explosive weapons in heavily populated areas. Third, Spagat’s research into data fabrication and sampling flaws in casualty recording initiated a change in transparency policy among survey organisations around the world. Each of these channels of impact is discussed below.

Informing and influencing public debate and policy through research

Spagat’s research informs two key conclusions and policy recommendations of The Chilcot Report in 2016 (a public inquiry into the role of the UK in the Iraq War commissioned by the British Government). The report uses evidence from Spagat’s research to highlight the need to minimise the damage of military operations on civilians and to improve the accuracy of civilian casualty recording. These recommendations were adopted by the Ministry of Defence in 2018 (E2).

The Chilcot Report build-up to the conclusion on civilian damage includes the following direct quote from Spagat’s research (Reference 2): “ It seems clear from these findings that to protect civilians from indiscriminate harm, as required by international humanitarian law … military and civilian policies should prohibit aerial bombing in civilian areas unless it can be demonstrated – by monitoring of civilian casualties for example – that civilians are being protected.” (E1, p. 212)

The Report also echoed Spagat’s research on the quality of casualty recording (Reference 4) by referring to it directly: “ Researchers at Oxford University (Mr Sean Gourley and Professor Neil Johnson) and Royal Holloway, University of London (Professor Spagat) issued a press release on 19 October, claiming that there were “serious flaws” in the methodology used by the Lancet study which acted to inflate its casualty estimate.” The subsequent Report’s recommendations on civilian casualty recording follow directly from the conclusions informed by Spagat’s research. For example, “ 277. The Inquiry considers that a Government has a responsibility to make every reasonable effort to identify and understand the likely and actual effects of its military actions on civilians.”, “ 280. The Government should be ready to work with others, in particular NGOs and academic institutions, to develop such assessments and estimates over time.”, and “ *281. The Government should take account of those assessments and estimates in developing its strategy and plans as well as in its military tactics and use of ordnance, in order to minimise, to the extent possible, the effects on civilians.*” (E1, p. 210)

In response, the UK Ministry of Defence established a “Chilcot Team” which, among other duties, engages with NGOs to develop policies for recording and reducing civilian casualties suffered during UK military operations. Two NGOs, Every Casualty Worldwide and Action on Armed Violence, for which Spagat is Board Chairman, engage with this process.

The Chilcot Report adopted the term “casualty recording,” coined by Every Casualty Worldwide to denote the data collection aspect of civilian protection policy, leading to Parliamentary support for casualty recording as manifested through written questions on June 8, 2016, June 27, 2018, September 24 & 25, 2019 and in a cross party motion tabled by 28 MP’s on June 26, 2019. On 27, February 2018 Earl Howe, Minister of State, Ministry of Justice announced to Parliament the UK’s first ever commitment to a policy of civilian casualty recording (E2): “ *Recognising the important work being done by a number of UK registered charities, including Every Casualty Worldwide, … , to ensure that all lives lost to armed violence anywhere in the world are properly recorded, the Ministry of Defence is making a commitment to increase transparency by publishing the number of all civilians admitted to UK military field hospitals.*”

Spagat’s research also influenced the conclusions of The Chilcot Report with respect to the justification for the invasion of Iraq, in particular in relation with arguments made about economic sanctions and child mortality. Tony Blair testified before the Chilcot Inquiry, relying on a UN-sponsored survey called the “ICMMS”, that the invasion of Iraq had helped the Iraqi people, arguing that “ In 2000 and 2001 and 2002 they had a child mortality rate of 130 per 1,000 children under the age of five … The figure today is not 130, it is 40. That equates to about 50,000 young people, children, who, as a result of a different regime that cares about its people – that’s the result that getting rid of Saddam makes.” (E3, pp. 233-234)

But The Chilcot Report cited Spagat’s Reference 6 to counter that argument and to conclude otherwise. “ In September 2010, Professor Michael Spagat reported that the child mortality estimates reported by the ICMMS were between two and three times higher than those reported by three other major UN- sponsored surveys (the Iraq Living Conditions Survey 2005, the Multiple Indictor Cluster Survey in Iraq 2007 and the Iraq Family Health Survey 2008).” (E1, p. 175) In particular, the Report highlights Professor Spagat’s methodological objections to Saddam Hussein’s government data that was used to make the claims about child mortality.

Influencing practice and policy of advocacy groups

Professor Spagat’s methodological contributions and findings in the cited research influenced how advocacy groups record casualties and have informed their public campaigns.

The advocacy group Iraq Body Count, whose event-data methodology was strengthened, validated and disseminated through References 1, 2 and 3 has become the most credible source on casualties related to the Iraq War and its aftermath. For example, their records are cited 44 times in The Chilcot Report.

Iain Overton, Executive Director of the NGO Action on Armed Violence writes (E4) that references 2 and 3 “ documenting the disproportionate impact of explosive violence on children and female civilians in the Iraq war provided some of the key initial inspiration for the INEW campaign,” that is, the International Network of Explosive Weapons, which works “ to prevent human suffering through the use of explosive weapons in populated areas.” Overton adds that “ this work [references 2 and 3] has been heavily cited within the campaign, particularly in its formative years but also continuing down to this day” and elaborates that “ Notable successes of the campaign include bringing together almost 100 countries to speak out about the use of explosive weapons in populated areas, creating the groundwork for a political commitment to avoid the use of explosive weapons in populated areas”.

Effecting change in survey organisations to raise standards in survey research

Professor Spagat’s work on data fabrication in the Iraq conflict influenced two important stakeholder groups to improve their survey research standards. First, the research motivated the US State Department (USSD) to closely scrutinize the work of a survey data provider they were heavily using in Iraq, leading eventually to an upgrade of their methods to combat data fabrication. Second, this research, as it highlighted transparency shortcomings in widely cited research on civilian deaths in Iraq, has also triggered a process of improving transparency standards in surveys, adopted by the American Association for Public Opinion Research (AAPOR).

Steven Koczela, a USSD poll analyst who in 2010 looked into data fabrication in a large number of public opinion surveys the conducted by the USSD during the Iraq war states in his testimonial letter (E5) that “ Professor Spagat’s work on Iraq death estimates was a major inspiration for us to consider fabrication as a potential explanation for unusual patterns in the microdata in some of the USSD surveys (…) Following these events, the Department of State has completely overhauled their approach to data quality. This series of events triggered an industry-wide focus on the issues of data fabrication”. According to Mr Koczela, who eventually left the USSD and ran several conferences on data fabrication between 2014 and 2017, Professor Spagat’s work has been instrumental to “ set in motion and help steer a long overdue conversation on fabrication. This conversation has brought significant benefit to public and private entities which conduct international survey research around the world.”

The violations of transparency standards exposed in reference 5 also influenced AAPOR to launch the Association’s Transparency Initiative (E6) that publicly rewards survey organizations for achieving high transparency standards set by the Association. These standards guide until today the work of 94 survey organizations and thousands of surveys conducted all over the world.

Spagat’s formal complaint to AAPOR’s Standards Committee (E7, based on an early version of reference 5) triggered a Committee investigation that resulted in the formal censure (E8) of the researcher behind the survey critiqued in reference 5. The Association’s outgoing president then devoted his entire Presidential Address (E9) to this censure, provoking a major discussion it among AAPOR members. These events have had enduring influence For example, in E10, The University of Florida Survey Research Center cites in a 2017 article the censure as the cause of AAPOR’s Transparency Initiative (which the Center joined in 2015 as a charter member).

5. Sources to corroborate the impact

  1. Chilcot, J., 2016. “The Report of the Iraq Inquiry.” Volume 12, Section 17, London: Her Majesty’s Stationery Office, pp.170-219. [online]; https://webarchive.nationalarchives.gov.uk/20171123122743/http://www.iraqinquiry.org.uk/the-report/.

  2. An Archive of parliamentary activity on casualty recording for events occurring on June 8, 2016, February 27, 2018 June 27, 2018, June 26, 2019 and September 24 & 25, 2019.

  3. Rt Honourable Tony Blair Transcript as part of his testimony to the Chilcot Inquiry, January 29, 2010.

  4. Testimonial letter from Iain Overton (Action on Armed Violence), May 20, 2020

  5. Testimonial letter from Steve Koczela (formerly of the US State Department), May 19, 2020.

  6. AAPOR’s Transparency Initiative, launched October 2014. [online]; https://www.aapor.org/Transparency_Initiative.htm.

  7. Correspondence on Spagat Complaint to AAPOR, 2008.

  8. AAPOR Censure of Gilbert Burnham. February 4, 2009. [online]; https://www.aapor.org/Communications/Press-Releases/AAPOR-Finds-Gilbert-Burnham-in-Violation-of-Ethics.aspx.

  9. Kulka, R. 2009, “Presidential Address: Plus Ça Change, Plus C'est La Même Chose? AAPOR's Standards Code and Procedures.” The Public Opinion Quarterly, Vol. 73, No. 3, pp. 610-628.

  10. Porter, C and J. Tipery, 2017, “How Was That Survey Conducted? The Need for Transparency of Research Methods”, University of Florida Survey Research Center.

Additional contextual information