Skip to main

Impact case study database

The impact case study database allows you to browse and search for impact case studies submitted to the REF 2021. Use the search and filters below to find the impact case studies you are looking for.
Waiting for server

The potential of Contribution Analysis to alcohol and drug policy strategy evaluation

1. Summary of the impact

The researchers won a tender to use Contribution Analysis (CA) methodology to review the Welsh Government’s policy on alcohol and other drugs and policy implementation.

  1. The review has made a substantive contribution to the framing of the Welsh Government Consultation Document ‘Substance Misuse Delivery Plan 2019-2022’ (Welsh Government, 1 July 2019 [5.1] and the subsequent Substance Misuse Delivery Plan [published 22 October 2019], **[5.2]**).

  2. The CA methodology has subsequently been adopted by the Welsh Government for evaluation of the Minimum Pricing for Alcohol Policy based on the evaluation [5.5].

  3. The research further contributed to the establishment of the Welsh Centre for Alcohol and other Drugs (Policy, Practice and Research) (hereafter referred to as Welsh Centre) in which service user advocacy is a key theme [5.6].

2. Underpinning research

Livingston, Madoc-Jones, and colleagues from Figure8 Consultancy Services and Glyndwr University undertook a policy evaluation in 2017-2018 that reviewed the Welsh Government’s Substance Misuse policy, Working Together to Reduce Harm: The Substance Misuse Strategy 2008-2018 [3.2]. Livingston and colleagues adopted a Contribution Analysis method, a preferred approach highlighted by the Welsh Government in the tender documentation associated with the initial commissioning of the research. Following publication of the report [3.2] the research team also published a reflective methods article in the journal Drugs Education Prevention and Policy [3.1].

Contribution Analysis (CA) is increasingly favoured by commissioners, governments and researchers as a policy evaluation tool. In addition to the research presented here, it has been applied in a number of other substance use evaluations (e.g. ‘Drink Wise Age Well’ and Minimum Pricing for Alcohol in Wales and Scotland).

CA was developed as an analytical tool for situations where adopting experimental or quasi-experimental approaches to test cause were either impractical or impossible, and so it is an approach to evaluation particularly suitable to explore complex, multi-level programmes of work where direct causal attributions are rarely possible. Our study was deemed necessary within the policy cycle to inform the Welsh Government’s next strategic approach. We, consistent with the method, explored existing knowledge and gathered quantitative and qualitative ‘evaluative evidence’ from a range of sources to tell the ‘performance story’. This then outlined how successful the strategy had been in contributing to particular outcomes in the short, medium and long-term [3.1].

Our research evaluated Wales’s second substantive devolved substance misuse policy. The pace of devolution meant that by 2018, within Wales, twenty-seven different Treatment Frameworks and Guidance, and four successive delivery plans had been developed from 2008 to provide the implementation detail to support this strategy (Livingston et al. 2018). The scope of the evaluation was considerable, as is noted in the Appendices to the report [3.2].

A particular problem faced by us here was that CA largely focuses on known outcomes. It is a theory of change evaluation method, that assumes a theory of activity to output is established at the onset. However, in this instance, the team had to establish this in retrospect and have provided presentations of a revised application of the methodology that others might find useful [3.1 and 3.4]. The evaluation team thus established a theory of change equation to an outcome associated with anticipated reduction in harms, and five logical models consistent with the core strands of the policy. Through a mixture of exploring existing literature, and data sets, we were able to establish the extent to which any planned activities related to measurable outcomes (i.e. attributable deaths, hospital admissions, numbers completing treatment programmes). Within the CA evaluation, we then undertook a series of interviews and focus groups to account for other influencing factors and to test out the emerging performance story. This work included senior and frontline staff, and service users – approximately 120 people [3.2 - appendices].

The key findings of the study identified significant achievements, mainly in the areas of harm reduction, treatment interventions for harmful users and partnership working. Continued momentum in active Service User Involvement was noted as one of the clear achievements of this strategy period. However, the findings also led the researchers to comment on how ensuring that this is inclusive, representative and definitely not tokenistic, remains a challenge.

The report included a recommendation for greater involvement of service users in developing policy nomenclature. This aspect of the review was informed by Livingston’s wider activity and deliberations with regard to Participatory Action Research [3.3]. Much of this has articulated the need to develop projects which establish the inclusion of substantive advocacy, involvement, and voice of those with lived experience in policy and research deliberations, as much as they are in the environs of roles in peer-led service provision.

3. References to the research

References 1 and 3 are published in peer-reviewed journals. Reference 2 was commissioned and published via the Welsh Government.

  1. Livingston, W. Madoc-Jones, I and Perkins, A (2020) The potential of Contribution Analysis to alcohol and other drug policy strategy evaluation: an applied example from Wales – Drugs, Education, Prevention and Policy 27(3), pp.183-190. doi: 10.1080/09687637.2019.1645093

  2. Livingston, W., Perkins, A., McCarthy, T., Madoc-Jones, I., Wighton, S., Wilson, F. & Nicholas, D. (2018). Review of Working Together to Reduce Harm: Final Report. Cardiff: Welsh Government GSR report number 21/2018 (Project Ref - C93/2016/2017). Available at:

Page Link - https://gov.wales/review-working-together-reduce-harm-substance-misuse-strategy-2008-2018-0

Full Report - https://gov.wales/sites/default/files/statistics-and-research/2019-06/180419-review-working-together-reduce-harm-en.pdf

Appendices - https://gov.wales/sites/default/files/statistics-and-research/2019-06/180419-review-working-together-reduce-harm-appendix-en.pdf

Summary - https://gov.wales/sites/default/files/statistics-and-research/2019-06/180419-review-working-together-reduce-harm-summary-en.pdf

  1. Livingston, W., Perkins, A. (2018), 'Participatory action research (PAR) research: critical methodological considerations', Drugs and Alcohol Today 18(1), doi: 10.1108/DAT-08-2017-0035 https://www.emerald.com/insight/content/doi/10.1108/DAT-08-2017-0035/full/html

  2. Livingston, W (2020) Minimum Pricing Policy Evaluation (thoughts) from experiences thus far. Global Alcohol Policy Conference 2020, 10th March, Dublin

Abstract available at https://www.gapc2020.org/; full presentation available on request (w.livingston@glyndwr.ac.uk).

4. Details of the impact

( a) Delivery Plan

This research was funded by Welsh Government, to review its own policy. In this sense, they are the primary beneficiaries. Funding was gained via public commissioning process; Glyndwr University was the lead research partner in a proposal submitted by Figure8 Consultancy Services Ltd. Welsh Government used the conclusion of the research as a foundation for the subsequent development of a consultation document that then led to the next Delivery Plan, being issued October 2019. The Welsh Government Consultation Document referred to the CA report’s highlighting of the importance of increasing the role of service users across all aspects of policy and practice implementation [5.1, p.6]. This was then explicitly highlighted in the Delivery Plan, which states (of the prior CA):

“… that whichever direction new policies took in the future they should be focused on continued support for harm reduction and useful accountability of activity… it is important that that [sic] any future approaches to dealing with the harms associated with the misuse of substances, continue to develop the significant improvements that were seen during the 2008-18 strategy in partnership working” [5.2, p8].

The CA report was the only external document referredFootnote:

[5.2 p.8] names the CA document and offers a footnote to a hyperlink. The hyperlink however erroneously refers to an earlier 2013 evaluation of Welsh Government Strategy and not the intended CA evaluation [3.2]. to in the delivery plan [5.2 p.1 and p.8], and as such, the Welsh Government further went onto reiterate its value as:

“The report stressed the importance of building on the platform of an increasing role for service users across all aspects of policy and practice implementation. The evaluation confirmed that the Key Aims of preventing harm; support for individuals – to improve their health and aid and maintain recovery; supporting and protecting families; and tackling availability and protecting individuals and communities via enforcement activity that underpinned the previous strategy remain relevant. Therefore, whilst this delivery plan sets out new priorities and actions for the 2019-22 period designed to combat today’s challenges it does so with the context and a continuation of the key themes of the previous 10 year strategy.” [5. 2, p.8].

The addition of ‘partnership working’ as an explicit aim in the Delivery Plan (as opposed to its implicit status in the previous strategy) highlights the direct impact of our review, with Welsh Government utilising our five evaluation strands in the new plan (preventing harm, support for individuals, support for families, tackling substance availability, and stronger partnerships, including those with service users) [5.2, p.4]. Within this context, the Delivery Plan identifies a number of key partners who share responsibility for this agenda and as such are impacted upon in terms of the continued policy approach, namely; Area Planning Boards, Health Boards, Local Authorities, Police and Public Health Wales.

The plan responds to the narrative of our review (continuity and enhancement) by translating the five aims into fifteen outcomes. The delivery of these outcomes has shaped the way in which alcohol and drug service provision has been delivered since the review.

(b) CA methodology/Minimum Pricing

The CA report made three key recommendations; (i) adoption of a minimum pricing policy for alcohol, (ii) greater involvement of service users in developing policy nomenclature and (iii) a revised approach to hosting of data and research reports on Welsh Government website. All three of these recommendations have been advanced by Welsh Government:

  1. A commitment to implementing Minimum Pricing for Alcohol was situated in the Delivery Plan and a Bill has subsequently come into force on 2nd March 2020 [5.4], (Livingston, Madoc-Jones and colleagues have been awarded three separate contracts to support the evaluation of this, including one explicitly using Contribution Analysis [5.3, 5.5 and 5.10] and have published the first findings **[5.6 and 5.7]**).

(c) Welsh Centre/Service User Advocacy

  1. The delivery plan enhanced explicit expectations of more service user involvement [5.1, p.6]. Conversations between Livingston and Welsh Government about increased service user involvement have contributed significantly to the establishment of The Welsh Centre: informing policy and practice on alcohol and other drugs [5.8]. This centre began meeting in Autumn 2020. It has a specific aim to establish a ‘Public Patient Involvement’ panel for Wales/alcohol and other drugs. It has been actively attended by Welsh Government Substance Misuse Branch staff.

  2. Welsh Government have indicated work is underway in reorganising the hosting of Welsh drug data sets and other related material, to improve accessibility [5.9]. (There is also some cross over activity for the Welsh Centre in this context [5.8])

The first experiences of Contribution Analysis in Welsh alcohol and drug policy evaluation have been subsequently published [3.1]. The Welsh Government have responded positively, such that they have adopted it as the overall methodological approach for evaluation of the Minimum Pricing for Alcohol Policy: Contract for Evaluation of Minimum Price for Alcohol, Lot 1 Contribution Analysis, Contract number C171/2018/2019 [5.3].

5. Sources to corroborate the impact

5.1 Welsh Government (2019) Substance Misuse Delivery Plan 2019-2022 Consultation Document, Number: WG38273. Welsh Government, 01 July 2019. [https://gov.wales/sites/default/files/consultations/2019\-06/consultation\-document.pdf\]

5.2 Welsh Government (2019) Substance Misuse Delivery Plan 2019-2022. Cardiff, Welsh Government [ https://gov.wales/substance-misuse-delivery-plan-2019-2022-0]

5.3 Welsh Government (2019) Contract Award Letter: Contract for Evaluation of Minimum Price for Alcohol, Lot 1 Contribution Analysis, Contract Number C171/2018/2019

5.4 PUBLIC HEALTH (MINIMUM PRICE FOR ALCOHOL) (WALES) BILL Explanatory Memorandum incorporating the Regulatory Impact Assessment and Explanatory Notes [https://senedd.wales/laid%20documents/pri\-ld11577\-em/pri\-ld11577\-em\-e.pdf\]

5.5 Figure8 Consultancy Services / University of South Wales: Collaboration agreements for Welsh Government Contract Number C171/2018/2019 (Contracts for Evaluation of Minimum Price for Alcohol): Lot 3 Qualitative Work With Services (and Service Users) (lead partner Figure 8 Consultancy Services) and Lot 4 Wider Population of Drinkers (lead partner University of South Wales).

5.6 Livingston, W.; Holloway, K., May, T., Buhociu, M., Madoc-Jones, I and Perkins, A. (2020) Adapting existing behaviour: perceptions of substance switching and implementation of minimum pricing for alcohol in Wales. Nordic Studies on Alcohol and Drugs. First Published December 20, https://doi.org/10.1177/1455072520972304

5.7 Holloway, K.; May, T.; Buhociu, M.; Livingston, W.; Perkins, A. and Madoc-Jones. I. (2019) Research into the potential for substance switching following the introduction of minimum pricing for alcohol in Wales: Final Report Cardiff: Welsh Government GSR report number 48. /2019. https://gov.wales/sites/default/files/statistics-and-research/2019-10/research-potential-substance-switching-introduction-minimum-pricing-alcohol.pdf

5.8 Consultation documentation, mission statements and minutes of meetings of the Welsh Centre for Alcohol and other Drugs (Policy, Practice and Research; contact Livingston w.livingston@glyndwr.ac.uk)

5.9 Informal Meeting 29/01/2020 between Livingston and 4 members of Welsh Government Substance Misuse Branch (contact: Head of Substance Misuse, Policy and Finance, Welsh Government) [corroborator #1]

5.10 Principal Research Officer (Health Economics, Welsh Government) [corroborator #2]

Additional contextual information

Grant funding

Grant number Value of grant
C93/2016/2017 £12,000