Impact case study database

Two distinct areas of impact related to Bolton’s evaluation research are evident in the period: Impact on public policy -* policy debate has been stimulated and informed by Bolton’s research, leading to a change in policy direction.

*Impacts on practitioners - the way projects are delivered and measured has improved due to the introduction by Bolton and her team of a Theory of Change methodology to Sport Wales funded programmes.

Impact on Public Policy

Bolton’s research created a shift in sport policy and its delivery by helping Sport Wales and its partners align their management approaches with the delivery of the Wellbeing of Future Generations Act and the recommendations of the 2017 Independent Review of Sport Wales commissioned by WG.

The 2017 Independent Review [E1] had noted that Sport Wales was viewed as “an organisation at a crossroads in its strategic direction and the role it plays in the delivery of wellbeing goals and government policy…However, the increasing focus on sport’s contribution to the wider policy priorities, in a climate of reduced public expenditure budgets, requires Sport Wales to lead the sector through transformational change.” The review recommended the preparation of a new strategy that would lead the sector and reflect a wider agenda to include new ways of working.

Drawing on the C4A evaluation undertaken by Bolton and her team, the Independent Review noted that C4A was a “refreshing departure” from the norm. The C4A evaluation had from the outset established a focus on organisational learning, that formed a key tenet of the “real time” evaluation process which featured significantly in the first report to the Sport Wales Board in January 2016. In November 2018, Sports Wales’ Director of Community Engagement presented to the Board on key learning from the evaluation of C4A and its influence on the emerging Sport Wales national strategy [E2]. The report specifically referred to Governance and Partnerships, the thematic area led by Bolton and noted that “learning to date has been used to inform the approach adopted by the Healthy & Active Fund (HAF)”, a £5.4m WG investment fund delivered jointly by Sport Wales and Public Health Wales. Minutes from the meeting confirm that “learning from C4A would be embedded within the new strategy” [E3].

Bolton’s team also undertook two other significant research evaluations. Firstly, the Free Swimming (FSI) review (June 2018) [R4] concluded “evidence suggests that the current approach to free swimming is not the most cost-effective way to use the £3 million pa allocated to it…it is no longer fit for purpose.”. This led to dialogue between Sport Wales, WG and key stakeholders resulting in a revised scheme for Free Swimming and the diversion of some funds to the Healthy & Active Fund. Secondly, the recommendations of the 2019 Young People Programmes Review (YPPR) [E4] were publicly endorsed in the Sport Wales 2018-19 Annual Report [E5]. Concomitant developments associated with the launch of the Healthy Weight: Healthy Wales national strategy and an agreed new fund of £1.3 million to support an over 60s Active Leisure Scheme, emerged directly from Bolton’s evaluation research findings [E6].

Impact on Practitioners

The use of a Theory of Change approach in the evaluation of C4A was the first time such a technique was utilised in relation to the evaluation and design of national sport funded programmes. Importantly, it was highly successful in assessing the extent to which the original policy drivers had been met. Furthermore, it had a positive impact on the organisations sponsored through C4A, allowing them to learn from each other and by facilitating cross-fertilisation of an enhanced experience for both the organisations themselves and, indirectly, their diverse end users [E7]. Its use had been driven by the team’s initial evaluation of the FSI [R3] which had highlighted that the existence of multiple objectives posed a significant challenge when evaluating the initiative. When developing a system of evaluation for C4A the team identified the use of Theory of Change as a way to provide a common approach which shifted the focus to outcomes rather than inputs and outputs, hitherto commonplace in Sport Wales performance management.

The use of Theory of Change techniques has subsequently become an expected norm for Sport Wales and its principal partners and has become a core tool used in the development and evaluation of its programmes. This has included the 2020 Free Swimming Independent Progress Monitoring Exercise (£60k), the invitation to tender for which states an expectation of the use of a Theory of Change approach to “ *help us to understand what sort of data we need…to help us learn.*”; the 2019 evaluation of the Healthy & Active Fund (£5.4m) which required the use of Theory of Change at both programme and project level; the 2019 Review of National Lottery Community Grant Scheme , which adopted a Theory of Change approach in order to focus on outcomes; and, the development of both the 2020 60+ Active Leisure Scheme (£1.3M) and the 2020 Sport Resilience Fund (£8M) which also used a Theory of Change approach. [E8]

Collectively, Bolton’s research has had significant impact across a number of Sport Wales and Welsh Government policy initiatives with its ongoing impact evidenced through the funding secured for two KESS2 projects (totalling £118k). The first is examining barriers to participation for young disadvantaged females. The second is examining the cross-sectional working, required by the WBFGA, of Sport Wales, Public Health Wales and Natural Resources Wales to achieve sustainable gains in the areas of health and physical activity.