Skip to main

Impact case study database

The impact case study database allows you to browse and search for impact case studies submitted to the REF 2021. Use the search and filters below to find the impact case studies you are looking for.
Waiting for server

Improving Risk Management in Scottish Charity Regulation

1. Summary of the impact

Public trust in charities is at an all-time low in the UK following a series of scandals over the past decade. Under substantial pressure to do more with less, charity regulators have been looking for ways to adopt a more targeted approach to regulation. Our work with the Scottish Charity Regulator (OSCR) has allowed it to reduce the administrative burden on Scotland’s 24,500 charities by introducing innovations that focus their regulatory activities more efficiently on relatively high-risk organisations, demonstrated by changing patterns in regulatory investigations. OSCR has used our findings in their move to targeted regulation when redeveloping their risk indicators, data collection, and data processes.

2. Underpinning research

Recent high-profile charity scandals, from Kids Company in 2015 (governance), to Olive Cooke in 2016 (fundraising), and most recently Oxfam in 2018 (safeguarding), have brought attention to risk, and led to pressure on regulators to focus their diminishing resources on the riskiest of charities. For many charities, regulation can be a burden, requiring extensive paperwork and reporting. But the misbehaviour of a few can have wider knock-on effects, as we have seen in media coverage and the dramatic fall in the general public’s trust in charities.

The Scottish Charity Regulator (OSCR) has powers to investigate and take regulatory action against charities who break the law. Funded through a GBP275,000 ESRC-OSCR-Scottish Council for Voluntary Organisations civil society data partnership, we designed our programme of research with OSCR to co-produce impact via better regulator data use. This has been achieved through both through the research findings and through the close involvement of OSCR staff in the research process.

Our research examined the ways in which ‘risk’ is operationalised through data by charity regulators, and experienced by charities. It used novel administrative data and meta-data from the work of the regulators in Scotland, England, and Wales to explore the effectiveness of quantitative performance indicators as measures of underlying risk. Our unprecedented access to the internal administrative data from charity regulators allowed us to conduct innovative and detailed analyses of risk for the first time. Four peer-reviewed papers reflect an ongoing programme of research using ‘big data’ drawn from multiple regulators in order to more effectively target their limited resources. Together this body of research demonstrates the power of linking different administrative data sources on charities and their interactions with regulators to understand how risk is measured, operationalised, and mitigated.

Research finding: Developing new models of risk can help regulators to target their interventions in a way that is proportionate and efficient

[R1] demonstrated that there is a disconnection between the organisations most likely to be the subject of complaints and the complaint, identifying a risk that required regulatory action. We used administrative data on 25,611 charities, and examined 2,109 regulatory investigations, including complaints from the public and concerns from professional advisors. Modelling the probability of both complaints and investigations, we showed that many of the risk indicators used by the regulator in assessing the risk of misconduct were not predictive of misconduct. We identified the strong association between the source of the complaint and the likelihood of regulatory action being required, demonstrating that complaints from the general public about large charities were less likely to reflect underlying risks.

[R2] explored the characteristics of charities engaging with the regulator’s “serious incidents” scheme, where charities are encouraged to voluntarily report risky incidents that pose a threat to their operations. We analysed data from 20,400 charities for patterns in the reporting of incidents and the resulting regulatory actions. We showed that serious incident reporting scheme “early adopters” were more likely to report incidents, and do not accurately reflect risk distribution across the sector. Older, larger charities (less likely to have regulatory enforcement action taken against them) were also the most likely to proactively report risks to the regulator.

Research finding: Regulators can and should use a range of measures drawn from their data when estimating the size and health of the charity sector

Regulators are often asked by both policy makers and practitioners to report on the size and health of the sector. [R3] describes the challenges in measuring the charitable-sector growth using the data available to regulators and compares different measures for understanding sector health using regulatory data. Using administrative data from charities in different jurisdictions, we illustrated the effect of different measures. This showed regulators should not rely on just one measure but should instead make best use of the range of measures available to them in the data they hold.

Research finding: Regulatory data can be used in evaluating the effectiveness of regulatory actions in order to design more effective interventions

[R4] examined a tool of regulatory compliance, ‘naming and shaming’, used by the Fundraising Regulator in England & Wales. We analysed data on 4,147 English and Welsh charities eligible to join the Fundraising Regulator, using an innovative quasi-experimental regression discontinuity design to estimate a regulator-action causal effect. This showed that as a method to encourage compliance with a voluntary scheme, this policy was effective. But we argue the potential for longer-term negative consequences for trust in the new regulator. We also demonstrated the use of administrative data from charity regulators for policy evaluation.

3. References to the research

All peer-reviewed Publications

[R1] **McDonnell D & Rutherford AC (2017) “The Determinants of Charity Misconduct” Nonprofit and Voluntary Sector Quarterly 47 (1), pp. 107-125. DOI: 10.1177%2F0899764017728367

[R2] **McDonnell D & Rutherford AC (2019) “Promoting charity accountability: Understanding disclosure of serious incidents” Accounting Forum 43 (1), pp.42-61. DOI: 10.1080/01559982.2019.1589903

[R3] Pennerstorfer A & Rutherford AC (2019) “Measuring growth of the nonprofit sector: The choice of indicator matters” Nonprofit and Voluntary Sector Quarterly, 48 (2), pp. 440-456. DOI: 10.1177%2F0899764018819874

[R4] Rutherford, AC, McDonnell, D & Hogg, E (2020) “Incentivising Regulatory Participation: Effectiveness of a Fundraising Levy” Public Administration Review. DOI: 10.1111/puar.13176

* Please note, McDonnell was not a Stirling employee when these papers were published; Rutherford was.

4. Details of the impact

Our work “ contributed to [OSCR’s] strategic objectives of ‘helping the public to have more confidence in charities’ and ‘ensuring that registration and reporting is straightforward and proportionate’ [S1, Head of Casework, OSCR].

Our research underpinned the Scottish Charity Regulator’s (OSCR) regulatory process improvements; increased their regulatory efficiency; reduced the administrative burden on 24,500 Scottish charities; and developed their conceptualisation and operationalisation of risk from 2014 to the present (figure 1).

Our work made a significant contribution, as “ without the work carried out [by] the University of Stirling, [OSCR] would not have had the skills or capacity to carry out such in-depth analysis to further our understanding of risk in charities” [S1].

Figure 1: Impact Model, showing how the research informed the linked impacts

Embedded image

Impact: Moving to risk-based processes has streamlined the regulator’s work, allowing more efficient use of the regulator’s limited resources.

Based on our work, OSCR has created a dedicated Risk Team [S1] with staff from across the organisation, reflecting the importance of combining data across the organisation [R2, R3] in evaluating risk. Using our research data and indicators, this team have developed and piloted a Risk Assessment Tool [S5].

“Most significantly, the collaborative work contributed to the development of a dedicated Risk Assessment Pilot Team and Risk Assessment Tool. The tool features key fields identified in the research programme, such as source of evidence, as relevant to predicting case outcomes.” [S1]

Embedded image Figure 2: OSCR Risk-based case classification[S7]

Our insights [R1, R2] allowed OSCR to use their limited resources more efficiently to focus on risk in the charitable sector. Following the introduction of risk-based triaging in 2017, the percentage of investigations resulting in a “Not Appropriate” outcome fell from 50% to less than 20% (Figure 2, [S7]), allowing OSCR to focus their resources on the riskiest concerns within their remit.

“[A] real change that resulted from [the research] has been the redeployment of resources and the creation of ‘Not Appropriate’ case files when the subject of a reported concern is not within our regulatory remit. This is a much more streamlined area of work which allowed more focus on the complex cases.” [S4, Engagement Manager, OSCR]

Impact: The Scottish Charity Regulator has changed their complaint-handling processes to be more targeted, based on our models of risk.

Our research [R1, R2] changed OSCR’s complaint prioritisation and the decision-making about which complaints should lead to full inquiries and regulatory action. We showed that some types of charity were much more likely to receive complaints that were not substantiated, and that the source of the complaint was a good predictor for the likelihood of regulatory action. This allowed OSCR to streamline their processes and better prioritise their responses to the complaints about charities that they received.

The effect of targeted regulation can be seen in the investigations and enforcement actions that OSCR undertakes. Measuring the direct contribution of our research on these changes is challenging. However, using our insights, OSCR now takes more consideration of the source of a complaint in assessing the appropriate action. Table One, below, shows the percentage of complaints that resulted in some form of regulatory action by the source of that complaint, comparing the period 2008-2014 to 2015-2018. It shows that following our research [R1], complaints about charities from professionals (e.g. auditors, accountants, etc.) increased in their likelihood of resulting in regulatory action, while the regulatory response to anonymous reports fell, and public reports was stable.

Table One: Proportion of Complaints Leading to Regulatory Action by Source [S6]

Complaint Source 2008 to 2014 2015 to 2018
Professional 46% 63%
Other body 45% 49%
Internal to Charity 35% 44%
Anonymous 28% 18%
Public 27% 29%

Source: Internal OSCR data on complaints and investigations (2019)

Compared to the proportion of complaints acted on pre-2015, this would suggest that post-2015 an additional 14 regulatory actions were undertaken based on Professional complaints, and 14 fewer regulatory actions on the basis of Anonymous complaints. This is not a causal analysis of a change in the regulator’s behaviour, but reflects the process of moving to a data-driven risk-based approach to regulation that was strongly informed by our research. This is confirmed by the Head of Casework for OSCR:

“That [R1, R2] has been really quite influential over the past 18 months where we have been trying to operationalise that targeted-regulation, in terms of how we go about inquiries and the risk assessment phase. It’s influential in two ways. One is what kind of weight we give different sources of intelligence, […]. That’s then coalesced into thinking about how we deal with the journey for people who raise concerns with us, and about the purpose of our inquiries.” [S3, Head of Casework, OSCR]

Impact: The Scottish Charity Regulator has developed their conceptualisation and operationalisation of risk, connected to the academic literature.

Our research [R1, R2, R3, R4] and the collaborative working supported OSCR in developing their conceptualisation of risk, and how it could be operationalised in their work. In particular, in setting up a cross-department risk team and tool [S5], they moved from a focus on financial measures of risk to measuring risk more broadly with the data they have available.

“Robust analysis and insight from an academic perspective helped us identify and evidence priorities for information gathering and targeted monitoring that gave us confidence to move to targeted regulation.” [S1, Head of Casework, OSCR]

Our research has underpinned the building of evidence-led practice within OSCR, with strong participation in knowledge exchange and data sharing across the organisation. We developed this through regular meetings, presentations, workshop participation and practitioner reports.

“[University of Stirling] kept up regular visits to talk to SMT members and business managers about the research findings. The practitioner reports are incredibly helpful. I look at [the research] sometimes so that we can pull out directions of what we’ve learned, feed that into things when we are reviewing the risk framework, say ‘what have we got on this’.” [S4, Engagement Manager, OSCR]

As well as specific examples of modelling risk, our work [R1, R2, R4] allowed OSCR to critically reflect and refine their rationale for collecting and linking data about charities, and consider an evidence-based approach to assessing the impact of regulatory actions.

“It’s been really useful to have external perspectives drawing on our information, acting as a critical friend, doing a ‘deep dive’ into the data. It’s really enriched the scope we’ve had to think about the practical application of our risk-led approaches, and it’s allowed us to test some thoughts and hunches that we had with more rigour behind it.” [S2, Chief Executive, OSCR]

Impact: By prioritising the data to be collected, the annual reporting burden on charities has been reduced.

Charities (n=24,500) must submit Annual Returns to OSCR. There is a difficult balance between collecting the data needed to measure risk, and over-burdening charities with information requests. Using our research [R1, R3], OSCR now reviews the data they collect, and the way they use that data to measure risk. Collecting the right data improves the efficiency of the regulator and reduces the reporting burden on charities.

“We’ve had to consider are we collecting the right information […] in our Annual Returns and other sources. In some cases we realised that some of the data we were collecting was of very little value, so we’ve reduced that reporting burden on charities. Sometimes we’ve started collecting new data.” [S2, Chief Executive, OSCR]

We [R1] considered the key performance indicators that OSCR were using to measure risk, and how predictive these were of the negative outcomes that the regulator was concerned about. We identified which indicators were the more targeted measures; this underpinned the review of the risk measures used by the regulator and the data which was to be collected from charities in the Annual Return, described by OSCR’s Engagement Manager:

“On the financial vulnerability and indicators, there were things we had a hunch were not working well […]. Now we have made changes to the Annual Return, we have the benefit of the completed research [R1, R2, R3], it has really helped us think about how we use information, how we keep data, and how we structure the Annual Return.” [S4, Engagement Manager, OSCR]

Using data from across the organisation, including financial information; risk indicators; complaints and investigations; and serious incident reporting, we explored how these disparate data sources could be combined to better understand different forms of risk in the charitable sector, and OSCR’s engagement with charities. Our research [R1, R2, R3] allowed OSCR to understand the relative value of different data sources, and informed their decisions about how to collect the right data more efficiently, in order to inform their casework:

“That aspect of the research [R1, R2, R3] has been really useful in getting us thinking about how we treat these things, how do we weigh the concerns that come through the door. The essence of being risk led is having the confidence in your perception of risk and you can’t do this unless you have the data and a useful analysis of that to back it up, otherwise you’re just going on instinct. We were collecting a lot of data, but it wasn’t giving us what we needed in terms of analysable data. We’ve streamlined the data collection, we can get a snapshot of what’s going on at any time.” [S3, Head of Casework, OSCR]

Finally, the overall value that OSCR has placed on our work is further evidenced by the fact that “ the findings have been discussed with other UK and international regulators to share our [OSCR’s] experience on the subject matter of risk [S1, Head of Casework, OSCR] .

5. Sources to corroborate the impact

| S1. Testimonial Letter from Head of Casework, OSCR || | --- | --- | --- | | S2. Interview with former Chief Executive, OSCR (2011 to 2019) || | S3. Interview with Head of Casework, OSCR || | S4. Interview with Engagement Manager: Policy & Research, OSCR || | S5. OSCR Risk Assessment Tool | | S6. Complaints Data Analysis | | S7. Risk-based Case Classification Analysis |

Additional contextual information

Grant funding

Grant number Value of grant
ESRC-OSCR £275,000