Impact case study database

Our research, software and training have achieved impact in terms of capacity building of advanced statistical skills in non-academic and non-specialist users, both nationally and internationally, as well as in directly improving the information underlying evidence-based policy making in the UK public sector and Government.

Building capacity of professional statisticians and quantitative researchers to apply multilevel modelling techniques in their work

Our flagship MLwiN software [5] package now has over 15,000 users, and since 1^st August 2013 it has been purchased by a further 193 new non-academic organisations [A], including the UK Home Office [F] and US Food and Drug Administration Office of Acquisitions and Grants Service. MLwiN is part of a suite of software packages including Stat-JR, Realcom and MLPowSim which are either supplied with MLwiN or as freeware. The linked on-line training course Learning Environment for Multilevel Modelling (LEMMA) contains 15 graduated modules starting from an introduction to quantitative research and progressing to multilevel modelling of continuous, binary, ordinal, and nominal data. There are now over 37,000 users of LEMMA with over 25,000 new users since 1 August 2013, 80% of who are non-UK based. Since August 2013 there are over 10,000 new non-academic users [B].

Since 2008 we have led annual courses with the Royal Statistical Society in London, with five courses between August 2013 and 2018, during which time we have trained 79 participants of whom 21 were government statisticians and 11 industry statisticians, including participants from the Swedish National Agency for Education, the UK Government Department for Health, Google and Virgin Media [C]. Each year, the course has received excellent feedback with an average score of 9.4 out of 10 agreeing the course will benefit their professional work [C]. Following one of these courses the Eurofound European Agency in Dublin requested a bespoke course which supported them to “[use] a multilevel approach to analysis of employment status in Europe which allows it to separate micro-/workplace-level and contextual effects. The conclusions of this research contribute towards the on-going EU policy debate on job quality and non-standard employment.” [C]. Both this example, and feedback from Ofsted [D], point to the importance of this robust analysis to inform policy makers; ‘the training course was very helpful for us in building our skills in data analysis we undertake on a regular basis which seeks to improve the overall quality of education and training in the UK, and inform policy makers about the effectiveness’ [D].

Our software packages (MLwiN and Stat-JR) are also used for training courses delivered by other researchers around the world. Since 2013 there have been over 30 courses open to non-academic participants, including 19 through the National Centre for Research Methods in the UK, and courses run in Australia, Canada and Germany. As one example, over past three years Ian Dohoo from Canada has presented three courses to 67 participants using MLwiN, open to professional non-academic participants, including 20 participants at a course in University of Queensland, Australia in 2018 [Ei, Eii].

Our latest software development, funded by the British Academy [iii], is new functionality in our StatJR software to automate the creation of training materials. This has encouraged others to work with us, including in 2019 the UK Data Service (UKDS) who have used StatJR to produce online training materials using their national data resources. This has impacted directly on their own training strategy and they recognise that the materials produced will have significant impact on researchers in terms of learning basic statistics techniques using key datasets held by the UKDS. As the Director of User Support and Training for the UKDS, stated: *“The materials produced […] will make this dataset more visible to our wide community of existing researchers, help our training effort and attract new users to our resources.*” [J].

Informing educational and social policy development by UK and International Public Sector & Government Services

Since August 2013, our statistical research [2-5] and software [1, 6] has been cited in over 25 national and international government and NGO reports in the areas of education and children’s services, community services, welfare services, public health, energy and climate change and agriculture and defence, including by several UK government departments, the World Health Organisation, Public Health England, Defence Research and Development Canada, and Ofqual. This demonstrates our research has been applied extensively by many public sector and government researchers to analyse complex data sets and has informed policy decisions.

In 2017, the Home Office approached CMM to apply our expertise in multilevel modelling to analyse the geographical predictors of crime and incidents. This contributed to the development of a key policing policy and resulted in an extensive report (not currently in the public domain) informing national policing strategy [F]. This commission from the Home Office [ii] also involved training government statisticians to ensure the analysis can be replicated with each new year of data, as well as applying multilevel modelling techniques to future policing policy. The Head of Policing and Police Resources Team, Crime and Policing Analysis, Home Office confirmed that “ The capacity of our analytical team to undertake advanced multilevel statistical analysis has been markedly improved” and in addition “Our analysts have been able to apply these techniques to at least one other project, unconnected to the original commission.” [F].

Prior to 2014, we worked with the Higher Education Funding Council for England (HEFCE, now Office for Students) to support their use of MLwiN [1, 5] to address fair admission to university for different ethnic groups. In the current REF period, HEFCE asked Browne to assist with changes to the UniStats website used by all prospective university students to compare courses, utilising his sample size calculations work on the MLPowSim package [6]. Browne was approached in 2014 to help HEFCE with a consultation about reducing threshold levels below which data cannot be shown on the Unistat website (available until end of 2019, but now replaced by updated DiscoverUni). He was able to supply research-based statistical support to the proposal to reduce the threshold from 23 to 10 students, as well as indicating the need for more transparency in the methods and statistical uncertainty in the presentation of data [G, H]. These recommendations were implemented by HEFCE, with the Head of Research Analysis, Office for Students, stating: “These changes mean that not only do prospective students have more data available on more courses at more institutions, but they are also supplied with additional resources to aid them in understanding the data presented to them.” [G, H].

Finally, our research has been used to improve the accuracy of information underlying the evidence base for policy development. For example, our research on missing data [4], has been used by others to improve the statistical methodology employed to evaluate the effectiveness of at least three school-based randomised control trials carried out by the Education Endowment Foundation [I].