Impact case study database
Answering educational, economic and social policy questions with novel methods and training for robust statistical assessment
1. Summary of the impact
Complex multilevel datasets are a challenge to analyse, especially for non-specialists, but are fundamental to answer questions in diverse settings. The University of Bristol’s Centre for Multilevel Modelling (CMM) addressed this by incorporating its cutting-edge statistical research into our widely used MLwiN multilevel modelling software, face-to-face training and training materials. This has significantly increased the number of non-academics and non-specialists equipped to apply these techniques in their work. These users have transformed the application of new statistical practices within government departments (such as the UK Home Office) and other non-academic organisations (such as the UK Office for Students and the Eurofound European Agency in Dublin) in the UK and overseas, empowering them to improve the accuracy of evidence-based decision making. Since August 2013, MLwiN has been purchased by 191 non-academic organisations internationally. This includes the UK Home Office, who commissioned bespoke training from CMM researchers, enabling multilevel modelling techniques to be applied to crime and policing policy.
2. Underpinning research
Often the data underlying decision making in policy, companies and agencies (particularly in social science contexts) has a complex hierarchical, nested or clustered structure. For example, educational outcomes might be impacted by national, local authority, school and teacher effects in complex interacting dependencies. These effects need to be accounted for to allow reliable decision making.
Since 2004, the Centre for Multilevel Modelling has developed new statistical methodology to fit such complex models. Much of the fundamental underpinning work is summarised in Goldstein’s seminal book [1], which is widely used across social, medical and biological sciences, receiving 12631 citations (Google Scholar, January 2021).
Examples of novel statistical developments in CMM work include:
extending multilevel models to account for spatially correlated effects (e.g. student outcomes) [2],
introduction of re-parameterisation techniques within Markov chain Monte Carlo (MCMC) algorithms for fitting multi-level models, increasing statistical efficiency and hence enabling significantly larger and more complex data structures to be analysed (e.g. survival analysis) [3],
novel multilevel modelling formulations for complex non-hierarchical data structures (e.g., combination of schools and neighbourhoods on pupil performance) [4],
methodological research on the problems of missing data and measurement errors (e.g., in birth cohort studies and longitudinal surveys) [5],
advanced multilevel modelling for analysis of longitudinal data with multilevel structure (e.g., school league tables) [6]
A key aspect of CMM’s work is translating these statistically novel but computationally intensive methodology developments into user-friendly statistical software packages (such as MLwiN, Stat-JR, Realcom and MLPowSim) to allow direct access to new methods for applied researchers [1-6]. This translation is performed by the programming team, currently led by Charlton. These packages are accompanied by extensive user documentation and training disseminated via face-to-face user workshops and online training materials, including the Learning Environment for Multilevel Modelling (LEMMA) online multilevel modelling course developed as part of ESRC LEMMA grants [vii] which brings the methodology and software to a very wide audience.
3. References to the research
[1] Goldstein H. (2011) Multilevel Statistical models (4th Ed.). Chichester: Wiley. [Book available on request].
[2] Browne WJ & Goldstein H. (2010) MCMC sampling for a multilevel model with non-independent residuals within and between cluster units. Journal of Educational and Behavioural Statistics, 35(4), 453-473. DOI: 10.3102%2F1076998609359788
[3] Browne, W.J., Steele F. A., Golalizadeh, M., and Green M.J. (2009) The use of simple reparameterizations to improve the efficiency of Markov chain Monte Carlo estimation for multilevel models with applications to discrete time survival models. Journal of Royal Statistical Society, A, 172, 579-598. DOI: 10.1111/j.1467-985X.2009.00586.x
[4] Steele, F. A., Washbrook, E., Charlton, C. & Browne, W. J. (2016) A Longitudinal Mixed Logit Model for Estimation of Push and Pull Effects in Residential Location Choice. Journal of the American Statistical Association. 111, 515, p.1061-1074 DOI: 10.1080/01621459.2016.1180984
[5] Goldstein H, Carpenter JR, Browne WJ. (2014) Fitting multilevel multivariate models with missing data in responses and covariates that may include interactions and non-linear terms. Journal of the Royal Statistical Society: Series A. 177(2), 553-564. DOI: 10.1111/rssa.12022
[6] Leckie G, Goldstein H. (2009) The limitations of using school league tables to inform school choice. Journal of the Royal Statistical Society, A, 172, 835-851. DOI: 10.1111/j.1467-985X.2009.00597.x
Research Grants:
[i] Browne WJ et al. (2018-2019) Borrowing Strength – a collaborative software development for Small Area Estimation, ESRC: GBP99,000
[ii] Browne WJ, Leckie G, Goldstein H. (2017) Consultancy project on crime data, Home Office: GBP45,000
[iii] Browne WJ et al. (2016-2018) Using Statistical E-books to teach undergraduate students quantitative methods and statistical software, British Academy: GBP115,000
[iv] Browne WJ et al. (2013-2017) The use of interactive electronic-books in the teaching and application of modern quantitative methods in the social sciences, ESRC: GBP786,000
[v] Browne WJ et al. (2009-2012) e-STAT –NCeSS quantitative node, ESRC: GBP1,100,000
[vi] Browne WJ. (2006-2009). Sample Size, Identifiability and MCMC Efficiency in Complex Random Effect Models, ESRC: GBP174,000
[vii] a) Rasbash J et al. (2005-2008) Learning Environment for Multilevel Modelling Applications (LEMMA), ESRC: GBP650,000
b) Rasbash J et al. (2008-2011) STRUCTURES for Building, Learning, Applying and Computing Statistical Models (LEMMA II), ESRC: GBP1,200,000
c) Steele F et al. (2011-2013) LEMMA III, ESRC: GBP1,400,000
[vii] Goldstein H et al. (2003-2005) Developing Multilevel Models for Realistically Complex Social Science Data, ESRC: GBP300,000
4. Details of the impact
Our research has strongly built capacity for advanced statistical skills in non-academic and non-specialist users and directly informed policy development in the UK public sector and Government.
Building capacity of professional statisticians and quantitative researchers to improve robustness of decision making by applying multilevel modelling techniques
Our flagship MLwiN software package [5] now has over 15,000 users. Since 1st August 2013 it has been purchased by a further 191 new non-academic organisations [A], including the UK Home Office [F] and US FDA Office of Acquisitions and Grants Service. MLwiN is part of a suite of software packages including Stat-JR, Realcom and MLPowSim which are either supplied with MLwiN or as freeware.
The linked LEMMA training course contains 15 graduated modules starting from an introduction to quantitative research and progressing to multilevel modelling of continuous, binary, ordinal, and nominal data. There are now over 37,000 users of LEMMA with over 25,000 new users since 1 August 2013, 80% of whom are international. Since August 2013 there are over 10,000 new non-academic users [B].
Since 2008, we have led annual courses with the Royal Statistical Society in London during which we have trained 79 participants (of whom 21 were government statisticians and 11 industry statisticians) including participants from the Swedish National Agency for Education, the UK Government Department for Health, Google and Virgin Media [C]. Each year, the course has received excellent feedback, with an average score of 9.4 out of 10 agreeing the course will benefit their professional work.
Our model supports policy makers in robust analysis to inform decision making. For example, following one of these courses the Eurofound European Agency in Dublin requested a bespoke course to support analysis of employment status in Europe. The successful delivery of this course gave their workers the skillset to “[use] a multilevel approach … which allows it to separate micro-/workplace-level and contextual effects. The conclusions of this research contribute towards the on-going EU policy debate on job quality and non-standard employment.” [C]. Both this example, and feedback from Ofsted [D], point to the importance of this robust analysis to inform policy makers; ‘the training course was very helpful for us in building our skills in data analysis we undertake on a regular basis which seeks to improve the overall quality of education and training in the UK, and inform policy makers about the effectiveness’ [D].
Our software packages (MLwiN and Stat-JR) are also used for training courses delivered by other researchers around the world. Since 2013 there have been over 30 courses open to non-academic participants, including 19 through the National Centre for Research Methods in the UK. Courses have run in Australia, Canada and Germany [E], for example at the University of Queensland, Australia open to non-academic participants [E].
A recent British Academy grant [iii] supported CMM in further developing our Stat-JR software to automate the creation of training materials. This encourages others to work with us. For example, recently the UK Data Service have used Stat-JR to develop online training using their own national data resources, with a significant effect on their internal training strategy. They expect the materials produced to benefit researchers through learning basic statistics techniques using key UK datasets. Their Director of User Support and Training stated: *“The materials produced […] will make this dataset more visible to our wide community of existing researchers, help our training effort and attract new users to our resources.*” [J].
Informing educational, economic and social policy development by UK Public Sector & Government Services
Since August 2013, our statistical research (including [1-6] above) and software have been cited in over 25 national and international government and NGO reports in the areas of education and children’s services, community services, welfare services, public health, energy and climate change and agriculture and defence. These reports include those by several UK government departments, the World Health Organisation, Public Health England, Defence Research and Development Canada, and Ofqual. This demonstrates the extensive application of Bristol research to analyse complex data sets, and has fed into policy and practice decisions.
In 2017, the Home Office approached CMM to apply our expertise in multilevel modelling to analyse the geographical predictors of crime and incidents. This was part of the development of a key policing policy and resulted in a substantial report (the report is confidential and not currently in the public domain) informing national policing strategy [F]. This commission from the Home Office [ii] also involved training government statisticians to ensure the analysis can be replicated with each new year of data, as well as applying multilevel modelling techniques to future policing policy. The Head of Policing and Police Resources Team, Crime and Policing Analysis, Home Office confirmed that “ The capacity of our analytical team to undertake advanced multilevel statistical analysis has been markedly improved” and in addition “Our analysts have been able to apply these techniques to at least one other project, unconnected to the original commission.” [F].
We have worked with the Higher Education Funding Council for England (HEFCE, now Office for Students) to answer several questions, including fair admission to university for different ethnic groups, with results included in their reports on these topics. In 2014, Browne was approached (based on his sample size calculations work on the MLPowSim package) to help HEFCE with a consultation on the UniStats website about reducing threshold levels below which data cannot be shown. We were able to supply research-based statistical support to the proposal to reduce the threshold from 23 to 10 students, as well as indicating the need for more transparency in the methods and statistical uncertainty in the presentation of data [H]. Our recommendations were implemented by HEFCE, with the Head of Research Analysis, Office for Students, stating: “These changes mean that not only do prospective students have more data available on more courses at more institutions, but they are also supplied with additional resources to aid them in understanding the data presented to them.” [G, H].
Finally, our impact can be more indirect. For example, our research on missing data [5], has been used by others to improve the statistical methodology employed to evaluate the effectiveness of at least three school-based randomised control trials carried out by the Education Endowment Foundation [I].
5. Sources to corroborate the impact
[A] MLwiN user database (2020). Download and sales figures report
[B] LEMMA training materials user database (2020). Registered users report
[C] i) Royal Statistical Society (2018). Executive Director - Factual statement
ii) Eurofound (2020). Research Officer - Factual statements
[D] Ofsted (2020). Factual statement
[E] University of Queensland, Australia (2018). MLwiN Training Course Flyer
[F] Home Office (2018). Head of Policing and Police Resources Team - Factual statement
[G] Office for Students (2018) Head of Profession and Head of Research Analysis - Factual statement
[H] HEFCE/Unistats (2015). i) Data publication thresholds and aggregation on Unistats
Technical advice from Browne (para 41), and consequent proposal to lower the publication threshold (para 63) & provide contextual information on methods & statistical uncertainty (para 64).
ii) Outcomes of the consultation on data publication thresholds and aggregation on Unistats and for the NSS
[I] Education Endowment Foundation (2014). Chatterbooks Evaluation Report and Executive Summary
[J] UK Data Service (2019). Director of User Support and Training - Factual statement
Additional contextual information
Grant funding
| Grant number | Value of grant |
|---|---|
| n/a | £99,000 |
| n/a | £45,000 |
| n/a | £115,000 |
| ES/K007246/1 | £786,000 |
| ES/G034834/1 | £1,100,000 |
| RES-000-23-1190-A | £174,000 |
| n/a | £650,000 |
| ES/F031904/1 | £1,200,000 |
| ES/I025065/1 | £300,000 |
| n/a | £300,000 |