Impact case study database
Informing higher education policy on how to understand and improve graduate outcomes
1. Summary of the impact
Professor Vignoles and colleagues researched ways to measure the benefits of higher education (HE), both in terms of learning gains and subsequent labour market outcomes, with the aim of providing information to students and policy makers that might guide their decision-making. Their research also showed the sizeable socio-economic gap in graduate outcomes.
Research into student learning gains led to the development of a framework and measuring tool that informed Dame Shirley Pearce’s independent review of the Teaching Excellence Framework (TEF), and the subsequent Government response. Research into the employment and earnings of graduates catalysed the creation of the Longitudinal Educational Outcomes dataset (LEO), downloaded by over 32,000 individuals. LEO data is used in the TEF, informed the Augur Review and more broadly has shaped government policy on graduate employment and focussed attention of universities on it.
2. Underpinning research
Background
There is a socio-economic gap in graduate outcomes in the UK, which policy makers and institutions consider a priority to understand and reduce [ R1]. Professor Vignoles co-published a paper which demonstrated how earnings of graduates around 10 years into the labour market vary with gender, institution attended, subject and socioeconomic background [ R2]. This study has been highly influential in the debate on social mobility, as it was the first to provide detailed insights into the way that students’ family origins continued to influence their earnings after graduation, even after taking account of the subject studied and the institution attended. This work has promoted wider understanding that while there are things universities could do to help narrow this gap, and student choices are also key, there are wider structural factors that also impact on social mobility.
Graduate Earnings
Students need high quality information on employment outcomes when making choices about their degree, particularly students whose families do not have experience or knowledge of HE. Previous measures of employment outcomes focused on whether students were employed 6 months after graduation - arguably too soon to measure the real benefits of HE. Vignoles and colleagues used data on students’ earnings to construct a much higher quality measure of employment outcomes and published a paper which verified the superior scale, quality, and duration of the data [ R3]. This dataset was a first in the UK, and Professor Vignoles was Principal Investigator of the project that proved its value. This research catalysed the linking of Department for Education and HMRC datasets to create the new LEO dataset, enabling higher quality information to be made available to policy-makers and students.
Subsequently Vignoles and a large multi-institution team received funding from the Department for Education for further research on graduate earnings using the LEO dataset [ R4, R5].
Learning Gain
In 2015 the Higher Education Funding Council in England (HEFCE) funded research into developing consistent sector-wide measures of student learning. The TEF relied on student satisfaction measures (e.g. the National Student Survey) which provide only a partial picture of the student experience and which could incentivise institutions to maximise student satisfaction by reducing academic expectations. The Faculty’s 3-year study (the LEGACY project) developed a theoretically informed, usable measure of HE learning gain, and was developed with significant input from students [ R6].
The research generated:
A new conceptual framework and measurement tool to measure learning gain with six main aspects: cognitive, metacognitive, affective, and socio-communicative components, as well as attitude to research and moral aspects.
Empirical evidence, based on a three-round longitudinal survey of over 4000 students, of the robustness of a low cost measurement tool that captures these aspects.
The research concluded that any measurement tool might be gamed if used in high stakes accountability systems, with league table style comparisons across subjects and institutions. However, as a tool for institutions to measure the learning gain of their students and enhance their own provision, it would be valuable.
3. References to the research
Key reports and papers:
[ R1] Britton, J., Dearden, L., Shephard, N., & Vignoles, A. (2019). Is improving access to university enough? Socio-economic gaps in the earnings of English graduates . Oxford Bulletin of Economics and Statistics, 81(2), 328-368. [DOI]
[ R2] Britton, J., Dearden, L., Shephard, N., Vignoles, A. (2016). How English domiciled graduate earnings vary with gender, institution attended, subject and socio-economic background. Institute for Fiscal Studies Working Paper W16/06. [Link]
[ R3] Britton, J., Shephard, N., Vignoles, A. (2019). A comparison of sample survey measures of earnings of English graduates with administrative data . Journal of the Royal Statistical Society, Series A, 182(3), 719-754. [DOI]
‘Read paper’ discussed at Royal Statistical Society October 2018. [Link], [Video]
[ R4] Belfield, C., Britton, J., Buscha, F., Dearden, L., Dickson, M., van der Erve, L., Sibieta, L., Vignoles, A., Walker, I., Zhu, Y. (2018). The relative labour market returns to different degrees. Department for Education/Institute for Fiscal Studies. [Link] - main paper. [Link] - appendix.
[ R5] Belfield, C., Britton, J., Buscha, F., Dearden, L., Dickson, M., van der Erve, L., Sibieta, L., Vignoles, A., Walker, I., Zhu, Y. (2018). The impact of undergraduate degrees on early-career earnings. Department for Education/Institute for Fiscal Studies. [Link]
[ R6] Vermunt, J. D., Ilie, S., Vignoles, A. (2018). Building the foundations for measuring learning gain in higher education: A conceptual framework and measurement instrument. Higher Education Pedagogies, 3(1), 266-301. [DOI]
Key grants:
- Vignoles, A., Shephard, N., Dearden, L. and Briton, J. - Estimating the human capital of graduates - Nuffield Foundation - 01 Oct 2013 - 30 Sep 2015 - GBP163,601
Peer-reviewed application. Outputs include papers [ R2] and [ R3] above.
- Vignoles, A. (consortium led by Institute for Fiscal Studies including University of Cambridge University of Westminster, University of Dundee, University of Bath) - An analysis of the employment and earnings returns to higher education using administrative data - Department for Education - 01 Aug 2017 - 28 Oct 2018 - GBP16,506.01
Peer-reviewed application. Outputs include paper [ R4] above.
- Vermunt, J., Vignoles, A., Ilie, S. - The measurement of learning gains in Higher Education across disciplines - HEFCE - Oct 2015 - Sep 2018 - GBP287,000
Peer-reviewed application. Outputs include paper [ R6] above.
[ R1, R6] were peer-reviewed and published in international recognised journals. [ R2, R3, R4, R5] were the outcomes of peer-reviewed grants. The research therefore meets the 2* minimum threshold.
4. Details of the impact
Overview
The combined impact of these research projects is to improve graduate outcomes in England (over two million students a year) by helping students and policymakers make better informed choices. Most importantly, the research has resulted in the provision of information which will help to close the gap in outcomes of graduates from advantaged and disadvantaged socio-economic backgrounds, improving social mobility across England [ S1]. It has provided universities with information to help improve their guidance and information practices, and the investment they make in supporting their graduates’ careers.
Policy impact:
Vignoles’ extensive dissemination activities include contributing to Alan Milburn’s Independent Review on Social Mobility & Child Poverty and joining the cross department Social Mobility Transparency Board, chaired by the then Deputy Prime Minister, Nick Clegg MP and Lord David Willetts [ S2]. On the Board, she contributed to a strategy to enable a legal gateway to link HMRC tax data to educational attainment data, leading to the development of the LEO dataset [ S3].
David Willetts, Minister for Universities and Science wrote:
“I have no doubt that Professor Vignoles’ work has had real world impact on higher education in England…she has developed a long-standing and trusted advisory relationship with the DfE and BEIS over a number of years.” [ S2]
A 2016 government report [ S1] cited Vignoles’ 2015 report as a key piece of research informing the creation of the Small Business and Enterprise Act 2015, which linked higher education and tax data to create the Longitudinal Education Outcomes (LEO) dataset.
In 2016 the government announced that it planned to use the LEO dataset to improve the quality of the labour market data [ R5] that the Teaching Excellence Framework relied on in its quality assessment [ S1].
In 2015, Universities Minister Jo Johnson MP said , “Information of this sort will be incredibly useful for young people choosing courses or jobs that are most suitable for them. It will also enable education providers to assess their effectiveness in delivering positive labour market outcomes for their students. [ S4]
Vignoles’ evidence of a socioeconomic gap in earnings of around 10% following graduation [ R6] led the Office for Students to recommend that institutions should be required to provide more support to help students, particularly those from low income backgrounds, to transition into the labour force [ S5].
In 2017, the Universities Minister announced that LEO data would be incorporated in the subject-level TEF [ S4], with the report on value for money [ S4] and pilot level specification document [ S4] also citing Vignoles’ [ R2] IFS report. Updated metrics that took some, but not all concerns, regarding the use of LEO into account were published [ S1], and the LEO metrics were extended to the pilot subject-level TEF in 2018 [ S1]. The potential for LEO data to be used in the TEF was supported by the [ R5] IFS research report for the DfE of which Vignoles was co-author. This report was directly cited by then Universities Minister Sam Gyimah in his speech launching the extension to LEO as having “ important and far-reaching ramifications for the debate on value for money in Higher Education.” [ S6].
Dr Tim Leunig, Analyst and Chief Scientific Adviser, DfE wrote, “LEO is an important step forward in the information available to government to make policy in this area and Vignoles and her team not only played a pivotal role in its creation but also produced research which has influenced the government’s approach.” [ S7]
Vignoles and her team have stressed the need for appropriate use of these data, particularly in the TEF [ S8]. She has highlighted the need to use them to improve practice, rather than to create league tables that are misleading. Vignoles continued to work with government to try to influence the way that such data are incorporated into the regulatory framework. She met in 2016 with the Minister for Higher Education; she was invited to present to the Department for Education (Oct 2017; Feb 2018); and was a member of the Universities UK Academic Reference Group for the Teaching Excellence Framework (2016). Vignoles regularly advised parliamentarians and politicians; in 2016 she spoke to The All Party Parliamentary University Group and also met with the Secretary of State for Education to discuss social mobility and issues regarding access to HE. In 2017 she met with Oliver Cardinali, advisor to the SoS and in 2018 she provided evidence on the underpinning research to the House of Lords Economic Affairs Committee [ S3] and the Education Committee [ S8]. Her expert testimony was extensively referenced in the House of Commons Select Committee report on value for money in Higher Education [ S8] including supporting her argument that LEO data should not be used inappropriately as a measure of teaching quality. She worked actively to ensure analyses of the LEO data informed the Augur Review and was an advisor to Dame Shirley Pearce for her review of the Teaching Excellence Framework [ S8].
The evidence from LEO encouraged Augur to recommend a rebalancing of investment between Further and Higher Education as set out in the recent White Paper [ S9] **.**The White Paper recognises the benefits of “Investing in higher-level technical qualifications that provide a valuable alternative to a university degree”, a conclusion based on LEO data. The White Paper guarantees financial support 4 years of post-18 education from 2025, reflecting the need to better support the non-graduate skills route.
Both the use of LEO and the concept of learning gain, including the need to recognise their limitations and appropriate use, influenced Dame Shirley Pearce’s review of the TEF.
Dame Shirley noted that Vignoles’ “…specific research expertise was important, for example in guiding recommendations as to how LEO data should be used, and just as importantly how it should not be used. In addition, her research expertise in the field of learning gain informed my key recommendations.” [ S10]. The Government has endorsed the statistical recommendations from the Review [ S10] and has emphasised the importance of measuring learning gain, rather than just the economic outcomes from degrees, noting that it is vital “that the TEF ratings are based on an assessment of high quality, nationally gathered metrics and data (e.g., Graduate Outcomes, Longitudinal Education Outcomes and non-continuation data) and contextual qualitative information. It should use more than just earnings and should take account of regional variations”.
As of the end of May 2019, the original subject-level LEO dataset had been downloaded by over 32,786 individuals as of October 2020 [ S11], thereby suggesting a strong student demand for more information on which to base degree choices. LEO data is now being used in the official information used by the Office for Students and Unistats, as well as the newly developed Discover Uni. Nicola Dandridge, CEO of the Office for Students noted:
“'This important research demonstrates that, for the vast majority of students, there are clear financial benefits of getting a degree…. This research provides a wealth of data to let students understand how the course and institution they choose may affect their earnings potential.” [ S12]
5. Sources to corroborate the impact
[ S1] Government reports referencing graduate earnings research and 2016 IFS paper:
(2016) Success as a Knowledge Economy: Teaching excellence, social mobility, and student choice ( link, p8,12,42); Teaching Excellence Framework, Subject-level pilot specification (2017) ( link, p9), Year two and beyond (2016) ( link), Teaching Excellence Framework and Student Outcomes Framework Specification (2017) ( link); (2019) Independent panel report to the Review of Post-18 Education and Funding ( link, particularly p85-89).
[ S2] Universities Minister (2010-2014) speech (December 2013) ( link) and testimonial (December 2019)
[ S3] House of Lords Economic Affairs Committee (2018) Treating Students Fairly: The Economics of Post-School Education ( link)
[ S4] Universities Minister (2015-2018)Teaching at the heart of the system (2015) ( link); On delivering value for money for students and taxpayers (July 2017) ( link); On embracing accountability and promoting value for money in HE (September 2017) ( link).
[ S5] Office for Students’ response to the report by the House of Commons Education Committee (2018) on value for money in higher education.
[ S6] Universities Minister (2018) Delivering value for money in the age of the student, promoting the 2018 IFS publication (June 2018) (3 in Section 3, Link).
[ S7] Testimonial from the Chief Scientific Adviser at the Department for Education, 2019
[ S8] House of Commons Education Committee (March 2018) Value for money in higher education, oral evidence ( link); Report ( link); Pearce Review https://www.gov.uk/government/publications/independent-review-of-tef-report
[ S9] Augur White Paper https://www.gov.uk/government/publications/skills-for-jobs-lifelong-learning-for-opportunity-and-growth
[ S10] Testimonial from Chair of Governors at LSEGovt response to Pearce Report https://www.gov.uk/government/publications/government-response-to-the-independent-review-of-tef
[ S11] IFS personal communication, 2020 on the download figures for [ R4]
[ S12] Office for Students responds to new research on graduate earnings 27 November 2017 Link
Additional contextual information
Grant funding
| Grant number | Value of grant |
|---|---|
| EDU/41336 | £163,601 |
| UNIVERSITY OF WARWICK (FB HEFCE) | £287,000 |
| GV00124 | £16,506 |