Impact case study database

Context

‘*Sudan has long been a conflict-affected country with very little support from the international community. More than three million children were not in school in Sudan in 2015-2016 – one of the highest numbers in the world; and those in school, often did not complete it. While access had been a big issue, the quality of education was also very poor, because of a weak system which struggled with a severe shortage of trained teachers, textbooks, learning materials, school buildings and equipment. Most importantly, there was no clear assessment of student learning making it difficult for the policy makers to plan any intervention. A learning assessment system was certainly necessary.*’ Country Lead, Global Partnership for Education (GPE) [E1].

The National Learning Assessments (NLA)

Following a pilot study in Khartoum State in 2014, the first national assessment of literacy and numeracy levels amongst Sudanese children was rolled out across all 18 States in the Republic of Sudan in 2015, encompassing some 18,000 students, against the background of civil unrest and conflict in some states. The reading proficiencies of 9,090 learners in the third grade of schooling (typically 8 or 9 years old) were assessed, as was the mathematics proficiencies of a further 9,055 learners in the third grade. Johnson was responsible for the research design and, employing methods established in R2, R4 and R5, guided the processes of data cleaning and analysis, and took responsibility for the analysis and the writing of the final report [E2].

The 2015 National Learning Assessment (NLA) report [E2], presented to the Minister of Education in Sudan, highlighted the reading proficiencies of children in Sudan compared to other countries in the Middle East and North Africa. The study found that Sudan had the highest proportion of children unable to read (40%) compared to that in Yemen, Egypt, Jordan, Morocco, and Iraq (17-27%), and that average reading speed in Sudan was substantially slower (15 words per minute compared to 21-27 wpm in Morocco, Jordan, Iraq, and Egypt). Younger children in Sudan were not being taught to read in ways that are seen to be policy-salient internationally. There has been much debate about the teaching of synthetic phonics in reading and this is now a fundamental policy plank for most high achieving countries in Europe and North America. According to the Senior Education Specialist at the World Bank, ‘ What was missing was the capacity to diagnose, country-wide, the situation and also to have a dialogue about the roles of the local government or local education authorities’ [E3]. Hence, for the first time, the diagnostic nature of the report’s results enabled the Ministry of Education to develop national-level interventions for the teaching of reading in the early grades which included synthetic phonics.

The findings of the 2015 report [E2] were incorporated into both the Sudanese government’s Education Sector Strategic Plan (2018) [E4] and the World Bank and GPE’s collaborative USD76,500,000 Sudan Basic Education Recovery Project (BERP) [E5]. This grant, according to the GPE’s Country Lead for Sudan, ‘was the biggest grant, perhaps the only big grant, to the education sector in Sudan during the period 2013-19’ [E1]. The World Bank’s Senior Education Specialist added that ‘ the results of the first assessment in 2015 changed the way that we targeted the other interventions in the [BERP] project. Originally all of the targeting for school construction or school grants was based purely on equity, on need, without any direct evidence of learning outcomes. After 2015 it changed’ [E3]. The NLA identified 6,000 of the worst performing schools, and consequently the BERP re-orientated its interventions to target these schools. A total of USD1,250,000 funding was given in grants to these schools, to be managed autonomously at school-level by management committees, led by key stakeholders such as teachers, parents, and community leaders, and used to fund teaching and learning materials as well as school facilities and equipment [E6, E7]. These investments contributed ‘ to building an education system for Sudan capable of providing [education] services and ensuring their continuity’, according to the Federal Minister of Education [E8].

The NLA was expanded and repeated as part of the BERP in 2018, overseen by Johnson, and the subsequent report [E9] showed an overall improvement in learning in the country. The title of this report, ‘Are more children learning, and are they learning more?’ was aimed at policymakers, drawing attention to the value of repeat studies. While the report was unable to identify which specific interventions had the biggest individual effects, it was clear that the overall package of interventions that the government had put in place after the release of the 2015 report had had some positive impact across states and schools, in that the average percentage of those unable to read dropped from 40% to 39% across the country in the period between the two assessments.

The findings showed significant impact on student learning in those schools which had received grants. In 2015, 47% of students in these schools were not able to read any words in a passage of connected text, higher than the national average of 40%. In 2018 the share of non-readers in the intervention schools had reduced to 42% [E7, E9]. Although the share of non-readers in these schools was still higher than the national average, the improvement is significant. The average reading rate in the intervention schools increased between the 2015 and 2018 assessments, from 12.18 to 15.49 words read per minute, a significant increase that has gone beyond the national average of 14.4 words per minute [E7]. The 2018 assessment also showed a 3.5 percentage increase in the proportion of students reading modestly well, from 35% in 2015 to 38% in 2018, which compares well to the picture at the national level [E9].

The 2018 study revealed a ‘peace dividend’ with states that had been identified as the worst performers, because of ongoing low-level conflict and the constraints on service delivery, making the most overall progress. Central Dafur, West Dafur, East Dafur, Blue Nile, North Dafur, North Kurdufan, River Nile, Gezeera, and West Kurdufan made substantial progress in reducing the number of non-readers and also made progress in raising the number of words read on average in one minute [E9]. As the GPE country lead stated, ‘the results from Johnson’s research clearly showed that the intervention schools were doing much better than the non-intervention schools. This enthused the Government to make learning assessment an integral part of the next intervention’ [E1].

As a result of the success of the BERP, an additional USD61,000,000 of funding from 2020 onwards has been allocated by the World Bank for the Sudan Basic Education Support Project (BESP) to build on the achievements of the BERP [E10]. The BERP completion report noted the importance of the NLA programme as ‘a major input into the M&E [monitoring and evaluation] system providing learning outcome information to guide policies and program implementation’ [E6, p. 30] and in institutionalising evidence-led decision making and policy in the Ministry of Education [E1]. Central to the BESP programme is a focus on the improvement of reading. Johnson was also instrumental in a pilot, involving 42 schools, and later scale-up to 2,000 schools, of a reading intervention as part of the World Bank initiative. Consequently ‘Every state will have one locality which will implement this reading improvement programme in the new [BESP] project’ [E3]. The design of this reading intervention builds on recommendations from both reports and adopts a synthetic phonics approach, using graded supplementary readers. These reading materials, selected by Johnson, will be provided by Oxford University Press to the government of Sudan [E3]. Johnson also designed recording sheets to accompany the readers to allow for effective monitoring and assessment of reading progress [E3].

Johnson oversaw the training of the assessors in the Khartoum pilot and the NLA assessments, furnishing the country with a body of professionals able to continue the process of continuous assessment [E3]. A third NLA, again led by Johnson, will be carried out in 2021. The three NLA assessments, combined with a commitment to undertake an additional three NLA cycles until 2027, will establish a trend line of student performance within Sudan and allow the effective monitoring of interventions.

‘Professor Johnson’s input helped convince the Government of Sudan about the importance of learning assessment and its enormous contribution to designing program intervention. It also helped institutionalization learning assessment in Sudan. His professional expertise also gave confidence to us that the education sector in Sudan was moving in the right direction with our support. His contribution has benefitted the Government and, of course, the children of Sudan through a country-wide intervention.’ GPE Country Lead [E1].

The Introduction of the National Learning Assessment in Nigeria

The impact of Johnson’s research on the use of national learning data to drive educational policy [R2, R6] has also led to a return to Nigeria. As part of the World Bank’s State Education Program Investment Project [E11] he was commissioned to carry out Nigeria’s first ever National Learning Assessment, which was concluded in December 2019. The formal report was expected to be published late 2020 but the initial fieldwork has provided positive internal results and will be used by the World Bank to develop their Education Sector Support Plan for Nigeria [E3]. It has also been used by UNESCO and the World Bank to update Nigeria’s Human Capital Index (HCI) score which has, up to now, relied on data from selected states and was last reported in 2010 [E3].