Impact case study database

Accurate educational measurement leads to effective English curriculum design, language learning, teaching, assessment, and better informed, more equitable decision-making in society. The immediate impact of CRELLA’s cutting-edge research in test validation can be seen in improved measurement of proficiency in several high-stakes tests of English language proficiency. The socio-cognitive framework (SCF) has enabled numerous English language test providers around the world to improve test quality and to clarify the proficiency levels underpinning their tests. The effect has been to enhance their reliability and reputation, leading to increased transparency for employers who use the tests. Through validating digital language assessment, CRELLA’s work has provided prospective students and jobseekers a wider and more affordable range of English language tests from which to choose, beyond traditional face-to-face, in-person testing. In short, CRELLA’s work on assessment has opened more doors thus affording greater educational and employment opportunities to successful candidates. Below we present the examples of two of the UK’s leading providers of English language assessments, Cambridge Assessment English (CAE) and the British Council (BC): both base their test design principles on SCF.

The Assistant Director (Assessment) at CAE observes that, “Over the past 20 years, Cambridge English has established a close working relationship with CRELLA and has embedded the socio-cognitive framework (SCF) for language test development and validation in its operation” ( 5.1). For example, SCF’s contextual and cognitive validity features for reading are now used to tag items in CAE’s database of test material, ensuring that individual test papers consistently and accurately represent distinctions between proficiency levels and present a similar experience across for users. The impact of the accurate measurement of English proficiency is substantial: “Cambridge Assessment English qualifications are taken by more than 7 million people every year in 130 countries and recognised by more than 25,000 universities, employers and governments” ( 5.1).

SCF has also provided the theoretical foundations for new digital CAE tests launched in the current REF period, such as Linguaskill, released in 2017. “As of December 2020, this new digital test has been administered over 375,000 times and is recognised by 450 organisations across 52 countries” ( 5.1). The practical value of SCF for assessment design and validation has led to its use in other specialised fields; Cambridge Assessment Admissions Testing employs SCF for the biomedical admissions test (BMAT) ( 5.2), recognised by 40 medical universities worldwide.

Like CAE, BC has implemented SCF in developing, validating, reviewing, and revising its English language tests. Since 2012, SCF has provided the conceptual basis for the digitally based Aptis testing service, used by businesses, governments, educational institutions and NGOs in more than 85 countries, and is used for gatekeeping and diagnostic purposes. According to the Senior Researcher for the BC’s Assessment Research Group, “In the 2019-2020 financial year, approximately 300,000 tests were delivered in 100 countries” ( 5.3). He notes that SCF has helped BC to ensure that the tests exhibit appropriate contextual and cognitive features at different language proficiency levels through:

• test specifications that guide item writers in targeting their questions to learners at different proficiency levels

• rating scales used in awarding scores on the writing and speaking components

• examiner training procedures designed to ensure accurate and consistent judgement

• a research programme designed to ensure that the tests are and remain fit for purpose.

The British Council’s internal continuing professional development programme is based on SCF ( 5.3, 5.4) and SCF provides the theoretical grounding for its ‘How Language Assessment Works’ initiative, which includes a free online FutureLearn course on assessment issues for a global audience. This has been taken by 45,315 people in more than 150 countries to date ( 5.3).

In addition to the CAE and BC testing programmes, CRELLA has used SCF in conducting, supporting and advising on a range of test development initiatives, for new markets in the UK and around the world. Examples include:

• EnglishScore: an innovative English language test designed by CRELLA and now available in 150 countries, administered for free via mobile devices and scored in under 40 minutes. According to the BC EnglishScore Ltd CEO, ( 5.5), since launching in 2019, over 1.7 million users have downloaded the EnglishScore app from the GooglePlay store, approximately 80,000 tests are taken each month and the organisation now employs 19 members of staff (headcount: 19, FTEs: 18). He adds that, “Without the input and expertise of CRELLA there is no way that we could have made the progress that we have.”

• The National English Adaptive Test (NEAT) and the Plan Ceibal Online Speaking Test in Uruguay: Commissioned by the Uruguay government in 2018, CRELLA re-developed NEAT and designed a new online speaking test to better measure the progress of primary and secondary school children. Since 2019, NEAT has been administered to over 75,000 children and the Plan Ceibal Online Speaking Test to over 6,000 children nationwide. According to the programme’s director,

“The application of the SCF informed quality assurance procedures for the design, piloting, validation and large-scale digital administration of the tests. It helped us to ensure that test tasks were suitable for online delivery, appropriate for young learners in Uruguay, and that the rating scales used to judge performance reflected the local context.” ( 5.6).

Furthermore, using the components and parameters of the SCF, CRELLA devised an effective methodology for comparing different English language tests. This research was commissioned by the General Medical Council (GMC) to help them evaluate the suitability of various English language tests available for use in licensing overseas trained doctors to operate in the UK. As a direct result of CRELLA’s research, the GMC now recognises another test alongside IELTS: the Occupational English Test (available at more than 145 test venues in 44 countries) ( 5.7).

The wider impact of the SCF is evidenced by various educational outreach initiatives taken by CRELLA staff, supporting quality assessments for new markets, and developing the assessment skills of teachers and assessment professionals in the UK and globally. Weir, Green, Taylor, Field, Khabbazbashi and Nakatsuhara have offered SCF-informed assessment training courses and webinars to 24 international institutions. These include the Association of Language Testers in Europe (ALTE: an association of test providers for all major European languages including the Goethe-Institut, Alliance Francaise and Instituto Cervantes) ( 5.8), the International Language Testing Association (ILTA), and the International Association of Teachers of English as a Foreign Language (IATEFL).

Using SCF as the informing theory, two large-scale EU-funded projects, in which CRELLA was a partner, have enhanced the assessment literacy of English teachers in Eastern Europe and the Russian Federation. With partners across Europe, the Erasmus+ TALE project (2015-1-CY01-KA201-011863: EUR268,884, c.GBP235,000) produced an on-demand, open-access course for language teachers designed to help them make effective use of assessment in the classroom. Green and Hamp-Lyons authored three of the eight course modules. The project was awarded the BC Innovation in Assessment Prize in 2019. In a series of workshops in Ukraine conducted in association with Taras Shevchenko University, teacher trainers were trained by CRELLA staff in testing principles and new university tests in English were developed.

Similarly, in an EU (Tempus) funded project that ran from 2011 to 2014 (517114-TEMPUS-1-2011-1-UK-TEMPUS-SMHES, EUR910.860, c.GBP790,000), led by Green, eleven Russian universities sent teacher trainers to CRELLA to work together in designing a new course in language assessment for preservice and in-service teachers. The handbook for the course was published by Cambridge University Press ( https://www.beds.ac.uk/media/kmrjwwzw/proset_lta-course-brochure_en.pdf). Participants also upgraded their test writing and development skills to improve the quality of the Russian national state examination in English. The Dean of the Faculty of Foreign Languages at Mordovia State University and the coordinator for the Russian partners in the project ( 5.9) testifies that the SCF and CRELLA’s research “has had a profound and sustained impact on language education in the Russian Federation.” The training courses are still run regularly by all the project partners: “In the 2014-2020 period, over 20,000 teachers have received in-service assessment training through programmes developed by the project.”