Attachment J.19 - Oral Language Module Pilot Test .pdf

Northern Education Initiative Plus Oral Language Module Pilot Test

March 2021

Submission Date: October 26, 2018 Contract Number: AID-620-C-15-00002 October 26, 2015 – October 25, 2020 COR: Olawale Samuel

DISCLAIMER

This document was produced for review by the United States Agency for International Development. It was prepared by Creative Associates International.

Submission Date: October 26, 2018

Contract Number: AID-620-C-15-00002

October 26, 2015 – May 25, 2021

COR: Nura Ibrahim

Amendment 0002 - Solicitation Number: 72062021R00001 - LEARN to Read Activity Attachment J.19

- Oral Language Module Pilot Test

Table of Contents Table of Figures

Table of Tables

Executive Summary

Study Design

Results Summary

Sample

OLM and EGRA Scores

Discussion

Introduction

Program Description

What is the Oral Language Module?

Purpose of the OLM Pilot

Sample

Research Instruments

OLM

EGRA

Learner Questionnaire

Head Teacher Questionnaire

Translation, Review, and Programming

Data Collection

Assessor Training

Data Analysis

OLM Results and Performance of the OLM Instrument

Obtained Samples

OLM Story Retell

Inter-rater Reliability (IRR) of Story Retells

Internal consistency of two Story Retells

OLM story inferences

IRR for story inference

Internal consistency of story inferences

OLM Receptive Vocabulary

OLM Expressive Vocabulary

Internal consistency of Expressive and Receptive Vocabulary

Summary of OLM Internal Consistency and Correlation of Subtasks

EGRA Results

EGRA Syllable Identification

EGRA Familiar Word Reading

EGRA Oral Reading Fluency

Reading Comprehension

EGRA Scores Among Pupils Who Succeeded and Did Not Succeed at Story Retells

Correlations Between OLM and EGRA Subtasks

Correlations between OLM and child contextual data

Discussion

OLM Results Overall

OLM subtasks

Grade level appropriateness

Receptive and Expressive Vocabulary

Interrater reliability (IRR)

Receptive and Expressive Vocabulary items and associated images

EGRA

Annex 1. Additional analyses

Receptive Vocabulary by gender

Expressive Vocabulary

EGRA syllable identification by sex

EGRA Familiar Words by Sex

EGRA ORF by Sex

EGRA Comprehension by Sex

Annex 2. OLM and EGRA Instrument and Learner Questionnaire

Annex 3. Receptive Vocabulary Images

Annex 4. Expressive Vocabulary Image

Annex 5. Head Teacher Questionnaire

Annex 6. Agenda for Assessors Training

Annex 7. Additional Data analyses

Table of Figures Figure 1. Story Retell scores by grade and home language

Figure 2. Story Retell by state (P2 and P3 combined): contrasting score distributions

Figure 3. Story inference scores by grade and home language

Figure 4. Story inference by state (P2 and P3): contrasting score distributions

Figure 5. Score distribution of Receptive Vocabulary, all learners (N=353)

Figure 6. Receptive Vocabulary: comparison of P2 to P3 performance on ten items

Figure 7. Score distributions for Expressive Vocabulary, all learners (N=353)

Figure 8. Expressive Vocabulary: Comparison of overall P2 to P3 percentages correct on ten items

Figure 9. P2 story retell scores by sex by state

Figure 10. P3 story retell scores by sex by state

Figure 11. P2 story inference scores by gender by state

Figure 12. P3 story inference scores by gender by state

Table of Tables Table 1. P2 and P3 Story Retell and inference scores by home language

Table 2. P2 and P3 Receptive and Expressive Vocabulary by home language

Table 3. P2 and P3 EGRA scores by home language

Table 4. Total designed sample

Table 5. P2 n values by home language by gender by state

Table 6. P3 n values by home language by gender by state

Table 7. Number of P2 pupils who were given the Receptive and Expressive Vocabulary subtasks by language, gender, and state

Table 8. Number of P3 pupils who were given the Receptive and Expressive Vocabulary subtasks by language by gender by state

Table 9. Receptive Vocabulary scores among P2 pupils by language by state and sex

Table 10. Receptive Vocabulary scores among P3 pupils by language by state and sex

Table 11. Expressive Vocabulary scores among P2 pupils by home language by state and by sex

Table 12. Expressive Vocabulary scores among P3 pupils by home language by state and by sex

Table 13. Internal consistency of OLM subtasks (all learners tested)

Table 14. Correlations between OLM subtask scores

Table 15. P2 Syllable Identification scores by home language, by state, and by sex

Table 16. P3 Syllable Identification scores by home language by state and sex

Table 17. Familiar Word Reading scores by home language by state and sex among P2 pupils

Table 18. Familiar Word Reading scores by home language by state and sex among P3 pupils

Table 19. Oral Reading Fluency scores by home language by state and by sex among P2 pupils

Table 20. Oral Reading Fluency scores by home language by state and by sex among P3 pupils

Table 21. Reading Comprehension scores by home language by state and sex among P2 pupils

Table 22. Reading Comprehension scores by home language by state and sex among P3 pupils

Table 23. EGRA Scores Among P2 Pupils Who Succeeded at Story Retell and Those Who Did Not Succeed at Story Retells

Table 24. EGRA Scores Among P3 Pupils Who Succeeded at Story Retell and Those Who Did Not Succeed at Story Retells

Table 25. Correlations between OLM and EGRA subtasks

Table 26. Correlations between media index and OLM scores

Table 27. Associations between attending nursery and OLM scores

Table 28. Receptive Vocabulary scores among P2 pupils by sex by state

Table 29. Receptive Vocabulary scores among P3 pupils by sex by state

Table 30. Percentage of P2 pupils answering each Receptive Vocabulary item correctly by home language

Table 31. Percentage of P3 pupils answering each Receptive Vocabulary item correctly by home language

Table 32. Expressive Vocabulary scores among P2 pupils by sex by state

Table 33. Expressive Vocabulary scores among P3 pupils by sex by state

Table 34. Percentage of P2 pupils answering each Expressive Vocabulary item correctly by home language

Table 35. Percentage of P3 pupils answering each Expressive Vocabulary item correctly by home language

Table 36. Language each Expressive Vocabulary question was answered in by P2 other language speakers, among those answering correctly

Table 37. Language each Expressive Vocabulary question was answered in by P3 other language speakers, among those answering correctly

Table 38. Syllable Identification scores by sex by state among P2 pupils

Table 39. Syllable Identification scores by sex by state among P3 pupils

Table 40. Familiar Word Reading scores by sex by state among P2 pupils

Table 41. Familiar Word Reading scores by sex by state among P3 pupils

Table 42. Oral Reading Fluency scores by sex by state among P2 pupils

Table 43. Oral Reading Fluency scores by sex by state among P3 pupils

Table 44. Reading Comprehension scores by sex by state among P2 pupils

Table 45. Reading Comprehension scores by sex by state among P3 pupils

Table 46. P2 story retell and inference scores by home language by state and sex

Table 47. P3 story retell and inference scores by home language by state and sex

Executive Summary Funded by the United States Agency for International Development (USAID), the five-year Northern Education Initiative Plus activity (the Initiative) strengthens the ability of Bauchi and Sokoto states to provide access to quality education—especially for girls, orphans, and children enrolled in non-formal schools—and to improve reading skills for more than one million school-aged children and youth.

At the request of USAID, the Initiative has piloted a version of the Oral Language Module (OLM), which was developed to measure pupils’ ability to comprehend and express themselves in the spoken language in which they receive reading instruction and assessment. The OLM has been proposed as useful in contexts with low performance on basic Early Grade Reading Assessment (EGRA) subtasks, which has been used throughout the life of the Initiative, to measure pupils’ early grade reading skills, including reading syllables, words, and passages; reading comprehension; and dictation/writing. Though an EGRA was conducted in tandem with the OLM pilot to help determine the OLM’s effectiveness, it is important to emphasize that the sample for the OLM pilot was designed to serve the piloting objectives and was not intended to provide a representative EGRA sample at state levels or to serve as an Initiative end line that could be comparable with baseline and midline EGRA results.1

The OLM consists of three subtasks: Story Retell, Receptive Vocabulary, and Expressive Vocabulary. In Story Retell, pupils are read two stories. After each story the pupil is asked to retell it in their own words and to explain its meaning. In Receptive Vocabulary, assessors pronounce a word, and the pupils point to the corresponding image on a sheet or to something in their environment that the word signifies. In Expressive Vocabulary, the assessor points to a part of a picture, asking what a depicted object or action is, and the pupil responds verbally. In this pilot study, in a context where Hausa is the official language of instruction in early primary, but a minority of pupils do not speak Hausa at home, all questions were initially asked in Hausa, but when a pupil whose home language was Fulfulde or some other language was unable to respond correctly, the instructions and items could be repeated in the child’s home language. Pupils could provide verbal responses in any language (including English).

The OLM had not previously been used in northern Nigeria, nor in Hausa. The pilot test aimed to determine whether the tool can be useful in the context for diagnosing weaknesses in pupils’ oral and/or listening competencies in Hausa and the degree to which these weaknesses may constitute obstacles to pupil progress in pre-reading skills. To perform this function, the instrument (or at least one of its subtasks) would ideally be sensitive enough to indicate a range of ability levels in Hausa oral expression and would correlate with EGRA scores. Prior to and during the training of data collectors, local partners assisted in refining the instrument for the context in order to maximize the probability of it performing as desired and thus the likelihood of the pilot itself already yielding useful insights into whether early primary pupils, and especially those not speaking Hausa at home, might require additional support in oral Hausa as a foundation for EGR progress.

1 Lengthy school closures in 2020 precluded comparability of an originally scheduled end line EGRA to baseline and midline EGRAs, and throughout the year the level of persistent COVID transmission risks in the context were difficult to assess. Following the “Guidance for USAID Education Sector Implementing Partners: Monitoring, Evaluation, And Learning (MEL) During the Covid-19 Pandemic”, and in consultation with USAID Nigeria, it was decided that under these conditions it would not be appropriate or responsible to devote resources to a full-scale EGRA administration. Resources were instead devoted to the OLM pilot, which had been long requested of the Initiative by USAID. A purposeful sampling strategy could meet the pilot objectives while substantially reducing the numbers of schools visited, pupils tested, and assessors trained, while still allowing collection of some indicative EGRA data.

Study Design The OLM pilot was conceived to be cross-sectional, as learners were assessed once. A combination of stratified, purposeful, and random sampling was used. Within each state, four LGEAs and four schools within each of those LGEAs were purposefully selected based on local partners’ suggestions of where a relatively high proportion of Fulfulde speakers would be found. In each school, 35 students were sampled, with the aim of having an equal number overall of Hausa and Fulfulde speakers, of P2 and P3 students, and of males and females (creating eight language/class, and sex combinations.

In previous versions of the OLM the Story Retell was the final subtask. However, the study team decided to begin the assessment with the Story Retell and to reserve the other two subtasks only for those children who did not perform well on it. The pilot field team reasoned that, since the Retell makes by far the greatest cognitive demands, a child who performs well on it would also perform well on the easier subtasks (Receptive and Expressive Vocabulary).2 For children who did not retell the stories well, the other subtasks could be used to determine their level of more basic oral language skills. This decision, as well as the Story Retell scoring rubric, was validated by the assessors during assessor training, since they have extensive experience in education in Bauchi and Sokoto. Further, certain items were modified based on assessors’ feedback during training and initial practice in schools before real data collection.

To enhance interrater reliability, training included intensive practice on scoring the Story Retell items against a four-point scale.

To investigate correlations between OLM and EGRA subtasks, the four Hausa EGRA subtasks used throughout the initiative (Syllable Identification (SI), Familiar Word Reading (FWR), Oral Reading Fluency (ORF), and Reading Comprehension (RC)) were administered to the same pupils following the OLM. In addition, pupils were asked contextual questions, including about reading practices at home and school reopening after pandemic-related closures. As part of the data collection, head teachers were also surveyed about the school’s situation in light of the pandemic.

Results Summary

Sample Fewer Fulfulde speakers than expected were found in the sampled schools. A few pupils who spoke neither Hausa nor Fulfulde at home were found and were included in the sample, because trained assessors were available who were able to administer relevant questions in their home languages if needed. Consequently, Fulfulde home speakers and the few speakers of other languages are combined in this report into one “Other” category. The small number of combined other language speakers (49 in P2 and 39 in P3 compared to 499 Hausa speakers in P2 and 538 Hausa speakers in P3) should be taken into account when interpreting the results by language.

2 The decision was made by the field team, mainly out of consideration for the burden on pupils and to reduce test administration time and thus contact hours and COVID risk at the schools. It is recognized that exempting pupils from the assumedly easier subtests, based on their success on the more difficult subtask, came at the cost using the full sample and full range of scores on the easier subtasks to confirm internal consistency and/or correlations among subtasks, and to identify problematic items. We affirm that the full purposes of the pilot were nonetheless fulfilled, since even those pupils who were less successful on the Retell provided a useful distribution of scores on the easier subtasks.

When schools reopened following pandemic-related closures, 75.0% of schools in Bauchi continued with the third term, while 62.5% of Sokoto schools started with the first term of the following school year, with all learners being promoted to the next class. This presents a limitation to comparability of pupils between states within a given grade, since pupils with supposedly the same amount of contact hours more often remained in a lower grade in Bauchi while they more often moved to a higher grade in Sokoto. It similarly complicates comparison of grades. However, such comparisons were not central aims of the pilot, and despite these issues the data allows both analysis of the instrument’s overall performance and initial conclusions about oral language competencies among the sampled pupils.

OLM and EGRA Scores

As shown in Table 1, most students correctly or partially correctly retold the stories and provided correct inferences. Retell scores were similar by language, showing that they are of equal appropriateness for students of different home languages.

Table 1. P2 and P3 Story Retell and inference scores by home language

P2 P3

Short Story Long Story Short Story Long Story

Hausa (n = 499)

Other (n = 49)

Hausa (n = 499)

Other (n = 49)

Hausa (n = 538)

Other (n = 39)

Hausa (n = 538)

Other (n = 39)

Retell

Correct 42% 43% 34% 33% 51% 51% 39% 33%

Partially correct

30% 25% 38% 27% 30% 28% 39% 33%

Minimally correct

14% 12% 15% 22% 14% 8% 14% 13%

Incorrect 6% 14% 5% 10% 4% 8% 4% 10%

No response

8% 6% 8% 8% 2% 5% 5% 10%

Inference

Correct 49% 57% 47% 51% 64% 59% 59% 49%

Incorrect 28% 31% 30% 41% 20% 26% 25% 36%

No response

24% 12% 23% 8% 16% 15% 16% 15%

Only students who did not score at least a “partially correct” on the retell on both stories were given the Receptive and Expressive Vocabulary subtasks. As a result, 194 P2 pupils (35%) and 159 P3 pupils (28%) were administered the Receptive and Expressive Vocabulary subtasks, with the proportions being slightly higher among other language speakers (43% in P2 and 36% in P3) than among Hausa speakers (35% in P2 and 27% in P3).

High Receptive and Expressive Vocabulary scores among both language groups shown in Table 2 and Table 3 support the choice to put the Story Retell subtask first: since students who did not succeed at the Story Retell still managed to score well on Receptive and Expressive Vocabulary, it is assumed that students who were successful on the Story Retell would also demonstrate mastery of these subtasks.

Table 2. P2 and P3 Receptive and Expressive Vocabulary by home language

P2 P3

Hausa Other Hausa Other n Mean n Mean n Mean n Mean

Receptive Vocabulary (total correct, out of 10)

173 7.4 21 7.1 145 8.4 14 8.3

Expressive Vocabulary (total correct, out of 10)

173 7.7 21 6.8 145 8.2 14 8.6

Table 3. P2 and P3 EGRA scores by home language

Hausa Other Hausa Other n Mean n Mean n Mean n Mean

Syllable Identification (correct per minute) 496 2.9 49 2.1 537 9.3 39 12.1

Familiar Words (correct per minute) 497 1.4 48 1.7 535 6.0 39 6.8

Oral Reading Fluency (correct per minute) 495 1.6 49 1.4 538 6.6 38 7.0

Reading Comp (total correct, out of 5) 495 0.11 49 0.10 538 0.49 38 0.55

Discussion EGRA scores were very low; on each subtask, over half of pupils in both grades did not respond to a single item correctly. Such low scores could be influenced by the pandemic-related school closures, lasting from March to October 2020. Many schools, especially in Sokoto, started the new academic year when schools reopened; in P2, some students, in the absence of school closures, would have been in P1, while some P3 students would have been in P2. Further, the sample of schools used for this pilot is not representative of all schools in Bauchi in Sokoto; rather, it is a purposeful sample to achieve the goals of the OLM pilot.

EGRA scores were similar between language groups. Particularly in P3, EGRA scores were higher among pupils who performed well on the story retells. In this context, the OLM may be helpful in determining where gaps in pupils’ skills lie.

Over two-thirds (69%) retold both OLM stories at least partially correctly (65% for P2 and 72% for P3).

The observed level of improvement between P2 and P3 suggests that the subtask would also be appropriate both for lower grades (at least P1), as well as higher grades (where an English version would also potentially come into play).

Given the quite top-heavy frequency distributions of both the Receptive and Expressive Vocabulary subtasks, which would be even more top-heavy if problematic items are corrected, it appears that these subtasks would be more useful at pre-primary and P1 levels, especially as this would assist in catching struggling pupils as early as possible. In addition, the high scores on the Receptive and Expressive Vocabulary subtasks support the chosen protocol of administering the Story retell first (treating it as the more challenging task) and using the easier subtasks only on those who are less successful, for finer sorting of their oral and listening competencies.

In both grades, reliability between the two retell scores, the two inference scores, the Receptive Vocabulary items, the Expressive Vocabulary items, and the Receptive and Expressive Vocabulary total scores were most often high, and medium where not high. These pilot data do not reveal a major gap between Hausa and non-Hausa that could be considered a major factor impeding mean EGR progress in the states.

Interrater reliability for story retells and inferences should remain a focus for improvement, despite some encouraging signs. Towards this end, serious consideration should be given to whether the inference portion as currently designed is adding value to the subtask, or whether it potentially even could add value. The guidance for judging a correct inference appears inadequate, with the required content of a “correct” inference being insufficiently distinguished from the content expected for a correct retell. Further, the moderate correlations between the retell and inference scores suggest some redundancy.

In Receptive and Expressive Vocabulary, there were issues with inadequate clarity of some images. Such issues can be addressed through cognitive testing.

Introduction

Program Description Funded by the United States Agency for International Development (USAID), the five-year Northern Education Initiative Plus (the Initiative) activity strengthens the ability of Bauchi and Sokoto states in northern Nigeria to provide access to quality education—especially for girls, orphans, and children enrolled in non-formal schools—and to improve reading skills for more than a million school-aged children and youth. The Initiative uses a holistic approach to address critical supply and demand factors that affect learning, teaching, management, parental participation, and responsiveness to children’s needs.

What is the Oral Language Module?

The Oral Language Module (OLM) developed by USAID/Washington, measures pupils’ ability to comprehend and express themselves in a spoken language, in this case Hausa. Throughout the life of the Initiative, the Early Grade Reading Assessment (EGRA) has been used to measure pupils’ early grade reading skills, including reading syllables, words, and passages; reading comprehension; and dictation of words. In contexts with low performance on EGRA, the OLM may be as useful for investigating whether pupils may require prior or simultaneous support in speaking and listening competencies to support EGR learning.

The OLM consists of three subtasks: Story Retell, Receptive Vocabulary, and Expressive Vocabulary. The OLM consists of three subtasks: Story Retell, Receptive Vocabulary, and Expressive Vocabulary. In Story Retell, pupils are read two stories. After each story the pupil is asked to retell it in their own words and to explain its meaning. In Receptive Vocabulary, assessors pronounce a word, and the pupils point to the corresponding image on a sheet or to something in their environment that the word signifies. In Expressive Vocabulary, the assessor points to a part of a picture, asking what a depicted object or action is, and the pupil responds verbally.

In this pilot study, in a context where Hausa is the official language of instruction in early primary, but a minority of pupils do not speak Hausa at home, all questions were initially asked in Hausa, but when a pupil whose home language was Fulfulde or some other language was unable to respond correctly, the instructions and items could be repeated in the child’s home language. Pupils could provide verbal responses in any language (including English).

Purpose of the OLM Pilot The OLM had not previously been used in northern Nigeria nor in Hausa. The pilot test aimed to determine whether the tool can be useful in the context for diagnosing weaknesses in pupils’ oral and/or listening competencies in Hausa and the degree to which these weaknesses may constitute obstacles to pupil progress in pre-reading skills. To perform this function, the instrument (or at least one of its subtasks) would ideally be sensitive enough to indicate a range of ability levels in Hausa oral expression and would correlate with EGRA scores.

Prior to and during the training of data collectors, local partners assisted in refining the instrument for the context in order to maximize the probability of it performing as desired and thus the likelihood that the pilot itself would already yield useful insights into whether early primary pupils, and especially those not speaking Hausa at home, might require additional support in oral Hausa as a foundation for EGR progress. The data collection experience and the results can inform further decisions on potential modifications to the instrument and protocols.

Because Fulfulde is the second most widely spoken language in northern Nigeria, and because earlier EGRA data from the Initiative confirmed the presence of pupils reporting that they mainly speak Fulfulde at home, the pilot was planned to include Fulfulde speaking pupils as well as assessors with the language skills to facilitate their assessments3.

3 In data collection for the Initiative’s midline EGRA, which used a sample representative at the state level, 6.4% of pupils sampled in Bauchi reported speaking Fulfulde in the home, while 2.6% reported this in Sokoto. However, for the baseline EGRA in 2016 fully 16% of pupils sampled in Bauchi reported speaking Fulfulde at home, with an additional 3% reporting speaking other languages. In Sokoto, the percentage was just 2.7% at baseline.

Methodology

Sample It is important to emphasize that the sample for the OLM pilot was designed to serve the piloting objectives and was not intended to provide a representative EGRA sample at state levels or to serve as an Initiative end line that could be comparable with baseline and midline EGRA results.4

A combination of purposeful, stratified, and random sampling was used, with available budget and health and safety issues also influencing the feasible sample size and design. The sample was first stratified by state (Bauchi and Sokoto). In each state, four Local Government Education Authorities (LGEAs) were purposefully selected based on the expectation of finding a relatively high proportion of pupils who speak Fulfulde at home in their schools. Within each LGEA, four schools were purposefully selected, also prioritizing those in areas known to have a relatively high proportion of pupils who speak Fulfulde at home. In each state, the Initiative’s Monitoring, Evaluation and Research (MEAR) Technical Working Group members (comprised of state partners) led the sampling process, and the Initiative MEL staff provided technical support in the parameters to be considered, including rural and urban location, population of pupils, and desired proportion of Fulfulde speakers. Additional Initiative staff advised on safety and security issues.

Within each school, 35 pupils were sampled. The number to sample (either 17 or 18) from each grade, sex and language was specified in order to yield equal targets in the overall sample for: P2 and P3, boys and girls, and Hausa and non-Hausa home speakers. However, the targeted number of Fulfulde home speakers could not be found in any school, so all Fulfulde speakers found in the P2 and P3 classrooms were sampled, and a random sample of Hausa speakers was selected to make up the remainder of the school’s targeted 35 pupils5. Pupils who spoke neither Hausa nor Fulfulde at home were originally intended to be excluded from the sample, since assessors who spoke other languages fluently had not been intentionally recruited for the purposes of this pilot. However, three pupils who spoke another language at home were in the end included since assessors were in fact available who spoke their languages and could appropriately administer the assessment according to the protocol.

For the competencies tested with the OLM, it would have been of interest to also include P1 learners and perhaps even younger, pre-primary children in the sample. However, the decision was taken by the field team and state partners to limit the sample to P2 and P3, since these have been the grades at which EGRA has been administered in formal schools throughout the Initiative; focusing on these grades would efficiently cover both parts of the assessment (OLM and EGRA), avoiding contact with additional groups of pupils.

4 Lengthy school closures in 2020 precluded comparability of an originally scheduled end line EGRA to baseline and midline EGRAs, and throughout the year the level of persistent COVID transmission risks in the context were difficult to assess. Following the “Guidance for USAID Education Sector Implementing Partners: Monitoring, Evaluation, And Learning (MEL) During the Covid-19 Pandemic”, and in consultation with USAID Nigeria, it was decided that under these conditions it could not be appropriate or responsible to devote resources to a full-scale EGRA administration. Resources were instead devoted to the OLM pilot, which had been long requested of the Initiative by USAID. A purposeful sampling strategy could meet the pilot objectives while substantially reducing the numbers of schools visited, pupils tested, and assessors trained, while still allowing collection of some indicative EGRA data.

5 The unexpected difficulty in finding Fulfulde speaking pupils in schools, especially in Bauchi, would be worthy of further investigation. It may suggest that this language minority group returned to school in lesser numbers than Hausa speakers after the COVID-closures.

Table 4. Total designed sample

States 2

LGEAs per state 4

Schools per LGEA 4

Pupils per school 35

Total pupils 1,120

Research Instruments

OLM

The OLM, which can be found in Annex 2, has three subtasks as described above: Story Retell, Receptive Vocabulary, and Expressive Vocabulary. Story Retell has two items: a shorter story and a longer story.

For the short story, the pupil had 30 seconds to retell the story. However, if the pupil did not respond within 15 seconds, the assessor offered to reread the story. In these cases, the 30-second timer is restarted after the rereading. Following the attempted retell, the assessor asked the pupil to explain the meaning of the story (referred to as the “inference” portion of the item), and this portion was untimed.

The same procedure was repeated for the longer story, but with the pupil having 60 seconds for the retell portion.

To promote interrater reliability, a rubric for assessing the story retell and inference was developed by the MEL team and state partners (including training facilitators) and refined and endorsed by assessors during training. This rubric was derived from the rubric in the OLM draft originally provided by USAID.

On the rubric, a pupil’s story retell was scored according to the following ratings and criteria:

• Correct – child retells story plot with beginning, middle, and end.

• Partially correct – child retells story events but out of logical narrative order or missing key information.

• Minimally correct – child lists one or more people, objects, and events from the story.

• Incorrect – incorrect response (people, objects, or events not listed in the story).

• No response – the child did not provide any response.

The pupil’s inference was rated simply as “Correct”, “Incorrect”, or “No response”. The definition of a correct inference was as follows:

• Child gives inferential conclusion to the story by identifying characters and causal connections.

During the assessors’ training, significant time was spent on ensuring common understanding of these criteria and applying them to rate the pupils’ retells and inferences. Many examples were presented by the facilitators, scored, and discussed, including real pupils’ responses both in the training hall and during practice visits to schools.

In previous versions of the OLM, the Story Retell was the final subtask. However, the study team decided, for this pilot, to begin the assessment with the story retell and to reserve the other two subtasks only for those children who did not perform well on it. The pilot field team reasoned that, since the retell makes by far the greatest cognitive demands, a child who performed well on it would also perform well on the easier subtasks (Receptive and Expressive Vocabulary). For children who did not retell the stories well, the other subtasks could be used to determine their competence in more basic oral language skills. This decision was made by the field team, mainly out of consideration of the assessment burden on pupils and to reduce test administration time and thus contact hours and COVID risk at the schools.

It is recognized that exempting pupils from the assumedly easier subtests, based on their success on the more difficult subtask, came at the cost of using the full sample (and full range of scores) on these easier subtasks to confirm their internal consistency, to test correlations among subtasks, and to identify problematic items. It is important to stress, however, that the obtained results on the two easier subtasks, just from the pupils who were unsuccessful on the Retell, were in fact very high, especially on the items whose image stimuli6 were the clearest. An adequate distribution was obtained to serve the purposes of the pilot.

In the Receptive Vocabulary subtask, the assessor pronounced a word, and the pupils were instructed to either point to the corresponding image on a stimuli sheet or to something in their environment that the word signified. There were 10 items: 5 body parts and 5 objects. If the child reported speaking Fulfulde at home and answered the first five items incorrectly (when asked in Hausa), the assessor repeated the instructions, the practice item, and the first five items in their home language. If the child still answered all five incorrectly, the remaining five were skipped, and the pupil proceeded to Expressive Vocabulary.

If the child answered one of the five correctly when asked in their home language, the assessor asked items 6-10 in the same language. If the child reported speaking Hausa at home and answered the first five items incorrectly, the remaining five were skipped, and the pupil proceeded to Expressive Vocabulary.

In Expressive Vocabulary, the assessor pointed to a specific detail in an image of a classroom (see Annex

3) asking the pupil to verbally name a depicted object or action. There were 10 items: 6 objects (nouns) and 4 actions (verbs). Multiple answer options could be counted as correct; for example, in the first item, the assessor pointed to the teacher in the picture and asked, “Who is this?”. Answer options included the Hausa and Fulfulde equivalents of “teacher”, “woman”, and “lady”, as well as the English terms. If the child answered correctly, the assessor marked the answer provided as well as the language of the response. The procedure in terms of the language the questions were asked in and the potential skipping of the last 5 items were the same as for Receptive Vocabulary.

Each OLM subtask began with a practice item. For example, at the beginning of the Story Retell subtask, the assessor read a practice story and asked the child to retell and explain its meaning. If the child did not retell or explain the meaning completely correctly, the assessor provided the correct answer for the practice item.

EGRA

Learners’ foundational reading skills in Hausa were tested in the following subtasks, which have been used in all Initiative EGRA administrations: syllable identification (SI), familiar word reading (FWR), oral reading fluency (passage reading) (ORF), and reading comprehension (RC). In previous EGRA administrations, word dictation was also included. To reduce the length of the full assessment for the child, since the OLM was included, dictation was eliminated from this pilot.

6 Image stimuli are pictures used in the OLM. In Receptive Vocabulary, the image stimuli are pictures of objects and body parts. In Expressive Vocabulary, the image stimulus is a picture of a classroom.

These subtasks of the Hausa instrument were the same as those administered to formal school learners in the Initiative’s formal school midline EGRA, as well as those administered at the beginning and end of the program for cohort 4 at non-formal learning centers. In the case of SI and FWR, the items were reordered. The essential structure and content of each instrument had originally been developed in collaboration with Nigerian state partners, either prior to the Initiative or during it in consultation with the TWGs for Reading and for Monitoring, Evaluation, Assessment, and Research (MEAR).

Learner Questionnaire A questionnaire was administered to each sampled learner immediately upon completion of his or her OLM and EGRA testing. The learner questionnaire elicited basic demographic data, as well as information on a range of contextual factors hypothesized to potentially affect basic literacy progress, including levels of support for reading in the home.

Head Teacher Questionnaire At each school, the head teacher was surveyed about school reopening after pandemic-related closures.

Questions pertained to: whether a new school term started or the previous term continued upon reopening; pupil and teacher attendance after the school reopened; health and safety measures the school is taking in light of the pandemic, known cases of COVID-19 among school staff and pupils, and concerns among school staff, parents, and pupils about COVID-19. The questionnaire can be found in Appendix 3. The results from this questionnaire will be presented fully in a separate report, but have informed some interpretation of the OLM and EGRA results

Translation, Review, and Programming The instruments were originally provided by USAID in English, and a rigorous process was undertaken to translate them into Hausa. Two language translators, who were also experienced EGRA trainers and among the assessor training facilitators, each independently translated the tools. These translators, along with two other Hausa experts on the Initiative M&E team, reviewed the translations together.

Their revision was then discussed with the two original language translators.

During assessor training, the assessors made suggestions to enhance translation and clarity. All of these issues were resolved. In a few instances, the enumerators advised that two different words or phrases be used on the instrument with a slash (/) between them to accommodate both Sokoto and Bauchi dialects of Hausa.

There were also issues of Receptive and Expressive Vocabulary images not clearly representing what was intended. For example, the images of nose and mouth were replaced with clearer, more representative images. In addition, in the Expressive Vocabulary picture, some enumerators believed that the window, which is asked about in one of the items, looked more like a door. Ultimately it was agreed that the image was clearly enough a window rather than a door since the bottom of it was not touching the ground.

The OLM and EGRA instruments and the learner questionnaire were combined into one assessment form and programmed into tablets using the application SurveyToGo. The complete form can be found in Annex 2. The head teacher questionnaire was programmed into a different form, which can be found in Annex 5.

Data Collection Data collection for the OLM and EGRA instruments and the learner questionnaire was conducted concurrently with administration of a head teacher questionnaire and a parent questionnaire regarding the effects of the COVID-19 pandemic on schooling and daily life.

Assessor Training Consistent with the system strengthening agenda of the Initiative, the assessors were drawn from state education institutions, in particular the State Universal Basic Education Board (SUBEB) and LGEAs. Most of the assessors had undergone multiple trainings since the beginning of the Initiative, including for the Years 1, 2, and 4 non-formal center EGRAs and the formal school baseline and midline EGRAs, and had taken part in those data collections. To handle the expected complexity of achieving high inter-rater reliability (IRR) for the OLM Story Retell, only particularly strong assessors were invited to participate in this pilot. Further, even though only 20 enumerators per state were required for data collection, 35 strong enumerators per state were invited to participate in the training, and the strongest 20 of them were chosen based on their performance during training.

A four-day training was conducted with 35 assessors in Sokoto from January 6-9, 2021 and with 35 assessors in Bauchi from January 20-23, 2021. Because the assessors were already familiar with the EGRA and because the head teacher and parent surveys were relatively straightforward, the majority of the training was focused on the OLM. The training included an overview of the purpose of the data collection and the instruments, including those pertaining to COVID-19; familiarization with the tools, including practice reading the instructions and questions and using the SurveyToGo application; role play to practice tool administration; a discussion of whether any of the tool items should be revised as discussed in the “Research Instruments” section; and data collection protocol and logistics. The full training agenda can be found in Annex 6.

The second and third days of each training included a practice test in non-sampled schools. During each school visit, enumerators practiced on students individually as well as conducting an IRR test (in one of the two schools) in pairs and in groups of three. The purpose of the IRR tests was for assessors to discuss similarities and difference between their judgements of the same responses by the pupils.

On the third day of training, two plenary IRRs were conducted and pupils for these IRRs were invited to the training venue. During the last IRR, which was used to select the final assessors, one of the trainers was used as the gold standard who scored the pupil together with the trainee assessors. The agreement between his judgment of each item and each assessor’s judgement (using the coding: agreement = 1, not in agreement = 0) was recorded and correlated. IRR between each enumerator and the gold standard was computed, and the 20 enumerators with the highest IRR scores were selected in each state. In both states, only one of the Fulfulde enumerators made the list of the best 20. Because of the importance of the Fulfulde speaker in each enumerator group (each group was assigned a different LGEA), the researchers decided to select four extra enumerators who were the best Fulfulde speaking assessors so that there was one extra (assessor) per group, making 24 enumerators in total. The four extra Fulfulde enumerators were the strongest of those who spoke Fulfulde. All assessors met the 90% standard on the final IRR test, with correlation coefficients of 0.93-0.97 in Bauchi and 0.93-0.99 in Sokoto).

The last sessions of the training were dedicated to the data collection protocol. In each state, the 24 assessors were broken up into four teams of six. Each team was assigned to one LGEA and visited a different school during each day of data collection. Data collection took place from January 11-14, 2020 in Sokoto and from January 25-28, 2020 in Bauchi.

Each team consisted of one Fulfulde speaker to assess children whose home languages were Fulfulde.

For each team, a leader was designated who was responsible for ensuring proper protocols, completing a summary form for each school visited, and communicating with the designated state-level supervisor and Initiative staff. The state-level supervisors were drawn from members of the state MEAR TWGs and other ministry staff.

Data Analysis All collected data were cleaned and analyzed by EDC M&E staff using SPSS. Analyses included frequencies, descriptive statistics, independent samples t-tests, and effect size (Cohen’s d) calculations.

Associations between OLM subtask scores and demographic variables as well as EGRA subtask scores were conducted using Spearman’s correlations and Wilcoxon-Mann-Whitney tests. Internal consistency (Cronbach’s alpha) was calculated to measure internal consistency of OLM items and subtasks. All analyses were disaggregated by grade. As appropriate, they were also disaggregated by home language, state, and sex.

OLM Results and Performance of the OLM Instrument

Obtained Samples The attained samples (after data cleaning) for P2 and P3 are shown in Table 5 and Table 6 below.

Despite the efforts to target Fulfulde home speakers through purposeful selection of LGEAs and schools, the vast majority of the sample was comprised of Hausa speakers7. However, as targeted, there were roughly equal numbers of males and females, learners in each state, and P2 and P3 learners in the sample.

Table 5. P2 n values by home language by gender by state

Bauchi Sokoto Total

Male Female Total Male Female Total Male Female Total

Hausa 114 129 243 121 135 256 235 264 499

Other 18 12 30 9 10 19 27 22 49

Total 132 141 273 130 145 275 262 286 548

Table 6. P3 n values by home language by gender by state

Male Female Total Male Female Total Male Female Total

Hausa 132 137 269 128 141 269 260 278 538

Other 13 8 21 5 13 18 18 21 39

Total 145 145 290 133 154 287 278 299 577

OLM Story Retell

Figure 1Table 5 presents the P2 and P3 retell scores by home language, for both states combined. The distributions of scores suggest that the subtask’s level of difficulty was well aligned with these grade levels, effectively distinguishing levels of expressive language (and listening) competencies. Moreover, the subtask was sufficiently sensitive to show progression from P2 to P3 performance, which conforms to expectation of children’s maturation, even without any explicit educational intervention targeting these competencies. This improved performance was particularly the case with the short story, where the percent of pupils rated with a correct retell increased by roughly 8 percentage points between P2 and P3, with associated decreases in those rated entirely incorrect and those attempting no response.

Improved performance in P3 on the long story is evident as well, though less pronounced, showing roughly a 6 percentage points increase in those rated as either correct or partially correct. Using mean scores out of 5 for statistical comparison of P2 to P3, the differences are significant at the .001 level, with an effect size of .29 for the short story and .16 for the long story.

Also tending to validate the subtask’s design was that the results suggest greater difficulty of the long story in comparison to the short story. This result confirms the hypothesized greater demands of the long story for listening, retention of story components, and accurate retelling. Use of the two stories, 7 About 8% of the total sample were pupils who reported speaking Fulfulde, or another language other than Hausa, at home. By comparison, the Initiative’s baseline and midline EGRAs, which sampled pupils randomly (i.e., made no special effort to find non-Hausa home speakers), included about 9% and 5% of such pupils respectively across the two states.

instead of just one, could therefore be justified to accomplish finer sorting of pupils by expressive and listening competencies.

Because the number of non-Hausa home speakers in the sample is small, caution must be exercised in drawing conclusions from their results. It appears, however, that the performance of the sampled non-

Hausa home speaking pupils largely mirrored that of their Hausa classmates. Because the subtask was sensitive to progression by age/grade, as noted above, it would have been expected to similarly detect any widespread difficultly of language minorities in oral Hausa, but major differences are not evident.

However, the limited data do in fact suggest that the weakest performers – those responding incorrectly or unable to respond – consistently appear in higher percentages among non-Hausa home speakers: for both of the story retells, and in both grades. Note especially that in P3, incorrect and no response scores occurred among “other” speakers at more than twice the rate than they did among Hausa speakers (albeit in relatively small percentages in both groups).

Figure 1. Story Retell scores by grade and home language.

Inter-rater Reliability (IRR) of Story Retells As noted in the methodology section, IRR for the Story Retells (as well as for the inferences) was a major focus of the training. We can credit the documented experiences and guidance from the previous piloting by Save the Children with assisting the present pilot to improve IRR.

When looking at available IRR data overall for the retell, the picture appears encouraging. As described above, IRR tests for the full suite of instruments were conducted near the end of the training in each state. For the 24 assessors in Bauchi who were selected to go the field, on two IRR tests near the

42%

43%

51%

51%

34%

33%

39%

33%

30%

25%

30%

28%

38%

27%

39%

33%

14%

12%

14%

8%

15%

22%

14%

13%

6%

14%

4%

8%

5%

10%

4%

10%

8%

6%

2%

5%

8%

8%

5%

10%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Hausa (N=499)

Other (N=49)

Hausa (N=538)

Other(N=39)

Hausa (N=499)

Other (N=49)

Hausa (N=538)

Other(N=39)

P

P

P

P

Sh o rt s to ry

Lo n g st o ry

Correct Partially correct Minimally correct Incorrect No response conclusion of training, 74% perfect agreement with the gold standard was recorded across the two short story and two long Story Retells. Every instance of disagreement on these IRR tests in Bauchi was by only one scale point (either correct versus partially correct or minimally correct versus incorrect). In Sokoto, the full 35 assessors trained achieved 89% perfect agreement with the gold standard across the four retells in their IRR tests. All instances of non-agreement were by one scale degree.

To the extent practical, IRR data was also collected while assessors were in the field for real data collection. Following the recommendations of the EGRA toolkit, the first assessment of each day was administered in pairs, with one assessor marking the results as “real data” and the second as “IRR”.