Ref 8. RWM Midline EGRA Report.docx

Early Grade Reading Assessment, Midline Report

USAID READ WITH ME PROJECT

Photo credit: USAID Read With Me

July 30, 2019

This publication was produced for review by the United States Agency for International Development. It was prepared by School-to-School International and Chemonics International Inc.

Early Grade Reading Assessment, Midline Report

USAID READ WITH ME PROJECT

Contract No. AID-176-C-16-00003

DISCLAIMER

The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States government.

Acknowledgments Many organizations and individuals contributed to the successful completion of the Read With Me Project (RWM) Midline Early Grade Reading Assessment (EGRA). The Ministry of Education and Science (MoES), National Testing Center, Academy of Education, Education Development Institute under the Academy of Education, Republican Methodological Center, Republican Teachers’ Training Institute with its five regional teacher training institutes and the district and city education departments provided insight, expertise, and guidance throughout the assessment process from the review of assessment tools to data collection monitoring. We would like to extend special gratitude to Mr. Nuriddin Said, Minister of Education and Science, and Ms. Latofat Naziri, Deputy Minister of Education and Science, for their guidance and support, as well as to Mr. Abdujabbor Aliev, RWM Coordinator from MoES, for his assistance throughout the assessment.

This report would not have been possible without the support of the American people through the United States Agency for International Development (USAID) RWM project. USAID provided critical input on the EGRA process through Mr. Andrew Colburn, USAID Regional Education Advisor; Mr. Paul Richardson, USAID Tajikistan Country Office Director; Mrs. Mavjuda Nabieva, USAID Tajikistan Contracting Officer’s Representative, Rebeca Martinez, Reading and Instructional Program Advisor; and Melissa Chiappetta, Senior Research and Learning Advisor.

The EGRA was implemented with the technical assistance of School-to-School International (STS) and Chemonics International. The assessment was planned and administered with the support of staff members in the United States and in the USAID RWM project office in Tajikistan. The authors appreciate RWM project staff’s collaboration and support. We also appreciate the hard work of many other individuals at STS, including Elizabeth Odenwald, Drew Schmenner, and Melyssa Sibal for contributions to tool development, programming, data collection training, data analysis, and producing this EGRA midline report. We thank Laura Zasoski for her editing.

We thank the 49 data collectors, whose efforts to reach schools in every region of Tajikistan made this report possible. Finally, we thank the school directors, teachers, and students who welcomed data collectors to their schools and participated in this study.

The following persons played a major role in the process:

Contents

Acronyms

List of Tables

List of Figures

Executive Summary

Study Purpose This is the midline report of a five-year study that seeks to understand trends in early grade reading performance in Tajikistan at the national level and among schools benefiting from the United States Agency for International Development (USAID) Read with Me (RWM) project. Findings from this study aim to inform the implementation of RWM and support the Ministry of Education and Science (MoES) improve early grade reading in Tajikistan.

Project Background RWM is a five-year (2016–21) project implemented by Chemonics International that aims to improve reading outcomes for 75 percent of all primary grade students nationwide over the course of implementation. It is designed to achieve this goal through improved reading instruction in grades 1 through 4, improved access to and availability of quality materials that support the development of reading in grades 1 through 4, increased innovations and partnerships supporting literacy outcomes, and increased government support to improve reading. RWM accomplishes these goals both by working with the MoES at different levels in the system and schools to support the improvement of reading outcomes.

RWM is the latest USAID-funded project intervening nationwide in primary education in Tajikistan. It builds on the gains made most recently by two previous education projects: the 2007–12 Quality Learning Project (QLP) and the 2013–17 Quality Reading Project (QRP).

Design, Methods, and Limitations Study Design and Purpose This is a five-year study with data collection taking place in spring 2018 as a baseline, spring 2019 as a midline, and spring 2021 as an endline.[footnoteRef:1] [1: This five-year study originally set out to hold baseline data collection in spring 2017. Due to timelines external to the project, data collection occurred in spring 2018. In addition, full implementation of the intervention was staggered throughout the 2018-2019 school year. Midline results from spring 2019 thus show results from less than a full year of project implementation. ]

The purpose of the report is twofold:

1. To provide a nationally representative “snapshot” of reading performance in Tajikistan over time (from baseline to midline), and

2. To understand which aspects of the RWM intervention are associated with improved student reading outcomes.

Methods RWM drew a representative sample of schools in the project’s second of four cohorts.[footnoteRef:2] The sample represents the full population of schools in Tajikistan based on region, language of instruction (Russian and Tajik), and location (urban and rural). Students were sampled at the class level by sex. [2: More information can be found in the Methods section of this report.]

Students were assessed using Tajik- and Russian-language adaptations of the Early Grade Reading Assessment (EGRA), which measures student performance on the basic foundational skills required for fluency in reading. Alongside the EGRA, students took a brief survey that collected information about their background and home environment. Additionally, school-level data were collected using Snapshot of School Management Effectiveness (SSME) tools, including a school director questionnaire, a teacher questionnaire, a school inventory checklist, a classroom inventory checklist, and a lesson observation. The purpose of the student survey and SSME data was to understand student and school contextual factors that may influence students’ reading performance.

Midline data collection took place in April and early May 2019. Data were collected by trained enumerators, many of whom were proposed by the MoES, who passed three assessor accuracy quizzes. In total, enumerators visited 207 schools—146 in the Tajik sample and 61 in the Russian sample, given the proportionate population sizes; they collected data from 1,948 grade 2 students—1,385 in the Tajik sample and 563 in the Russian sample—and 1,912 grade 4 students—1,376 in the Tajik sample and 536 in the Russian sample. Details of sampling and the census approach are found in Methods and Limitations.

Prior to analysis, the oral reading fluency (ORF) subtask from the midline EGRA tools was equated with—or brought to a common scale as—the EGRA tools used in this study’s 2018 baseline data collection and in the QRP endline data collection. Equating was done to allow for appropriate comparisons with data reported at baseline. Sampling weights were calculated and applied to the assessment data to produce more representative estimates in the sample of students. After applying the equating function to 2019 ORF data and the weighting functions to all 2019 data, mean scores, zero scores—the proportion of students unable to answer a single item correctly—and reading benchmark results across subtasks over time were analyzed for each of the disaggregated groups of interest. Differences within each group were analyzed using linear regression for mean score comparisons and logistic regression for zero score and reading benchmark comparisons.

Data from the student survey and SSME questionnaires, inventories, and observation were correlated with the ORF scores to identify student and school characteristics that are linked to the subtask scores. Those items with statistically significant correlations with the ORF subtask scores were included in linear regression models to understand the extent to which they predicted the students’ reading performance.

Limitations The following are key limitations that should be kept in mind while reviewing EGRA results:

1. The non-experimental nature of this study design limits the level of attribution of the RWM intervention on reading outcomes as measured by EGRA. The EGRA uses a non-experimental design because RWM is primarily an implementation project that does not randomly assign schools to treatment conditions or cohorts. As there is no counterfactual, this study cannot attribute results to RWM.

2. As with most EGRA studies, the results are not appropriate for cross-linguistic comparisons. That is, the report does not statistically compare students’ reading skills in Tajik with students’ reading skills in Russian. Acquisition of language and reading development depends on several factors, including the different levels of orthographic transparency, visual complexity, and phonology.

3. This design assumes that cohort 2 schools are representative of the RWM school population in order to generalize results to schools participating in the other cohorts.

4. Russian-language schools were not sampled but treated as a census. Given the decrease in number of Russian schools from baseline to midline, differences in Russian EGRA results between baseline and midline should be treated with caution.

Findings Key findings from the report are presented here, including comparison with the 2018 baseline; more detailed results are presented in the report’s EGRA Results and Contextualizing EGRA Results sections. Because this cohort of schools received RWM interventions in a staggered rollout, this midline report provides results on school performance after receiving the intervention for less than one academic year. See Annexes E–H for comprehensive results from the EGRA, student survey, and SSME.

National-Level Results Snapshot Overall, the midline EGRA results demonstrate strong reading outcomes for students studying in Tajik and Russian (Figure 1 and Figure 2). Similar to baseline, the ORF subtask mean scores at midline for Tajik grade 2 students, Russian grade 2 students, and Russian grade 4 students approached the established benchmarks for their respective grades—40 correct words per minute (CWPM) for grade 2 and 80 CWPM for grade 4. Regardless of language or grade, the percentage of students who received zero scores on the ORF subtask was extremely low—less than one percent—indicating that nearly all students had some ability to read.

Figure 1. Baseline (2018) and Midline (2019) Oral Reading Fluency Results by Grade and Year, Tajik

Note: One asterisk (*) denotes differences that are statistically significant at p < 0.05. No asterisk indicates that differences between baseline and midline were not statistically significant.

Figure 2. Baseline (2018) and Midline (2019) Oral Reading Fluency Results by Grade and Year, Russian

Note: Differences between baseline and midline were not statistically significant for either grade.

For both Tajik- and Russian-language students, performance at midline relative to baseline varied by grade and language depending on the EGRA subtask (see Table 1). For Tajik grade 2 students, midline mean scores were significantly lower than baseline mean scores on five subtasks, including ORF. For Tajik grade 4 students, midline mean scores were significantly higher than baseline mean scores on two subtasks. For Russian grade 2 students, midline mean scores were significantly higher for one subtask—reading comprehension—but significantly lower on three subtasks. For Russian grade 4 students, midline mean scores were significantly higher than baseline mean scores on two subtasks.

Table 1: Trends in Grade 2 and Grade 4 Student Reading Performance by Language and Subtask

Note: An up arrow (↑) indicates that the mean score for students at midline was significantly higher than the mean score for students at baseline; a down arrow (↓) indicates that the mean score for students at midline was significantly lower than the mean score for students at baseline; and a left-right arrow (↔) indicates that the mean score for students in both years was comparable. For all comparisons, significant differences are reported at the p<0.05 level.

2019 Subgroup Results by Sex, Location, and Region Girls generally outperformed boys, especially in Tajik-language schools. Among Tajik grade 2 and grade 4 students, girls scored significantly higher than did boy across most EGRA subtasks at both baseline and midline. Among Russian grade 4 students, girls scored significantly higher than did boys on two subtasks at midline—including ORF. Among Russian grade 2 students, girls and boys performed at statistically similar levels across most subtasks at both baseline and midline (see Figure 3 and Figure 4 for ORF results at midline).

Figure 3. Midline (2019) Oral Reading Fluency Results by Grade and Sex, Tajik

Note: Three asterisks (***) indicates a statistically significant difference between girls and boys at p < 0.001.

Figure 4. Midline (2019) Oral Reading Fluency Results by Grade and Sex, Russian

Note: Two asterisks (**) indicates a statistically significant difference between girls and boys at p < 0.01. No asterisk indicates that the difference between girls and boys was not statistically significant.

Students from urban areas outperformed students from rural areas on a variety of subtasks and by wide margins, suggesting the potential benefits of focused interventions in rural areas. This trend was true at both baseline and midline across grades and languages. Tajik grade 2 students and Russian grade 2 students from urban areas outperformed students from rural areas on nearly every subtask. Figure 5 and Figure 6 present ORF results at midline, and similar patterns are seen for other subtasks.

Figure 5. Midline (2019) Oral Reading Fluency Results by Grade and Location, Tajik

Note: Two asterisks (**) indicates a statistically significant difference between urban and rural at p < 0.01. Three asterisks (***) indicates a statistically significant difference between urban and rural at p < 0.001.

Figure 6. Midline (2019) Oral Reading Fluency Results by Grade and Location, Russian

Note: Three asterisks (***) indicates a statistically significant difference between urban and rural at p < 0.001.

Students from schools located in Dushanbe and the Gorno-Badakhshan Autonomous Oblast (GBAO) generally showed the highest mean scores relative to students from other regions (see Figure 7 for ORF results at midline). In Tajik-language schools, students from Dushanbe and the GBAO typically had the highest mean scores relative to students from other regions. In Russian-language schools, Dushanbe students typically had the highest mean scores relative to other regions. These results were consistent across grades at both baseline and midline.

Figure 7. Mean Correct Words per Minute on the Midline (2019) Oral Reading Fluency Subtask by Language, Grade, and Region

Tajik Grade 2

Tajik Grade 4

Russian Grade 2

Russian Grade 4

Note: The Russian sample did not include schools from the GBAO because Russian-language schools were not present in this region at the time of sampling or data collection. Additionally, results from Russian-language schools in Khatlon-Kulob are not reported due to insufficient sample size.

Teacher Instructional Practices In Tajik-language schools, instructional practices in grade 4 classrooms were more advanced than instructional practices in grade 2 classrooms (Figure 8). Across most focus areas of the lesson observation, the percentage of teachers demonstrating positive instructional practices was significantly higher in Tajik grade 4 classrooms than in Tajik grade 2 classrooms. Instructional practices in Russian grade 2 and Russian grade 4 classrooms were, on average, statistically comparable (Figure 9).

Figure 8. Midline (2019) Overall Lesson Observation Scores for Teachers by Grade, Tajik

Note: One asterisk (*) indicates a difference between grade 2 and grade 4 teachers at p < 0.05. Three asterisks (***) indicate a statistically significant difference between grade 2 and grade 4 teachers at p < 0.001.

Figure 9. Midline (2019) Overall Lesson Observation Scores for Teachers by Grade, Russian

Note: No asterisk indicates that differences between baseline and midline were not statistically significant.

Student and School Background Factors At the student level, several factors had positive associations with reading outcomes, including parents’ ability to read in the LOI, practicing reading with someone at home, the availability of books at home, the ability to take books home from school for pleasure reading, and preschool attendance.

At the school level, several factors had positive associations with reading outcomes, including schools with a female director, parent involvement in the schools, the availability of books other than textbooks in the classroom, and class size.

Conclusions and Recommendations Among the midline report findings, several conclusions stand out as having implications for RWM implementation. These conclusions are highlighted below.

Some gaps in performance identified from earlier assessments remain. Similar to the baseline results, the midline results show that, on average, students’ comprehension skills are generally not at the same level as their fluency skills. This is especially true for Tajik grade 2—where students read 36.66 CWPM but answered less than 2 out of 5 comprehension questions correctly—and for Tajik grade 4—where students read 59.86 CWPM but answered just over 2 out of 5 comprehension questions correctly. For all grades and languages, girls continue to do better than boys on most subtasks. Students from rural areas continue to lag behind their urban peers. Disparities remain between regions, with Dushanbe—an urban region—and GBAO outperforming the other regions. Consistent with recommendations from baseline, RWM should be cognizant of these gaps and seek to address them through developing teachers’ understanding of how to build students’ explicit reading comprehension skills, providing teachers with strategies to close the girl-boy performance gap, and providing extra support to teachers and schools in rural areas—especially in the DRS, Khatlon-Bokhtar, Khatlon-Kulob, and Sughd.

A second key finding with implications for RWM implementation is that performance for Tajik grade 2 students declined on several skill areas from baseline to midline. On three out of the four timed subtasks—letter name identification, familiar word reading, and ORF—Tajik grade 2 students scored significantly lower at midline than at baseline. This is notable because the timed subtasks, along with initial sound identification, are the EGRA subtasks that are most comparable from one time point to another.[footnoteRef:3] By contrast, baseline and midline results for all timed subtasks were statistically similar for Tajik grade 4 and Russian grade 4 students, and the only timed subtask with a statistically significant difference from baseline to midline for Russian grade 2 was letter name identification. The declines for Tajik grade 2 students may be the result of teachers for that language and grade struggling to adapt and implement teaching strategies prescribed by the RWM intervention—a possibility that is supported by the fact that teachers in Tajik grade 2 classrooms received significantly lower overall scores on lesson facilitation, checking for understanding, and providing student feedback than teachers in Tajik grade 4 classrooms (see Figure 8 above). RWM should consider making lesson observations part of routine project monitoring to gain greater understanding if the decline in Tajik grade 2 results is due to teacher instruction or other factors. [3: Across languages and grades, the ORF subtask used at midline was statistically equated to the ORF subtask used at baseline. For letter name identification, familiar word reading, and nonword reading, the midline tools used the same items used at baseline, only the items were re-randomized to mitigate potential test leakage. For the reading comprehension, silent reading comprehension, and listening comprehension subtasks, statistical equating was not possible due to the number of items on each subtask (five) and nature of the tasks. Therefore, comparisons of results from baseline to midline for these subtasks should be made with caution.]

A third key finding with implications for implementation is the continued importance of fostering a culture of reading to help students improve their abilities. Analysis of student survey data revealed that having reading books at home, having reading books at school that can be taken home, and practicing reading aloud with someone at home were predictors of student reading performance. Through the provision of reading books and engagement with parents and the community on the importance supporting students’ reading, RWM has an opportunity to leverage its current and future activities to address these student-level factors. RWM’s future work with schools and communities should also aim to ensure that books are provided for students to take home and to encourage adults to read with students outside of school.

Introduction

Read With Me Project Background The Government of the Republic of Tajikistan has implemented reforms to improve the quality of its education system since independence and the country’s five-year civil war in the 1990s.[footnoteRef:4] The Government of Tajikistan and the Ministry of Education and Science (MoES) view inclusive access to quality education as a requirement of the country’s transition to a market economy and is committed to reforming the education sector as outlined in its National Strategy for Education Development (2012–20) and Midterm Development Strategy (2016–20).[footnoteRef:5] Investment in education—particularly early childhood education and the inclusion of girls, rural students, and students with disabilities—supports building the capacity of Tajikistan’s workforce and contributing to the economic development of the country.[footnoteRef:6] Although Tajikistan’s education budget has steadily increased as a percentage of the gross domestic product, the level of financing is inadequate to meet system needs, according to a funding gap analysis conducted by the MoES.[footnoteRef:7] [4: Government of the Republic of Tajikistan, National Strategy for Education Development for 2012-2020. (2012), 3–4. http://planipolis.iiep.unesco.org/sites/planipolis/files/ressources/tajikistan_ed_sector_plan_2012-2020.pdf and United Nations in Tajikistan. Annual UNDAF Results Report. (2016). https://untj.org/files/Publications/UNDAF/UNDAF_Annual_Report_English_Version_2016.pdf] [5: Ibid] [6: Government of the Republic of Tajikistan, National Strategy for Education Development for 2012-2020. (2012), 3, 9–12.] [7: The World Bank Group, Tajikistan Partnership Program Snapshot (October 2015), 5. http://pubdocs.worldbank.org/en/645741444794465533/Tajikistan-Snapshot.pdf ]

Aid agencies and international organizations in Tajikistan help to fill this gap through initiatives like early childhood education, inclusive education, quality improvements in education, and child-friendly classrooms, as well as other educational programming. The United States Agency for International Development (USAID) has provided support to the government’s reforms to primary education for years through efforts such as the 2007–12 Quality Learning Project (QLP) and the 2013–17 Quality Reading Project (QRP).[footnoteRef:8] USAID is now supporting the third initiative—the Read with Me project (RWM)—to build on the gains made by these two projects in reading skills, including comprehension. [8: RWM expands on QRP’s work, which reached over 60 percent of schools in Tajikistan. QRP supported reading instruction in grade 1 through 4 through in-service training and classroom-based mentoring for educators, as well as Tajik and Russian-language reading materials and other school-based reading activities and governmental policy support to reading. See American Institutes for Research, USAID Quality Reading Project Republic of Tajikistan: Final EGRA and Impact Report, 2013–2017, (2017).

RWM is a five-year (2016–21) project that builds on both QLP and QRP, as well as other education programs in Tajikistan; it supports the National Strategy for Education Development 2012–20 and the Midterm Development Strategy for 2016–20. USAID’s RWM project involves both the MoES and individual schools to support the improvement of reading outcomes, as well as reaches out to communities as well as the private sector. It looks to introduce the role of science, technology, innovations, and partnership in supporting reading outcomes. The RWM project provides supports directly to schools in a series of four cohorts from 2018 through 2021.

RWM aims to improve reading outcomes for students in grades 1 through 4 in targeted schools in Tajikistan by

a. Increasing the availability of age-appropriate reading materials in Tajik and Russian languages, which will develop students’ core reading skills;

b. Providing educators with reading-specific in-service training, supplementary materials, and the integration of assessments to monitor reading progress;

c. Increasing innovations and partnerships supporting literacy outcomes; and,

d. Increasing government support for reading through capacity building in reading instruction and mentoring as well as increasing dialogue on the reading environment.[footnoteRef:9] [9: Chemonics International, Inc, USAID Read With Me Project Annual Report Year 1. (2017). https://pdf.usaid.gov/pdf_docs/PA00N7RF.pdf ]

RWM aims to support 75 percent of all primary school teachers nationwide (approximately 24,000 teachers) and 75 percent of all primary grade students nationwide (approximately 524,000 students) in approximately 2,800 schools over the course of implementation. RWM is active in every region of Tajikistan and expands USAID’s support of early grade reading education to the Gorno-Badakhshan Autonomous Oblast (GBAO).

Early Grade Reading Outcomes in Tajikistan The RWM Early Grade Reading Assessment (EGRA) builds on a foundation of research in early grade reading outcomes in Tajikistan. Previous USAID-funded projects have administered multiple EGRAs, which provide the most extensive research into early grade reading outcomes in Tajikistan. In 2011, the first USAID-funded EGRA in Tajikistan analyzed results from students in Tajik and Russian in grades 2 to 4 and included a qualitative study of early grade reading pedagogy in Tajikistan.[footnoteRef:10] This study, implemented in the last year of QLP, found that students generally had strong foundations in early grade reading skills, including the alphabetic principle and recognition of sight words; however, they had a weakness in decoding unfamiliar words and comprehending texts. The report suggested links between classroom practices and these outcomes, as pedagogy at that time focused more on rote memorization of words and speed reading rather than on phoneme segmentation, decoding skills, and comprehension strategies.[footnoteRef:11] [10: Mirka Tvaruzkova and Duishon Shamatov, Review of Early Grade Reading Teaching and Skills: The Kyrgyz Republic and Tajikistan Final Report.