Attachment J.22 EGRA EGMA Reports.pdf

PAQUED: DRC

Projet d’Amélioration de la Qualité de l’Education (PAQUED) 2014 Endline Report of Early Grade Reading Assessment (EGRA) and Early Grade Mathematics Assessment

(EGMA)

October 2014 This publication was produced for review by the United States Agency for International Development and by Education Development Center. It was prepared by RTI International.

PAQUED: DRC

Projet d’Amélioration de la Qualité de l’Education

(PAQUED)

Endline Report of Early Grade Reading Assessment (EGRA) and Early Grade Math Assessment (EGMA)

Contract: AID-623-A-09-00010 EDC Project: 11224 Date: October 2014

Prepared for USAID/DRC and EDC

Prepared by Jennae Bulat, Aarnout Brombacher, Timothy Slade Jennifer Iriondo-Perez, Maureen Kelly, Susan Edwards RTI International 3040 Cornwallis Road Post Office Box 12194 Research Triangle Park, NC 27709-2194

The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

RTI International is one of the world’s leading research institutes, dedicated to improving the human condition by turning knowledge into practice. Our staff of more than 3,700 provides research and technical services to governments and businesses in more than 75 countries in the areas of health and pharmaceuticals, education and training, surveys and statistics, advanced technology, international development, economic and social policy, energy and the environment, and laboratory testing and chemical analysis. For more information, visit www.rti.org.

RTI International is a trade name of Research Triangle Institute.

Projet d’Amélioration de la Qualité de l’Education (PAQUED) iii

Table of Contents Page

List of Figures ............................................................................................................................. vi

List of Tables .............................................................................................................................. ix

Acronyms ................................................................................................................................... xx

Executive Summary

Evolving Donor, Country, and DRC Education Policy Context: 2009-2014 Study Design and Limitations Data Collection

Timing of Studies Tools Used to Measure Impact

Findings Impact on Student Reading Performance Discussion of Mathematics Outcomes

Recommendations

Introduction

Brief Chronological History of Impact Evaluation Activities under PAQUED Structure of the Report

Chapter A: PAQUED Project Context and Evolution; Study Design, Objectives, and Limitations

1. Project Context and Evolution

Project Implementation pre-2012 Realignment of PAQUED’s Focus for 2012–2014 Elements of the PAQUED Intervention pre-Realignment

PAQUED Inputs Targeting Students PAQUED Inputs Targeting Teachers

Elements of the “Reading Program” Intervention Post-Realignment Reading Program Inputs Targeting Students Reading Program Inputs Targeting Teachers

2. Study Design and Objectives

Research Design The EGRA Tool The EGMA Tool Student Context, Teacher, and Head Teacher Interviews Assessor Training

3. Limitations

Misalignment of the Intervention and Assessment Curricular Expectations of Grade 2 Students’ Reading Abilities iv Projet d’Amélioration de la Qualité de l’Education (PAQUED)

Changes in Sampling Design

Chapter B: Results and Analysis of Student Reading Performance in Reading Program Schools

1. Descriptive Statistics

2014 Sample

2. EGRA Subtask Outcomes

Vocabulary Initial Sound Identification Listening Comprehension Grapheme Recognition Familiar Word Reading Invented Word Reading Oral Reading Fluency Reading Comprehension Dictation

Student Characteristics Analyses Teacher Characteristics Analyses Head Teacher Characteristics Analyses

3. Summary and Conclusions

Chapter C: Results and Analysis of Student Reading Performance in Accessible PAQUED and Accessible Control Schools

1. Student Performance on EGRA Measures in Accessible PAQUED and Accessible Control Schools

Descriptives 2014 Sample

2. EGRA Subtest Outcomes

Vocabulary Initial Sound Identification Listening Comprehension Grapheme Recognition Familiar Word Reading Invented Word Reading Oral Reading Fluency Reading Comprehension Dictation

Student Characteristics Analyses Teacher Characteristics Analyses Head Teacher Characteristics Analyses

3. Summary and Conclusions

Chapter D: Results and Analysis of Student Reading Performance in PAQUED and Control Schools

Projet d’Amélioration de la Qualité de l’Education (PAQUED) v

1. Student Performance on EGRA Measures in PAQUED and Control Schools

Descriptives 2014 Sample

2. EGRA Subtest Outcomes

Vocabulary Initial Sound Identification Listening Comprehension Grapheme Recognition Familiar Word Reading Invented Word Reading Oral Reading Fluency Reading Comprehension Dictation

Student Characteristics Analyses Teacher Characteristics Analyses Head Teacher Characteristics Analyses

3. Summary and Conclusions

Chapter E: Results and Analysis of Student Mathematics Performance in Reading Program, Accessible PAQUED, and Accessible Control Schools

1. EGMA Results in Reading Program, Accessible PAQUED, and Accessible Control Schools

EGMA Bandundu EGMA Equateur EGMA Orientale

2. EGMA Results Disaggregated by Sex in Reading Program, Accessible PAQUED, and Accessible Control Schools

EGMA Bandundu Accessible PAQUED EGMA Equateur Accessible PAQUED schools EGMA Orientale Accessible PAQUED schools EGMA Bandundu Reading Program schools EGMA Equateur Reading Program schools EGMA Orientale Reading Program schools

Chapter F: Results and Analysis of Student Mathematics Performance in PAQUED and Control Schools

1. EGMA Results in PAQUED and Control Schools

EGMA Bandundu EGMA Equateur EGMA Orientale

2. EGMA Results Disaggregated by Sex in PAQUED and Control Schools

EGMA Bandundu PAQUED Schools vi Projet d’Amélioration de la Qualité de l’Education (PAQUED)

EGMA Equateur PAQUED Schools EGMA Orientale PAQUED Schools

Chapter G: Conclusions and Recommendations

Discussion of Reading Outcomes Discussion of Mathematics Outcomes Recommendations

Annex I: Details on Study Methodology

Annex II: The Early Grade Reading Assessment (EGRA)/Early Grade Mathematics Assessment (EGMA) Data Collection Process

Annex III: About the Assessment Instruments

Annex IV: Reading Program Schools: 2012-2014 Comparisons

List of Figures Figure ES1. Subdivisions Containing Schools Assessed at Endline and Relative Size of the Study Populations

Figure ES2. Performance of Students in Reading Program Schools Relative to DRC Benchmarks

Figure ES3. Performance of Students in Accessible PAQUED Schools Relative to DRC Benchmarks

Figure ES4. Performance of Students in PAQUED Schools Relative to DRC Benchmarks

Figure A1. Subdivisions Containing Schools Assessed at Endline and Relative Size of the Study Populations

Figure B1. Grade 2 Scores on Vocabulary at Endline by Province

Figure B2. Grade 4 Scores on Vocabulary at Endline by Province

Figure B3. Grade 2 Scores on Initial Sound Identification at Endline by Province

Figure B4. Grade 4 Scores on Initial Sound Identification at Endline by Province

Figure B5. Grade 2 Scores on Listening Comprehension at Endline by Province

Figure B6. Grade 4 Scores on Listening Comprehension at Endline by Province

Figure B7. Grade 2 Scores on Grapheme Recognition at Endline by Province

Figure B8. Grade 4 Scores on Grapheme Recognition at Endline by Province

Figure B9. Grade 4 Scores on Familiar Word Reading at Endline by Province

Figure B10. Grade 4 Scores on Invented Word Reading at Endline by Province

Figure B11. Grade 4 Scores on Oral Reading Fluency at Endline by Province

Projet d’Amélioration de la Qualité de l’Education (PAQUED) vii

Figure B12. Grade 4 Scores on Reading Comprehension at Endline by Province

Figure B13. Grade 4 Overall Correspondence between Oral Reading Fluency and Reading Comprehension

Figure B14. Grade 4 Scores on Dictation at Endline by Province

Figure B15. Performance of Students in Reading Program Schools Relative to DRC Benchmarks

Figure C1. Grade 2 Scores on Vocabulary at Endline by Province

Figure C2. Grade 4 Scores on Vocabulary at Endline by Province

Figure C3. Grade 2 Scores on Initial Sound Identification at Endline by Province

Figure C4. Grade 4 Scores on Initial Sound Identification at Endline by Province

Figure C5. Grade 2 Scores on Listening Comprehension at Endline by Province

Figure C6. Grade 4 Scores on Listening Comprehension at Endline by Province

Figure C7. Grade 2 Scores on Grapheme Recognition at Endline by Province

Figure C8. Grade 4 Scores on Grapheme Recognition at Endline by Province

Figure C9. Grade 4 Scores on Familiar Word Reading at Endline by Province

Figure C10. Grade 4 Scores on Invented Word Reading at Endline by Province

Figure C11. Grade 4 Scores on Oral Reading Fluency at Endline by Province

Figure C12. Grade 4 Scores on Reading Comprehension at Endline by Province

Figure C13. Grade 4 Overall Correspondence between Oral Reading Fluency and Reading Comprehension

Figure C14. Grade 4 Scores on Dictation at Endline by Province

Figure C15. Performance of Students in Accessible PAQUED Schools Relative to DRC Benchmarks

Figure D1. Grade 2 Scores on Vocabulary at Endline by Province

Figure D2. Grade 4 Scores on Vocabulary at Endline by Province

Figure D3. Grade 6 Scores on Vocabulary at Endline by Province

Figure D4. Grade 2 Scores on Initial Sound Identification at Endline by Province

Figure D5. Grade 4 Scores on Initial Sound Identification at Endline by Province

Figure D6. Grade 6 Scores on Initial Sound Identification at Endline by Province

Figure D7. Grade 2 Scores on Listening Comprehension at Endline by Province

Figure D8. Grade 4 Scores on Listening Comprehension at Endline by Province

Figure D9. Grade 6 Scores on Listening Comprehension at Endline by Province viii Projet d’Amélioration de la Qualité de l’Education (PAQUED)

Figure D10. Grade 2 Scores on Grapheme Recognition at Endline by Province

Figure D11. Grade 4 Scores on Grapheme Recognition at Endline by Province

Figure D12. Grade 6 Scores on Grapheme Recognition at Endline by Province

Figure D13. Grade 4 Scores on Familiar Word Reading at Endline by Province

Figure D14. Grade 6 Scores on Familiar Word Reading at Endline by Province

Figure D15. Grade 4 Scores on Invented Word Reading at Endline by Province

Figure D16. Grade 6 Scores on Invented Word Reading at Endline by Province

Figure D17. Grade 4 Scores on Oral Reading Fluency at Endline by Province

Figure D18. Grade 6 Scores on Oral Reading Fluency at Endline by Province

Figure D19. Grade 4 Scores on Reading Comprehension at Endline by Province

Figure D20. Grade 6 Scores on Reading Comprehension at Endline by Province

Figure D21. Grade 4 Overall Correspondence between Oral Reading Fluency and Reading Comprehension

Figure D22. Grade 6 Overall Correspondence between Oral Reading Fluency and Reading Comprehension

Figure D23. Grade 4 Scores on Dictation at Endline by Province

Figure D24. Grade 6 Scores on Dictation at Endline by Province

Figure D25. Performance of Students in PAQUED Schools Relative to DRC Benchmarks

Figure E1. EGMA Bandundu: 2014 Mean Percentage Scores by Subtask, Grade, and Treatment Group

Figure E2. EGMA Equateur: 2014 Mean Percentage Scores by Subtask, Grade, and Treatment Group

Figure E3. EGMA Orientale: 2014 Mean Percentage Scores by Subtask, Grade, and Treatment Group

Figure E4. EGMA Bandundu Accessible PAQUED Schools: 2014 Mean Percentage Scores by Subtask, Grade, and Sex

Figure E5. EGMA Equateur Accessible PAQUED Schools: 2014 Mean Percentage Scores by Subtask, Grade, and Sex

Figure E6. EGMA Orientale Accessible PAQUED schools: 2014 Mean Percentage Scores by Subtask, Grade, and Sex

Figure E7. EGMA Bandundu Reading Program Schools: 2014 Mean Percentage Scores by Subtask, Grade, and Sex

Projet d’Amélioration de la Qualité de l’Education (PAQUED) ix

Figure E8. EGMA Equateur Reading Program Schools: 2014 Mean Percentage Scores by Subtask, Grade, and Sex

Figure E9. EGMA Orientale Reading Program Schools: 2014 Mean Percentage Scores by Subtask, Grade, and Sex

Figure F1. EGMA Bandundu: 2014 Mean Percentage Scores by Subtask, Grade, and Treatment Group

Figure F2. EGMA Equateur: 2014 Mean Percentage Scores by Subtask, Grade, and Treatment Group

Figure F3. EGMA Orientale: 2014 Mean Percentage Scores by Subtask, Grade, and Treatment Group

Figure F4. EGMA Bandundu Treatment: 2014 Mean Percentage Scores by Subtask, Grade, and Sex

Figure F5. EGMA Equateur Treatment: 2014 Mean Percentage Scores by Subtask, Grade, and Sex

Figure F6. EGMA Orientale Treatment: 2014 Mean Percentage Scores by Subtask, Grade, and Sex

Figure A1.1 Subdivisions Containing Schools Assessed at Endline and Relative Size of the Study Populations

List of Tables Table ES1. Study Populations and Analyses Permitted

Table ES2. Subtask Composition of Assessment Instruments

Table ES3. Reading Performance in Subset of Reading Program Schools (n=20), in Mean Scores

Table ES4. Student Performance in Reading Program Schools Relative to National Benchmarks

Table ES5. Reading Performance in Accessible PAQUED Schools, in Mean Scores, by Province

Table ES6. Student Performance in Accessible PAQUED Schools Relative to National Benchmarks

Table ES7. Reading Performance in PAQUED Schools, in Mean Scores, by Province x Projet d’Amélioration de la Qualité de l’Education (PAQUED)

Table ES8. Student Performance in PAQUED Schools Relative to National Benchmarks

Table ES9. Change in Mathematics Performance in Reading Program Schools (n =

20) over Time, in Percentage of Items Correct

Table ES10. Change in Mathematics Performance in Accessible PAQUED Schools over Time, in Percentage of Items Correct, by Province

Table ES11. Change in Mathematics Performance in PAQUED Schools over Time, in Percentage of Items Correct, by Province

Table A1. PAQUED Intervention—Inputs Targeting Students

Table A2. PAQUED Intervention—Training Inputs Targeting Teachers

Table A3. PAQUED Intervention—Materials Inputs Targeting Teachers

Table A4. Reading Program—Inputs Targeting Students

Table A5. Reading Program—Additional Inputs Targeting Teachers

Table A6. Study Populations and Analyses Permitted

Table B1. Intended Sample by Province and Grade

Table B2. Actual Sample by Province and Grade

Table B3. School Management Type, by Province (n = 43)

Table B4. General Characteristics of the Overall Student Sample (n = 1,103)

Table B5. Student SES Indicators (n = 1,103)

Table B6. Student Responses to Select Demographic Questions (n = 1,097)

Table B7. Student Indication of Language Spoken in the Home

Table B8. General Characteristics of the Teacher Sample (n = 86)

Table B9. Teacher Responses to Select Survey Questions (n = 86)

Table B10. General Characteristics of the Head Teacher Sample (n = 42)

Table B11. Head Teacher Responses to Select Survey Questions (n = 42)

Table B12. Overall Percent Zero Scores and Mean Scores by Grade and Subtask at Endline

Table B13. Vocabulary Zero Scores and Percentage of Items Attempted, by Grade and Province

Table B14. Vocabulary Mean Scores by Type of Vocabulary Word, by Grade and Province

Table B15. Comparison of Vocabulary Mean Scores by Grade, Province, and Sex

Projet d’Amélioration de la Qualité de l’Education (PAQUED) xi

Table B16. Initial Sound Identification Zero Scores and Percentage of Items Attempted by Grade and Province

Table B17. Comparison of Initial Sound Identification Mean Scores by Grade, Province, and Sex

Table B18. Listening Comprehension Zero Scores and Percentage of Items Attempted by Grade and Province

Table B19. Comparison of Listening Comprehension Mean Scores by Grade, Province, and Sex

Table B20. Grapheme Recognition Zero Scores and Percentage of Items Attempted by Grade and Province

Table B21. Comparison of Grapheme Recognition Mean Scores by Grade, Province, and Sex

Table B22. Familiar Word Reading Zero Scores and Percent Attempted for Grade 4 by Province

Table B23. Comparison of Familiar Word Reading Mean Scores for Grade 4 by Province and Sex

Table B24. Invented Word Reading Zero Scores and Percent Attempted for Grade 4 by Province

Table B25. Comparison of Invented Word Reading Mean Scores for Grade 4 by Province and Sex

Table B26. Oral Reading Fluency Zero Scores and Percent Attempted for Grade 4 by Province

Table B27. Comparison of Oral Reading Fluency Mean Scores for Grade 4 by Province and Sex

Table B28. Reading Comprehension Zero Scores and Percent Attempted for Grade 4 by Province

Table B29. Comparison of Reading Comprehension Mean Scores for Grade 4 by Province and Sex

Table B30. Dictation Zero Scores and Percent Attempted, for Grade 4 by Province

Table B31. Comparison of Dictation Mean Scores for Grade 4 by Province and Sex

Table B32. Chi-Squared Analyses of Grade 2 Student Characteristics with Student High Performance on the Grapheme Recognition Subtask

Table B33. Grade 2 Student Characteristics Logistic Regressions

Table B34. Chi-Squared Analyses of Grade 4 Student Characteristics with Student High Performance on the Grapheme Recognition Subtask xii Projet d’Amélioration de la Qualité de l’Education (PAQUED)

Table B35. Grade 4 Student Characteristics Logistic Regressions

Table B36. Teacher Characteristics Logistic Regressions

Table B37. Student Performance in Reading Program Schools Relative to National Benchmarks

Table C1. Intended School Sample by Province

Table C2. Intended Student Sample by Province and Grade

Table C3. Actual Sample by Province and Grade

Table C4. School Management Type, by Province (n = 82)

Table C5. General Characteristics of the Overall Student Sample (n = 1,965)

Table C6. Student SES Indicators (n = 1,965)

Table C7. Select Student Responses to Demographic Questions (n = 1,097)

Table C8. Student Indication of Language Spoken in the Home

Table C9. General Characteristics of the Teacher Sample (n = 154)

Table C10. Select Teacher Responses to Survey Questions (n = 86)

Table C11. General Characteristics of the Head Teacher Sample (n = 81)

Table C12. Select Head Teacher Responses to Survey Questions (n = 42)

Table C13. Overall Percent Zero Scores and Mean Scores by Subtask, Grade and Group at Endline

Table C14. Vocabulary Zero Scores and Percent Attempted by Grade and Province

Table C15. Vocabulary Difference-in-Differences Analyses by Grade, Region, and Group

Table C16. Comparison of Vocabulary Mean Scores by Grade, Province, Group, and Sex

Table C17. Initial Sound Identification Zero Scores and Percent Attempted by Grade and Province

Table C18. Initial Sound Identification Difference-in-Differences Analyses by Grade, Region, and Group

Table C19. Comparison of Initial Sound Identification Mean Scores by Grade, Province, Group, and Sex

Table C20. Listening Comprehension Zero Scores and Percent Attempted by Grade and Province

Table C21. Listening Comprehension Difference-in-Differences Analyses by Grade, Region, and Group

Projet d’Amélioration de la Qualité de l’Education (PAQUED) xiii

Table C22. Comparison of Listening Comprehension Mean Scores by Grade, Province, Group, and Sex

Table C23. Grapheme Recognition Zero Scores, Percent Attempted, and Mean Scores by Grade and Province

Table C24. Grapheme Recognition Difference-in-Differences Analyses by Grade, Region, and Group

Table C25. Comparison of Grapheme Recognition Mean Scores by Grade, Province, and Sex

Table C26. Familiar Word Reading Zero Scores and Percent Attempted for Grade 4 by Province

Table C27. Familiar Word Reading Difference-in-Differences Analyses by Region and Group

Table C28. Comparison of Familiar Word Reading Mean Scores for Grade 4 by Province, Group, and Sex

Table C29. Invented Word Reading Zero Scores and Percent Attempted for Grade 4 by Province

Table C30. Invented Word Reading Difference-in-Differences Analyses by Region and Group

Table C31. Comparison of Invented Word Reading Mean Scores for Grade 4 by Province, Group, and Sex

Table C32. Oral Reading Fluency Zero Scores and Percent Attempted for Grade 4 by Province

Table C33. Oral Reading Fluency Difference-in-Differences Analyses by Region and Group

Table C34. Comparison of Oral Reading Fluency Mean Scores for Grade 4 by Province, Group, and Sex

Table C35. Reading Comprehension Zero Scores and Percent Attempted for Grade 4 by Province

Table C36. Reading Comprehension Difference-in-Differences Analyses by Region and Group

Table C37. Comparison of Reading Comprehension Mean Scores for Grade 4 by Province, Group, and Sex

Table C38. Dictation Zero Scores and Percent Attempted for Grade 4 by Province

Table C39. Dictation Difference-in-Differences Analyses by Region and Group

Table C40. Comparison of Dictation Mean Scores for Grade 4 by Province and Sex xiv Projet d’Amélioration de la Qualité de l’Education (PAQUED)

Table C41. Chi-Squared Analyses of Grade 2 Student Characteristics with Student High Performance on the Grapheme Sound Knowledge Subtask, Accessible PAQUED and Accessible Control Schools

Table C42. Chi-Squared Analyses of Grade 4 Student Characteristics with Student High Performance on the Grapheme Sound Knowledge Subtask, Accessible PAQUED and Accessible Control Schools

Table C43. Grade 4 Student Characteristics Logistic Regressions

Table C44. Teacher Characteristics Logistic Regressions

Table C45. Student Performance in Accessible PAQUED Schools Relative to National Benchmarks

Table D1. Intended School Sample by Province and Grade

Table D2. Intended Student Sample by Province and Grade (n = 5,616)

Table D3. Actual School Sample by Province and Grade

Table D4. Actual Student Sample by Province and Grade (n = 5,255)

Table D5. School Type, by Group (n = 145)

Table D6. General Characteristics of the Overall Student Sample (n = 5,255)

Table D7. Student SES Indicators (n = 5,255)

Table D8. Select Student Responses to Demographic Questions (n = 5,204)

Table D9. Student Indication of Language(s) Spoken in the Home

Table D10. General Characteristics of the Teacher Sample (n = 389)

Table D11. Select Teacher Responses to Survey Questions (n = 389)

Table D12. General Characteristics of the Head Teacher Sample (n = 142)

Table D13. Select Head Teacher Responses to Survey Questions (n = 142)

Table D14. Overall Percent Zero Scores and Mean Scores by Grade and Subtask at Endline

Table D15. Vocabulary Zero Scores and Percent Attempted, by Grade and Province

Table D16. Vocabulary Difference-in-Differences Analyses by Grade, Province, and Group

Table D17. Comparison of Vocabulary Mean Scores by Grade, Province, Group, and Sex

Table D18. Initial Sound Identification Zero Scores and Percent Attempted, by Grade and Province

Projet d’Amélioration de la Qualité de l’Education (PAQUED) xv

Table D19. Initial Sound Identification Difference-in-Differences Analyses by Grade, Province, and Group

Table D20. Comparison of Initial Sound Identification Mean Scores by Grade, Province, Group, and Sex

Table D21. Listening Comprehension Zero Scores and Percent Attempted by Grade and Province

Table D22. Listening Comprehension Difference-in-Differences Analyses by Grade, Province, and Group

Table D23. Comparison of Listening Comprehension Mean Scores by Grade, Province, Group, and Sex

Table D24. Grapheme Recognition Zero Scores and Percent Attempted by Grade and Province

Table D25. Grapheme Recognition Difference-in-Differences Analyses by Grade, Province, and Group

Table D26. Comparison of Grapheme Recognition Mean Scores by Grade Province, Group, and Sex

Table D27. Familiar Word Reading Zero Scores and Percent Attempted by Grade and Province

Table D28. Familiar Word Reading Difference-in-Differences Analyses by Grade, Province, and Group

Table D29. Comparison of Familiar Word Reading Mean Scores by Grade, Province, Group, and Sex

Table D30. Invented Word Reading Zero Scores and Percent Attempted by Grade and Province

Table D31. Invented Word Reading Difference-in-Differences Analyses by Grade, Province, and Group

Table D32. Comparison of Invented Word Reading Mean Scores by Grade, Province, Group, and Sex

Table D33. Oral Reading Fluency Zero Scores and Percent Attempted by Grade and Province

Table D34. Oral Reading Fluency Difference-in-Differences Analyses by Grade, Province, and Group

Table D35. Comparison of Oral Reading Fluency Mean Scores by Grade, Province, Group, and Sex xvi Projet d’Amélioration de la Qualité de l’Education (PAQUED)

Table D36. Reading Comprehension Zero Scores and Percent Attempted by Grade and Province

Table D37. Reading Comprehension Difference-in-Differences Analyses by Grade, Province, and Group

Table D38. Comparison of Reading Comprehension Mean Scores by Grade, Province, Group, and Sex

Table D39. Dictation Zero Scores and Percent Attempted by Grade and Province

Table D40. Dictation Difference-in-Differences Analyses by Grade, Province, and Group

Table D41. Comparison of Dictation Mean Scores by Grade, Province, Group, and Sex

Table D42. Chi-Squared Analyses of Grade 2 Student Characteristics with Student High Performance on the Grapheme Recognition Subtask

Table D43. Chi-Squared Analyses of Grade 4 Student Characteristics with Student High Performance on the Grapheme Recognition Subtask

Table D44. Grade 4 Student Characteristics Logistic Regressions

Table D45. Chi-Squared Analyses of Grade 6 Student Characteristics with Student High Performance on the Grapheme Sound Knowledge Subtask

Table D46. Student Performance in PAQUED Schools Relative to National Benchmarks

Table E1. EGMA Bandundu: 2014 Percent Zero Scores and Mean Percentage Scores by Grade

Table E2. EGMA Bandundu: 2012 vs. 2014 Difference-in-Differences of Mean Percentage Scores by Grade

Table E3. EGMA Equateur: 2014 Percent Zero Scores and Mean Percentage Scores by Grade

Table E4. EGMA Equateur: 2012 vs. 2014 Difference-in-Differences of Mean Percentage Scores by Grade

Table E5. EGMA Orientale: 2014 Percent Zero Scores and Mean Percentage Scores by Grade

Table E6. EGMA Orientale: 2012 vs. 2014 Difference-in-Differences of Mean Percentage Scores by Grade

Table E7. EGMA Bandundu Accessible PAQUED Schools: 2014 Percent Zero Scores and Mean Percentage Scores by Sex and Grade

Projet d’Amélioration de la Qualité de l’Education (PAQUED) xvii

Table E8. EGMA Equateur Accessible PAQUED Schools: 2014 Percent Zero Scores and Mean Percentage Scores by Sex and Grade

Table E9. EGMA Orientale Accessible PAQUED Schools: 2014 Percent Zero Scores and Mean Percentage Scores by Sex and Grade

Table E10. EGMA Bandundu Reading Program Schools: 2014 Percent Zero Scores and Mean Percentage Scores by Sex and Grade

Table E11. EGMA Equateur Reading Program Schools: 2014 Percent Zero Scores and Mean Percentage Scores by Sex and Grade

Table E12. EGMA Orientale Reading Program Schools: 2014 Percent Zero Scores and Mean Percentage Scores by Sex and Grade

Table F1. EGMA Bandundu: 2014 Percent Zero Scores and Mean Percentage Scores by Grade

Table F2. EGMA Bandundu: 2010 vs. 2014 Difference-in-Differences of Mean Percentage Scores by Grade

Table F3. EGMA Equateur: 2014 Percent Zero Scores and Mean Percentage Scores by Grade

Table F4. EGMA Equateur: 2010 vs. 2014 Difference-in-Differences of Mean Percentage Scores by Grade

Table F5. EGMA Orientale: 2014 Percent Zero Scores and Mean Percentage Scores by Grade

Table F6. EGMA Orientale: 2010 vs. 2014 Difference-in-Differences of Mean Percentage Scores by Grade

Table F7. EGMA Bandundu PAQUED Schools: 2014 Percent Zero Scores and Mean Percentage Scores by Sex and Grade

Table F8. EGMA Equateur PAQUED Schools: 2014 Percent Zero Scores and Mean Percentage Scores by Sex and Grade

Table F9. EGMA Orientale PAQUED Schools: 2014 Percent Zero Scores and Mean Percentage Scores by Sex and Grade

Table G1. Study Populations and Analyses Permitted

Table A1.1: 2010 Sample: Geographic Distribution by Treatment Type

Table A1.2: 2012 Sample Geographic Distribution by Treatment Type

Table A1.3: 2014 Sample A Number of Schools Sampled by Geographic Distribution and Treatment Type

Table A1.4: 2014 Sample A Number of Students Sampled by Geographic Distribution, Treatment Type, and Grade xviii Projet d’Amélioration de la Qualité de l’Education (PAQUED)

Table A1.5: 2014 Sample B Number of Schools Sampled by Geographic Distribution and Treatment Type

Table A1.6: 2014 Sample B Number of Students Sampled by Geographic Distribution, Treatment Type, and Grade

Table A1.7: 2014 Sample A Frame and Sample Counts: Subdivisions (Stage 1)

Table A1.8: 2014 Sample A Frame and Sample Counts for PAQUED Schools (Stage 2)

Table A1.9: 2014 Sample A Frame and Sample Counts for Control Schools (Stage 2)

Table A1.10: 2014 Sample B Frame and Sample Counts for Subdivisions (Stage 1)

Table A1.11: 2014 Sample B Frame and Sample Counts for Accessible PAQUED Schools (Stage 2)

Table A1.12: 2014 Sample B Frame and Sample Counts for Accessible Control Schools (Stage 2)

Table A1.13: Chapter B Bonferroni Adjustment Families for Reading Program Study, EGRA Results

Table A1.14: Chapter C Section 1 - Bonferroni Adjustment Families for Accessible PAQUED Study, EGRA Results

Table A1.15: Chapter C Section 2 - Bonferroni Adjustment Families for PAQUED Study, EGRA Results

Table A1.16: Chapter D Sections 1 and 2 - Bonferroni Adjustment Families for Accessible PAQUED Study, EGMA Results

Table A1.17: Chapter D Sections 3 and 4 - Bonferroni Adjustment Families for Accessible PAQUED Study, EGMA Results

Table A3.1: Correlation Matrix for EGRA Tasks

Table A3.2: Correlation Matrix for EGMA Tasks

Table A3.3: Reliability Analysis of EGRA Tool

Table A3.4: Reliability analysis of EGMA Tool

Table A3.5: Comparison of EGRA and EGMA Instruments

Table A3.6: Example of Application of the Means Equating Method

Table A4.1: Reading Program Schools Change Over Time, Grade 2

Table A4.2: Reading Program Schools Change Over Time, Grade 4

Table A4.3: Reading Program Schools (n=20) Change Over Time, Mathematics, Grade 2

Projet d’Amélioration de la Qualité de l’Education (PAQUED) xix

Table A4.4: Reading Program Schools (n=20) Change Over Time, Mathematics, Grade 4 xx Projet d’Amélioration de la Qualité de l’Education (PAQUED)

Acronyms CELTA Centre de Linguistique Théorique et Appliquée cgpm correct graphemes per minute cnonwpm correct non-words per minute (invented words) cwpm correct words per minute (familiar words)

DEFF design effect

DRC Democratic Republic of Congo

EDC Education Development Center

EGMA Early Grade Mathematics Assessment

EGRA Early Grade Reading Assessment

FPC Finite Population Correction

IAI interactive audio instruction

IRR inter-rater reliability

MEPSP Ministry of Primary, Secondary, and Professional Education orf oral reading fluency

PAQUED Projet d’Amélioration de la Qualité de l’Education (PIEQ)

PIEQ Package for Improving Education Quality (PAQUED)

PCA principal components analysis

RTI RTI International

SBD subdivisions

SES socioeconomic indicators

TLM teaching and learning material

USAID U.S. Agency for International Development

WT weight

Projet d’Amélioration de la Qualité de l’Education (PAQUED) 1

Executive Summary Funded by the U.S. Agency for International Development (USAID) and led by the Education Development Center (EDC), the Projet d’Amélioration de la Qualité de l’Education (PAQUED) project was a five-year initiative that aimed to raise student learning through improved teaching and school environments in the Democratic Republic of the Congo (DRC). As a partner in PAQUED, RTI International (RTI) was responsible for adapting and applying the EGRA and EGMA instruments to assess the program impact on student learning outcomes.

This document presents a discussion of PAQUED’s impact on three distinct populations of schools. Each discussion reflects a different stage in the significant evolution of PAQUED’s focus and scope that took place between project design in 2009 and a formal realignment that occurred in January 2013. These three populations cannot be compared directly to each other, so each is dealt with in a separate chapter of the report.1 In the narrative introduction below they are presented in chronological order of their emergence as the point of focus for the PAQUED project because this is most conducive to understanding the evolution of the project. In both the Executive Summary and the body of the report, however, the actual results are presented in reverse chronological order, reflecting the degree to which each was expected to produce meaningful impact.

Evolving Donor, Country, and DRC Education Policy Context: 2009-2014 The PAQUED project design has evolved significantly since implementation began in 2009. The most significant characteristic of the project evolution was the measured shift away from an emphasis on primary education quality broadly construed toward a focus on improving early grade reading outcomes. As is often the case, this project evolution both influenced and was influenced by the evolving government of the DRC (GDRC) and donor education priorities.

An important contextual development during this period was USAID’s release of its 2011-2015 Education Strategy, which strongly emphasized early grade reading. The GDRC also concurrently followed up on its participation in the 2012 Global Program for Education (GPE) All Children Reading Conference in Kigali by committing to put in place the policy framework and investments necessary to reach ambitious targets for increasing the number of readers in the DRC.

The original PAQUED teacher training and classroom interventions conformed to the Ministry of Education’s (MEPSP) requirement that they align with the existing curriculum and textbooks. However, in response to the changing context, redesigned teacher training and classroom interventions were implemented in 45 Reading Program

1 Please see Annex I for a detailed discussion of the sampling approaches and why they must be treated separately.

2 Projet d’Amélioration de la Qualité de l’Education (PAQUED) schools in the final academic year of the project. These new approaches included approaches characteristic of successful reading programs in other contexts.

Study Design and Limitations The goals of the 2014 studies and this report were to describe (1) changes in student performance in reading and mathematics that took place in PAQUED schools between 2010 and 2014, (2) similar changes in the Accessible PAQUED schools between 2012 and 2014, and (3) the current state of student learning outcomes in PAQUED’s Reading Program schools.

The impact of the intervention in the project’s 3,000 target schools (1,000 each in the provinces of Bandundu, Equateur, and Orientale) was evaluated using a sample of 145 schools. Of this number, 36 were control schools and were not expected to have received any intervention and 109 were treatment schools (called “PAQUED” schools throughout this report) that were expected to have received the intervention. The results presented in this report for PAQUED schools are generalizable to the provincial level. The findings of this study are presented in Chapter D of this report.

The impact of the intervention in the 618 schools out of the original 3,000 that PAQUED deemed most easily accessible—and therefore most likely to have received the foreseen inputs—was evaluated using a sample of 75 schools. Of this number, 36 were “Accessible Control” that were not expected to have received any intervention and 39 were treatment schools (called “Accessible PAQUED” schools throughout this report) that were expected to have received the intervention. The results presented in this report for Accessible PAQUED schools are generalizable to schools which meet the same criteria for accessibility.2 The findings of this study are presented in Chapter C of this report.

The impact of the intervention in the subset of 45 schools (out of the 618 Accessible PAQUED schools) on which PAQUED focused intensely from January 2013 through June 2014—and to which it provided a modified, far more robust set of inputs, including student texts, regular coaching, and peer-led continuous professional development—was evaluated by attempting to assess all 44 schools (only 43 were reached). These are called “Reading Program” schools throughout this report, and their results are not generalizable to any other population. There was no counterfactual or control group for these schools.

The majority of the analysis for this population focuses on performance at endline in 2014. However, a subset of Reading Program schools (20 out of 44) were assessed in both 2012 and 2014; with caveats in place about small sample sizes, a brief discussion of change in performance over time for those 20 schools is included as well. The findings of this study are presented in Chapter B of this report.

2 The accessibility criterion specified that the schools had to be located within approximately 20 km of an urban center. For a subdivision to be eligible for inclusion in the sample population for the 2012 study which serves as the baseline for the Accessible PAQUED evaluation, there had to be a minimum of six intervention schools within the subdivision that were within 20 km of an urban center.

Projet d’Amélioration de la Qualité de l’Education (PAQUED) 3

Table ES1 below provides a visual representation of the studies whose results are presented in this report.

Table ES1. Study Populations and Analyses Permitted

Study Treatment

Groups 2010 2012 2014 Evaluation

PAQUED

(Grades 2, 4, 6)

PAQUED

Baseline Endline

Relative growth in performance in PAQUED vs. Control over the period from 2010-2014 Control

Accessible PAQUED (Grades 2, 4)

Accessible PAQUED

Baseline Endline

Relative growth in performance in Accessible PAQUED vs.

Accessible Control over the period from 2010-2014 Accessible Control

Reading Program (Grades 2, 4) Reading Program Snapshot Performance in 2014

Figure ES1, on the next page, provides two visual representations of the school populations assessed during the 2014 endline Early Grade Reading Assessment (EGRA)/Early Grade Mathematics Assessment (EGMA). The top portion of the image includes a map of the educational subdivisions in which schools were assessed, which is color-coded according to the population(s) whose schools were located in that subdivision. The bottom of the image provides scale representation of the relative sizes of the populations being studied and the samples drawn (i.e., the ratio of the area of the light grey Population: 3,000 PAQUED Schools rectangle to the area of the dark grey Population: 618 Accessible PAQUED Schools is 3,000:618, or ~4.85:1.) In the map and visualization, Study A refers to the PAQUED and Control schools; Study B refers to the Accessible PAQUED and Accessible Control schools; and the Reading Program schools are nested within the Study B shape because they were a subset of the Accessible PAQUED schools.

4 Projet d’Amélioration de la Qualité de l’Education (PAQUED)

Figure ES1. Subdivisions Containing Schools Assessed at Endline and Relative Size of the Study Populations

Limitations

The current report presents data and analyses that are subject to a few limitations. Some are due to the differences between the original (2009) vision for the project and the post-realignment (2013) vision;; some are a function of how the DRC educational system’s standards have evolved since 2010, when adaptation of the EGRA instrument was

Projet d’Amélioration de la Qualité de l’Education (PAQUED) 5 conducted; and others are due to the change in sampling design between baseline (2010) and midterm (2012).

Data Collection

Timing of Studies Data for the 2014 Endline EGRA/EGMA was collected from May 1, 2014, through June 9, 2014, by teams of assessors using Tangerine® electronic data collection software.

Teams spent an average of two days per school sampling 13 students per grade in Grades 2, 4, and 6 (the PAQUED schools study) or Grades 2 and 4 (the Accessible PAQUED and Reading Program studies). Teams also conducted interviews with the sampled students’ classroom teachers and the school’s headteacher.

Tools Used to Measure Impact PAQUED’s impact on student learning outcomes was measured using the EGRA and the

EGMA.

The EGRA instrument is a 15- to 20-minute orally administered set of subtasks designed to assess foundational literacy skills crucial to becoming a fluent reader. The EGRA assessment is designed to be a method-independent approach to assessment, in that the instrument does not reflect a particular method of reading instruction (i.e., “whole language” or “phonics-based” approach). Rather, EGRA measures basic skills that a child must possess to eventually be able to read fluently and with comprehension—the ultimate goal of reading. The EGRA subtasks are based on research regarding a comprehensive approach to reading acquisition across languages, including five essential components:

phonemic awareness, phonics, reading fluency, vocabulary, and comprehension.3

Similarly, the EGMA is an orally administered instrument that individually tests children and is designed to assess student performance on foundational mathematics competencies which they are expected to master in the early grades. The instrument includes subtasks which are designed to do the following:

x include key skills that developing country and developed country curricula have determined should be acquired in early grades;

x reflect those skills that are most predictive of future performance, according to available research and scientific advice;

x represent a progression of skills that lead toward proficiency in mathematics; and x target both conceptual understanding and procedural fluency.

3 The definitions are adapted from U.S. Department of Health and Human Services, National Institutes of Health, Eunice Kennedy Shriver National Institute of Child Health and Human Development. (2000).

Report of the National Reading Panel—Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction: Reports of the subgroups (NIH Publication No. 00-4769). Available at http://www.nichd.nih.gov/publications/nrp/smallbook.cfm

6 Projet d’Amélioration de la Qualité de l’Education (PAQUED)

The particular subtasks included in the EGRA and EGMA used in a given context are determined through a consultative adaptation process. Subtasks are selected according to the skills that are of interest to the stakeholders involved in the study, the curricular expectations of the country, and the particular characteristics of the language(s) being assessed.4 Table ES2 describes the subtasks included in the 2014 EGRA/EGMA tools.

Table ES2. Subtask Composition of Assessment Instruments

Instrument Subtasks Grade

2 4 6

EGRA

Vocabulary X X X

Initial Sound Identification X X X

Grapheme Recognition X X X

Familiar Word Reading X X

Invented Word Reading X X

Oral Reading Fluency X X

Reading Comprehension X X

Listening Comprehension X X X

Dictation X X

EGMA

Rote Counting X X X

Rational Counting X X

Number Identification X X X

Quantity Comparison X X X

Missing Number X X X

Word Problems X X X

Calculations

Addition X X X

Subtraction X X X

Multiplication X X

4 At the time of the adaptation of the tools in the DRC, the policy regarding the use of national languages was in question. In consultation with the donor and government counterparts, it was decided to use French for the project interventions and as the language of assessment for the EGRA instruments.

Projet d’Amélioration de la Qualité de l’Education (PAQUED) 7

Instrument Subtasks Grade

2 4 6

Division X X

Findings Despite the complexity of the situation being examined, the findings can be summed up fairly simply: these studies do not permit us to make a positive statement regarding the characteristics of an intervention—whether in terms of its components or its intensity— required in order to realize meaningful improvements in student performance in reading or mathematics.5 However, these studies do allow us to state with confidence that the support provided to Accessible PAQUED and PAQUED schools was not sufficient to generate learning gains as measured by the DRC EGRA and EGMA instruments that are either significant or substantive.

Impact on Student Reading Performance In the study of 3000 PAQUED schools, only two comparisons revealed statistically significant differences: (1) while Grade 4 male students in Control schools in Equateur initially outperformed their female classmates on the Vocabulary subtask in 2010, by 2014 the difference had disappeared, and (2) while Grade 6 male students in PAQUED schools had not outperformed their female classmates on the Familiar Word Reading subtask in 2010, by 2014 they were doing so. (While performance by both boys and girls appears to have decreased over time, girls’ performance decreased much more, thus accounting for the difference.) There were no statistically significant differences in performance between students in PAQUED intervention schools and Control schools, whether cross-sectionally or over time. The intervention did not measurably improve student reading performance between the baseline assessment in 2010 and the endline assessment in 2014. Given that anecdotal sources and qualitative studies, such as the 2012 Midterm Report conducted by School-to-School International6, highlighted the complexities and challenges of operating in the DRC and raised serious questions about whether the intervention was actually implemented with any fidelity to speak of in these schools, this result was not surprising.

5 In 2013-2014 EDC carried out an evaluation of the growth over time in the content knowledge and pedagogical practices of teachers in PAQUED schools. Those teacher data were matched with student performance data collected using different instruments than those used for the RTI EGRA/EGMA studies.

The report on that study suggests that the Reading Program intervention had a significant and positive effect on teacher pedagogical practice as well as some student learning outcomes. Please see Louge, N.

(2014). 2014 final evaluation report teachers’ literacy knowledge, instructional practices, and their students’ reading performance in PAQUED supported schools in the Democratic Republic of Congo.

Education Development Center.

6 Lynd, M. (2012, September). Final report: Mid-term review: Package for improving educational quality (PIEQ) project. Washington, DC: USAID

8 Projet d’Amélioration de la Qualité de l’Education (PAQUED)

The Accessible PAQUED study began in 2012 was based on a revised sampling frame drawn from the population of schools the project estimated would be most likely to have received the intervention’s inputs. As was the case with the PAQUED schools, the study of Accessible PAQUED schools did not reveal any discernible impact on student reading performance. While an absence of robust data on fidelity of implementation cautions against making definitive statements, this suggests that the package of inputs that constituted the original PAQUED intervention was inadequate to the task of improving student reading, and that the lack of results in either of these first two study populations was not simply a matter of poor implementation.

The intensified, focused intervention delivered to the Reading Program schools in response to changing USAID and GDRC priorities certainly bore the hallmarks of the sort of well-designed reading-improvement strategies that have demonstrated success elsewhere, i.e., books in the hands of students, including decodable and leveled texts;

read-aloud books and teacher guides for the instructors; comprehensive training for teachers on effective reading strategies and the use of the materials provided; at least semi-regular coaching and mentoring by someone trained in reading pedagogy; and— crucially—adequate, daily instructional time focused on reading.

While scores on most reading subtasks appeared to improve slightly from 2012 to 2014, relatively few gains were shown to be statistically significant.7 Such statistical confirmation would be particularly valuable because the Vocabulary, Initial Sound Identification, Grapheme Recognition, Familiar Word Reading, Invented Word Reading, Oral Reading Fluency, Reading Comprehension, and Dictation subtasks—that is, every subtask save Listening Comprehension—included at least one instance of student performance in Reading Programs schools that appeared to decline, rather than improve, from 2012 to 2014.

Some evidence emerged from these studies that support for students appears to improve student performance. For instance, for students in Reading Program schools, having a teacher who engaged actively with the intervention (as measured by attendance at school-and cluster-level forums, participating in training, and receiving visits by PAQUED personnel trained in reading) correlated positively and significantly with being a high performer (top quintile of all students in the same grade in the same study) on the Grapheme Recognition subtask. It is encouraging that something over which the school can exert influence could have such an impact. (This is in contrast to other factors that also correlated with being a high performer, such as speaking predominantly French at home or having a family member at home who also knows how to read.)

The following section provides summaries of student performance in reading for each of the populations of interest. The first table in each case summarizes the change in

7 More insight on the outcomes of the intervention in the Reading Program schools can be gleaned from Louge, N. (2014). 2014 Final Evaluation Report: Teachers’ Literacy Knowledge, Instructional Practices, and Their Students’ Reading Performance in PAQUED Supported Schools in the Democratic Republic of Congo. Education Development Center.

Projet d’Amélioration de la Qualité de l’Education (PAQUED) 9 performance over time from the population’s baseline (2010 or 2012) to endline (2014).

The second table compares student performance at endline to national benchmarks8 for the various reading subtasks, and the figure provides a visualization of the same.

Table ES3 provides a summary of student performance in reading for the Reading Program population, disaggregated by Province.

Table ES3. Reading Performance in Subset of Reading Program Schools (n=20), in Mean Scores9

Subtask Group Grade 2 Grade 4

2012 2014 2012 2014

Vocabulary Control - - - -

Treatment 8.21 9.34 * 11.08 12.07

Initial Sound Identification

Control - - - -

Tr…