Attachment J.28 PAQUED Final Evaluation.pdf

2014 final evaluation report:

Teachers’ literacy knowledge, instructional practices, and their students’ reading performance in PAQUED-supported schools in the Democratic Republic of Congo

September, 2014

Submitted by Education Development Center, Inc.

Agreement #: AID-623-A-09-00010

Executive Summary

The Project d’Amélioration de la Qualité de l’Education (PAQUED), funded by the United States

Agency for International Development (USAID) and led by Education Development Center

(EDC), is a five-year program spanning 2009-2014 that is focused on improving the quality of basic education in 3,000 schools in the Democratic Republic of Congo (DRC). In its initial stages, the project centered around three objectives: improving the quality of teaching and teachers’ mastery of content; improving student mastery of subject content; and improving the school learning environment. The project’s interventions included the introduction of over 600

Interactive Audio Instruction (IAI) programs for reading and math, which mirrored the national curriculum; the training of over 30,000 teachers in French and math content; the production and distribution of cluster-directed professional development modules; the rehabilitation and construction of training centers; the distribution of teaching and learning kits; and the training of 3,000 communities on school governance and on improving the school learning environment. In response to USAID’s new strategy (launched in 2012 with a goal of 100 million children demonstrating improvements in reading by 2015) and to external midterm review findings that revealed project activities to be spread too thinly given the vast terrain and numbers of schools outlined in the initial project design, PAQUED realigned in January 2013 to focus primarily on improving student reading outcomes. Selected components of the PAQUED program (IAI, self-directed training, community support, and kit distribution) were continued, and a robust experimental reading program was introduced in 45 PAQUED schools. This reading program combined intense training, coaching, and the production of teaching and learning materials, as well as community mobilization activities centered on reading.

This report presents the results of a comparative evaluation study that was conducted post-realignment, between March 2013 and May 2014. The study focused on three groups of teachers in Grade 1 to 6: experimental school teachers, IAI-only teachers, and control teachers.

It endeavored to understand how teachers were using the various PAQUED interventions available to them and how their knowledge of teaching reading and their literacy instructional practices may have changed as a result of these interventions. The study also sought to understand whether there was any difference in how Grade 1 and 2 students performed in reading as a result of their teachers’ participation in the interventions and acquisition of literacy knowledge and practice.

In summary, the findings from this study show that experimental teachers’ knowledge of how to teach reading and writing is more closely aligned with sound literacy instruction than their

IAI-only and control counterparts. Experimental teachers’ practice also changed significantly within the year in which they used the reading program. As a result, the performance of the students of these experimental teachers in key reading skills like letter identification and fluency showed dramatic differences in comparison to their control counterparts. Linear regression analysis establishes significant links between teachers’ application of PAQUED interventions and student performance. Specifically, experimental teachers’ IAI usage, their fidelity of implementation of the reading program, and their participation in continuing professional activities and visits from coaches were shown to contribute to changes in teacher practice, teacher knowledge of literacy instruction, and student performance. Many of these findings are supported by the separate PAQUED 2014 Endline Report of EGRA and EGMA produced by Research Triangle Institute (RTI).

Table of contents

EXECUTIVE SUMMARY

FIGURES AND TABLES

ACRONYMS

INTRODUCTION

STUDY PARTICIPANTS, THEORY OF CHANGE, AND RESEARCH QUESTIONS

CHAPTER 1: GRADE 1 AND 2 STUDENT AND TEACHER RESULTS

PAQUED READING PROGRAM DESCRIPTION

GRADE 2 STUDENT READING PERFORMANCE

STUDENT PERFORMANCE RELATIVE TO NATIONAL BENCHMARKS

STUDENT PERFORMANCE AND PAQUED INTERVENTIONS

Student Performance and Teacher Practices

Student Performance and Teacher Knowledge

GRADE 1 AND 2 TEACHERS’ KNOWLEDGE OF LITERACY INSTRUCTION

Phonemic, Phonological and Alphabetic Awareness

Fluency

Vocabulary

Comprehension

Writing

GRADE 1 AND 2 TEACHER PRACTICE RESULTS

Phonemic, Phonological and Alphabetic Awareness

Fluency

Vocabulary

Comprehension

General classroom and literacy practices

CHAPTER 2: GRADE 3 TO 6 TEACHER RESULTS

TEACHER KNOWLEDGE OF LITERACY INSTRUCTION: GRADE 3, 4, 5 & 6 TEACHERS

Phonemic and Phonological Awareness

Fluency

Vocabulary

Comprehension

Writing

GRADE 3 TO 6 TEACHER PRACTICE FINDINGS

Phonemic and Phonological Awareness

Fluency

Vocabulary

Comprehension

General Instructional and Literacy Practices

RECOMMENDATIONS FOR POLICY AND PRACTICE

Training modalities

Materials development:

Community mobilization

Research and evaluation

Institutional Capacity Building:

ANNEX A. METHODOLOGY

Observation (practice) and interview (knowledge) tools:

Reading assessment:

DATA ANALYSIS

STUDY LIMITATIONS:

REFERENCES

Figures and Tables

Table 1. Frequencies of sampled Grade 1 and 2 teachers by province and status at endline

Table 2. Frequencies of sampled Grade 3 and 4 teachers by province and status

Table 3. Frequencies of sampled Grade 5 and 6 teachers by province and status

Table 4. Frequencies of sampled Grade 1 and 2 teachers by sex and status

Table 5. Frequencies of sampled Grade 3 and 4 teachers by sex and status

Table 6. Frequencies of sampled Grade 5 and 6 teachers by sex and status

Table 7. Mean class size disaggregated by sex per Grade 1 and 2 teacher sampled by status

Table 8. Mean number of students disaggregated by sex per Grade 3 and 4 teacher sampled by status

Table 9. Mean number of students disaggregated by sex per Grade 5 and 6 teacher sampled by status

Table 10. Number of schools sampled by sub-division

Table 11. Summary descriptive statistics of Grade 2 student performance in reading sub-tests by status………..17

Figure 1. Percentage of students with zero scores by status

Table 12. Summary descriptive statistics of Grade 2 student performance in reading sub-tests by status omitting zero scores

Table 13. Summary descriptive statistics of Grade 2 student performance in reading sub-tests sub-tests by province and status

Figure 2. Mean scores, by province and status

Figure 3. Students performance in WCPM against national benchmarks set for 3rd Grade

Table 14. Fidelity of implementation data for Grade 1 and 2 teachers by school treatment status

Figure 4. Teachers’ fidelity of implementation (FOI) of the reading program and their students’ mean performance in number of words read correctly

Figure 5. Teachers’ fidelity of implementation (FOI) of the reading program and their students’ WCPM

Figures 6. Experimental teachers’ application of yocabulary practices at endline and their students’ WCPM

Figure 7. Experimental teachers’ application of P4 at endline and their students’ WCPM

Figure 8. Experimental teachers’ total mean knowledge of literacy instruction and their students’ mean WCPM26

Figure 9. Experimental teachers’ total mean knowledge of literacy instruction and their students’ mean WCPM26

Figure 10. Experimental teachers’ responses to Question 3.1 and their students’ mean reading accuracy

Figure 11. Experimental teachers’ total mean knowledge of teaching writing and their students’ mean WCPM 27

Figures 12. Correlations between teachers’ responses to Question 1.3 and student performance

Figure 13. Experimental teachers’ responses to Question 5.2 and theirr students’ mean reading accuracy

Figure 14. Experimental teachers’ responses to Question 5.2 and theirr students’ mean WCPM

Figure 15. Experimental teachers’ fidelity of implementation (FOI) of the reading program and their total mean knowledge of literacy instruction at endline

Table 15. Summary of the Grade 1 and 2 teacher knowledge results of comparison of means at endline between groups (higher percentages convey “sound” knowledge)

Figure 16. Teachers’ mean knowledge of teaching reading by component skill, at endline

Table 16. Item analysis of the Grade 1 and 2 teacher endline knowledge results (means) comparison by groups (percentages reflect agreement)

Figure 17. Experimental teachers’ fidelity of implementation (FOI) of the reading program and their total mean knowledge of teaching fluency, at endline file:///C:/Users/nlouge.EDC_MASS/Desktop/PIEQ/reading%20program/M&E/endline%20reports/findings_write-up_all%20together(SFR)_NL_graphs.docx%23_Toc401654956 file:///C:/Users/nlouge.EDC_MASS/Desktop/PIEQ/reading%20program/M&E/endline%20reports/findings_write-up_all%20together(SFR)_NL_graphs.docx%23_Toc401654958 file:///C:/Users/nlouge.EDC_MASS/Desktop/PIEQ/reading%20program/M&E/endline%20reports/findings_write-up_all%20together(SFR)_NL_graphs.docx%23_Toc401654959 file:///C:/Users/nlouge.EDC_MASS/Desktop/PIEQ/reading%20program/M&E/endline%20reports/findings_write-up_all%20together(SFR)_NL_graphs.docx%23_Toc401654960 file:///C:/Users/nlouge.EDC_MASS/Desktop/PIEQ/reading%20program/M&E/endline%20reports/findings_write-up_all%20together(SFR)_NL_graphs.docx%23_Toc401654961 file:///C:/Users/nlouge.EDC_MASS/Desktop/PIEQ/reading%20program/M&E/endline%20reports/findings_write-up_all%20together(SFR)_NL_graphs.docx%23_Toc401654962 file:///C:/Users/nlouge.EDC_MASS/Desktop/PIEQ/reading%20program/M&E/endline%20reports/findings_write-up_all%20together(SFR)_NL_graphs.docx%23_Toc401654964 file:///C:/Users/nlouge.EDC_MASS/Desktop/PIEQ/reading%20program/M&E/endline%20reports/findings_write-up_all%20together(SFR)_NL_graphs.docx%23_Toc401654965 file:///C:/Users/nlouge.EDC_MASS/Desktop/PIEQ/reading%20program/M&E/endline%20reports/findings_write-up_all%20together(SFR)_NL_graphs.docx%23_Toc401654966 file:///C:/Users/nlouge.EDC_MASS/Desktop/PIEQ/reading%20program/M&E/endline%20reports/findings_write-up_all%20together(SFR)_NL_graphs.docx%23_Toc401654971

Figure 18. Experimental teachers’ fidelity of implementation (FOI) of the reading program and their responses to Question 2.1, at endline

Table 17. Summary of the Grade 1 and 2 teacher practice results of comparison of means between the baseline and the endline disaggregated by status

Table 18. Item analysis of the Grade 1 and 2 teacher practice results of comparison of means between the baseline and the endline disaggregated by status

Figure 19. Grade 1 and 2 teachers’ change in literacy instructional practices from baseline to endline

Table 19. Summary results of linear regression for the Grade 1 and 2 teacher change of instructional practices using adherence to teachers participation in CPD and IAI dosage as predictors

Figure 20. Percent of variance in experimental teachers’ instructional practices explained by their IAI listenership rates 44

Figure 21. Percent of variance in experimental teachers’ instructional practices explained by their IAI listenership rates 45

Figure 22. IAI-only teachers’ IAI usage and their gains in Practice 4

Figure 23. IAI-only teachers’ IAI usage and their gains in Practice 22

Figure 24. Experimental teachers’ IAI usage and their gains in fluency-building practices

Figure 25. Experimental teachers’ IAI usage and their gains in Practice 9

Figure 26. Experimental teachers’ IAI usage and their gains in Practice 10

Figures 27. IAI-only teachers’ IAI usage and their gains in instructional practice

Figure 28. Experimental teachers’ IAI usage and their gains in vocabulary building practices

Figure 29. Experimental teachers’ IAI usage and their gains in comprehension building practices

Figure 30. Experimental teachers’participation in CPD and their gains in Practice 20

Figure 31. Experimental teachers' IAI usage and their gains in Practice 21B

Table 20. Summary of the Grade 3 and 4 teacher knowledge results of comparison of means at endline

Figure 32. Item analysis of the Grade 3 and 4 teacher knowledge results of comparison of means at endline disaggregated by status

Table 21. Summary of the Grade 5 and 6 teacher knowledge results of comparison of means at endline disaggregated by status

Figure 33. Item analysis of the Grade 5 and 6 teacher knowledge results of comparison of means at endline disaggregated by status

Table 22. Summary of the Grade 3 and 4 teacher practice results of comparison of means between the baseline and the endline disaggregated by status

Table 23. Item analysis of the Grade 3 and 4 teacher practice results of comparison of means between the baseline and the endline disaggregated by status

Figure 34. Grade 3 and 4 teachers’ change in literacy instructional practices from baseline to endline

Table 24. Summary of the Grade 5 and 6 teacher practice results of comparison of means between the baseline and the endline disaggregated by status

Table 25. Item analysis of the Grade 5 and 6 teacher practice results of comparison of means between the baseline and the endline disaggregated by status

Figure 35. Grade 5 and 6 teachers’ change in literacy instructional practices from baseline to endline

Figure 36. Percent of variance in experimental teachers’ instructional practices explained by their IAI listenership rates 68

Table 26. Item analysis results of linear regression for the Grade 5 and 6 teacher observation of instructional practices using IAI dosage as a predictor

Figure 37. Experimental teachers’ IAI usage and their gains in Practice 1

Figure 38. Experimental teachers’ IAI usage and their gains in general literacy practices

Acronyms

CPD Continuing Professional Development

CRS Catholic Relief Services

CTB Cooperation Technique Belge (Belgian Technical Cooperation)

CWPM Correct Words per Minute

EDC Education Development Center

EGRA Early Grade Reading Assessment

IAI Interactive Audio Instruction

MEPSP Ministère de L’Enseignement Primaire, Secondaire et Professionel

PAQUED Projet d’Amelioration de la Qualité de L’Education

RTI Research Triangle Institute

USAID United States Agency for International Development

Introduction

In 2014, the DRC Ministry of Education (Ministère de L’Enseignement Primaire, Secondaire et

Professionel, MEPSP) launched new policy initiatives intended to improve the quality of literacy teaching and learning. In February 2012, the National Reading Commission, established by the Ministry of

Education, proposed new performance standards for reading and writing for all six primary school

Grades, in French and in national languages. The new performance standards were part of a “roadmap”

(feuille de route) of key tasks to develop, implement and effectively monitor progress toward achieving the goal of improving the state of literacy education in the country. The reading roadmap includes the development of a new reading curriculum and pedagogical tools to support implementation.

The five-year USAID-funded Projet d’Amelioration de la Qualité de L’Education (PAQUED) project aligned with these national initiatives by developing and implementing a research-based reading instructional approach for Grades 1 and 2 in selected project schools. The experimental reading program aimed to provide a platform for testing key inputs from the roadmap. These inputs include the content standards, an evidence based instructional sequence, and text-leveling criteria and guidelines, which the Reading

Commission has developed as part of the proposed new national reading curriculum. The training approach designed for the experimental program also provided multiple opportunities for teachers to learn and reflect upon the new approaches.

PIEQ began the process of developing and testing the new program by identifying 45 experimental schools in the three provinces where the project operates. 16 schools were identified in Bandundu, 16 in

Equateur, and 13 in Orientale. Grade 1 and 2 teachers in these schools benefited from ongoing training, coaching, a detailed daily lesson structure and accompanying activity guide, and leveled reading materials. The leveled reading materials, developed for both classroom and student use, drew upon themes and content within the official DRC language curricula (both for national languages and for

French). These texts were developed according to provisional benchmarks and text-leveling criteria developed and adopted by the National Reading Commission, which was established late in 2012 by the

Ministry of Education. Classroom activities and strategies outlined in these materials mirror the

Ministry-validated student learning standards. Experimental school teachers also continued to benefit from PAQUED’s Interactive Audio Instruction (IAI) and other project inputs (e.g., video training modules).

618 additional PAQUED project schools (referred to in this report as the IAI-only schools) benefited from

IAI; project training on IAI use and on French and math content knowledge; self-directed learning modules for professional development; occasional visits from a PAQUED project team member; and materials like student kits, classroom materials (chalk, rulers, math kits), mp3 radios, and teacher guides, but they did not receive the other supports associated with the reading program. The remaining 2,382

PAQUED project schools were provided with IAI programs, training, and kit materials but were less likely to receive visits from a PAQUED agent, as they were largely inaccessible due to distance, security, and limited transport options. These 2,382 schools were not included in the study described in this document, since the project’s realigned focus had shifted to the experimental and IAI-only schools.

A comprehensive study was undertaken to identify how teachers in the experimental and IAI-only schools evolved over the course of the program in terms of their classroom practices, knowledge about literacy instruction, dispositions (i.e., attitudes) toward literacy and literacy instruction, and changes in student performance. Initial findings show marked improvement in teachers’ knowledge and skills over baseline, as well as improved student performance on letter-sound, vocabulary, and fluency measures.

Data collected via individual interviews, classroom observations, and focus group interviews of Grade 1 and 2 teachers show improvements in their knowledge and practice for developing a range of student skills, including letter-sound knowledge, decoding/encoding, vocabulary, fluency and comprehension.

These results suggest that the reading program, including the integrated use of IAI instruction, positively impacted teacher knowledge and practice in support of the development of students’ literacy skills.

This report presents the results of the study and highlights key elements of the reading program intervention that are believed to have contributed to those results. First, we describe the study sample and the theory of change on which this study was based. Thereafter, the discussion is divided into two sections: the first focusing on the results for Grade 1 and 2 students and teachers, who were the primary targets of the reading program, and the second focusing on Grade 3 to 6 teachers, whose exposure to the program came through cluster training with Grade 1 and 2 peers and IAI literacy materials for

Grades 3-6.

Chapter 1 (on Grades 1 and 2 of the intervention) is presented into two parts: The first part presents student reading performance data and the possible linkages to students’ exposure to PAQUED intervention and the changes in their teachers’ practice and knowledge. The second part dives deeper into the findings related to teacher knowledge of how to teach reading and writing and changes in teachers’ literacy instructional practices from baseline to endline. These results are linked to the various

PAQUED interventions made available to teachers. Chapter 2 explores Grade 3 to 6 teachers’ knowledge of reading and writing and their changes in practice from baseline to endline, linking these results to

PAQUED interventions.

The report concludes with a discussion of recommendations and lessons learned for future projects and policies derived from a data and results workshop attended by the DRC Ministry’s National Reading

Commission in August 2014.

Study Participants, Theory of Change, and Research Questions

This study examined 3 distinct groups of teachers:

teachers who benefited from PAQUED’s intensive daily reading program (experimental school teachers) plus Interactive Audio Instruction (IAI); teachers who benefited only from PAQUED’s IAI programming (this group represents 98% of PAQUED intervention schools); and teachers who did not benefit from the

PAQUED program at all (control school teachers). The study was designed as a matched pair study (see

Annex A) to permit both longitudinal and cross-sectional analysis. Teachers within each school were selected randomly from the PAQUED teacher database at baseline in February 2012.

At baseline, the study participant sample size was pre-determined based on a matched-pair design using a one tail, .5 effect size (=.025, =.8) providing the following breakdown of teachers to be surveyed and observed:

Taking into account general attrition, teacher mobility across Grade-levels, and subsequent replacement teachers selected to participate in the study, the distributions changed over the course of endline and baseline. The tables and figures below provide an overview of our sampled population of teachers disaggregated by Grade level, status, and province, and their average classroom sizes. The distribution of teachers by province and status were fairly even, with the exception of Orientale, where IAI-only teachers represent a greater percentage of the sample across Grade levels.

TEACHERS Experimental IAI-only Control

Grade 1-2 35 35 35 Grade 3-4 35 35 35

Grade 5-6 35 35 35 total 105 105 105 315

TEACHER SAMPLE BY PROVINCE AND GRADE

In terms of gender differences in teachers sampled, it is interesting to note the drop-off of female teachers in Grade 5 and 6 for IAI-only and control schools. This is consistent with the DRC-based stereotype male teacher for older students. For experimental schools though, this trend wasn’t as pronounced.

TEACHER SAMPLE BY SEX AND GRADE

TABLE 1. FREQUENCIES OF SAMPLED GRADE 1 AND 2

TEACHERS BY PROVINCE AND STATUS AT ENDLINE

Status Province N

Control Bandundu 34

Equateur 38

Orientale 53

Experimental Bandundu 29

Equateur 30

Orientale 25

IAI-only Bandundu 30

Equateur 43

Orientale 56

TABLE 2. FREQUENCIES OF SAMPLED GRADE 3 AND 4 TEACHERS BY

PROVINCE AND STATUS

Status Province N

Control Bandundu 42

Equateur 30

Orientale 36

Experimental Bandundu 30

Equateur 30

Orientale 28

IAI-only Bandundu 33

Equateur 46

Orientale 62

TABLE 3. FREQUENCIES OF SAMPLED GRADE 5 AND 6 TEACHERS BY

PROVINCE AND STATUS

Status Province N

Control Bandundu 39

Equateur 32

Orientale 26

Experimental Bandundu 26

Equateur 27

Orientale 23

IAI-only Bandundu 29

Equateur 29

Orientale 53

TABLE 4. FREQUENCIES OF SAMPLED GRADE 1

AND 2 TEACHERS BY SEX AND STATUS

Status Sex N

Control F 36

M 53

Experimental F 54

M 15

IAI-only F 66

M 32

TABLE 5. FREQUENCIES OF SAMPLED GRADE 3

AND 4 TEACHERS BY SEX AND STATUS

Status Sex N

Control F 21

M 53

Experimental F 29

M 38

IAI-only F 59

M 46

TABLE 6. FREQUENCIES OF SAMPLED GRADE 5

AND 6 TEACHERS BY SEX AND STATUS

Status Sex N

Control F 8

M 53

Experimental F 18

M 21

IAI-only F 21

M 46

In addition to teacher demographics, it is also important to consider teachers’ mean class sizes because larger class sizes are often correlated with teacher and student performance. Interestingly, the sample revealed slightly bigger mean class sizes in experimental schools than in IRI-only and control schools.

However, this does not mean that experimental schools necessarily had higher enrollment rates. This measure was captured at the beginning of every classroom observation, when the enumerator would draw a map of the class and count the number of boys and girls. Therefore, this suggests that student attendance may be better in experimental schools than in IAI-only and control schools.

MEAN CLASS SIZE BY STATUS AND GENDER

Grade 1 to 6 teachers were sampled from the 3 PAQUED intervention provinces. Within these provinces, data was collected from randomly selected schools in the sub-divisions as follows:

TABLE 10. NUMBER OF SCHOOLS SAMPLED BY SUB-DIVISION

Bandundu Orientale Equateur

Kikwit (N=13) Kisangani (N=17) Mbandaka (N=14)

Bandundu-ville (N=5) Bunia (N=9) Boende (N=5)

Gungu(N=5) Isiro (N=4) Gemena (N=4)

Masi-Manimba (N=5) Zongo (N=3)

Kenge (N=4) Gbadolite (N=5)

Approximately half of the sample was drawn from RTI’s midline evaluation schools in order to permit triangulation of results between student performance and teacher practice and knowledge. The remaining half of the sample was selected based on school cluster divisions; that is, if an experimental school was selected in the RTI midline sample, those schools that were already designated as

“clustered” with those schools were also selected to be experimental. This is consistent with PAQUED’s

Objective 2 theory of change, which posits:

TABLE 7. MEAN CLASS SIZE DISAGGREGATED BY SEX PER GRADE 1

AND 2 TEACHER SAMPLED BY STATUS

Status Sex Mean Total mean

Control Girls 14 30

Boys 16

Experimental Girls 19 37

Boys 18

IAI-only Girls 17 35

Boys 18

TABLE 8. MEAN NUMBER OF STUDENTS DISAGGREGATED BY SEX

PER GRADE 3 AND 4 TEACHER SAMPLED BY STATUS

Status Sex Mean Total mean

Control Girls 14 27

Boys 13

Experimental Girls 25 44

Boys 19

IAI-only Girls 19 39

Boys 20

TABLE 9. MEAN NUMBER OF STUDENTS DISAGGREGATED BY SEX

PER GRADE 5 AND 6 TEACHER SAMPLED BY STATUS

Status Sex Mean Total Mean

Control Girls 12 26

Boys 14

Experimental Girls 25 40

Boys 15

IAI-only Girls 16 34

Boys 18

This theory of change is based upon teacher use and application of PAQUED tools and resources. More specifically, under “quality of teaching improved,” we explored what type of approach would foster this improved quality of teaching, and how to discern the potential of this approach for sustained and internalized improvement in teaching beyond the length of the PAQUED program, as shown in the graphic below:

In order to confirm or to disconfirm this theory of change, this study explored the following specific questions:

1. How are Grade 2 students in control and experimental schools performing in reading at the end of school year 2013/14?

2. How are teachers applying the PAQUED interventions available to them?

3. How do teachers’ classroom practices link to their students’ performance in reading?*

4. How do teachers’ understanding of effective reading instruction link to their students’ performance in reading?*

5. How do teachers’ use of the PAQUED interventions link to their students’ performance in reading?*

6. How did teachers’ classroom practices change over 1.5 school years AND are these changes linked to their use of the PAQUED interventions available to them?

7. What do teachers understand about effective reading instruction AND is this knowledge linked to their use of the PAQUED interventions available to them?

*This question is limited to Grade 2 students and teachers only

The first five questions are addressed in the first section on Grade 2 student reading performance. The following two questions are addressed in the sections on teachers’ practices and teachers’ knowledge.

In chapter 2 (on Grade 3 to 6 teachers), only questions 2, 6, and 7 are answered, as the study did not collect reading performance data for Grades 3 to 6 students.

Improving quality of teaching in reading

If we give teachers an explicit reading program to follow in their classrooms

AND

We give them opportunities for learning and reflection (including collective reflection/exchange with their peers)

THEN

Teachers will gain an understanding of literacy learning needs and processes among their students

AND will appropriately apply instructional techniques and strategies in the classroom.

AND will change their dispositions vis a vis reading and writing instruction

Quality of teaching improved in reading

Increase in the number of students in DRC with improved reading skills

Chapter 1: Grade 1 and 2 Student and Teacher Results

PAQUED Reading Program Description

The reading program was designed to provide stakeholders (most importantly, the Ministry) with a robust model for transforming teachers’ instructional practices and knowledge of how to teach reading;

thereby improving student performance. The program included the following components:

Government-validated standards and benchmarks from which all materials were designed

Comprehensive face-to-face teacher training on effective reading strategies and on the use of instructional materials

Teacher activity guide keyed to curriculum, teaching strategies, and materials, presented in simple language (French) that was accessible to the teachers

Teacher read-aloud books (1/week per class)

Decodable /leveled texts (1/week per class)

30-minute IAI lessons focused on developing reading skills (1/week per class)

Monthly in-class coaching/mentoring by a coach trained in reading

Adequate, dedicated daily teaching time focused on reading

Teacher-led weekly meetings in a school-based learning circle focused on reading

Teacher-led monthly meetings with peers in a multiple-school learning circle focused on reading

Community support/participation through reading clubs or Espace Communautaire d’Eveil en

Lecture (ECEL).

The training was designed to launch within the PAQUED project’s final year of operations. Therefore, it aimed to quickly respond to Grade 1 and 2 teachers’ needs for knowledge and skills development in three areas: subject matter (literacy), pedagogy (the teaching and learning process) and instructional practice in reading and writing. The scope and depth of teachers’ identified needs at baseline presented a challenge: how could the program develop teachers’ knowledge and skills quickly, following a comprehensive, research-based pedagogical approach, while at the same time supporting rapid improvement of learners’ skills in reading and writing? PAQUED addressed this problem by developing a series of structured classroom teaching and learning activities which repeated themselves weekly, to help teachers master strategies and continue to practice them. These activities shared a basic lesson structure, beginning with the development of learners’ knowledge of letters and sounds and how to apply this knowledge to decode and encode new words. The program facilitated rapid teacher mastery of instructional strategies by repeating instructional activities several times during the week. This approach aimed to promote the development of teachers’ understanding of literacy learning needs and processes among early Grade learners, on the one hand, and their ability to effectively apply appropriate instructional techniques and strategies, on the other hand. This practice-based approach designed to foster teacher change via ongoing application and reflection1 was vital to the program’s success.

The training component of the program entailed two training workshops, regular mentoring support and teacher learning (i.e., discussion) forums/learning circles. In an initial weeklong face-to-face training, participating teachers learned the basic steps to correctly execute literacy lesson activities via lesson demonstrations and group discussion. The project then reinforced teachers’ skill development through regular mentoring or “coaching” classroom visits and teacher-led discussion forums. At the beginning of every week, teachers also participated in peer-to-peer coaching and lesson preparation, to further enhance their capacity to correctly execute activities and apply techniques and strategies for developing learners’ skills. A second five-day face to face training workshop was given midway through the year to help teachers better understand, improve on, and add to the activities they had become comfortable implementing. Overall, these trainings and ongoing teacher support contributed to teachers’ motivation and confidence in implementing the structured program in their classrooms and provided them with forums for sharing their students’ progress and continuing difficulties. The section that follows presents results of Grade 2 student performance after benefiting from one year of the reading program intervention.

Grade 2 Student Reading Performance

Although student reading performance was captured in RTI’s PAQUED 2014 Endline of EGRA and EGMA performance, the Grade 2 reading data largely focused on pre-reading skill testing, skills that were chosen by a Ministry committee at test adaption in 2009. In order to capture more advanced reading skills targeted in the Grade 1 and

2 PAQUED reading program, a short reading assessment tool was developed by EDC to measure fluency (accuracy and automaticity) and alphabetic awareness. The sub-tests employed were letter identification, high frequency words, and connected-text subtests adapted from existing EGRA tools from Mali. Students tested were randomly selected from the classrooms of teachers who participated in the study (see sampling and methodology in Annex

A). This was to offer the opportunity to triangulate teacher practice, knowledge, and fidelity of implementation with student performance results. Only a limited number of IAI-only students were tested in this study, which explains their omission from this section of the discussion and analysis. RTI’s more extensive PAQUED 2014 Endline of EGRA and EGMA performance also shows positive trends in different reading sub-tests for Grade 2 experimental school students. For example, Grade 2 experimental school students progressed significantly in their identification of graphemes from baseline to endline. Below is a summary of student results disaggregated by status (experimental and control) and disaggregated by province and status.

1 The approach is based upon Schon’s (1987) “knowledge-in-action”, in which teachers develop the knowledge and skills for effective reading and writing instruction while applying research-based instructional strategies in the classroom.

TABLE 11. SUMMARY DESCRIPTIVE STATISTICS OF GRADE 2 STUDENT PERFORMANCE IN READING SUB-TESTS BY STATUS

Sub-task Status Mean SD p-value Cohen’s D Effect size

Number of letters read (out of 26)

Experimental (N=169)

20.96 5.4 .000 -2.11 0.73

Control (N = 82)

10.2 6.71

Number of high frequency words read (out of 8)

Experimental (N=169)

4.39 2.63 .000 -1.46 0.59

Control (N = 82)

1.21 2.005

Number of words read in a text (out of 26)

Experimental (N=169)

11.24 9.25 .000 -1.27 0.54

Control (N = 82)

2.22 5.014

Words Correctly read Per Minute

Experimental (N=169)

9.8 13.73 .000 -1.03 0.46

Control (N = 82)

1.22 3.69

The table above reveals that Grade 2 students in experimental schools performed significantly better than their control and counterparts in all sub-tasks (p=.000) at endline in May 2014. The graph below illustrates the differences in zero scores across subtests, that is, students who could not identify or read a single letter or word. For number of letters read, all Grade 2 experimental students were able to identify at least one or more letters whereas 1.2% of control students were not able to identify a single letter. For high frequency word reading, only 7.7% of Grade 2 experimental students were unable to read a single word out of eight whereas 53.7% of control students were unable to do so. In terms of percent accuracy in reading a connected text, only 17.2% of Grade 2 experimental students were unable to read a single word out of eight whereas almost 59% of control students could not read one word.

FIGURE 1. PERCENTAGE OF STUDENTS WITH ZERO SCORES BY STATUS

C o n tr o l

C o n tr o l

C o n tr o l

C o n tr o l

E x p e ri m e n ta l

E x p e ri m e n ta l

E x p e ri m e n ta l

E x p e ri m e n ta l

0%

20%

40%

60%

80%

100%

Number of letters read (out of 26)

Number of high frequency words read (out of 8)

Number of words read in a text (out of

26)

Words Correctly read Per Minute

1.2%

0% 7.7%

58.5% 53.7%

17.2%

55%

15.2%

Given the large numbers of zero scores, it is useful to look at results omitting non-reader’s scores in order to capture a realistic view of reader’s performance. The table below summarizes results omitting the zero scores. Overall, omitting these scores drives up mean scores slightly in each subtest with the exception of numbers of letters read for experimental school students for whom none had zero scores in that subtest. Despite omitting zero scores, experimental students still significantly outperformed their control counterparts across subtests (p=.000).

TABLE 12. SUMMARY DESCRIPTIVE STATISTICS OF GRADE 2 STUDENT PERFORMANCE IN READING SUB-TESTS BY STATUS, OMITTING ZERO SCORES

Sub-task Status Mean SD p-value Cohen’s D Effect size

Number of letters read (out of 26)

Experimental (N=169)

20.96 5.4 .000 -1.71 0.71

Control (N = 81)

10.32 6.65

Number of high frequency words read (out of 8)

Experimental (N=156)

4.75 2.4 .000 -1.36 0.56

Control (N = 38)

2.61 2.25

Number of words read in a text (out of 26)

Experimental (N=140)

13.57 8.45 .000 -1.55 0.61

Control (N = 34)

5.35 6.66

Words Correctly read Per Minute

Experimental (N=117)

11.56 14.22 .000 -.99 0.44

Control (N = 27)

2.72 5.17

Student performance by province

Grade 2 experimental school student performance varied significantly from province to province. The table below shows a summary of scores across all subtests for each province.

TABLE 13. SUMMARY DESCRIPTIVE STATISTICS OF GRADE 2 STUDENT PERFORMANCE IN READING SUB-TESTS

BY PROVINCE AND STATUS

Province Sub-task Status Mean SD p-value Cohen’s D Effect size

BANDUNDU

(N= 107)

Number of letters read (out of 26)

Experimental (N=81)

19.26 6.23 .000 -1.69 0.65*

Control (N=26)

11.77 5.631

Number of high frequency words read (out of 8)

Experimental (N=81)

3.9 2.9 .000 -1.29 0.54*

Control (N=26)

1.31 2.478

Number of words read in a text (out of 26)

Experimental (N=81)

7.67 8.6 .002 -0.78 0.36

Control (N=26)

3.04 5.67

Words Correctly read Per Minute

Experimental (N=78)

4.75 8.6 .001 -0.75 0.35

Control (N=17)

1.04 1.5

EQUATEUR

(N=55)

Number of letters read (out of 26)

Experimental (N=32)

20.69 .403 .000 -2.51 0.78*

Control (N=23)

9.61 6.31

Number of high frequency words read (out of 8)

Experimental (N=32)

4.31 1.91 .000 -1.97 0.7*

Control (N=23)

1.26 1.3

Number of words read in a text (out of 26)

Experimental (N=32)

12.28 7.78 .000 -2.7 0.8**

Control (N=23)

1.04 1.64

Words Correctly read Per Minute

Experimental (N=19)

5.77 3.51 .000 -3.14 0.84**

Control (N=18)

0.33 0.35

ORIENTALE

(N=89)

Number of letters read (out of 26)

Experimental (N=56)

23.57 3.42 .000 -3.2 0.84**

Control (N=33)

9.36 7.67

Number of high frequency words read (out of 8)

Experimental (N=56)

5.14 2.44 .000 -1.91 0.68*

Control (N=33)

1.09 2.07

Number of words read in a text (out of 26)

Experimental (N=56)

15.82 8.85 .000 -1.85 0.67*

Control (N=33)

2.39 5.9

Words Correctly read Per Minute

Experimental (N=41)

21.27 17.47 .000 -1.82 0.67*

Control (N=25)

1.99 5.54

*effect size is medium **effect size is large

For the letter reading and high frequency words, sub-test variation between provinces remained stable.

However, for connected text reading, experimental school students in Orientale significantly outperformed not only their control counterparts in that province but also their experimental counterparts in Bandundu and Equateur in their percent accuracy and in their number of words read per minute (p=.000). This can be explained by several factors that were found to positively and significantly correlate with student results. These factors include teachers’ fidelity of implementation of the reading program and teachers’ knowledge and classroom practices. These relationships will be discussed further below.

FIGURE 2. MEAN SCORES, BY PROVINCE AND STATUS

*performance for Orientale experimental schools for WCPM is p=.000

Student Performance Relative to National Benchmarks

The DRC government set provisional benchmarks for different reading competencies in February 2012 for both national languages and French. Because students in Grade 1 and 2 are intended to learn to read in national languages, no benchmarks were set for reading fluency in French for Grade 2. However, benchmarks were set for Grade 3. Figure 3 below shows the proportion of Grade 2 experimental and control schools who are below the Grade 3 benchmark, at benchmark, and above the benchmark. 12% of Grade 2 experimental students read above fluency benchmark for French set for Grade 3, 9% read at benchmark and 78% read below the benchmark. In contrast to this, only 2% of Grade 2 control students showed to read at benchmark for fluency and 98% read below benchmark.

10.52

14.56 16.41

6.5 5.78

21.27

6.6

7.9

0.5

7.1

N u m b er o f w o rd s

Experimental

Control

Number of words read correctly in a connected text (out of 26)

Number of words read correctly per minute

FIGURE 3. GRADE 2 STUDENTS’ PERFORMANCE IN WCPM AGAINST NATIONAL BENCHMARKS SET FOR 3RD GRADE

Student Performance and PAQUED Interventions

As mentioned earlier, analysis revealed student performance to be positively and significantly linked to several factors to teachers’ fidelity of implementation of the reading program, teachers’ knowledge and teachers’ classroom practices. The data specifically revealed that student performance was significantly linked to two factors: rates of IAI listenership and their teachers’ fidelity of application of the reading program. Below is an outline of the degree to which teachers applied or participated in certain PAQUED interventions available to them.

TABLE 14. FIDELITY OF IMPLEMENTATION DATA FOR GRADE 1 AND 2 TEACHERS BY SCHOOL TREATMENT STATUS Experimental

N= 69 IAI-only N= 96

Fidelity of implementation of reading program rate

Participation in CPD2 IAI listenership IAI listenership

Mean St. Dev. Mean St. Dev. Mean St. Dev. Mean St. Dev.

.88 .13 .71 .15 .86 .067 .51 .28

Overall, experimental teachers used and/or followed the various elements of the PAQUED intervention as designed. Despite this, experimental teachers’ employment of only one of these interventions correlated significantly with student performance: fidelity of implementation of the reading program.

Linear regression showed that 21% of the variation in mean student performance in connected text

CPD=Continuing professional development. This is a composite score including rates of teacher participation in school-based and cluster based meetings available to them and number of monthly coaching visits from facilitators or PAQUED staff.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Students reading <15

WCPM

Students reading 16 to

29 WCPM

Students reading above

29 WCPM

Experimental

Control

21% of students at the end of 2nd Grade in experimental schools attained or surpased the WCPM benchmarks set for French reading in 3rd Grade compared to 2% of students in control schools.

78%

98%

9%

2%

12%

0% reading correlated positively and significantly (p=.016, d=1.01, ES=0.45) with their teachers’ fidelity of implementation of the reading program.

In addition to this, 23.8% of the variation in student’s mean words correct per minute can be explained by teachers’ following the reading program the way it was designed (p=.021, d=1.09, ES=0.479).

This correlation points to the importance of teachers following a program as it is designed. On average, teachers in experimental schools followed 88% of the reading program activities as they were devised.

The rate was higher for Orientale and Equateur provinces, where teachers applied more than 90% of the reading program appropriately. However, due to low numbers of teachers as matched with students

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0 5 10 15 20 25 M e an p e rc en ta ge o f re ad in g p ro gr am F

O I

Students' mean number of words read correctly

FIGURE 4. TEACHER’S FIDELITY OF IMPLEMENTATION OF THE READING PROGRAM (FOI)

AND THEIR STUDENTS' MEAN PERFORMANCE IN NUMBER OF WORDS READ CORRECTLY

R2=.21 sig=.016

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0 10 20 30 40 M ea n p er ce n ta ge o f re ad in g p ro gr am F

O I

Students' mean WCPM

FIGURE 5. TEACHER’S FIDELITY OF IMPLEMENTATION OF READING PROGRAM (FOI) AND

THEIR STUDENTS' MEAN WCPM

R2=.238 sig=.021 tested, correlations of fidelity of implementation of the reading program and mean student performance cannot be presented by province.

Other PAQUED interventions such as IAI listenership were assumed to have an impact on student performance in reading. This is because the programs were broadcast directly into the classroom and were designed to engage the students just as much as they were to provide continuous training to the teachers. However, linear regression analysis found no significant correlation between IAI usage and student performance. A possible explanation for this lack of significant correlation is twofold: a lack of data on student absenteeism and time on task (students may not have been receiving a sufficient dosage of the programming), and the very low exposure to IAI programming that was called for in Grade

1 and 2. Student absenteeism is a major issue in the DRC because attendance is directly linked to the student’s payment of fees (exam fees, enrollment fees, etc). When students are not able to pay these fees, they are barred from attending school. Therefore, even if teachers are present to listen to the IAI, it is not certain that every student benefited equally. Secondly, in Grade 1 and 2, only one 30-minute program was provided for reading per week. On average, teachers in experimental schools used more of the IAI programs available to them than their IAI-only counterparts, and variation of usage was also much lower for experimental school teachers. This is likely due to IAI’s integration in the reading program’s weekly activity calendar. Still, given the low dosage of IAI programming per week, it is little surprise that IAI listenership was not significantly correlated with student reading performance.

However, because this sample size is fairly small, conclusions are difficult to draw. The 2014 PAQUED

EGRA/EGMA report produced by RTI, a larger scale study, establishes the relationship between student performance on the grapheme recognition subtask and PAQUED interventions. This report showed that teacher participation in continuing professional development (CPD) activities was found to have a substantial impact on student performance in this sub-task (p = 0.0387). This correlation is consistent with teachers’ assertions in focus groups, which revealed that they believed IAI to be a useful tool for oral language development and engaging students in numerous pre-reading activities like stretching out words to hear individual sounds, cutting up words by syllable, etc. Taken together, these data suggest that IAI, when used regularly and in the context of a robust reading program, can bear positive results and provide sound models of teaching reading. This conclusion is consistent with conclusions drawn by the Ministry Reading Commission’s analyses of data collected from various sources (EDC, RTI, and

Mukendi, 2014).

Student Performance and Teacher Practices

In addition to teachers’ application of PAQUED interventions, we were also interested in better understanding how PAQUED teachers’ practice and their knowledge of teaching reading links to student reading performance. According to PAQUED’s theory of change, improvement in teacher classroom practices concerning literacy will influence student performance. Research suggests that teachers’ explicit modeling and instruction of the component skills of reading and writing will benefit students’ reading acquisition. In the reading program, activities often involved a combination of reading and writing to develop skills like phonological and alphabetic awareness, fluency, vocabulary building and comprehension. However, linear regression analysis indicated that only teachers’ application of vocabulary activities was strongly correlated with students’ reading performance on certain sub-tests.

The graph below shows that 25.1% of the variance in students’ mean fluency (WPCM) can be explained by a teachers’ application of vocabulary activities in the classroom (p=0.021, d=1.13, ES= 0.49).

While vocabulary activities may not seem directly linked to improving reading fluency, the specific types of vocabulary activities that correlated significantly with student performance expose students to text, thereby providing opportunities for students to develop familiarity with sight word vocabulary. For example, a teacher’s application of pre-reading activities like making predictions and discussing illustrations and new vocabulary embedded within a text explained 20.1% of the variance in student’s fluency (WCPM) (p=.021, d=0.978, ES=0.44). Such activities inevitably engage students with the reading of new words, which can be linked to developing decoding skills necessary for building fluency.

Student performance in reading a connected text was also significantly correlated with their teacher’s application of engaging students in correcting their spelling. For example, the graph below demonstrates that experimental teachers’ engaging in the practice of asking students to correct their invented spellings can explain 30.9% of the variation in students’ mean WCPM (p=.009, d=1.3, ES=0.55). In the reading program, students are asked encode words that contain a phonics pattern studied that week.

This is to help them apply their knowledge of letter-sound relationships. When teachers ask students to correct their spellings, students are helped to reinforce these letter-sound relationships, which will help them decode words that contain those same patterns.

0 10 20 30 40

M ea n p er ce n ta ge o f ti m e al lo ca te d t o vo ca b u la ry a ct iv it ie s

Students' mean WCPM

FIGURE 6. EXPERIMENTAL TEACHERS' APPLICATION OF VOCABULARY PRACTICES AT ENDLINE

AND THEIR STUDENTS' MEAN WCPM

R2=.251 sig=.021

It had been expected that analysis would reveal more significant links between teacher practice and mean student performance.