ATTACHMENT_J.5_EGR_Benchmarking_Workshop_MALAWI.pdf

Proposing Benchmarks for Early Grade Reading in Malawi

Background National early grade reading performance in

Chichewa has been assessed annually in Malawi since 2010. The data from these surveys, summarized in the table below, show very low levels of reading skill development among students in primary standards 1 through 3, but with some improvement.

*denotes significant differences in means from 2010 to 2012

The Early Grade Reading Assessments (EGRA) in

2010 and 2012 were conducted during the first term of the 2010-11 and 2012-13 school year in standards 2 and 4, so the results are really indicative of the learning levels achieved by students who had completed standard 1 and standard 3 in the previous school years.

Even basic skills, such as recognizing letters, were very under-developed, with students who had completed standard 1 only correctly naming about

6 letters per minute. While those who had completed through standard 3 did much better, on average naming about 27 letters per minute, their level of performance was well below what would be considered automatic for such a fundamental skill.

Regarding reading fluency and comprehension, the skills that indicate whether children can actually read and understand simple text, on average students who had completed standard 1 were essentially not reading. Those who had completed through standard 3 were well below what is needed for independent reading at just over 15 words per minute of oral reading fluency (ORF).

What pulls down the averages across all the skill areas were the high percentages of students scoring zero (no items correct). Even with the slight improvement from 2010 to 2012, the vast majority of students at the start of standard 2 were scoring zero – over 70% on syllables, over 80% on familiar and non-words, and over 90% on ORF and comprehension. There were many fewer zero scores among students who had completed through standard 3. In fact most of the improvement seen in the average scores resulted from reductions in the percentage of students scoring zero, as can be seen in the chart below.

While the low levels of performance revealed by the national surveys are disappointing, a reading program introduced in Standard 1 through the

Malawi Teacher Professional Development Support program (MTPDS) demonstrated that reading skills

Standard 2 Standard 4 2010 2012 2010 2012

Letter name knowledge (correct letters/min) 2.3 5.7* 21.3 26.7*

Syllable reading (correct syllables/min) 1.4 3.2* 18.9 26.3*

Familiar word reading (correct words per min) 0.8 1.9* 11.3 16.9*

Non-word reading (correct non-words/min) 0.5 1.2* 7.7 11.4*

Oral reading fluency (correct words/min of text) 0.8 1.3* 11.3 15.4*

Reading comprehension (out of 5 questions) 0.0 0.0 0.4 0.7*

0%

20%

40%

60%

80%

100% Standard 4 % of students scoring zero

2010 2011 2012 can be improved. Comparison of baseline and end line data from the MTPDS reading pilot (introduced in two districts) show how much students skills increased following a year of improved instruction.

The chart below shows that foundational skills of students in pilot schools improved dramatically when compared to those in control schools.

While the MTPDS pilot achieved impressive improvements, the overall level of performance in all skill areas was still below what would be considered a true demonstration of the desired skills. And if children who have completed standard 1 are on average only achieving oral reading fluency of 7 words per minute, they are still a long way from being able to easily make their way through text.

This begs the question, “What is an acceptable level of reading achievement for Malawian students in the early grades of primary school?” In 2012, the

EGRA Coordinating Committee in Malawi attempted to answer this question by defining benchmarks for different areas of reading skill. The committee discussed what they thought were reasonable expectations for each of the skill areas and attempted to project improvement in the percentages of students meeting benchmarks in standards 1 and 3 for the period 2012 to 2017.

With additional assistance from the USAID’s

Education Data for Decision Making Project

(EdData) reading benchmarks were revisited, this time applying a rigorous methodology to the available EGRA results to define data-based benchmarks. Technical experts from across the education sector participated in this exercise during a workshop at the Malawi Institute of Education on the 12th and 13th of November, 2014.

National Reading Strategy In addition to the accomplishments of the MTPDS pilot, there are two other reasons to be optimistic regarding the future of early grade reading performance in Malawi. First, the reading intervention piloted under MTPDS in two districts is now being implemented in 11 districts and is progressively being extended to cover standards 2 and 3 as well as standard 1. Second, in the interest of supporting the eventual national implementation of this kind of effort to improve reading, the Ministry of Education, Science and

Technology (MoEST) has developed and officially approved a National Reading Strategy (NRS).

The NRS emphasizes that learning to read with understanding is perhaps the most critical skill children can acquire; it is necessary for all other academic learning. And it stresses the importance of the five components of reading (phonemic awareness, phonics, fluency, vocabulary, and comprehension), adding two more areas of emphasis – oral language and writing. The National

Reading Strategy is seen as an important step to ensuring that the outcomes of the Education Sector

Implementation Plan (ESIP II) for 2013-2017 are realized. The NRS also lays out the following requirements to achieve improved reading:

Ensuring that sufficient time is allocated to the teaching of reading and that class sizes are reduced;

Defining standards that explicitly state what children will learn by the end of each grade level in each language:

Aligning curriculum with such standards:

Developing materials that support student learning and the teaching of reading;

Letters Syllables Words Non-words ORF

Control Baseline Control Endline

Treatment Baseline Treatment Endline

Enacting an assessment plan that informs the reform effort;

Ensuring teacher preparation and continuous professional development that builds teachers’ knowledge and skills to teach and assess reading;

Involvement of families and communities to foster a culture of reading in and out of school.

Regarding the second bullet point above, the

National Reading Strategy further calls for setting standards that are “clearly defined, quantifiable, anchored in research and that prioritize the main components of reading.” The benchmarking workshop was designed to develop benchmarks that can meet these requirements, and in addition, ensure that benchmarks and objectives would be grounded in the available data on Malawian learners’ reading performance in Chichewa.

THE BENCHMARK SETTING WORKSHOP

A two day workshop on November 12 and 13, 2014 brought together 26 education stakeholders, including representatives of the Department for Inspection and Advisory Services, The Department of Basic Education, the Department for Teacher Education and Development, the Malawi Institute for Education, Domasi College of Education, Teacher Training Colleges, and the Centre for Education, Research and Training. Staff from the USAID Malawi Early Grade Reading Activity also participated as did representatives of USAID/Malawi.

The objectives of the workshop were to:

Orient and engage a cross section of Malawian education stakeholders in a participatory process for setting reading benchmarks in Chichewa for grades 1, 2 and 3.

Expose education sector specialists to a process that could be used to set reading benchmarks for English, as well as other learning outcome benchmarks in other subjects.

The first half day of the workshop allowed all participants to review the existing data – including the results of the national EGRA surveys from 2010, 2011 and 2012, as well as the baseline and end line EGRA results from the MTPDS pilot (comparing intervention and control schools).

Data from the US, from other OECD countries and from Tanzania and Ghana were shared to enable participants to see what are considered acceptable levels of reading performance in other settings.

Following the presentation, participants were engaged in a discussion of benchmarks – what they are and how to set them by combining empirical data, the curriculum expectations expressed in the National Reading Strategy, analysis of the relationships between different areas of reading skill development, and the participants working knowledge of Malawi’s education sector.

A process for reviewing the available data and deriving from that information proposed benchmarks was mapped out, and the participants were arranged into three groups. Each group took on the challenge of analyzing the available information, discussing and debating what seemed possible, and then defining an initial set of benchmarks for each skill area in Standard 3.

Those results were shared and discussed and consensus was negotiated, prior to moving on to setting benchmarks for the other standards.

THE READING SUBTASKS

The workshop facilitator suggested that participants focus on setting benchmarks for a select set of skill areas based on what the Malawian data showed were the strongest predictors of reading success.

After discussion participants agreed on the following skill areas assessed by EGRA for which benchmarks would be set:

Reading comprehension (the number or percent correct out of five questions).

Oral reading fluency (ORF) (the number of words of text correctly read per minute).

These two tasks were addressed first because there is a strong relationship between oral reading fluency and comprehension and because these two skills together represent what is intuitively understood to mean that a child is able to read.

Familiar word reading (number of familiar words correctly read per minute).

Familiar word reading indicates the extent to which students are building their vocabularies and their ability to automatically recognize common words, allowing them to devote effort to decoding unfamiliar words they may encounter in a text. This skill area was included because familiar word reading correlates strongly in Malawi with ORF.

Syllable reading (number of common syllables correctly read per minute).

Syllable reading was deemed an important skill to include in the benchmarks because it demonstrates children’s understanding of grapheme-phoneme correspondence as well as their ability to decode.

Since syllables are important units of meaning in

Chichewa, learning to recognize and read syllables is an important skill for reading with understanding.

Again, this skill was included because it also correlates strongly with ORF.

Letter naming (number of letters correctly named per minute).

Some participants advocated for including letter naming as another skill area in which to set benchmarks. Because this skill has the weakest relationship with a child’s eventual ability to read fluently and with comprehension, some argued against including it. The group did agree that it is an important initial skill, perhaps only with relevance for standard 1. Benchmarks were not set for letter naming, but this report includes recommendations for a standard 1 letter naming benchmark based on analysis of the available data.

THE BENCHMARKING RESULTS

For each subtask, the working groups defined the following three things:

The benchmark value for the indicator for that subtask, The percentage of students that would be meeting that benchmark in five years, and

The percentage of students who would be scoring zero on that indicator in five years.

As mentioned above, the groups began working in standard 3 (using the EGRA data from students assessed at the beginning of standard 4) on

1 Participants raised the point that test results from the beginning of standards 2 and 4 probably are lowered by the fact that students could well have lost some of their skill during the long vacation between the end of one school year reading comprehension and ORF, developing benchmarks for the two together, since different levels of comprehension align with different levels of ORF. Participants made use of box and whisker plots and tables showing the range of oral reading fluency scores that align with each level of reading comprehension to complete this portion of the benchmarking process.

Once a benchmark was selected, the groups identified the current percentage of students reaching that level of skill (using cumulative distribution tables) as a basis for discussing what they thought could be the percentage meeting the benchmark in five years.

Lastly, each group looked at the trend over the three year period (2010-2012) in the percentage of students scoring zero in these skill areas as a basis for setting the percentage scoring zero that could be achieved in five years.

Once benchmarks and objectives were set for reading comprehension and ORF in standard 3, the groups worked in the same way on standard 1.

Assessment data from the surveys done at the beginning of standard 2 were used as indicative1 of what students were capable of after having completed standard 1. When working on standard

1 benchmarks and objectives, in addition to the national data, participants took into account the

MTPDS results.

With standard 3 and 1 complete for reading fluency and comprehension, participants filled in standard 2 benchmarks and objectives in a way that made sense both as a logical progression and based on their knowledge of how students develop reading skill.

The process was then repeated for the next skill area, and so on.

The benchmarks and objectives are presented in the tables on the next page. The tables are followed by justification for the workshop, participants’ recommendations, including and the start of the next. Therefore, it was agreed that one could assume a slightly higher level of performance for standards 1 and 3 than that which was reflected in the available data.

discussion of the main assumptions, considerations and debates that were taken into account.

Reading Comprehension Std 3 Std 2 Std 1

Average in 2012 14% -- 6%

Recommended benchmark 80% 80% 60%

% of students meeting benchmark (2012)

6% -- 7%*

Recommended objective:

% at benchmark in 5 years

50% 40% 35%

% of students scoring zero (2012)

51% -- 94%

Recommended objective:

% of zero scores in 5 years

10% 20% 30%

* This value is from MTPDS treatment schools end line.

For comprehension, participants stressed the importance of students progressing towards being independent readers by the time they finish standard 3. This implies full comprehension, which it was agreed among all participants corresponds to a score of 80% of better. A lower benchmark for standard 1 was seen as a way to accommodate that fact that some learners may still be struggling to achieve the levels of fluency one would associate with fuller comprehension. But increased emphasis on building those skills in standard 2 would mean that such students should meet the

80% comprehension benchmark.

Oral Reading Fluency Std 3 Std 2 Std 1

Average in 2012 15.4 -- 1.3

Recommended benchmark 50 40 30

6% -- 12%*

Recommended objective:

50% 50% 40%

(2012)

39% -- 89%

Recommended objective:

% of zero scores in 5 years

5% 10% 20%

* This value is from MTPDS treatment schools end line

2 See footnote 1.

The benchmark for ORF was justified by the data indicating that students achieving 80% or higher comprehension were for the most part reading between 43 and 51 correct words per minute

(cwpm). The group chose a value towards the upper end of that range, since the available data may underestimate performance slightly.2

Familiar Word Reading Std 3 Std 2 Std 1

Average in 2012 16.9 -- 1.9

Recommended benchmark 45 40 30

10% -- 2%

Recommended objective:

50% 50% 40%

29% -- 80%

Recommended objective:

% of zero scores in 5 years

5% 10% 20%

Participants agreed that word reading should track closely with ORF, with students achieving comparable levels for each in standards 1 and 2.

Hence the word reading and ORF benchmarks are the same in those two grades. However, they are different for standard 3, as less emphasis in the curriculum is placed on word reading per se, and more emphasis is given to developing greater levels of fluency in reading text, the benchmark for familiar word reading should be less than the one for ORF. The continuation of this trend should be taken into account when any work is done to consider benchmarks for standard 4.

For the three benchmarks discussed above – comprehension, ORF and word reading – participants felt that to get 50% of students to meet the benchmark by standard 3 was an important objective to establish. They felt that a target of

50% would be in keeping with the ESIP II commitment to have 50% of students meeting the learning outcomes goals established in that plan.

Across the board (including for syllable reading discussed below), the reduction of zero scores was seen as a critical component of how average reading performance will be improved. During the workshop the importance of moving students from the low end of the performance distribution into the middle ranges is one way to assure greater equity.3

Syllable Reading Std 3 Std 2 Std 1

Average in 2012 26.3 -- 3.2

Recommended benchmark 65 60 50

12% -- 1%

Recommended objective:

60% 55% 50%

20% -- 70%

Recommended objective:

% of zero scores in 5 years

5% 10% 15%

The groups felt strongly that syllable reading is an important skill to develop in students given the structure of Chichewa, wherein syllables are added or changed to change the meaning of words. And syllable reading is both more in line with teaching practice and a more reliable measure of phonemic awareness than say letter sounds in isolation

(which is harder to assess) or initial sound identification. The increase in the benchmark from

50 to 60 derives from continued skill development in reading increasingly complex syllables. A less significant increase (from 60 to 65) between standards 2 and 3 was justified in the same way as for other skills – emphasis in standard 3 shifting more to developing fluency in reading text.

As mentioned earlier, the groups did not have time, nor did they agree on the need, to develop benchmarks for letter naming. However, based on the 2012 MTPDS baseline and end line results for letter naming, it is possible to recommend a benchmark for standard 1. Students who were reaching oral reading fluency of 30 cwpm in treatment schools at end line, were naming 50 correct letters per minute (clpm). It would seem

3 In contrast to say increasing the proportion of students in the highest levels of performance, but leaving many still at the low end.

that a reasonable standard for the benchmark for standard 1 should be at least 50, or even better, 60 clpm (the equivalent of naming one letter per second). In 2012, 11% of students in MTPDS treatment schools were achieving 60 clpm or more.

Five years from now all children should be finishing standard 1 able to name that many letters per minute. An objective of 90% meeting the benchmark of 60 clpm is therefore recommended.

IMPORTANT ASSUMPTIONS

All of the benchmarks are well above what the available data indicate are the current average levels of performance of Malawian students. And the objectives, both in terms of the percentage of students meeting those benchmarks and reduction of zero scores are also ambitious. How high to set benchmarks and how ambitious (or cautious) to be in recommending objectives were often the basis of vigorous debate during the workshop. During that debate, participants were able to identify the underlying assumptions and current factors that guided their thinking. These are summarized here.

Implementation of the National Reading

Strategy. Given that the NRS had just been officially adopted, the group felt that there was cause to be optimistic about the MoEST commitment to implementing a reading improvement effort on a national scale. And the fact that a model, the Early Grade Reading

Activity, is already in operation in 11 districts, led participants to assume this activity (or something like it) would expand to cover the whole country.

Extended time in school and for reading. A priority component of the NRS is a policy adding an additional hour to the school day for standards 1 through 4, which would be used to increase the amount of instructional time devoted to reading. Instructional time being one of the strongest determinants of reading achievement, this policy bodes well for improved outcomes in the future.

Continued support from MoEST’s partners.

Participants were realistic in their recognition that Malawi struggles to have sufficient resources to meet the needs associated with improving education. However, important financial and technical partners appear to remain committed to assisting the education sector. Furthermore, the group felt that the approval of the National Reading Strategy provides MoEST the means to rally continued support.

Ability of the education system to support schools. The group felt that one can foresee improvement in learning outcomes under the assumption that MoEST will be able to adequately manage the provision of support to schools. This will need to include Primary

Education Advisors being able to visit schools and support teachers, the Department of

Inspection and Advisory services being able to monitor school performance, and MoEST being able to provide the resources for this.

Benchmarks need to be based on “grade-level” work. Up to now students in standards

1 through 4 have been assessed using materials at a single level, the equivalent of standard 2. Participants set benchmarks under the assumption that expectations for students should be in relation to material and content appropriate to each standard – i.e., standard 1 students should be assessed using standard 1 level items. This is partly how the groups justified setting higher benchmarks for standard

1 than may otherwise have been indicated by the available data.

As students move from standard 1 to standard 3, progress should be uneven.

When considering the sequence of benchmarks for standards 1 – 3 in each skill area, participants recognized that foundational skills will be emphasized in the first two years, and de-emphasized (except for students who need additional time/help) in standard 3. Conversely, higher order skills such as fluency and comprehension will be emphasized more as students move towards standard 3. For this reasons the increases in the benchmarks and/or in the objectives for syllable and word reading occur primarily between standards 1 and 2, while the increases for higher order skills for the most part come between standards 2 and 3.

Considering inclusion of special needs students. Participants engaged in an interesting discussion regarding how to take into account increased inclusion of special needs students in school. It was agreed that the same benchmarks should hold for all students, especially as MoEST pursues its policy of inclusive education. But in the near term, it may be unrealistic to assume that the education system will be able to meet the needs of all disadvantaged learners. Therefore the groups agreed that in all cases the percentages of students scoring zero would never be completely eliminated. Some 5 to 10 percent of learners would continue to struggle, and that among those would likely be students with disadvantages that the education system will still not be able to adequately address.

NEXT STEPS

This report documents the work of the 24 technical experts and education sector stakeholders who contributed to defining benchmarks for Chichewa reading. Participants agreed that they would review this report, verify its accuracy and provide feedback to ensure its completeness. Once the report has been finalized, it was agreed that DIAS should take forward the recommendations of the group to MoEST leadership, with the intention of having these benchmarks officially adopted as part of the National Reading Strategy.

In addition, the report will be shared with

USAID/Malawi so that they can use the benchmarks to evaluate the impact of Malawi

EGRA and other activities they are funding in support of improving reading.

By defining specific benchmarks and objectives for the percent of children reaching those benchmarks, it is hoped that all stakeholders in Malawi will be able to monitor the improved reading performance that should result from the implementation of the

National Reading Strategy.

Proposing Benchmarks for Early Grade Reading in Malawi

Benchmarking Workshop Participant List

1. Kevin Roberts, USAID

2. Ramsey Sosola, USAID

3. William Susuwele Banda, Malawi Institute of Education

4. Margaret Chilimanjira, Malawi Institute of Education

5. Henri Chilora, Malawi Institute of Education

6. Samson Khondiwa, Malawi Institute of Education

7. Miriam Katundulu, Malawi Institute of Education

8. Ezekiel Kachisa, Malawi Institute of Education

9. Joseph Chimombo, Department of Basic Education

10. Lizinet Daka, Department of Basic Education

11. Chrissie Soko, Department of Inspection and Advisory Services

12. Robert Nyambi, Department of Inspection and Advisory Services

13. Jeremiah Kamkuza, Department of Inspection and Advisory Services

14. Michael Lwanda, Department of Inspection and Advisory Services

15. Dorothy Matiti-Yassin, Department of Inspection and Advisory Services

16. Alfred Kamoto, Department of Teacher Education and Development

17. Edith Mmela, Domasi College of Education

18. Grace Chiuye, Center for Education Research and Training

19. Donna Namaona, Kasungu Teacher Training College

20. Nicholas Kalinde, Machinga Teacher Training College

21. Paula Green, Early Grade Reading Activity

22. Charles Gunsaru, Early Grade Reading Activity

23. Odala Banda, Early Grade Reading Activity

24. Mike Nkhoma, Early Grade Reading Activity

25. Sydonio Matope, Early Grade Reading Activity

26. Stephen Backman, Early Grade Reading Activity