Attachment_6.I_-_Honduras__Final_Report_1.30.15_RTI_Primary_Schoool_Reading_Study.pdf

EdData II

Task Order 30: Education Data for Decision Making (EdData II):

Primary School Reading Study for Honduras Final Report January 2015

EdData II Technical and Managerial Assistance Contract Number BPA-EHC-E-00-04-00004-00 Task Order: AID-522-BC-14-00011 RTI Project No. 0209354.030.000

This publication was produced for review by the United States Agency for International Development. It was prepared by RTI International.

Task Order 30: Education Data for Decision Making (EdData II):

Primary School Reading Study for Honduras Final Report January 2015

Prepared for Dr. Renán Rápalo USAID/Honduras Education Office Avenida la Paz, Frente a Embajada Americana Tegucigalpa, Honduras rrapalo@usaid.gov

Prepared by RTI International 3040 Cornwallis Road Post Office Box 12194 Research Triangle Park, NC 27709-2194

USA

RTI International is a registered trademark and a trade name of Research Triangle Institute.

The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

mailto:rrapalo@usaid.gov

Honduras EdData: Task Order 30 Final Report iii

Table of Contents Page

List of Annexes ...................................................................................................... v

List of Tables ........................................................................................................ vi

List of Figures ...................................................................................................... vii

Abbreviations ....................................................................................................... ix

Acknowledgements ............................................................................................... x

Executive Summary Desk Study Field Study Design Key Research Findings and Recommendations Effective Teacher Practices Use of Instructional Materials

Teacher Preparation and Support

Resumen ejecutivo Estudio teórico Diseño del estudio de campo Resultados clave de la investigación y recomendaciones Prácticas eficaces de los maestros Uso de materiales de instrucción

Preparación y apoyo para maestros

Project Description Structure of this Report

Desk Study Policy Assessment

Rendimiento Académico EGRA and PIRLS

Materials Review

Field Study Methodology

Sample Instruments Data Collection Data Analysis

Findings Demographics Use of Effective Instructional Strategies Profile of Enfoque Comunicativo iv Honduras EdData: Task Order 30 Final Report

Video Results Teacher Feedback and Assessment

Teachers’ Knowledge and Use of Informal Assessment Teachers’ Knowledge and Use of Differentiation

Instructional Materials Availability and Use of Curricular Materials Profile of Textbook Use Profile of Supplementary Materials Use

Teacher Preparation School and Classroom Environment

Additional Analyses Analysis of Profiles by Grade Analysis of Profiles and Performance

Recommendations and Implications Effective Teacher Practices Teacher Feedback and Assessment Differentiated Instruction Use of Instructional Materials School and Classroom Environment Teacher Preparation Recommendations from the Workshop

Honduras EdData: Task Order 30 Final Report v

List of Annexes

Annex A. Teachers’ Knowledge of the Enfoque Comunicativo ........................ A-1

Annex B. Profile of Differentiated Instruction ................................................... B-1

Annex C. Profile of Textbook Use .................................................................... C-1

Annex D. Profile of Supplementary Materials Availability and Use .................. D-1

Annex E. Demographic Information of the Professors Interviewed at the Escuelas Normales ..................................................................................... E-1

Annex F. Location and Types of Sampled Schools ......................................... F-1

Annex G. Desk Study ....................................................................................... G-1 vi Honduras EdData: Task Order 30 Final Report

List of Tables

Table 1. Summary of Goals with Potential to Directly Impact Literacy Outcomes

Table 2. 2013 Rendimiento Académico Cut Scores

Table 3. School Level Descriptive Statistics

Table 4. Teacher Level Descriptive Statistics, Overall and by CARSI Designation

Table 5. Teacher Level Descriptive Statistics, by School Performance Level

Table 6. Locations and Achievement Status of the Schools Included in the Video Sample

Table 7. Effective Classroom Practices

Table 8. Escuelas Normales Course Work in Teaching Early Literacy

Table 9. Teacher Focused Improvement Activities, by Sector Plan

Table 10. In-Service Training Received in the Last Year

Table 11. Personnel Conducting Support Visits

Table 12. Classroom Environment: Inventory Items

Table 13. Profile Summaries, by Teacher Grade

Table 14. Logistic Regression Estimates—Teacher Profiles and Low School Performance

Table 15. Percent of Top Profile Categories, by School Performance Level

Table A-1. Teachers’ Knowledge of the Enfoque Comunicativo ......................... A-1

Table B-1. Questions for Teachers’ Knowledge of Differentiated Instruction ........................................................................................ B-1

Honduras EdData: Task Order 30 Final Report vii

List of Figures Page

Figure 1. Rendimiento Académico Performance, by Grade and Subtask

Figure 2. Mean Percent of Time Spent on Each Content Area

Figure 3. Mean Percent of Time Spent on Each Content Area, by Level of School Performance

Figure 4. Mean Percent of Time Spent on Each Content Area, by Geographic Region

Figure 5. Average Percent of Time Spent Teaching Vocabulary in Context versus Isolated, by Geographic Area

Figure 6. Mean Percent of Time of Teacher Actions

Figure 7. Mean Percent of Time of Teacher Spends Explaining versus Listening According to Performance Levels

Figure 8. Profile of Teachers’ Use of the Enfoque Comunicativo, by Level of School Performance

Figure 9. Profile of Teachers’ Use of the Enfoque Comunicativo, by Geographic Area

Figure 10. Alignment to Best Practices in Comprehension Instruction

Figure 11. Vocabulary Instruction

Figure 12. Informal Evaluation of Students, by Level of Performance

Figure 13. Informal Evaluation of Students, by Geographic Area

Figure 14. Differentiated Instruction, by Level of Performance

Figure 15. Differentiated Instruction, by Geographic Area

Figure 16. Availability of Materials, by Level of School Performance

Figure 17. Use and Intention to Use Textbooks and Activity Books, by Level of Performance

Figure 18. Use and Availability of Supplemental Materials, by Level of Performance

Figure 19. Use and Availability of Supplemental Materials, by Geographic Area

Figure 20. Teachers’ Highest Level of Education, by School Performance Level

Figure 21. Frequency with Which Teacher Lesson Plans Are Reviewed

Figure 22. Frequency of Support Visits viii Honduras EdData: Task Order 30 Final Report

Figure 23. Major Activities during Support Visits

Figure 24. Classroom Environment, by Level of School Performance

Figure 25. Classroom Environment, by CARSI Designation

Honduras EdData: Task Order 30 Final Report ix

Abbreviations AIR American Institute for Research

CARSI Central America Regional Security Initiative

CIASES Centro de Investigación y Acción Educativa Social/Center for Educational Research and Social Action

CIDEH Consultores para la Investigación y el Desarollo en Honduras

DCNB el Diseño Curricular Nacional Básico

EdData II Education Data for Decision Making II

EGRA Early Grade Reading Assessment

INICE Instituto Nacional de Investigación y Capacitación Educativa

IRT Item Response Theory

MIDEH Mejorando el Impacto de Desempeño Estudiantil de Honduras/ Honduras Improving Student Achievement

MOE Ministry of Education

PIRLS Progress in International Reading Literacy Studies

PROHECO Programa Hondureno de Educacion Comunitaria/Community School Program in Rural Honduras

RQ research question

TPST teacher pre- and in-service training

UPNFM National Pedagogical University Francicso Morazan

UNAH Universidad Nacional Autónoma de Honduras

USAID United States Agency for International Development x Honduras EdData: Task Order 30 Final Report

Acknowledgements This study was carried out under a technical assistance contract awarded to RTI International from the United States Agency for International Development (USAID) under the Education Data for Decision Making (EdData II) project. The team that conducted this study consisted of two core partners: RTI International, and Consultores para la Investigación y el Desarollo en Honduras (CIDEH).

RTI would like to extend our gratitude toward CIDEH, in particular the leadership demonstrated by Dr. Mario Alas Solis, Dr. German Moncada, Aurora Genova Gonzalez, and Bessy Hernández. We recognize the team of assessors and field supervisors who gave their time and energy to collecting data across the country, often in hard-to-access locations. Their smiling faces while observing classrooms made the data collection process much easier for everyone. And, of course, we thank the teachers and children in Honduras, who consented to participate and allow a disruption in their schedules in order to participate in this important study.

Honduras EdData: Task Order 30 Final Report 1

Executive Summary This final report for the USAID/Honduras Primary School Reading study describes the activities, results and recommendations of a two part study that took place from July to December 2014. The purpose of the this study was to gain a better understanding of the current context of literacy instruction in primary schools by reviewing relevant documents such as curriculum materials and by investigating the instructional practices and materials use of teachers in primary schools. The second half of the study took place in the western districts and Central America Regional Security Initiative (CARSI) municipalities.

Desk Study RTI reviewed a variety of documents to better understand the recent policy issues around the improvement of reading outcomes, including reports from international assessments, curriculum documents used to support teachers in planning instruction, and the 2013 versions of the national assessment, the Rendimiento Académico. This review of documents gave insight into recent and current plans for improvement to the education sector in general and literacy instruction specifically, teacher preparation and knowledge, and student knowledge of literacy skills. It also provided a view of the alignment between the curriculum documents, assessment, and results of assessments. Using this information RTI was able to design instruments and plan data analysis for the field study that would further investigate what specific instructional practices were supporting students’ literacy development.

Field Study Design The field study collected data from 43 schools in Grades 2, 3, 5, and 6 in October 2014. These grades were chosen based on data from the Rendimiento Académico, which show that student performance is lowest in Grades 3 and 6. USAID was interested in discovering why scores were lower in these grades than in the preceding grades—thus RTI collected data in Grades 3 and 6 and in Grades 2 and 5. RTI drew a convenient sample of schools from western districts and the CARSI municipalities using the results from the Rendimiento Académico to classify schools as either high, average, or low performing in reading achievement. The study consisted of a teacher interview, a teacher beliefs and knowledge survey, classroom observation, classroom inventory, lesson plan inventory, and student interview. Data collected from these instruments were mostly quantitative with the idea of gaining a general understanding of the classroom environment and teachers’ instructional practices. Qualitative data were collected from 9 of the 143 classrooms to gain an in-depth understanding of the instructional practices that most specifically related to the skills tested on the Rendimiento Académico.

2 Honduras EdData: Task Order 30 Final Report

Key Research Findings and Recommendations The purpose of this report is to provide data on effective teacher practices and school environment and infrastructure related to early grade reading instruction in Honduras.

We organized the report around five key factors:

• Effective teaching practices

• Use of instructional materials

• Teacher evaluation and feedback

• Teacher preparation and support

• School and classroom environment

Below is a list of the most significant findings and recommendations that came out of this research which point to areas that warrant more investment and attention.

Effective Teacher Practices On average, we found a mix of skills in classrooms representing a variety of evidence-based content areas. Much class time was spent reading texts and on comprehension. Less time was spent on writing, vocabulary, and spelling, which are equally important.

• Recommendation 1: More research could be done to better understand how teachers are choosing to spend their instructional time and to help them balance the time across all content areas.

A significant amount of time was spent teaching grammar skills, which research shows is not essential for learning to read successfully. Teachers in higher performing schools in our sample spent the least amount of time on grammar instruction. CARSI school teachers appeared to spend more time on grammar than those in western district schools.

• Recommendation 2: Consider what grammar skills are essential and how much time is truly needed for this type of instruction. Time spent on grammar could be better spent on skills directly impacting reading development, especially in the earlier grades.

At present, decoding only occurred 1.5 percent of classroom reading instruction time.

• Recommendation 3: There should be more focused time on decoding or word study in Grades 2 and 3.

Although most teachers used the enfoque comunicativo (or said they used it and understood it), there was less evidence of teachers’ fully implementing this approach to teaching reading.

• Recommendation 4: More training in how to implement the enfoque comunicativo approach is likely needed. This approach requires teachers to be highly knowledgeable about their practice and their students. It also requires significant resources beyond a reading textbook for students to be able to construct knowledge rather than be given information solely from the teacher.

Honduras EdData: Task Order 30 Final Report 3

Use of Instructional Materials Teachers were more likely to have textbooks and teachers’ guides in their classrooms than the el Diseño Curricular Nacional Básico (DCNB) materials, Programaciones and the Estandares Nacionales. Most teachers reported that they used textbooks to plan their lessons.

• Recommendation 5: Efforts should be made to ensure that teachers have all the materials necessary for teaching reading, not just textbooks and teachers’ guides.

• Recommendation 6: Materials should be updated regularly and all materials should be aligned, including the textbooks. It is not clear from this study if the textbooks are fully aligned with the DCNB, and further research is needed.

• Recommendation 7: Adequate support in learning how to use these materials and incorporate them in daily lesson planning should be given to all teachers.

The National Standards and Pacing Guides include useful information, however they are very vague and repeat the same standards for several grades without explaining how these standards should become more complex across grades. The DCNB provides significantly more nuanced explanations of the specific skills and standards that are expected of students for each grade.

• Recommendation 8: The National Standards and Pacing Guides should be more explicit and precise with regard to expected increases of skills and standards across grades.

• Recommendation 9: More decoding and/or word study instruction should be added throughout the National Standards and Pacing Guides

• Recommendation 10: Due to the fact that the DCNB is now over 10 years old, it would be useful to review this document and ensure it is aligned with the most recent research.

• Recommendation 11: The standards in the DCNB for Grades 3 and 6 are significantly more difficult than in Grades 2 and 5. Due to the relatively poor performance of students in these grades, it is essential that teachers receive more support in planning for instruction to meet the standards.

Teacher Preparation and Support Approximately 60 percent of teachers have university degrees and say they are prepared to teach reading. Despite these high levels of education and preparedness, the results suggest that teachers lack specific skills and knowledge for teaching children how to successfully read.

• Recommendation 12: Increasing the level of education for all teachers, along with professional development on how to teach reading would greatly support the improvement of the quality instruction.

Teachers in high performing schools were more likely to report that their school director reviewed their lesson plans and spent time conducting support visits than teachers in low performing schools.

4 Honduras EdData: Task Order 30 Final Report

• Recommendation 13: School directors should be encouraged to spend more time with teachers in classrooms (including time spent reviewing lesson plans).

There is a lack of regular support received by teachers in the classroom. Teachers are supposed to receive regular pedagogical support visits; however, most teachers interviewed said that these visits either never happened or happened once a year or once every two months.

• Recommendation 14: It would be beneficial to increase the quantity and quality of school support visits. Further research or consideration of how such a system could better support and impact teacher performance is highly recommended.

Honduras EdData: Task Order 30 Final Report 5

Resumen ejecutivo El presente informe final del estudio de la Agencia de los Estados Unidos para el Desarrollo Internacional / Lectura en las Escuelas Primarias de Honduras (USAID / Honduras Primary School Reading) describe las actividades, los resultados y las recomendaciones de un estudio de dos partes, que se llevó a cabo de julio a diciembre de 2014. El objetivo del presente estudio fue obtener un mejor entendimiento del contexto actual de la enseñanza de lectoescritura en las escuelas primarias, mediante la revisión de documentos relevantes, tales como materiales de los planes de estudio, y mediante la investigación de las prácticas de instrucción y los materiales utilizados por los maestros en las escuelas primarias. La segunda mitad del estudio se llevó a cabo en los distritos occidentales y en las municipalidades de la Iniciativa Regional de Seguridad para América Central (CARSI, por sus siglas en inglés).

Estudio teórico RTI analizó una gran variedad de documentos para comprender mejor los asuntos normativos recientes en torno a la mejora de los resultados en el aprendizaje de la lectura, incluyendo informes de evaluaciones internacionales, documentos curriculares utilizados para apoyar a los maestros en la planificación de la instrucción, y las versiones de 2013 de la evaluación nacional, conocida como Rendimiento Académico. Esta revisión de documentos proporcionó una visión de los planes recientes y actuales para las mejoras del sector educativo en general, y la instrucción de la lectoescritura en particular, la preparación y el conocimiento del maestro, y el conocimiento del estudiante de las habilidades de lectoescritura. También proporcionó una visión de la alineación entre los documentos curriculares, la evaluación y los resultados de las evaluaciones. Por medio de esta información, RTI pudo diseñar instrumentos y planear un análisis de datos para el estudio de campo, para investigar más a fondo cuáles son las prácticas de instrucción específicas que apoyan el desarrollo de la lectoescritura de los estudiantes.

Diseño del estudio de campo El estudio de campo recopiló información procedente de 43 escuelas en los grados segundo, tercero, quinto y sexto en octubre de 2014. Estos grados fueron elegidos con base en información de la evaluación de Rendimiento Académico, la cual indica que el rendimiento estudiantil es más bajo en los grados tercero y sexto. USAID estaba interesado en descubrir por qué las puntuaciones en estos grados fueron inferiores a las de los grados precedentes, por lo que RTI recopiló información de los grados tercero y sexto, y los grados segundo y quinto. RTI obtuvo muestras convenientes procedentes de escuelas de los distritos occidentales y las municipalidades de CARSI, utilizando los resultados de la evaluación de Rendimiento Académico, para clasificar a las escuelas como de nivel alto, medio o bajo en el rendimiento en lectura. El estudio consistió en una entrevista con el maestro, un estudio sobre las creencias y los conocimientos del maestro, observación en el aula, 6 Honduras EdData: Task Order 30 Final Report inventario del aula, inventario del plan de lección, y entrevistas con los estudiantes.

La información obtenida de estos instrumentos fue en su mayoría cuantitativa, con la idea de obtener un entendimiento general del ambiente en las aulas y las prácticas de enseñanza de los maestros. Los datos cualitativos fueron obtenidos de 9 de las 143 aulas, para obtener una comprensión a fondo de las prácticas de enseñanza que están relacionadas más específicamente con las competencias que se evalúan en la evaluación de Rendimiento Académico.

Resultados clave de la investigación y recomendaciones El propósito de este informe es proporcionar información sobre las prácticas docentes eficaces, el ambiente escolar y la infraestructura, en relación con la enseñanza de la lectura en los primeros grados en Honduras. Hemos organizado el informe en torno a cinco factores clave:

• Prácticas de enseñanza eficaces

• Uso de materiales de instrucción

• Evaluación y retroalimentación del maestro

• Preparación y apoyo del maestro

• Ambiente escolar y del aula

A continuación se muestra una lista de las conclusiones y recomendaciones más importantes que surgieron de esta investigación, las cuales indican las áreas que requieren más inversión y atención.

Prácticas eficaces de los maestros En promedio, encontramos una mezcla de habilidades en las aulas, representativas de una variedad de áreas de contenido basadas en la evidencia. Gran parte del tiempo de instrucción se dedicó a la lectura y comprensión de textos. Se dedicó menos tiempo a la escritura, el vocabulario y la ortografía, los cuales son igualmente importantes.

• Recomendación 1: Se podría realizar más investigación para comprender mejor cómo los maestros eligen utilizar su tiempo de instrucción, y para ayudarlos a equilibrar el tiempo en todas las áreas de contenido.

Una parte considerable de tiempo se dedicó a la enseñanza de habilidades gramáticas que, de acuerdo a la investigación, no son esenciales para aprender a leer correctamente. Los maestros en las escuelas de mayor rendimiento de la muestra dedicaron la menor cantidad de tiempo a la enseñanza de la gramática. Los maestros de las escuelas de CARSI parecieron pasar más tiempo que los maestros de las escuelas del distrito occidental en la enseñanza de la gramática

• Recomendación 2: Considerar qué habilidades gramaticales son esenciales, y cuánto tiempo es realmente necesario para este tipo de instrucción. El tiempo dedicado a la enseñanza de la gramática podría ser mejor utilizado en la enseñanza de habilidades que impactan directamente el desarrollo de la lectura, especialmente en los primeros grados. En la actualidad, la decodificación sólo se lleva a cabo durante un 1.5 por ciento del tiempo de instrucción de lectura en el aula.

Honduras EdData: Task Order 30 Final Report 7

• Recomendación 3: Se debe dedicar más tiempo a la enseñanza de decodificación o el estudio de palabras en el segundo y tercer grado. Aunque la mayoría de los maestros utilizan el enfoque comunicativo (o dijeron que lo usaban y lo entendían), existen menos evidencias de que los maestros aplican plenamente este enfoque a la enseñanza de la lectura.

• Recomendación 4: Es probable que sea necesaria más formación sobre la implementación del enfoque comunicativo. Este enfoque requiere que los maestros estén muy bien informados sobre su práctica y sus estudiantes.

También es necesario utilizar recursos significativos, más allá de un libro de texto de lectura, para que los estudiantes sean capaces de construir su propio conocimiento, en lugar de solamente recibir información del maestro.

Uso de materiales de instrucción Los maestros eran más propensos a tener en sus aulas libros de texto y guías para maestros, que los materiales del Diseño Curricular Nacional Básico (DCNB), Programaciones y los Estándares Nacionales. La mayoría de los maestros indicaron que utilizan los libros de texto para planificar sus lecciones.

• Recomendación 5: Se deben hacer esfuerzos para asegurar que los maestros cuentan con todos los materiales necesarios para la enseñanza de la lectura, no sólo libros de texto y guías para maestros.

• Recomendación 6: Los materiales deben ser actualizados regularmente, y todos los materiales deben estar alineados, incluyendo los libros de texto. No queda claro a partir de este estudio si los libros de texto están totalmente alineados con el DCNB, y es necesario llevar a cabo más estudios de investigación al respecto.

• Recomendación 7: Los maestros deben recibir el apoyo adecuado para aprender a utilizar estos materiales e incorporarlos en la planificación de las lecciones diarias. Los Estándares Nacionales y las Guías de Ritmo incluyen información muy útil, sin embargo, son muy vagos y repiten los mismos estándares para varios grados, sin explicar cómo estos estándares deberían ser más complejos entre los distintos grados. El DCNB ofrece explicaciones mucho más minuciosas de las competencias y los estándares específicos que se esperan de los estudiantes para cada grado.

• Recomendación 8: Los Estándares Nacionales y las Guías de Ritmo debe ser más explícitos y precisos, en lo que respecta a la mejora prevista de competencias y estándares en todos los grados.

• Recomendación 9: Se debe añadir más tiempo de enseñanza de la decodificación y/o el estudio de palabras a los Estándares Nacionales y las Guías de Ritmo.

• Recomendación 10: Debido a que el DCNB tiene ahora más de 10 años de antigüedad, sería útil revisar este documento y asegurarse de que está alineado con la investigación más reciente.

• Recomendación 11: Los estándares en el DCNB para el tercer y sexto grado son significativamente más difíciles que los del segundo y quinto grado.

Debido al relativamente bajo desempeño de los estudiantes en estos grados, es esencial que los maestros reciban más apoyo en la planificación de la instrucción para cumplir con los estándares.

8 Honduras EdData: Task Order 30 Final Report

Preparación y apoyo para maestros Aproximadamente el 60 por ciento de los maestros tienen títulos universitarios y dicen que están preparados para enseñar a leer. A pesar de estos altos niveles educativos y de preparación, los resultados sugieren que los maestros carecen de las habilidades y los conocimientos específicos para enseñar a los niños a leer correctamente.

• Recomendación 12: El aumento del nivel educativo para todos los maestros, junto con el desarrollo profesional sobre cómo enseñar a leer, apoyaría en gran medida la mejora de la calidad de la enseñanza. Los maestros en escuelas de alto rendimiento fueron más propensos que los maestros en escuelas de bajo rendimiento, a informar que su director de la escuela revisó sus planes de estudio y dedicó tiempo a la realización de visitas.

• Recomendación 13: Los directores de las escuelas deben ser alentados a dedicar más tiempo a los maestros en las aulas (incluyendo el tiempo dedicado a la revisión de planes de estudio). Hay una falta de apoyo regular para los maestro en el aula. Los maestros deben recibir visitas de apoyo pedagógico regulares; no obstante, la mayoría de los maestros entrevistados dijeron que estas visitas o bien nunca ocurrieron, o sucedieron una vez al año o una vez cada dos meses.

• Recomendación 14: Sería beneficioso aumentar la cantidad y calidad de las visitas de apoyo escolar. Es muy recomendable llevar a cabo más estudios de investigación a fondo para determinar cómo este sistema podría apoyar mejor el desempeño docente.

Honduras EdData: Task Order 30 Final Report 9

Project Description As one of the countries with the fewest resources in Latin America, Honduras faces many challenges to implementing and maintaining a strong education system. Despite this the government of Honduras has put education, especially primary education, at the forefront of its priorities. Over the years the United States Agency for International Development (USAID) has supported Honduras in in its educational efforts, most recently through projects such as the Mejorando el Impacto de Desempeño Estudiantil de Honduras/Honduras Improving Student Achievement (MIDEH) project, which has worked with the MOE to create national assessments by grade level and write standards and pacing guides for teachers to use in planning their instruction. In 2014 USAID asked RTI to take stock of the efforts that have been put forth by Honduras to better understand the current context of literacy education in primary school. A two-part study was designed to gain an understanding of past efforts and current instructional practices in primary literacy. The first part was a desk study that reviewed policy and curriculum documents as well as assessment reports from several different literacy assessments conducted in recent years. The second part of the study was a field study that looked specifically at instructional practices in literacy of teachers in Grades 2, 3, 5, and 6. The study results will support the Government of Honduras (GOH) and USAID in considering next steps for further research and reforms. The following is the report of the study and its major findings.

Structure of this Report The purpose of this report is to provide data on effective teacher practices and school environment and infrastructure related to early grade reading instruction in Honduras.

To do this, we report information from classroom observations (both quantitative and qualitative), teacher and student interviews, a classroom observation, and a teacher knowledge measure. The report focuses on these key research questions (RQs):

1. What is the current status of literacy development based on existing assessments in the last three years?

2. How do the national standards and curriculum support literacy development in Grades 1–6?

3. How do instructional practices support literacy development in Grades 1–6?

4. How does the school environment support literacy development in Grades 1–6?

This analysis is guided through the prism of the core research questions that were developed by USAID. This report presents data that address each of the RQs and draws conclusions and recommendations based on those data. The report also identifies additional areas for further research or exploration that could be useful for decision makers.

To get started, high-level results from the desk study conducted in July 2014 are presented that detail the context of early grade reading in Honduras. Following that, 10 Honduras EdData: Task Order 30 Final Report descriptive characteristics of the study are provided. In-depth results are then presented for five key areas related to the research questions:

• Effective teaching practices

• Teacher evaluation and feedback

• Use of materials

• Teacher preparation

• School and classroom environment

Desk Study The desk study conducted by RTI took place in July 2014. The purpose of the study was to better understand the educational context in Honduras; it included a review of current national standards and assessments. This review served as the guiding document for the field study design and data analysis. The main questions answered by the desk study were:

1. What is the current status of literacy development based on existing assessments in the last three years?

2. How do the national standards and curriculum support literacy development in Grades 1–6?

RTI reviewed key documents in order to answer the RQs. The documents fell into the following four categories:

1. USAID strategy documents focused on the global Goal 1 and the Honduras strategy

2. Honduras policy documents that helped to gain understanding of the political context in which decisions and priorities around literacy development exist

3. Assessment reports and databases from various assessments conducted by the Ministry of Education ([MOE] with the support of USAID through MIDEH) in the last three years to elucidate the current context and issues for primary school children developing literacy skills in Honduras

4. Curriculum materials including the standards, curriculum, and formative assessment documents to explain the expectations and approach to reading instruction in Grades 1–6

Using all of these documents RTI was able to analyze the current context, what students are learning in reading, what the teachers know about teaching literacy, and what the curriculum and standards do to guide literacy instruction. Knowledge gained from reviewing each document was synthesized to explain how all these pieces impact the literacy skills that are seen in the assessment reports. The following is a summary of major findings from the desk study. More detailed findings can be found in sections to which they correspond.

Honduras EdData: Task Order 30 Final Report 11

Policy The bulk of the policy review focused on three national education plans and their goals and objectives.

1. MOE’s Plan de Educación 2010–2014 (Education Plan).

2. Fast Track Initiative (FTI) Education for All Honduras (EFA) 2003–2015 (FTI-EFA plan).

3. Plan Estratégico del Sector Educación Período 2005–2015 (Strategic Sector Plan).

The three plans each set forth at least one goal specific to improving academic performance, and several goals across the three plans were set to support this improvement. Table 1 shows a summary of the goals set forth across the three plans that could most directly impact literacy development and that are also associated with the RQs for this final report. The goals in Table 1 show all three plans having at least one goal to improve academic performance, which is the overall goal of all the reforms planned. These will be monitored in most cases using the data from the national assessments. All three plans set some goal to reduce illiteracy rates and increase achievement at Grades 3 and 6 selected because they are the end of each of the cycles of basic education. Coincidentally these grades are also the grades in which student outcomes on the national assessment conducted by the MOE with support of MIDEH seem to be lowest according to the MIDEH report of the national assessment data; they are also grades that can be associated with transitioning to more complicated text and academic work. Only one plan, the Strategic Sector Plan, set a specific goal around materials in classrooms, teacher training, and supervision.

Table 1. Summary of Goals with Potential to Directly Impact Literacy Outcomes

Focus: Student Assessment Materials Infra-structure Instruct-ion Time Teachers

Goal:

Reduce Illiteracy

Percent of Sixth Grade Graduates at Age 12

Increased Academic

Achievement:

Grades 3 and 6

Improved Materials

Distri-buted

Improved Infrastruc-ture 200 Days of School

Building Human Capital

Super-vision and Evaluation of Teachers

Education Plan 2010–

X

X X

FTI-EFA

2003–

X X X

Strategic Sector Plan 2005–

X X X X X X

12 Honduras EdData: Task Order 30 Final Report

These goals and objectives of the national plans became part of the Act for the Establishment of a National Vision and Adoption of a National Plan for Honduras.

This is a law and plan published in 2009 to support the development of Honduras.

There is one main educational goal in the Act that calls for raising the average years of schooling to nine throughout the nation. The plan also calls for the establishment of a system for evaluating the quality of primary schools; promoting the organization of competitions at the regional level in the areas of science, math, and Spanish; and making arrangements with schools and international research experts for the training of teachers and students.

Assessment The assessments reviewed in the desk study cover a range of skills and populations to reveal some insights into the state of literacy among primary students in Honduras.

Overall it is difficult to draw any comparisons, as the assessments each test different skills for different purposes; however, they do give some useful information on specific literacy skills. We first discuss analyses from the Rendimiento Académico, then quickly review results from the Early Grade Reading Assessment (EGRA) and the Progress in International Reading Literacy Studies (PIRLS) assessment.

Rendimiento Académico In 2012 and 2013 a sample of students were given the annual literacy assessment administered as a standardized research study by trained data collectors as part of the USAID-supported MIDEH project. The assessment was the same one given to all other students in the country; however, it was not administered by teachers, thus it was more formal and standardized and could be used to get a more accurate national picture of literacy development in Grades 1 to 9. Although this is a national assessment given to all students at the end of each school year, it is not meant to be a high stakes assessment.

One of the more interesting results reported from the national tests is a drop in academic achievement on the Rendimiento Académico from second to third grade and then again from fifth to sixth grade. This phenomenon has been widely discussed and the trend can be seen clearly in the national, grade-level data from 2013. The MOE’s Rendimiento Académico document from last year shows that the proportion of students designated as satisfactory and advanced in reading was significantly lower for Grades 3 and 6 than it was for Grades 1, 2, 4 and 5.

In order to explain this trend, it is necessary to understand the method used to determine cut scores for the assessment. According to documentation provided by MIDEH/American Institute for Research (AIR) staff, the Angoff method was used to divide student performance scores into four categories: unsatisfactory, needs improvement, satisfactory and advanced. Most simply, this means that experts were asked to evaluate how many test items they believed a student should be required to answer correctly in order to be in a given performance category. These individual cut points were then averaged across all experts. Finally, in order to strengthen the reliability of this approach, expert evaluations were conducted across several rounds.

Honduras EdData: Task Order 30 Final Report 13

These final averages were then used to determine the final cut scores. This is a reliable and well-established method but it does rely heavily on the standards for each category. The most interesting aspect of this is that while the cut scores for satisfactory and advanced performance were relatively consistent across Grades 1, 2, 4 and 5, they were noticeably higher for Grades 3 and 6. In other words, students needed to answer more questions correctly on the assessment in grades 3 and 6 than in the rest of the grades, in order to obtain a satisfactory designation. Therefore, it appears that there are three potential explanations for the Grade 3 and Grade 6 drops in performance when examined via performance levels. First, it could be that the tests for Grades 3 and Grade 6 were significantly easier (thus requiring higher scores in order to perform adequately against the standards). It is clear from the psychometric analyses conducted by AIR that the third and sixth grade assessments were easier relative to the second and fifth grade assessments. However, test equating accounts for the test difficulty across years and therefore precludes this as the explanation for differences in student performance. The second option is that students simply perform worse in Grades 3 and 6 than they do in the other grades. The third option is that the standards are significantly harder for those two grades. This latter option seems plausible given the fact that experts felt as though students needed to answer more questions correctly in order to be in the higher categories. In other words, the standards were set in such a way that students had to do extremely well on the test in order to be deemed satisfactory (relative to how they would have to do in other grades).

In correspondence with AIR/MIDEH staff, we were informed that the question of high standards was routinely discussed during the standard setting process in 2008 and that the panel experts felt strongly that the established cut scores accurately reflected the DCNB standards. The trend of lower performance has continued to be apparent in Grades 3 and 6 in each year since the assessment standards were set. Most recently, the 2013 data show very clearly that while the raw cut scores are higher for Grades 3 and 6, there are greater demands in performance relative to student ability as seen by examining the latent ability cut scores (ascertained through Item Response Theory (IRT) analyses). Table 2 shows that while the raw cut scores for satisfactory performance are higher for third and sixth grade then for second and fifth grade, respectively, there is also a significant difference in the associated student ability (theta) for these cut scores. For example, while a student ability of just below average (-0.09) is associated with satisfactory performance in Grade 5, an ability of .71 would be needed for satisfactory performance in Grade 6.

Table 2. 2013 Rendimiento Académico Cut Scores

RAW

THETA Form 1 Form 2 Form 3

Grade 2 Below Basic 7 7 7 -0.97

Satisfactory 17 16 17 0.16

14 Honduras EdData: Task Order 30 Final Report

RAW

THETA Form 1 Form 2 Form 3

Advanced 28 28 28 2.25

Grade 3

Below Basic 10 9 10 -0.80

Satisfactory 22 21 22 0.75

Advanced 28 28 28 2.40

Grade 5

Below Basic 9 9 9 -1.14

Satisfactory 17 17 17 -0.09

Advanced 25 26 26 1.52

Grade 6

Below Basic 15 16 15 -0.71

Satisfactory 25 25 24 0.71

Advanced 29 29 29 2.48

Ultimately, these higher demands in performance are likely due, in part, to the fact that Grades 3 and 6 are end-of-cycle grades—which bring about more stringent standards. While it is clear that these assessment standards are aligned with the DCNB standards at each grade level, AIR is currently in the process of reviewing and evaluating the vertical alignment of these standards. This could provide further insight into whether or not the increase in standard difficulty in Grades 3 and 6 is justifiably high. Regardless of what they find in these analyses, it seems clear that students in Grades 3 and 6 are not being adequately prepared to meet their grade-level standards.

This may be due to shortcomings in particular content areas or possibly just a result of the fact that teachers are not aligning their practice with the DCNB standards.

In an effort to determine the likelihood of the first hypothesis, we conducted additional analyses in order to examine what types of teacher practices and/or school environment factors might be negatively associated with student and classroom level performance in Grades 3 and 6 (but not in other grades), as well as item level analyses for content area difficulty. These results of these analyses are found below.

Initial analyses were conducted for assessment sub-topics, across all primary grades.

Figure 1 shows that there was no significant drop in scores for third and sixth grade students on the three major reading skills of comprehension, vocabulary, and types of text.

Honduras EdData: Task Order 30 Final Report 15

Figure 1. Rendimiento Académico Performance, by Grade and Subtask

Similar results were found when the analyses were limited to the students in schools sampled for the field study. However, it was still possible that while the overall scores were similar, the relationships between the scores and measures of teacher practice and classroom environment varied by grade. Therefore, correlations were run for student scores and all available measures from the field study instruments, with a focus on teacher practice (from the classroom observation), school infrastructure (from the school inventory and teacher questionnaire), and teacher profiles.

Analyses were run separately for all students as well as by grade. In both the overall and grade-level analyses, the majority of correlations were exceedingly small. As a matter of fact, while very few correlations across any of the measures were above 0.30, the highest was 0.45 (which falls below the conventional standard for strong correlation). This is due in part to the relatively small sample size (particularly in the grade-level analyses) but ultimately points to the fact that we do not have any specific evidence from differences in teacher practice or classroom environments that can provide an explanation for why scores appear to be dropping off for Grades 3 and 6.

Lastly, it was possible to analyze item-level data for each grade by examining the item difficulty parameters obtained from IRT analyses. The purpose of this analysis was to determine if there were certain content areas that tended to be more difficult in Grades 3 and 6, which could subsequently be used as areas of focus for future curriculum/materials development or teacher professional development.

Unfortunately, at the time that this report was written, IRT item analyses were only available for 2012, for Grades 1 to 3. Due to the interest in the drop in scores from Grade 2 to Grade 3, analyses were focused on those two grades. In both grades, the assessment questions were designed to cover three main components: comprehension, vocabulary, and types of text. Comprehension covered four specific standards: 1) understanding global ideas; 2) understanding main ideas; 3) understanding secondary ideas; and 4) understanding inferential ideas. Vocabulary was made up of two

16 Honduras EdData: Task Order 30 Final Report standards: 1) recognizing and understanding familiar words; and 2) interpreting unfamiliar words. Types of text aligned with just a single standard on reading and using text for various purposes. Both the second and third grade tests had a range of questions on each of these standards. Careful analysis of item difficulties showed that there was little difference in content areas that were found to be easier or more difficult across the two grades. For example, the 10 easiest questions in third grade covered the standards of global comprehension, inferential comprehension, main idea comprehension, secondary idea comprehension, familiar word recognition, and interpretation of unfamiliar words—whereas the easiest questions for second graders were split across the exact same categories. As for the most difficult items, the categories remained nearly identical in both grades, with the only real difference being types of text in lieu of familiar word recognition. In other words, the easiest and most difficult items in both Grades 2 and 3 covered a range of content areas and standards, with no discernible difference by grade1.

This finding leads to the conclusion that there is no single content area that must be focused on in order for third grade students to perform as well as their second grade counterparts. Instead, it is necessary for teachers to teach to a higher standard across all content areas or components in Grade 3, in order to more appropriately align their teaching with the DCNB standards and to adequately prepare their students for the national assessment.

EGRA and PIRLS In 2009 EGRA scores showed students in Grades 2 to 4 in rural Programa Hondureno de Educacion Comunitaria/Community School Program in Rural Honduras (PROHECO) schools with very low foundational reading skills, such as letter sound knowledge (average numbers of letter sounds per minute were 8.5 in Grade 2, 11.0 in Grade 3, and 12.3 in Grade 4) and word reading fluency (average numbers of words per minute were 24.8 in Grade 2, 42.1 in Grade 3, and 60.4 in Grade 4), leading to low comprehension scores (percentage of reading comprehension questions answered correctly were 42.8 in Grade 2, 63.8 in Grade 3, and 79.4 in Grade 4). While there are no research-based benchmarks for Honduras, a general rule would be that students should read 65 correct words per minute (cwpm) by the end of Grade 2 depending on the quality of instruction and access to text outside and inside school in a particular country.2 The Catholic Relief Services EGRA data from 2013 show similar results and patterns. This study was also a baseline for a project called Food for Education funded by the United States Department of Agriculture. The study covered 2,218 students in eight municipalities (La Esperanaza, Intibuca, San Francisco de Opalaca, Yamaranguila, San Juan, San Miguelito, Dolores, and San Isidro). One interesting note about the EGRA data is the improvement in word reading fluency both across grades and in both data sets, versus the lack of improvement in letter sound knowledge across grades. It would be expected that students improving in word

1 2013 data for grades 1 to 6 were made available to RTI just prior to report submission. Brief analyses of these data show similar trends to what is discussed for 2012. Difficult questions were spread across content areas for all grades, with no apparent difference between grades 2 and 3 or grades 5 and 6.

2 https://dibels.uoregon.edu/market/assessment/idelgoals/

Honduras EdData: Task Order 30 Final Report 17 reading skills are also improving in letter recognition skills. However these data do not show that to be the case.