ATTACHMENT_5_-_EDUCATION_PROJECT_PERFORMANCE_EVALUATION_-_2013.pdf

Projet AIDE Performance Evaluation 2009-2013

EVALUATION REPORT

FINAL

Prepared by

LTL Strategies

Evaluation Team Iain McLellan, BA (Team Leader)

Ilmi Awaleh Elabeh, MS

May 2013

LTL Strategies 4545 42nd Street NW - Suite 306 – Washington, DC 20016

Tel (202) 362-6800, Fax (202) 362-6881, www.ltlstrategies.com

4545 42nd Street, NW - Suite 306 - Washington, D.C. 20016

Tel: (202) 362 6800 - Fax: (202) 362 6881

Web site: www.ltlstrategies.com

TABLE OF CONTENTS

Executive Summary………………………………………………5

Introduction………………………………………………………11

Key Findings……………………………………………………...19

Lessons Learned…………………………………………………..34

Recommendations………………………………………………...36

Annex

LIST OF ACRONYMS

AFD French Development Agency

AGSP Ambassador’s Girls’ Scholarship Program

AIDE International Assistance for Education Development

ANEFIP National Agency for Employment Data, Training, and Job Placement

(Ministry of Labor)

CGE School Management Committee

COP Chief of Party

CFFB Balbala Women’s Training Center

CFPA Adult Center for Professional Training

CFPEN Center for Initial and In-Service Teacher-Training (Ministry of Education)

PC Pedagogical Counselor

CRIPEN Book Design and Publishing Center (MENFOP)

CRP Center for Pedagogical Resources

DPI Department of Planning and Information at MENFOP

ED* Assist FHI360’s master program and software, powering MENFOP’s databases

EGRA Early Grade Reading Assessment

EMIS Education Management Information System

EQUIP Education Quality Improvement Program

ESL English as a Second Language

FASE School Report Card (physical status of the school)

GORD Government of the Republic of Djibouti

GPE Education Partners’ Group

IFESH International Foundation for Education and Self-Help

MEDA Methods for Teaching Pupils with Difficulties

MENESUP Ministry of National Education and of Tertiary Education

MENFOP Ministry of National Education and Professional Training

NFQE Fundamental Quality of Education Levels

PASP Projet AIDE Scholarship Program

PTA Parent-Teacher Association

UNICEF United Nations Children’s Fund

USAID United States Agency for International Development

USG United States Government

Part of the Ministry of Education. Closed in May 2011.

2 Part of the Ministry of Education. Opened in May 2011.

Acknowledgements

The evaluation team would like to thank the USAID/Djibouti staff for its collaboration and guidance during this process. Particular thanks go to Saada Abdillahi, Education Team Leader, and Brian Frantz, General Development Officer, for their well thought-out scope of work, which set clear objectives and questions to be explored in the evaluation.

Randall Martin, Chief of Party of Projet AIDE, and his staff were very helpful managing the logistics for the field visit part of the evaluation, and were able to set up a full slate of stakeholders at nine schools on short notice. The project staff was instrumental in ensuring all the interviews, site visits, and focus group discussions were scheduled within the tight time frame.

Finally, the Secretary General of MENFOP, Mahdi Mahamoud, enabled the access to Ministry staff for the evaluation team, and also provided very useful insights and guidance to the team on two occasions.

USAID/Djibouti Education Project (AIDE)

Performance Evaluation

i. Executive Summary

Djibouti background. A country of limited natural resources, the economy of the Republic of

Djibouti is centered on the services sector, which accounts for 80% of GDP and 80% of employment. The relatively underdeveloped raw materials and industrial sectors represent 5% and 15% of GDP respectively. The Djiboutian services sector focuses primarily on port activities, road and rail infrastructures, and banking. The public administration’s role in providing value-added represents 27% of GDP, and transport and telecommunication represent

18%. With less than 0.3% of the 23,000 square kilometers of land being arable, and a low annual level of rainfall, agriculture plays a very limited role in the Djiboutian economy. Djibouti ranks

150 out of 172 countries in its Human Development Index. The unemployment rate is around

60%. Djibouti achieved positive growth rates over the past five years at an average of about 5 percent. The Government of Djibouti is exploring ways to diversify its economy through the development of the private sector and industries. However, with poor natural resources and with a relatively young population, Djibouti’s

Development will have to depend on its human resources.

Educational reform. Since 1999 the Government of the Republic of Djibouti (GORD) has demonstrated a strong political will to undertake an ambitious education reform. The program to reform the education system is in line with the education for all objectives by 2015.

Donors/organizations and local civil society funds are involved in helping the government to reach its goal for universal access to basic education. Currently, 28% of the national budget supports the educational system. Remarkable results have been achieved in terms of access, but the quality lagged behind.

The long-term goal is to develop Djibouti human capital to build the foundation of a democratic, secure and well-managed state. While access and coverage in education have improved, Djibouti still faces serious challenges in the quality of its education and relevance to the country’s development needs.

Evaluation team. LTL Strategies fielded an evaluation team comprised of a team leader, Iain

McLellan, and a Monitoring and Evaluation specialist, Ilmi Awaleh Elabeh, to conduct the final evaluation of Projet AIDE, which was funded as an associate award under the EQUIP2 Leader

Award. The implementing partner for this project, FHI 360, has focused on improving education systems and policies through working in collaboration with the Ministry of National Education and of Professional Education (MENFOP) and its related structures, including EMIS and

CRIPEN (Centre de recherche, d’information et de production de l’Education nationale, an educational book design and publishing center). The goal has been to improve the quality of primary education in Djibouti. The project has worked with all 100 schools in Djibouti. A cross-cutting gender initiative that integrates gender strategies into all components of the project has been implemented. The project’s activities are based on five results-oriented pillars of educational support: i) decentralized in-service teacher training; ii) decentralized planning systems at the school and regional levels; iii) community participation; iv) improving education management information systems; and v) training for out-of-school youth.

Decentralized In-Service Teacher Training. Projet AIDE has supported MENFOP and its regional offices to deliver effective decentralized teacher training through the development of national policy and training plans. It has also effectively used teacher resource centers (equipped with computer labs with prior USAID funding), improved school capacity to provide ongoing workshops, and improved school-based support focused on improving French language instruction.

Decentralized Planning Systems at the School and Regional Levels. Projet AIDE provided extensive training to pedagogical counselors, school directors, and PTAs in the utilization of improved EMIS data to plan, budget, and monitor school performance. The capacity of MENFOP to manage these reforms was also enhanced.

Community Participation. Projet AIDE built the capacity of PTAs to become involved in school improvement planning, and assist in the management of small grants (in-kind grants are focused on supporting improvements in reading and promoting girls’ retention in primary schools). The project was successful in stimulating community involvement on a large scale where there was virtually none before.

Improving Education Management Information Systems (EMIS). Projet AIDE supported the

MENFOP planning unit to install a new sustainable EMIS system (ED*ASSIST), produce the

Annual Education Data Report, and develop capacity to analyze data for decentralized planning and management. The quality of the data produced is extraordinary, and the technical capacity of the Ministry has been greatly improved.

Training for Out-of-School Youth. Projet AIDE built the capacity of MENFOP to deliver market-driven vocational education training (with functional literacy integrated), using local training centers with support from government organizations, NGOs, and private providers and private-sector employers who agreed to provide internships as part of the training. The Public-

Private Partnerships Projet AIDE implemented provide a model for innovative public-private partnerships, linking non-formal training programs to private sector firms. The private sector firms provide internships and on the job training for out-of-school youth, particularly girls.

Though not on a scale that would have a significant impact, the mechanisms established have good potential for replication, and the interventions correspond with GORD’s new education priorities, which place more emphasis on professional development.

Support to Production of Materials and Training of English Teachers in Middle School.

This project component is based on a request from GORD, and an aim to increase Djibouti’s competitiveness and workforce development in the region. It will build on previous English language material development by Projet AIDE and the British Council. The focus is on producing materials and training 88 middle-school English language teachers.

Intensive and Expanded Focus on Early Grade Reading. Project activities were modified to support a new strategy with new activities, including the specific reading goals outlined in Goal

One of the new USAID Education Strategy. This involved an intensive and expanded focus on early grade reading under the decentralized teacher education component, and the development of an Early Grade Reading Assessment (EGRA) impact evaluation.

Improved Education Systems. The project has been effective in improving education systems, particularly in supporting a system of decentralized training, strengthening MENFOP’s strategic information capacity, and increasing community participation. Less was achieved in terms of increasing opportunities for out-of-school-youth.

Sustaining Decentralized Teacher Training. Support made a difference, particularly in decentralized teacher training where the project engaged the districts, and met with measurable success. A more effective system has been created that is both popular and efficient. The only shortcoming is the very limited collaboration with in-service teacher training at the central level, which needs to be enhanced if the actually system has any chance of being sustained. This intervention certainly merits continued support until its sustainability can be ensured.

EMIS Analysis Capacity. Great progress has also been made in the establishment of Education

Management Information Systems (EMIS). New systems have been put in place with user-friendly software, and, most importantly, skills have been built among MENFOP staff to manage and report the results of the data collection. Further support will be needed to assist the Ministry in taking the analysis of the data to another level. At the moment, a rich trove of information is collected and reported annually, but there remains a need to develop interpretation skills so that data may guide policy-making.

Achieved program performance targets and results based on annual work plans, performance monitoring plans, and the cooperative agreement between FHI360 and

USAID. Projet AIDE met all the reporting requirements and, for the most part, surpassed the relatively modest indicators that were established by USAID. However, these indicators tell only a partial story of what was accomplished with this project, and tend to primarily “count heads”

(people trained, members of PTAs, etc.) without providing deeper and more significant data, including positive changes in education levels and community behaviors. With the notable exception of the EGRA, which provided a detailed analysis of the impact of the early grade reading, monitoring and evaluation outputs tended to be limited and superficial.

Improved Capacity and Skills for School Directors, Teachers, PTAs, and MENFOP Staff.

According to both the quantitative and qualitative research conducted by the evaluation team, the

USAID-supported activities of Projet AIDE resulted in an increase in the ability of school directors, teachers, and Pedagogical Advisors to conduct their work. Two thirds of those surveyed said they were able to do their work better as a result of the project. Parents, who were members of Parent-Teacher Associations (PTAs), and took part in Focus Groups Discussions, said they appreciated the inputs from the project, and were more involved in school activities than ever before. An insufficient amount of quantitative data was collected from MENFOP staff but in-depth individual interviews with staff from different Ministry departments collaborating with different project components revealed varying degrees of satisfaction. Those interviewed who indicated they had increased capacity and skills included EMIS and CRIPEN (Centre de recherche, d’information et de production de l’Education nationale). Staff at CFPEN (Center for

Initial and In-Service Teacher Training) felt strongly that they had not benefitted at all from the project.

USAID’s Education Strategy Affected Project Goals. The emphasis on Early Grade Reading in the last two years of the project came following the creation of the new USAID strategy, which also corresponded with the GORD education goals. Projet AIDE showed flexibility and technical competence in adopting and applying the new goals. The creation of concurrent goals by both

USAID and the GORD, prioritizing Early Grade Reading, contributed to accelerating the funding and establishment of programming.

Suspension of Project Activities Not Major Handicap. From December 2010 to July 2011, all project activities were halted due to the closing of the implementing organization, the Academy for Educational Development (AED). The project was picked up by FHI360, and the change did not appear to affect project performance, as project goals were met before and after the seven-month hiatus. The project proved to be versatile and flexible, and overcame both these challenges.

Projet AIDE Follows MENFOP Education Goals and Action Plan. According to Projet

AIDE staff, project planners were very careful to ensure that its goals and work plans corresponded to those set by the Ministry. Despite this, the project had some difficulties in getting its work plans approved by the Ministry, due to internal coordination challenges at the Ministry.

The project was also slowed by delays the Ministry experienced in finalizing policy, such as the policy on the decentralization of in-service teacher training. Both of these issues had an impact on the implementation of project activities. It should be noted that Ministry and project coordination has greatly improved in the past year, there are currently no obstacles, and monthly meetings among senior-level officials from the Ministry, USAID/Djibouti, and the project provide a forum for addressing any issues that emerge.

Evaluation Methodology

Document review. The two-person evaluation team, made up of a team leader and monitoring and evaluation specialist, reviewed a total of 50 documents, which included: USAID policy guidelines on evaluation, education, and gender (5), project documents including quarterly and semester reports (7), project outputs, including training guidelines, reports (12), and regional training documents (10). Individual in-depth interviews were held with seven key MENFOP informants at the national level, including the secretary-general and the heads of EMIS and

CFPEN, as well as with six key Projet AIDE staff.

School visits. Five days were spent conducting field visits to eight schools (including two in the capital, three semi-urban, and three rural schools in three regions and the capital). In the Regional

Offices and Pedagogical Resource Centers, the evaluation team visited three regional directors, and 27 Pedagogical Advisors were interviewed in Focus Group Sessions. Focus Group

Discussion sessions were also held with 39 teachers and 23 members of Parent Teacher

Associations. In-depth individual interviews were conducted with nine school directors.

Quantitative studies. Small quantitative studies were prepared, consisting of five to seven statements in which respondents were asked to indicate the degree to which they were in agreement with the statements. They were administered to 27 Pedagogical Advisors, 39 teachers, seven school directors, and seven MENFOP staff.

Evaluation objectives. Objectives set by USAID/Djibouti for the evaluation were as follows:

• Assess the performance to-date of Projet AIDE

• Provide project-level and broad-based recommendations for future education programing in alignment with the USAID Global Education Strategy, MENFOP’s 10-year strategic plan, and GORD’s increased focus on English-language learning

• Examine and determine if accomplishments are in line with expected results

Limitations to the evaluation. The amount of time the evaluation team was able to spend in the field conducting the evaluation was a constraint. Interviews with key informants were condensed into one week, and the second week was dedicated to the field visits. The schools visited were selected in a collaborative effort between the consultants and USAID/Djibouti, but relied heavily on FHI360 recommendations. This may have resulted in the evaluation team seeing the most successful interventions, most positive results, and the most cooperative project partners. The samples for the quantitative data collected were very small, and should be considered to be illustrative, and not scientifically representative.

Findings

Project design complex and challenging. The project seemed to respond well to the challenge of the diversity of its components.

Performance reporting adequate but limited. Expectations were very low for performance reporting, with very few and very limited indicators.

Focus remaining performance period on collecting lessons learnt and sustainability. This documentation is essential for sustaining interventions about any future project follow up or replication.

Ability to adapt to unanticipated changes in the implementation environment. The project showed resilience in keeping staff and not losing too much momentum considering the closing of the Academy for Educational Development and the takeover of its projects by Family Health

International, FHI360.

Changes might have been made to program design that would have increased its impact.

Greater collaboration between the USAID/Djibouti education and economic development offices could have potentially allowed for greater resources and technical expertise to be provided to this component.

Community engagement increased but fragile. The impact of project interventions remains fragile, especially in rural schools, where interventions requiring parental participation need to be systematized, intensified, prolonged, and better coordinated with the work of school performance inspectors and school directors.

Improved school physical environments. The most significant impact of the project is having supported the development of a role and the furnishing of a physical space for parent participation in the management of schools.

Change of mentality regarding parents’ role in schools. Even if some parents are less equipped than others to implement their end of parent-teacher relationship bargain, it is clear that

Projet AIDE has helped to instill the participatory parent mentality among the communities in which it has been implemented.

Promotion of community involvement. USAID/Djibouti’s support to Projet AIDE to promote community involvement in schools, through the broadcasting of roundtable discussions on television, has had a positive impact on communities’ perceptions of the value of parent participation.

Value of decentralized planning proven. The pilot project, like the School-Based Training, has been developed, and can be replicated in all regions and the capital now.

Implementation affected by limited MENFOP support. The speed of implementation varied depending on the particular project component.

Mix of community and learning objectives a valuable combination. The strategy of mixing the School-Based Training with teacher supervision and School Small Projects supported learning.

Education of girls normalized. Great progress has been made in maintaining the results of

Djibouti’s efforts to dramatically increase the percentage of girls attending school.

Recommendations

Prepare Projet AIDE close-out plan. This should include the development of case studies highlighting its successes, recommendations for making links with other donors who may be interested in continuing support for different components, and recommendations to the MENFOP on areas where they can take over ownership of key component parts.

Continue support to the decentralization of education services to the regions. Any future programming plans for sustaining interventions at the regional level should be built in from the start, and include a role for MENFOP at the central level to prepare it for post-project continuation.

Enhance policy-level collaboration. Future project designs should include content that ensures that MENFOP policies, regulations, and institutional structures are well understood by the implementer, and will contribute measurably to the achievement of project goals.

Examine possibility of work readiness project for youth. A review of the Rwanda model for work readiness and feasibility study of its applicability for Djibouti might be considered.

Shore up Monitoring and Evaluation. Any future follow-up project should have a more predominant Monitoring and Evaluation component included in the RFA.

Reinforcement and sustainability of parental involvement. Community participation in the maintenance and support of schools’ goals and interests could be dynamized and rendered sustainable through the inclusion of a community resource or liaison person.

Provision of school equipment. This could invigorate and encourage increased community participation in schools.

Award for most active school communities. Encourage schools with passive PTAs to become active, and to encourage diverse members of the community to participate in school activities (not just the same members every year).

Determine whether community participation supports educational goals. There are sufficient data on early-grade reading to recommend further support, but community participation should be examined further to determine its impact on education before further programming is developed.

1.0 Introduction

1.1 Background

Education a government priority. The Government of the Republic of Djibouti (GORD) has long demonstrated its commitment to improving access to education. In recent years, it has constructed new classrooms and invested in new textbooks. In addition, the national budget allocation for the sector has increased significantly since 1999, from 15% in 1999 to over 25% in

2010. Ongoing efforts to address workforce development, encourage investment in youth, and increase access to basic education remain active policy priorities. Further, basic education enrollment rates have steadily increased from 49.5% in 2003 to 78% in 2011, and the total number of public primary schools increased from 83 in 2003 to 119 in 2012.

Education systems quality still a challenge. Despite considerable political commitment and notable gains in access to education, Djibouti’s schools are challenged by declining quality. In its

2009 mid-term report, the National Ministry of Education and Professional Training (MENFOP) discovered that despite recent efforts, many contract teachers at the primary and middle schools remain untrained. Worse, national exam results in the second and fifth grades showed some of the lowest reading and writing skills within the Horn of Africa (HOA) region. Moreover, while efforts to increase the enrollment of girls and other under-represented groups are prominent and ongoing, additional steps by MENFOP are necessary to ensure that the recent enrollment gains are not lost, especially at critical transition points such as between primary and middle school.

Emphasis on education quality. In 2007, GORD launched the National Initiative for Social

Development Strategy for which education is a priority sector. This 10-year, government-wide strategy specifically links desired education outcomes with economic growth indicators. GORD recognizes that Djibouti is faced with an increasingly competitive and constantly changing local and global economic, political, and social environment. However, it also recognizes that improvements in the quality of education will positively impact human health, and reduce the incidence of poverty. GORD has placed quality education at the heart of its social and economic development strategies, and retains active partnerships with the donor community to achieve its

“education for all” goals.

Education action plan goals. To sustain GORD’s level of commitment to the education sector, MENFOP developed an action plan for 2010-2020 that focuses on improving quality and learning outcomes. This plan operates along six program areas goals:

1. Improve the quality and relevance of teaching and learning;

2. Reinforce access and participation in education;

3. Reduce disparities in enrollment and training;

4. Adjust the structure, vision and content of vocational and technical education;

5. Reinforce efficiency in higher education; and

6. Improve efficiency and efficacy of governance, piloting and resource management.

Strengthening systems and management capacity. With other donors, USAID/Djibouti actively supports these program goals through its basic education program, with a flagship project called Projet AIDE (Assistance Internationale pour le Dévelopepment de l’Education), a four-year, $9,500,000 cooperative agreement. The project is designed to strengthen systems and

MENFOP’s management capacity through:

1. Decentralized teacher training and community participation;

2. Strengthened strategic information and communication capacity through an Education

3. Education Management Information System (EMIS); and

4. Increased community participation and education and job opportunities for out-of-school youth.

Since its inception, Projet AIDE has worked in 56 primary public schools in the capital city of

Djiboutiville, and in 63 primary schools in rural and semi-rural areas throughout the country.

AED served as the implementing partner from 2009-2010, before the change resulting in the creation of FHI360 in July 2011.

1.2 Project Description

Decentralized Teacher Training. Projet AIDE objectives and activities include improving education systems through decentralized teacher training by supporting MENFOP in the development of a three-year in-service teacher training plan and the establishment and support of school clusters around the Teacher Resource Centers (TRC) and school districts. It also has developed MENFOP capacity to use information technology for teacher training and provided technical assistance to increase the capacity of TRCs to provide services to all education staff in all districts as well as develop and address gender disparities in this component as well as all the others.

Strengthen Djibouti’s Strategic Information Capacity. The development and installation of an

Education Management Information System is a key element of this component. It also includes support for the use of Fundamental Quality of Education Levels (FQEL) for school level, regional and national planning and the improvement of MENFOP systems capacity in a specific thematic areas as well as the development of management manuals in specific thematic areas.

Increase community participation and opportunities for out-of-school youth. The creation of a legal framework for Parent Teacher Associations and training parents and school directors in roles, responsibilities, and grant management was a key element which also involved assisting

PTAs to register with the Ministry of Interior. The provision of grants to PTAs for school improvements and award programs were developed that recognized school performance. Another part of this component was the strengthening of non-formal NGOs and privately-led initiatives that support Technical Vocational Education Training (TVET) and the non-formal education for out-of-school youth. An analysis was also conducted that allowed for the development of a comprehensive plan for support to public sector vocational training. On another front, the project provided incentives for disadvantaged girls to attend formal or non-formal schools that provide technical education leading to employment opportunities.

Address gender challenges. Each component had a gender element built in. Specifically, that involved the development of a community mobilization strategy around addressing increasing access and retention of girls in primary and secondary school, and the provision of scholarships to alleviate the economic burden of school fees on families. The provision of mentoring to empower and inspire girls to complete their education and train teachers in gender sensitive teaching methods were also part of the strategy.

Intensive and Expanded Focus on Early Grade Reading. Project activities were halted between December 2010 and July 2011. When they resumed under a new contractor, FHI360, they did not become fully operational until mid-2012, and were modified to support a new strategy with new activities, including the specific reading goals found in Goal One of the Global

Education strategy. This involved an intensive and expanded focus on early grade reading under the decentralized teacher education component, and the development of an Early Grade Reading

Assessment (EGRA) impact evaluation to design impact indicators for the last two years of the project. Other wrap-around activities were developed to reinforce early grade reading interventions, including support to produce materials and train English teachers in middle schools, and support to MENFOP to prepare drop-outs for transition to the workforce.

1.2.1 Support to Produce Materials and Train English Teachers in Middle School

This project component is based on a request from GORD, and an aim to increase Djibouti’s competitiveness and workforce development in the region. It will build on English language material development earlier by Projet AIDE and the British Council. The focus is on producing materials and training 88 middle school English-language teachers.

1.2.2. Project Design

Projet AIDE proved to be a versatile and adaptable planner when developing these multi-faceted interventions. The design was as diverse as it was responsive to changing priorities on the part of the government of Djibouti and USAID. The project was comprehensiveness from the start with its combination of community level components like the development of Parent Teacher

Associations, in-school components like the decentralized teacher training and Ministry level components like strengthening its information capacity. Then, important new components were planned, including primary reading and post-primary English language education.

A review of the planning documents revealed solid plans that were realistic and comprehensive, and have largely been implemented as planned. Changing circumstances and priories required some adaptations along the way, but this was due to external pressures rather than faulty planning. It is unclear as to why monitoring and evaluation was not given more attention in the project. USAID did require a monitoring and evaluation plan ,but Projet AIDE responded minimally and, according to USAID/Djibouti, did not hire sufficiently skilled staff to implement the plan. As a result, Monitoring and Evaluation did not receive sufficient attention in this project.

1.3 Project Goals and Objectives

At the completion of Projet AIDE, USAID anticipates that the project will result in:

Improved education systems through decentralized teacher training

a. 1,100 teachers trained in new methodologies focusing on literacy and numeracy;

b. MENFOP develops an annual and two-year teacher education plan;

c. Teacher Resource Centers support an in-service system using cluster and school-based teacher training;

d. An increasing number of teachers are supported by pedagogical Advisors and school directors and work as teams tin schools and clusters to master new techniques; and

e. The majority of teachers use the new curriculum with a particular focus on improving the teaching of literacy and numeracy.

Strengthened Djiboutian strategic information capacity

a. 100 MENFOP administrators and officials trained in strategic information management and on using data for planning FQEL;

b. A functional EMIS to collect annual data and report analyses on a timely basis;

c. MENFOP uses FQEL and school report cards as tools to use EMIS data for policy, planning and budgeting at school, regional and national levels; and

d. MENFOP has a functioning operational plan that is implemented to maintain facilities

(i.e., computers, Teacher Resource Centers, schools, etc.).

Increased community participation and opportunities for out-of-school youth

a. 100 primary schools have a PTA by the end of the program;

b. 50 PTAs receive small grants and implement their projects;

c. At least 110 out of school youth (50% girls) provided learning opportunities per year,

d. Twenty youth receive skills training leading to employment; and

e. Four public or private professional and vocational education centers are assisted every year.

1.4 Evaluation Methodology

Four components of evaluation methodology:

A document review of USAID guidelines on related policies, including evaluation, education, and gender, and project documents including quarterly and semester reports.

Interviews with key informants at the national and site levels, including implementing partner staff.

Focus group discussion sessions and in-depth individual interviews with selected community stakeholders and project beneficiaries.

Site visits.

Data triangulation. This approach allowed the team to gather information from a diversity of sources and provide multiple sources for triangulation of the data. The methodology was adapted based on the dictates of the situation on the ground and direction from USAID/Djibouti. For example the individual interviews with teachers and Pedagogical Advisors evolved into Focus

Group Discussion sessions.

Approach and stages of evaluation. An understanding of Projet AIDE and its objectives and goals was created through document review and briefing with project and USAID staff. A basic understanding of its achievements and the obstacles that were faced also came from those initial meetings. The evaluation tools were prepared including qualitative questionnaires and Focus

Group Discussion guides based on those discussions as well. The field visits were scheduled in a collaborative meeting with Projet AIDE, the USAID project officer, and the evaluation team to ensure a good balance between diverse settings and a reasonable sample size in the limited time frame available.

Interviews and field visit. In the first week, interviews were held with MENFOP staff, partners and interested parties such including UNICEF and French cooperation and Projet AIDE staff.

During the five-day field visit to the agreed-upon program sites, interviews and focus group discussions were held with key stakeholders such as school teachers, school directors, and PTA members, and representatives from the Ministry of Education, including regional Pedagogical

Advisors and resource center staff. Interviews provided an opportunity to examine how well activities were coordinated by local program implementing partners. Two zones in Djiboutiville and four schools were also visited by the evaluation team.

Multiple sources of data. The data gathered from the multiple sources enabled the evaluation team to develop a nuanced perspective the impact of the teacher training exercises, the effectiveness of the Education Management Information System (EMIS), the number and types of opportunities targeted for out-of-school youth, the effectiveness of early grade reading activities

(in relation to the project’s intended Intermediate Results), and the extent to which the overall program objectives and expected results were met. The evaluation team was able to identify and assess the impact of activities at the community level, but also to assess the current institutional capacity of partners and beneficiaries who were central to implementing and monitoring the education activities. Weaknesses and specific challenges were also identified which can inform future program planning.

Project Spending

Overall project spending. A total of $6,344,297 was spent over the four years of the project.

That amount breaks down into an 80% to 20% ratio of end-user focus versus project-specific focus.

Estimated allocation of expenses for the period December 1, 2009 - February 28, 2013*

Chart 1: Overall Project Spending

Categories Spent Percent

End-user Focus $4,974,892 80%

Project Specific Focus $1,269,505 20%

Total $6,244,397 100%

Project specific focus spent. The main project support items were the management, administration and finance which absorbed about three quarters of this expenditure. Overhead took another quarter. The relatively small amount spent on Monitoring and Evaluation way be an indication of why this project component was relatively weak.

Chart 2: Project Specific Focus

Management $327,266 26%

Admin & Finance $593,554 47%

M&E $19,554 2%

Overhead $329,131 26%

Total $1,269,505 100%

[Type a quote from the document or the summary of an interesting point. You can position the text box anywhere in the document. Use the

Drawing Tools tab to change the formatting of the pull quote text box.]

End user focus. The expenditures for each of the component parts reflect the diversity of this project, and the relatively even spread of resources among its component parts. Teacher training and information systems improvement took the lion’s share with 18% each, but support for out- of-school youth (15%), community development (14%), and early grade reading (13%) were not far behind. Finally, gender, girls’ scholarships, and middle school English took more modest percentages at 7% each.

Budget breakdown. An analysis of the breakdown of the specific focus of project spending shows that monitoring and evaluation at only 1 percent was extremely modest compared to similar projects and reflects the lack of data available for this project. The proportion of funds dedicated to the other areas is comparable with other projects, with the exception of the overhead, which is relatively modest compared to many other projects.

Chart 3: End User Focus

Early Grade Reading $697,829 13%

Middle School English $399,718 7%

Gender $405,954 7%

Girls Scholarships $405,954 7%

EMIS $994,978 18%

Out of School Youth $779,475 14%

Community Development $842,869 15%

Teacher Training $994,978 18%

Total $5,521,757 100%

Balanced approach. Teacher training and the

EMIS tool represent the largest proportion of the project intervention budget, followed by community development, out-of-school youth interventions, and Early Grade Reading.

Gender received the same amount if it is combined with the Girls’ Scholarships. Middle

School English had the least important percentage. This pattern reflects a very balanced use of financial resources for each of the project components.

Chart 3: End User Focus

2.0 Key Findings

2.1 Evaluation Key Issue: Program Results and Impact

2.1.1 Improve education systems through decentralized teacher training

Training indicators met. Projet AIDE largely met its indicators for training teachers/educators over the four years of the project. It exceeded them slightly in FY10 and FY12, but fell short in

FY11.

Number of teachers/educators trained with USG support

Year Target Actual

FY10 1,100 1,211

FY11 1,100 893

FY12 380 433

FY13 955

Total 3535 2537

Decentralized training on large scale. The most impressive numbers of teachers and school directors trained occurred in the training for the Early Grade Reading Assessment (EGRA) that was introduced to all pedagogical Advisors and 113 target primary schools. In FY12 and FY13 a total of 94 training sessions were attended by 5,023 teachers and 205 school directors were trained in 46 sessions in five regions and two districts in Djiboutiville. Included in those numbers are the teachers and directors who attended more than one session. Other training included 132 teachers who got special training in teaching reading recuperation for students with reading difficulties. The number of administrators trained both at the central and decentralized levels was twice as much as the FY12 target of 50.

Training developed using participatory methods. The School-Based Training was developed using participatory methods. The contents were developed in collaboration with five regions and two districts in the capital. Assessments were also done right down to the school level to identify teaching training priorities. This approach was universally appreciated by the Regional Directors, Pedagogical Advisors, and teachers interviewed. In the qualitative study conducted by the evaluation team, 64% of the 39 teachers who were surveyed said they agreed or agreed very much with the statement that the Projet AIDE-funded training had “increased their skill level for teaching and made them a better teacher.” One regional director pointed out that the identification and resolution of specific education problems had not previous been part of the general culture of education in Djibouti. He added that the participatory approach and local needs assessments were positive contributions of Projet AIDE.

Better reading instruction skills developed. Appreciation of the training in teaching reading skills was also reflected in the focus group discussion sessions held with teachers in five schools.

“We learnt different ways to teach reading,” and “We have more success when we are aware of how to break down into different stages the teaching of reading,” were typical of the comments.

The teachers found the Pedagogical Advisors who conducted the training to be well prepared and knowledgeable. They also found the training had a good balance between theory and practice.

“What works best is practice. We got to try the new methods during the training.” The teachers also liked the convenience of attending several two-day trainings near their schools rather than longer ones far from their homes during their vacation time that was the method used in the past.

Number of Pedagogical Advisors Trained

Year Number Note

FY10 60

FY11 60

FY12 16

FY13 12

Total 3535

Pedagogical Advisors appreciate school-based training. As can be seen from the above chart

148 Pedagogical Advisors were trained over the four years of the project. The AAs were trained by the project and then, in turn, trained the teachers and school directors. The evaluation team administered a quantitative study to 35 Pedagogical Advisors and asked them to consider how much they were in agreement with three questions related to the training. Three quarters of the

Pedagogical Advisors either agreed very much or agreed with the statement that teachers are

“better prepared with new teaching methods and supports” after undergoing Projet AIDE-funded training. The same percentage responded favorably to their own experience with the Projet AIDE training of trainers which they agreed was “useful and relevant,” “transferred easily” and had a

“clear impact on schools.” The third quantitative question was related to the training of teachers allowing them to “develop good technical teaching skills that can be immediately put to use and will have a lasting impact in the quality of learning in the future.” Again, three quarters of the

Pedagogical Advisors either agreed or agreed very much with the statement.

Participatory approach creates responsive model. In the four Focus Group Discussions with

Pedagogical Advisors in three different regions and one district in Djiboutiville the comments were also very positive regarding the participatory approach used in the decentralized training, and the impact it had on reading instruction. “We all benefit from the training because it meets the needs identified in advance,” and “Ideas are shared when preparing the modules” were representative comments.

Supervision follow-up adds value. The AAs were impressed by the application of the training which they were able to observe first hand when they did in-school supervision of the teachers.

“They take what they have learnt and apply it in the classroom,” and “the behavior of teachers changed,” and “they have lots of energy to bring to the classrooms and want to practice what they have learnt,” were common comments. In fact, the project supported supervision by the AAs in the schools using a competence based evaluation checklist contributed to enhancing the quality of teaching reading. “Teachers understand what is expected of them when they get their evaluation.”

Local responsibility for the training and follow-up adds greatly to the quality to the training.

Teachers are trained then tracked back at schools against an established basic competence then their success is charted against it enables fine-tuning of teaching skills.

“Everyone agrees and understands the challenge,” one teacher commented.

Compared to the previous system of centralized training, the teachers, CPs, school directors and regional directors almost universally preferred the decentralized approach. Participation is higher, the training is timely, and ample resources are available for local transport and local per diems.

The content is adapted to local needs. It was suggested by one regional director that it is more cost-effective to spend 100 liters of petrol in a region to reach remote areas than to spend it sending national staff to a regional capital or teachers to the capital.

Collaboration between project and CFPEN low. To be sure one of the reasons for the success of the schools based training was that Projet AIDE financial resources and technical assistance went directly to the regions. CFPEN doesn’t have the teacher trainers or the administrative and management capacity to deliver decentralized teacher training at present according to several key informants. The belated establishment of MENFOP policy on decentralized training let to a lack of clarity regarding the relationship between CFPEN and the project. There is some resentment on the part of CFPEN that it has played a insignificant role. The CFPEN director suggested that

Projet AIDE should support and not take the place of CFPEN and has not invested in building up its capacity. The low level of collaboration between the project and CFPEN will be a hamper to replication and sustainability.

Teacher training plans. Ongoing support has been given to MENFOP to develop teacher training plans. In FY12 a three-year in-service teacher training plan was developed. Though it does not factor in to the indicator, five regional school-based training plans were developed with project support.

Number of Teacher Training Plans Developed

FY10 1 1

FY11 1 2

FY12 1 1

FY13 1

Total 4 4

2.1.2 Strengthening Djibouti’s Strategic Information Capacity

Great progress made with strategic information. For each year of the project, the indicator for both the Fundamental Quality and Equity Level evaluation and other strategic information management was surpassed except in one instance. In fact, the FY2012 target of people trained in strategic information was set at 100 and almost doubled to 195. There were some problems with a high level of turnover of MINPOF staff assigned to work on strategic information which slowed progress. During the life of the project there were three different people in charge of collaborating with the project. Those challenges have been overcome with the creation of a permanent strategic information staff. Overall the Education Management Information System was developed and installed with good collaboration with MENFOP. According to all stakeholders, there has been an acceleration of the availability of the national annual statistics compared to previous years.

Decentralized planning systems at the school and regional levels. Projet AIDE also trained

Pedagogical Advisors, school directors, and PTAs to utilize the improved EMIS data to plan, budget, and monitor school performance using the FQEL (Fundamental Quality and Equity

Level) school planning tool and School Report Cards (performance monitoring tool). The project successfully built the capacity of the Ministry of Education to manage these reforms. The project also contributed to improving MENFOP systems capacity in specific thematic areas and developed management manuals in specific thematic areas.

Number of administrators trained both at the central and decentralized level on FQEL (Planning), disaggregated by gender; national and district.

Year Target Actual

FY10 100 102/F=28

FY11 50 61/ F= 02

FY12 50 105/ F= 46

FY13 50

Total 250 268/ F= 76

Number of people trained in other strategic information management

FY10 100 104

FY11 100 35

FY12 100 195

FY13 100

Total 400 334

Better analysis helpful. National data is in raw form, with little or no analysis at present.

Training MENPOF MIS staff on how to draw conclusions from the data, recognize trends and would add value to the annual reports. Special reports could also be prepared as well on topics of interest. For example, an analysis could be made on the progress on encouraging the education of girls based on the rich data collected on gender.

Software user-friendly. Software ED*Assist and tools used by regional focal points (OPAD) was considered user…