EQ2 End of Project Evaluation SOW - Request for Comments.docx

DRAFT

EQ2 End of Project Scope of Work STATEMENT of WORK

End of Project Evaluation Education Quality and Access for Learning and Livelihood Skills Project, Phase II

1. INTRODUCTION

The United States Agency for Development (USAID) in the Philippines seeks to conduct an evaluation that will assess the performance and impact of the second phase (EQ2) of the Education Quality and Access for Learning and Livelihood Skills (EQuALLS) Project in meeting its objectives: 1) to improve quality and access to basic education in areas most affected by poverty and conflict in the Philippines, specifically the ARMM and Regions 9 and 12 in Mindanao, and 2) to provide alternative learning opportunities, including livelihood training, to out-of-school youth in these regions. Working with the national and ARMM Departments of Education, EQuALLS increases learning opportunities by providing infrastructure and materials, improving instruction quality in teaching English, science and math, mobilizing local governments, communities, and other stakeholders to give greater support to basic education and workforce development.

The evaluation will inform the USAID/Philippines Country Development Strategy (CDCS) for FY 2011-2016 and will guide the design and development of future education programs. The evaluation will examine the extent the results of the EQ2 project have been achieved in targeted regions in the Philippines—the Autonomous Region in Muslim Mindanao and other conflict-affected areas.

USAID will seek the services of an evaluation team over a period of three months to implement the requirements defined in this statement of work.

2. BACKGROUND

In a 2003 visit to the Philippines, President George W. Bush committed USG support for basic education reform in the country. USAID/Philippines responded by initiating education programming to achieve the Mission strategic objective to “Increase access to quality education and livelihood skills in selected areas of the Philippines, particularly those most affected by conflict and poverty.” These efforts are aligned with the priorities of the Government of the Philippines (GOP), responding in particular to the GOP Medium Term Development Plan 2004-2010 which highlights the following critical steps towards improving education in the Philippines: upgrading math, science, and English teaching and learning in formal basic education; promoting school-based management; and expediting TVET programs and services in the Special Zone of Peace and Development in Mindanao.

USAID education programming in the Philippines focuses on the Autonomous Region in Muslim Mindanao (ARMM), and Mindanao Regions IX and XII which include Muslim and non-Muslim areas. These regions face similar challenges but education statistics relating to both access and quality are much lower in the ARMM, which falls under the jurisdiction of a separate ARMM Ministry of Education rather than the National Department of Education (DepED). A recent GOP study shows that 12% of the Mindanao population aged 10-64 years is illiterate compared to a 6.9% national average, and the rate rises to 22% in the ARMM.[footnoteRef:1] Among the illiterates, 47.2 % are women. The school drop-out rate is estimated at 6% nationwide, but 10% for Mindanao and reaches nearly 18% in the ARMM. Some Mindanao target communities report that 50% to 70% of school age children and youth are out of school. Studies on education outcomes in the Philippines point to poverty as the leading cause of low educational achievement, citing both direct costs (e.g., uniforms) and the opportunity cost of children working to support family livelihood.[footnoteRef:2] [1: GOP Functional Literacy, Education, and Mass Media Survey (2008).] [2: Sources include an Asian Development Bank Working Paper, “Education Outcomes in the Philippines” (2010).]

To address these challenges to learning in Mindanao, USAID issued a five-year, $30.1 million cooperative agreement in 2004 for the implementation of the EQuALLS Project. EQuALLS was designed to achieve four Intermediate Results (IRs): 1) increased learning opportunities for children and youth through community support for education and livelihood programming ( includes infrastructure refurbishment and/or construction, and community incentive grants); 2) strengthened capacity for teaching math, science, and English; 3) improved relevance of education and training for out-of-school children and youth; and 4) heightened government capacity to implement education policy.

The project was implemented through an umbrella grant with a lead implementing partner and eight additional lead actors, including local and international NGOs and private sector contributors linked through public-private alliances. The wide variety of lead actors involved in implementation was intended to promote innovation, tap into Mindanao-specific expertise, and leverage private resources. All actors sought to work in consultation with the national DepED, the Department of Education of the ARMM, the Techncial Education and Skills Development Authority (TESDA) and relevant local and regional government representatives.

After two years of implementation, USAID/Philippines determined that the diversity of approaches to implementation pursued by different lead actors varied too widely to permit the achievement of desired results. The Mission ended the project in early 2006, calling it EQuALLS Phase I (EQ1) (see Attachment 1, EQuALLs Program Description, for more detail on EQ1). A new 5-year, $59.3 million cooperative agreement was issued to Education Development Center, Inc., (EDC) in July 2006 to implement EQuALLS Phase II (EQ2) under a streamlined management structure that designated EDC as the lead, and required three sub-grantee partner groups to work under the technical direction of EDC.

The three (3) EQ2 implementing partner groups are the following:

1. Save the Children

2. Synergeia Foundation

3. International Youth Foundation/Education and Livelihood Skills Alliance (ELSA), with the following alliance members: Petron Foundation, Ayala Foundation; Consuello Foundation, and Philippine Business for Social Progress

In addition, EQuALLS 2 has the following Global Development Alliance (GDA) Partners: Petron Foundation, Brother’s Brother Foundation, Microsoft Philippines, and National Bookstore.

Under EQuALLS 2, other key technical changes include the following:

1) elimination of Intermediate Result (IR) 4 on government policy capacity strengthening;

2) deepening of coverage within target municipalities;

3) increased emphasis on out-of-school youth; and

4) increased engagement with community stakeholders through the introduction of a whole-school based approach to the implementation of all activities

In addition, EDC identified several cross-cutting interventions to support the achievement of the remaining IRs, including an emphasis on local stakeholder ownership and capacity building and a communications strategy to publicize and disseminate project achievements to national level stakeholders and other donors. The three remaining IRs remained unchanged and continue to be implemented, as follows:

IR1: Increased learning opportunities for children and youth through community support for education and livelihood programming. Interventions include:

· Increased community involvement in education and training for children and out-of-school youth, including through strengthened Local School Boards, use of Barangay Report Cards, and community incentive grants, etc.

· Improved physical learning environments in formal (school) and non-formal (community center) education and training sites

· Capacity building for Parents Teachers and Community associations (PTCAs)

· Capacity building for Madrasah education

· More private, faith-based schools accredited by the GOP and adhering to GOP teaching and learning standards (Madrasah Education)

IR2: Strengthened capacity to teach math, science, and English. Interventions include:

· Core training of teacher mentors/Learning Facilitators in math, science, and English as well as mentoring strategies

· Training of administrators and officials in school finance, management, and governance

· Supplemental training for all teachers in target areas to address challenges such as multi-grade classroom instruction, etc.

· Teaching and student materials development

· Teacher computer literacy

· Book donations (GDA partnership)

IR3: Improved relevance of education and training for out-of-school children and youth.

Interventions include:

· Education for out-of-school youth in literacy and other specialized courses on topics related to basic literacy and numeracy, and an Accreditation & Equivalency (A&E) program for those wishing to pursue further education

· Workforce development (WFD) training in collaboration with the GOP Technical Education Development Authority (TESDA)

· Vocational Scholarships for out-of-school youth attending WFD training

· Academic Scholarships for out-of-school youth who pass the A&E exam and wish to pursue further technical or academic degrees

The first year of EQ2 implementation was a transition year to ensure greater program coherence, standardize implementing partner interventions, and develop a more focused communications and branding strategy. Other efforts to enhance implementation included the introduction of baseline proficiency asessments in English, science and math to serve as a basis for the selection and qualification of teachers as EQ2 ‘master teachers’ under IR2.

By 2008, however, a mid-course internal review raised new Mission concerns about the pace of implementation, and effectiveness of management structures. In February 2009, the USAID Regional Inspector General conducted a formal program audit to “determine whether EQ2 was achieveing its main goals.” Per the audit summary, “The audit found that EQuALLS was making progress toward increasing access to quality education and improving the livelihood skills of out-of-school children and youth. However the extent of the progress is uncertain because reported results were unreliable, and key program interventions were not as successful as expected.” Additionally, the audit found that “the Mission and EDC need to provide more oversight to ensure that the program is being implemented as planned and that intended results are being achieved.” The audit documented a variety of other management and implementation challenges affecting each of the three result areas. (Audit Summary, Attachment 2).

On-going efforts to strengthen the teacher professional development component led to additional technical program modifications. Building on a previous partnership with the Basic Education Assistance for Mindanao (BEAM) Project, EQuALLS expanded IR2 to go beyond English, math, science, and mentoring, and into reading. Thus, the Whole School Reading Program was implemented as a pilot in 14 selected schools to strengthen the teaching of reading, and introduce new assessment tools to assess levels of reading competency at the early grades.

In September 2009 another important modification was introduced under IR1 to structure and institutionalize local stakeholder engagement. The concept of the Barangay(Village) Education Report Card (BERC) was introduced and obtained support from DepED. Field implementation began in January 2010 in 44 brangays. (Attachment 3. Barangay Education Report Card Briefer). A small community grant program was also incorporated under this IR to provide community stakeholders experience with resource management.

Finally a scholarship component was added under IR3 to enable out of school youth who passed the GPH high school equivalency exam to apply for post secondary technical training, and in a few cases, to higher education programs. This component is very important, as scholarship programs in the Philippines are available only to students who manage to initiate studies on their own.

The scheduled end of EQ2 in December 2011 and two other critical events drive the timing of this evaluation. First, in early 2010, a new reform-minded administration was elected in the Philippines and stated its intent to raise education spending levels and introduce systemic reforms, including the addition of two years to the current 10 year basic education cycle. Second, USAID/Washington issued new guidance for the formulation of USAID Country Development Cooperation Strategy (CDCS) in order to better align USAID funded programs with the Paris Declaration principles and U.S. President Obama’s administration priorities. The new political context and the ending of the EQ2 project present an important opportunity to factor EQ2 experience into the shaping of a new education sector strategy.

3. SCOPE OF WORK

The purpose of the evaluation is to assess the impact of USAID’s investment in EQuALLS 2’s intended beneficiaries at the national, regional and local community levels, and assess the performance of EQuALLS 2 in achieving its objectives of improving quality and access to basic education in areas most affected by poverty and conflict in the Philippines, specifically the ARMM and Regions 9 and 12 in Mindanao, and of providing alternative learning opportunities, including livelihood training, to out-of-school youth in these regions.

The evaluation will also look at the effectiveness of its three intermediate results -- increased learning opportunities, improved teaching in math, science, and English, and greater relevance of education and training for out-of-school children and youth. The evaluation will be expected to identify the mix of variables and conditions that affected the project results and outcomes, within the context of the Education Results Framework and its performance indicators.

The evaluation will identify factors that contributed to the achievement or underachievement of targets and deliverables and assess the efficiency of the implementing strategy. In evaluating efficiency, the assessment will focus on USAID’s mix and balance of approaches, effectiveness, sustainability, USAID’s comparative advantage, and implementing mechanisms.

If any stated results were not achieved, the contractor shall identify probable impediments. Where education indicators have remained low in project areas despite USAID assistance, the evaluation needs to provide a thorough assessment of key constraints to success and provide recommendations, as appropriate. Where education indicators have been exceeded, the evaluation needs to document best practices and develop success stories.

The evaluation will also assess the degree to which these results or other, unanticipated project outcomes enjoy stakeholders support and ownership at the local, district and/or the national government levels. In addition, the Contractor will assess EQ2’s public private partnerships and the project’s communications strategy. Finally, the Contractor will compile recommendations for ensuring the sustainability of project outcomes, and state how the evaluation results can be applied to improve the design and implementation of similar education programs.

The main audience for the evaluation is USAID/Philippines. Secondary but very critical audiences will be the GOP and the ARMM Regional Government (RG) at the national and/or sub-national levels, select school community stakeholders, and USAID/Washington. The evaluation shall also document best practices/lessons learned, develop success stories, and propose recommendations that will inform USAID’s future program directions and strategy and the Agency’s broader efforts to provide assistance in other similar, conflict-affected regions or countries.

In sum, the evaluation will cover:

· All interventions undertaken under EQuALLS2

· All interventions carried out under the three IRs

· All outcomes and impacts achieved by EQ2, whether intended or not

· All issues identified as relevant by the contractor

· Any significant differences in the results, outcomes or impacts achieved in target communities

· Implementing approach and arrangement/mechanism of the project

· Cost effectiveness of interventions undertaken under EQuALLS2.

To the maximum extent feasible, the contractor shall build upon and complement the data contained in the 2009 IG Audit summary (Attachment 2).

4. DETAILED WORK REQUIREMENTS

The evaluation will be guided by the following approaches –

1. Assess the over-all performance and impact of the project based on the Education Program Results Framework, the Monitoring and Evaluation Plan and its performance indicators, comparing baselines with final project results.

2 Describe and analyze the factors that affected project performance.

As guided by the Results Framework and the Monitoring and Evaluation Plan of the project, the evaluation shall cover the following areas,: EQuALLS Project as a whole, program approach, program relevance, project components, cross cutting strategies; and assessment of implementing partners’ performance.

A. EQuALLS Project as a Whole

B. Program Approach

C. Program Relevance

by stakeholders, and why? Which are perceived to be least useful and why?

D. Project Components

1. IR1: Increase Learning Opportunities for Children and Youth

opportunities for children and for out of school youth;

1.4 Examine the programming approach and determine if and how it promotes synergy of activities

2. IR2: Strengthen Capacity of Teachers to Teach Math, Science, and English

3. IR3: Improve Relevance of Education and Training for Out of School Youth

E. Cross-Cutting Strategies

1. Partnership building under Global Development Alliance

F. Assessment of Implementing Partners’ Performance.

G. Recommendation for Education Programming

1. What are the key programmatic recommendations for achieving the sustainability of EQ2 results and outcomes?

5. METHODOLOGY

The Evaluation Team shall propose a methodology that best meets the SOW and detailed work requirements to achieve the evaluation objectives, and has been proven to be effective in the evaluation of large social development projects.

Below are some data collection approaches that the Team may consider in choosing the most appropriate methodology. These approaches are not prescriptive and other forms of information collection may be proposed by the consultants. USAID will assist the consultant to the extent possible to ensure that all the appropriate and necessary inputs are obtained to maximize the results of this engagement.

1. Review of relevant project documents, contracts and reports; existing monitoring and evaluation data collected by the contract agent, including but not limited to project documents on current and immediately past USAID/Philippines education activities (Peace Corps Tudlo Mindanao Project, EQ1).

2. Key informant interviews or, as needed, workshop meetings, with USAID personnel, national and local public sector partners including but not limited to the national, regional and field officials of DepED, DepED ARMM, Technical Education and Skills Development Authority (TESDA) and TESDA ARMM, community learning centers and barangay officials; LGUs, donors, and selected project beneficiaries/partners.

3. Site visits to Mindanao. The areas to be visited shall depend on the most cost effective and practical method to gather information about the project without sacrificing the quality and validity of data.

4. Survey or any combination of evaluation methods, to measure perceptions of beneficiaries on the impact of the project and the effectiveness of development approaches and implementation mechanisms.

Given the above possible approaches, the Team however will be expected to interview at a minimum, the prime and local implementing partners, technical director(s) for each IR, select members of partner community level implementation teams; relevant local, district and regional and/or national level government officials, community level stakeholders, and select target beneficiaries in EQ2 areas, to be chosen by the Team in consultation with USAID/PRM, the office tasked with managing this Contract.

In addition to the beneficiary impact assessment, and the document review and analysis of completed evaluation reports, sectoral studies and assessments, and project reports, other evaluation methods may be employed by the Contractor, such as focus group or roundtable discussions to validate results of secondary and primary data analysis and case studies of selected beneficiaries and communities, among others.

The number of sites to be visited in each region will be decided by the Evaluation Team in consultation with USAID/PRM and the Office of Education.

6. GEOGRAPHIC COVERAGE

The EQ2 is implemented in Mindanao Regions IX and XII and in the ARMM . The Evaluation Team will be expected to conduct field visits to all three regions, but gather data from outside the project area for data comparison purposes. This will ensure that regional distinctions are factored into the assessment.

7. DELIVERABLES

This Scope of Work has the following deliverables:

1. Within the first five days of the consultancy, the Evaluation Team shall present a work plan of activities throughout the engagement to be approved by USAID. During the Entry Briefing /presentation, the Evaluation Team shall present the analytical instruments and/or tools, methodology and work plan.

2. A mid-project debriefing will be made by the Team to Mission staff to report on observations, insights, progress, problems and opportunities encountered at the midcourse. The results of the debriefing may determine the subsequent work or direction of the Team.

3. Initial Findings and Recommendations – The Evaluation Team will provide a brief presentation of initial findings and recommendations to Mission staff for feedback and comments to be incorporated in the first draft of the evaluation report.

4. First Evaluation Report. The Team is expected to submit a first draft the evaluation report at the end-of-consultancy debriefing. The report should contain the team’s key findings as presented during the final debriefing. It will be expected to include recommendations for future education programs and projects.

5. Final Report. The final report should incorporate Mission’s comments and recommendations and should be submitted by the team leader a week upon receipt of such comments and recommendation. The main body of the report (minus the appendices and tables) should not be more than 50 pages. At least five hard copies of the report and one electronic pdf file should be submitted to:

Chief, Program Resources Management Office USAID/Philippines 8th Floor, PNB Financial Center

The final report must be in Times New Roman 12 and must have relevant annexes, matrices and other visuals to consolidate and summarize data. The report format will be agreed upon with the Mission, but shall include the following elements of an Evaluation Report. Attached for the contractor’s reference is USAID’s Evaluation Report Checklist:

I. Executive Summary of findings, conclusions and recommendations, not to exceed five pages II. Introduction and Background III. Scope and Metholodogy IV. Body of Report

1. Major Findings (Evidence) for each component (supported by data)

2. Conclusions

3. Recommendations

4. Lessons Learned Annexes

6. Organize two (2) presentations of the report findings to education stakeholders that will include donors, host government agencies, partners, and project beneficiaries. One in Manila and one in Mindanao.

7. The contractor shall submit at least five (5) paper copies and five (USB) electronic copy in Microsoft Word format or PDF of all deliverables with supporting documentation in Word, Excel or other relevant software.

Deliverables Schedule:

12 weeks from start of contract