Attachment__2-Task_Order_2.pdf

ATTACHMENT 2 – TASK ORDER 2

SECTION C – DESCRIPTION / SPECIFICATIONS/STATEMENT OF WORK

C1. INTRODUCTION

Name of Activity to be Evaluated: Basic Education Program (BEP) Implementer: Family Health International Development 360 LLC (FHI 360) Award Number: 167-A-00-10-00101 Agreement Value: $9,791,000.00 and €5,000,000.00 of the GoK Trust Fund Life of Program: August 30, 2010 – July, 2016 Period to be Evaluated: August 30, 2010 - Present

C.2 BACKGROUND

During the BEP implementation the Ministry of Education science and Technology (MEST) adopted the Kosovo Education Strategic Plan for 2010-2015. At the heart of this plan are three key developments in the education sector. These developments are: 1) Decentralization of schools and municipalities; 2) A new Curricular Framework (KCF); and 3) A new teacher licensing system. USAID’s Basic Education Program (BEP) supports these positive and forward-looking initiatives in close collaboration with the Ministry of Education, Science and Technology (MEST), as well as other donors. With its activities, BEP directly contributes to KESP.

C.3 BEP PROGRAM DESCRIPTION

BEP was awarded in August 2010 as a five-year initiative, funded jointly by USAID and the Government of Kosovo. In 2015, BEP was given an eleven-month no-additional-cost extension to allow program activities to continue through the 2015-2016 school year. BEP was designed to benefit the public primary and lower secondary schools in Kosovo (serving grades 1 – 9). Over the course of implementation the project has worked in almost 650 schools in 31 of Kosovo’s current 37 municipalities. Approximately half of the schools received assistance in all three objective areas (stated below); the others received assistance in one or two of the objective areas.

BEP Components/Objectives:

It includes the following three components aimed at supporting reforms introduced by the Ministry of Education, Science and Technology (MEST):

1. Enhance School Management Capacities in the Decentralized Environment: BEP aims to improve the management skills of school directors, school boards, and Municipal Education Departments (MEDs) in the areas of planning, school management and quality assurance.

2. Strengthen the Assessment of Learning Outcomes: BEP aims to improve the capacity to develop and implement new school-based (internal) and potentially national (external) assessments tied to the new curricula at the local, municipal, and central levels. This will support the establishment of an effective and reliable assessment system of student learning outcomes that will increase the quality of education at the primary level.

3. Improve In-service Teacher Training: BEP assists the Ministry of Education, Science and

Technology (MEST) in providing in-service teacher training reforms. These reforms include certification requirements, supporting the MEST new Teachers’ Licensing foundation and establishing a program for teachers that will provide continuous professional development.

Development Hypothesis:

If Kosovo has a better educated population, well-trained in relevant skill areas, then the human resource needs of the economy will be more effectively met, the economic wellbeing of citizens better ensured, and citizens will be better equipped to engage effectively with Government.

BEP’s motto is “Developing students’ 21st Century skills with schools and communities”. BEP seeks to improve the capacity of Kosovo’s schools to provide relevant skills for its students. Its overarching goal is to improve the Government of Kosovo’s institutional capacity in the education sector and improve the quality of primary education.

Planned Outcomes:

Expected results of BEP include:

1. School directors have increased school management capacity (can be measured by surveys of teachers, students, parents and municipalities);

2. School boards contribute to school accountability and transparency, as well as to the improved quality of education services in their school;

4. Strengthened capacity of teachers to deliver quality instruction;

5. Increased capacity of MEST, municipalities, and schools to develop and implement a system of continuous professional development for teachers.

C.4 PURPOSE OF THE EVALUATION

The purpose of this performance evaluation is to provide USAID/Kosovo with an objective external evaluation of the management, performance and sustainability of BEP activities from August 30, 2010 to the present.

The Evaluation Team will examine the overall impact of the activities on the target institutions and validate/observe the progress made in achieving the results and objectives as specified in the award with

FHI 360.

The Evaluation Team will review actual versus planned progress in attaining the anticipated results; will identify and analyze problems, delays and other issues related to project implementation; and will document lessons learned.

C.5 TARGET STAKEHOLDERS

The target audience for this assessment is USAID/Kosovo and the Government of Kosovo, specifically the Ministry of Education, Science and Technology. Other stakeholders include USAID’s implementing partners, other international donors working in the education sector in Kosovo (such as the European Commission, World Bank, UNICEF, and GIZ), and local stakeholders and beneficiaries including Municipal Education Departments, , Kosovo primary school students, teachers and school administrators.

C.6 SCOPE OF WORK

The Contractor shall design and execute the evaluation to generate detailed knowledge about the performance of the BEP project, to measure accountability, project outcomes and benefits.

The Contractor shall disaggregate collected data by sex to the greatest extent possible in order to ascertain how the project impacted teachers, administrators, and students; how the activities affected the status and roles of women and men, girls and boys within the areas of intervention (for example roles in decision-making); how results of the work affected men and women differently; and what specific benefits of the program can be uniquely and specifically attributed to targeting women.

USAID/Kosovo will provide the Contractor with key documents and background material relevant to Kosovo’s education sector and the applicable USAID design and project documentation, as well as any available documents deemed necessary to the Contractor to be familiar with the BEP activities. Key documents include:

• BEP SOW

• Quarterly and Annual Reports, Work Plans, Activity Monitoring Plans

• Kosovo Education Strategic Plan,

• Project developed report assessment and surveys, such as initial, midterm and final reports of early grade reading assessment

All above reports will be made available to the contractor immediately upon award. Many relevant documents can also be found at the following link https://www.usaid.gov/kosovo/newsroom.

Since the project has ended, USAID cannot confirm the availability of project records beyond those listed above. However, USAID will request FHI 360 to make project records and personnel available to the evaluation team.

C.7 EVALUATION QUESTIONS

The Contractor must address the following key questions and may include others as necessary to meet the objectives of the evaluation. In addressing all evaluation questions the Contractor will do so in a manner and order that it determines to be most effective, efficient, and encompassing of all relevant stakeholders:

• To what extent has the program met its three stated objectives and what were the results ?

• What is the current situation of the Kosovo Pre-University Education System and to what degree did it benefit from the implementation of BEP’s three (3) program assistance components?

• Based on the review of BEP’s implementation and results, what recommendations are there for possible future USAID programming and/or other donors or governments in improving Kosovo education system? What recommendations are there for supporting the implementation of the reform process?

• To what extent did BEP interventions contribute to improvements in the reading abilities of Kosovo students. ?

https://www.usaid.gov/kosovo/newsroom

• To what extent are BEP’s achievement and results sustainable? How much are the counterparts taking on the responsibility to continue the work? What can be done to better ensure sustainability of project interventions?

C.8 METHODOLOGY

The Contractor shall design and execute an evaluation to generate detailed knowledge about the magnitude and performance of the BEP project and measure accountability and benefit. It is anticipated that the evaluation methods will include and rely on a mixture of methods, including documentation review, small surveys, and in-person or telephone interviews with key informants in the U.S. and in-person interviews in Kosovo. The Contractor will review all of the available documents made available by USAID/Kosovo prior to arrival in country. Upon review of the documentation, the Contractor will develop an evaluation framework (including a draft evaluation Work Plan) that is most appropriate and feasible to accomplish the goals outlined in the Scope of Work. In considering the evaluation design, the Contractor will incorporate diverse information gathering approaches in order to reach the widest possible sample of the main target audiences.

In preparing a data-gathering approach, questions should be tailored to reflect, as appropriate, the specific roles of the stakeholders. The data analysis plan will include how interview and/or focus group interviews will be transcribed and analyzed; what procedures will be used to analyze quantitative data from surveys and qualitative data from key informant and other stakeholder interviews; any methodological limitations;

and how the evaluation will weigh and integrate qualitative data with quantitative data. All data will be disaggregated by sex, as appropriate.

Data collection and analysis

USAID requests that the evaluator complete the following table as part of its detailed design and evaluation plan.1

Evaluation question Data source Data collection method (including sampling methodology, where applicable)

Data analysis method

Scheduling and period of performance Anticipated start date would be mid-October. Anticipated time for fieldwork in Kosovo is approximately three weeks. A six-day workweek is authorized while performing fieldwork in Kosovo.

1 Another format may be used if the table is not preferred, but any chosen format should contain all the information specified for each question.

C.9 DELIVERABLES

1. Work Plan – A Work Plan for the evaluation must be completed by the Contractor within 10 working days of the award of the contract and presented to the M&E Specialist and Contracting Officer’s Representative (COR) for approval prior to starting any data collection.

The evaluation design will include a detailed evaluation design matrix, draft questionnaires, other data collection instruments and known limitations to the evaluation design. The Work Plan must include the anticipated schedule and logistics, and delineate the roles and responsibilities of members of the evaluation team.

2. Initial briefings. The Contractor will meet with USAID/Kosovo upon arrival in Kosovo to review the objectives of the evaluation. The team is expected to be in country for three weeks.

3. Preliminary Draft: The team will submit a preliminary draft report and a presentation for the debriefing to the Mission M&E Monitoring and Evaluation (M&E) Specialist and to the USAID/Kosovo Economic Growth Office team at least one working day before Mission debriefing.

4. Debriefing with USAID: The team will present the major findings of the evaluation to

USAID/Kosovo after submission of the preliminary draft report and before the team’s departure from country. The debriefing will include a discussion of achievements and issues as well as any recommendations the team has for possible modifications to project approaches, results, or activities. USAID will submit written comments within five working days after debriefing.

5. Final Report: The Final Report will be provided to the USAID/Kosovo Mission Monitoring and Evaluation Specialist in electronic form within five (5) business days following receipt of comments from USAID. The Contractor is expected to send a revised version of the final report within 5 working days after receipt of comments from USAID/COR. The report shall include an executive summary and not exceed 30 pages (excluding appendices). The executive summary should be 3-5 pages in length and will summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The report shall follow USAID branding procedures.

The annexes to the report shall include, at a minimum:

• The Evaluation Scope of Work

• Any statements of differences” regarding significant unresolved difference of opinion by funders, implementers, and/or members of the evaluation team;

• All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides;

• Sources of information, properly identified and listed; and

• Disclosure of conflicts of interest forms for all evaluation team members, either attesting to a lack of conflict of interest or describing existing conflict of interest.

C. 10 TEAM COMPOSITION

The team shall be composed of two international subject area experts and two local Kosovar subject area experts. All fours embers shall be considered key personnel.

Evaluation Team Leader (Senior): A relevant education and experience in evaluating (preferably education programs). S/he should have a minimum of 10 years’ experience in project evaluation shall have a preferably working knowledge of basic education in developing countries. The individual shall have demonstrated excellent written and oral communication skills, as well as exceptional organizational and analytical abilities.

Basic Education Analyst (Senior): A relevant education in the area of Basic Education is required. S/he should have knowledge of basic education system reforms, and should have at least 7 years of experience in evaluation and analyses of project in the education sector.

Basic Education Analysts (2 – Local): Both Analysts should have knowledge of basic education system reforms in Kosovo, and should have at least 5 years of experience in implementation, development and/or evaluation of projects related to the education sector. They must have proven ability to collect, process, analyze, and report data collected during the evaluation process.

The team will develop and adopt an approach that elicits and analyzes information, provides key findings, conclusions, recommendations and lessons learned.

C. 11 RELATIONSHIPS & RESPONSIBILITIES

In accordance with USAID Policies, this task order will be managed by the USAID/Kosovo Program and Project Office. Primary point of contact is Melita Cacaj, Monitoring and evaluation Specialist, Program and Project Office mcacaj@usaid.gov.

The Mission’s Monitoring and evaluation (M&E) Specialist, or his/her designee, will be the designated Contracting Officer’s Representative (COR) for this award. Upon arrival in Kosovo, the Contractor shall meet with the M&E Specialist and representatives from the USAID/Kosovo Democracy and Governance Office prior to starting any work.

C.12 ADDITIONAL REQUIREMENTS

An acceptable report will meet the requirements as per USAID rules and procedures. The following considerations should also be included:

• The evaluation report should represent a thoughtful, well-researched and well-organized effort to objectively evaluate what program activities were most successful in achieving the desired results, what did not work and why;

• The evaluation report should address all evaluation questions included in the scope of work;

• The evaluation report should include the scope of work as an Annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline shall be agreed upon in writing by the USAID Mission M&E Specialist;

• Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex to the final report;

• Evaluation findings will assess outcomes and impacts using gender disaggregated data.

mailto:mcacaj@usaid.gov

• Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparative groups, etc.);

• Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions;

• Findings should be specific, concise and supported by strong quantitative or qualitative evidence;

• Sources of information need to be properly identified and listed in an Annex, including a list of all individuals interviewed;

• Recommendations need to be supported by a specific set of findings; and

• Recommendations should be action-oriented, practical and specific, with defined responsibility for the action.

All quantitative data collected by the evaluation team must be provided in an electronic file in easily readable format agreed upon with the COR. The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed.

[END OF SECTION C]