RFI_EDU_EVALUATIONS.docx

MERGEFORMAT

REQUEST FOR INFORMATION

Date of Issuance: November 29, 2017

Reference No.: AID-RFI-608-18- 000001

Activity Title: Performance Evaluation of the Reading For Success (RFS) Project and Midterm Evaluation of RFS-National Program for Reading (NPR) Activity

Deadline to Submit Responses: December 15, 2017; 12:00 PM, Rabat local time

The United States Government, represented by the Agency for International Development (USAID) Morocco is seeking comments on a draft Statement of Work (SOW) for the Evaluation of a planned activity under the Reading For Success Project and the Midterm Evaluation of RFS-National Program for Reading (NPR) Activity.

PURPOSE AND INSTRUCTIONS: The purpose of this RFI is to solicit and obtain input, feedback, questions and suggestions from interested parties on the attached draft Statement of Work.

Interested parties are requested to submit their comments, questions of the draft Statement of Work to the Senior Acquisition and Assistance Specialist for USAID/Morocco, Najia Mesfioui, at nmesfioui@usaid.gov and Craig Smith, Contracting Officer for USAID/Morocco, at csmith@usaid.gov .

Information should be submitted electronically by email in Microsoft Word. The Government is not responsible for any format changes due to the use of a newer version of Microsoft Office.

Responses are due no later than December 15, 2017, 2:00 PM, Rabat local time.

Please include a cover letter with contact information. Do not submit any proprietary information. Interested parties are encouraged to request clarifications of USAID/Morocco as part of their submissions. Questions and answers of general interest and applicability will be posted with the RFP. It is anticipated that the RFP will be posted on or about January, 2018 on www.fbo.gov.

THIS IS NOT A REQUEST FOR PROPOSALS. This RFI is for planning purposes only. The information provided in the RFI is subject to change and is not binding on the Government. Issuance of this notice does not constitute any obligation on the part of the Government to procure supplies or services or to issue a solicitation. In addition, the Government will not pay direct or indirect costs or charges that arise as a result of contractor submission of responses to this RFI. The release of this RFI shall not be construed as a commitment or as authorization to incur cost for which reimbursement would be required or sought. All submissions become Government property and will not be returned.

Thank you for your interest in USAID’s programs and we look forward to your comments.

Sincerely, //s// Craig Smith Contracting and Agreement Officer Middle East Regional Platform/MERP

Attachment: Draft Statement of Work

STATEMENT OF WORK

Performance Evaluation of the Reading For Success (RFS) Project and Midterm Evaluation of RFS-National Program for Reading (NPR) Activity

I. PURPOSE OF THE EVALUATION

The USAID/Morocco Education Office seeks to procure services of an Evaluation Firm to conduct two evaluations under one mechanism: 1) A performance evaluation of the Reading For Success (RFS) project; and 2) A midterm performance evaluation of the activity “National Reading Program” (RFS-NRP).

The goal of the performance evaluation of the RFS project is threefold: 1) To assess the extent to which all the activities under the RFS project together contribute to achieving the Education Development Objective; 2) to inform the implementation other ongoing or upcoming activities, including that of the National Program for Reading; 3) to inform the next Country Development Coordination Strategy design. The results of the RFS evaluation will be used by the USAID/Morocco Education Office to inform the design of the new strategy and make any adjustments needed to improve the implementation of the National Program for Reading and other activities, including those related to deaf education. The results will be shared with broader USAID audiences, including USAID/Washington and other missions, to highlight best practices and lessons learned, and to inform upper management of the results achieved by the Morocco mission in basic education and early grade reading. Finally relevant results will be shared with Government of Morocco (GOM) counterparts and other donors to further promote improving basic education in Morocco through enhanced reading instruction in early grades.

The goal of the RFS-NPR midterm performance evaluation is to determine if the activity design is addressing the identified needs, and assess how well the activity is being implemented to meet those needs. The result of the RFS-NPR evaluation will be used by the USAID/Morocco Education Office and the activity implementer to strengthen the activity design, implementation, and management.

II. SUMMARY INFORMATION OF ACTIVITIES FOR RFS PROJECT

Contract #

March 22, 2018 4 regions: Rabat-Kenitra; Oriental; Souss-Massa; Fes-Meknes

III. BACKGROUND

A. Description of the Problem and Development Hypothesis(es) Problem Statement: In the past decade, Morocco has made significant gains in making primary schooling nearly universal with net enrollment rates in grade one surpassing 97%. However, in terms of learning achievement, Morocco lags behind other lower middle income countries, and has nearly the lowest overall literacy rates in the MENA region. The youth literacy rate is 87% for males and 72% for females[footnoteRef:1]. These numbers hide significant disparities in literacy between rural and urban areas in Morocco, with a staggering illiteracy rate of over 51% among rural girls[footnoteRef:2]. Based on enrollment and literacy data, the United Nations Development Program (UNDP) considers Morocco to be 15 years behind Tunisia, 10 years behind Algeria, and 5 years behind Egypt in terms of overall educational outcomes[footnoteRef:3]. [1: UNESCO 2011, including the Education for all 2000 Assessment.] [2: Direction de la Lutte Contre l’Analphabetisme (DLCA), Ministry of Education, 2011. ] [3: World Bank Report No: 52749-MA, 20 April 2010.]

On international assessments, Moroccan students rank among the lowest achievers. In the 2011 Progress in International Reading Literacy Study (PIRLS) Moroccan fourth graders ranked last among the 48 participating countries, with an average scale score of 310 (PIRLS scale centerpoint is 500). In 2007 and 2011, Morocco ranked eighth out of the nine participating Arab countries, in the Trends in International Mathematics and Science Study (TIMSS)[footnoteRef:4]. [4: PIRLS and TIMSS 2011 Reports.]

A National Analysis of Student Performance (PNEA) conducted by the Higher Council of Education (a research and watchdog organization reporting directly to the King) showed that only 24% of fourth graders and 36% of sixth graders possess the basic reading skills mandated in the national curriculum. The study showed that less than 32% of students could read at grade level. At all four grade levels examined in the study, girls outperformed boys in reading but not in the other subject areas. It is hypothesized that girls seemed to better master the various reading sub-skills than their male peers. This national study associates the causes of poor levels of literacy to a multitude of factors including a weak reference framework for reading skills (standards and benchmarks), a rigid and overloaded curriculum that does not respond to children’s needs, inadequate reading assessment tools, a school environment not conducive to learning, and the persistence of inequality in access to school and education opportunities[footnoteRef:5]. [5: PNEA Report, May 2009, Higher Council for Education.]

In 2011, USAID funded an Early Grade Reading Assessment (EGRA) in the Doukkala Abda region. The EGRA mirrored previous findings, and showed that only 34% of students in grades two and three read well enough to fully comprehend a grade-two level text. Boys and girls equally lack basic letter naming skills and are not able to sound out simple words, and 33% of second graders and 17% of third graders (21% for girls and 29% for boys both grades combined) could not read a single word of text in Classical Arabic, the language of instruction in the formal education system[footnoteRef:6]. Only 2.5% of students tested responded correctly to five of the six reading comprehension questions on the EGRA. The assessment showed that factors inhibiting student performance are multifaceted and complex: [6: EGRA/EGMA/SSME Report, RTI, 2011.]

· Inadequate levels of teacher preparation: Both the Snapshot of School Management and Effectiveness (SSME) component of the study and a recent review of the pre-service teacher education curriculum showed that Moroccan teachers receive little to no specialized training in reading instruction and assessing students’ reading abilities, leaving them ill- equipped to help children excel in the early grades. The absence of supplemental reading material for all students at school severely constrains the overall learning environment.

· The absence of reading materials in the home environment: Many children do not experience a print-rich environment at home. It is well-documented that children whose family members read to them and who grow up in homes where reading is valued do better in school. The SSME showed that few grade two and three students read outside the school environment. Only 9.5% of students reported using their textbooks at home; an equal percentage reported reading at home every day. Over 52% of students reported never reading aloud at home, and 32% reported sometimes reading aloud at home.

· The lack of parental involvement in student and school performance: While parents reported meeting with teachers at least once a year, teachers largely reported parental involvement to be insufficient.

· The language context in Morocco adds further complexity to the learning environment. Classical Arabic is the official language of instruction. However, children’s first exposure to Classical Arabic happens at school. Moroccan students’ mother tongue is either Arabic Dialect (Darija) of one of the Amazigh Dialects.

· The EGRA report showed that there are only minimal achievement differences between boys and girls in the early grades. However, gender disparities in learning outcomes seem to progress as students get older and as cultural beliefs about gender roles particularly in rural areas are reinforced. The primary school dropout rate is higher for girls (2.7% compared to 1.2% for boys in 2012-2013). Girls begin to drop out in the fourth grade (1.8% for girls and 1.3% for boys), and their rate of dropout increases (9.9% for girls and 2.1% for boys in grade 6)[footnoteRef:7].The repetition rate in primary school was 10.5% with 12.6% for boys and 8.2% for girls in 2012-2013[footnoteRef:8]. The data show that gender disparities exist, with more boys repeating grades in primary school and more girls permanently leaving school at a young age. Although several factors explain why students drop out, numerous studies[footnoteRef:9] have found that the most effective way to reduce the number of children who will ultimately drop out is to provide the best possible classroom instruction from the beginning of their school experience through the primary grades. The road to academic success starts early in the education system. Early grade reading proficiency has been cited as indicators of future academic success. [7: Id.] [8: Id.] [9: Ekstrom et al., 1986 : Who drops out of high school and why?; Goldschmidt & Wang, 1999: When can schools affect dropout behavior? ; Rumberger, 1995: Dropping out of middle school: A multilevel analysis of students and schools. American Educational Research Journal, 32, 583-625; Swanson & Schneider, 1999: Students on the move: Residential and educational mobility in America's schools. Sociology of Education, 72, 54-67; Wehlage & Rutter, 1986: Dropping out: How much do schools contribute to the problem? Teachers College Record, 87,374-392]

· Reading is a foundational skill and the basis of most other academic learning. Thus, it is crucial to ensure that students acquire good reading and comprehension skills in the first three grades so that they have a more equal opportunity to master other skills and knowledge[footnoteRef:10]. The gap between poor and good readers continues to widen as they advance through primary and into middle school. Children who do not read well at the primary level are often on a lifetime trajectory of limited educational progress and therefore limited economic and developmental opportunity. Having all children reading well and completing primary school is foundational for a more equitable society and lower disparities in human and economic development across Morocco. Struggling readers will experience difficulty in later years and, because catching up is nearly impossible without a strong remedial support system, may opt to drop out instead. [10: Moats, 1999, Teaching Reading is Rocket Science: What Expert Teachers of Reading Should Know and Be Able to Do, American Federation of Teachers.]

At the primary level, 90% of students who start grade one complete the primary cycle (89% for girls)[footnoteRef:11]. However, primary school dropout begins, in earnest, starting at grade four, and proceeds to the point where only 18% of a cohort of first graders will reach Grade 12. Despite dropout prevention efforts, over 300,000 children and youth under the age of 15 leave the formal school system every year. Approximately 10% of primary school enrollment in 2012- 13 were students repeating at least one grade. Two-thirds of primary school repeaters were boys. While the MOE data does not indicate the numbers of students repeating a grade for the second time, officials in the School Dropout Prevention program in the MOE state that students who have repeated two grades or more are twice as likely as one-time repeaters to drop out of school completely. [11: Persistence to last grade of primary is the%age of children enrolled in the first grade of primary school who eventually reach the last grade of primary education. The estimate is based on the reconstructed cohort method. Source: United Nations Educational, Scientific, and Cultural Organization (UNESCO) Institute for Statistics.]

There is growing evidence that the quality of service delivered is an important factor in enrolling and keeping children in school. A study[footnoteRef:12] on grade repetition in primary schools shows that in order to effectively reduce grade repetition, reforms need to tackle the reasons behind poor attendance, the quality and relevance of teaching, and provide sufficient support for teachers. [12: Jonathan Penson, “Grade repetition in primary schools in Sub-Saharan Africa: An evidence base for change,” February 2008.

The Reading for Success project, a collaboration between USAID and the Ministry of Education, is designed to test new approaches to confront the persistent low reading scores, and the accompanying low levels of overall academic achievement.

B. Description of the Strategy/Project/Activity/Intervention to be Evaluated

The Reading For Success (RFS): The Reading for Success project is designed to enhance educational attainment for Moroccan children at the primary level. Specifically, the project supports the Ministry of National Education, Vocational Training, Higher Education, and Scientific Research (MOE) in its curriculum reform initiative by developing instructional reading materials in Arabic for grades 1-4, including students with disabilities; by providing training and coaching to teachers; and by implementing a community mobilization approach tailored to support student reading. In addition to developing and implementing a new reading instruction approach, a human and institutional capacity assessment is also necessary to help identify the Ministry's needs to carry out their reform effectively.

Conceived as a learning activity, the project works with Government of Morocco (GOM) counterparts to develop an evidence base for the design of effective project interventions aimed at improving reading skills, both in targeted primary schools and in non- formal education settings. The project supports a principal objective of USAID/Morocco’s Country Development Cooperation Strategy for FY 2014-2018, “Enhanced educational attainment for children at the primary level,” as well as the Agency-wide Education Strategy. The project comes at a time when the Ministry of Education (MOE) is developing a significant education reform, including a National Strategy for Reading. The ministry has specifically asked USAID to be its lead international partner to help find effective approaches at the community, school, and classroom level to improve the reading skills of Moroccan children. The project will be implemented over a five-year period (FY 2014-2018) at a total estimated cost of approximately $25 million.

The specific activities outlined below have been approved as part of the RFS Project and are designed around three principal streams as outline below.

· Improved Reading Skills in Grades 1-4

· Increased Community Support for Reading

· Improved Policy Environment to Support Reading

Reading for Success - SMALL SCALE EXPERIMENTATION (RFS-SSE), implemented by Chemonics: The activity was awarded as a 30 month Task Order (September 23, 2015 to March 22, 2018) with a total estimated cost of $5,820,752. The objective is to support the Ministry in its curriculum reform initiative by developing reading instructional materials in Arabic for grades 1-2; providing training and coaching to teachers; and, implementing a community mobilization approach tailored to support student reading. In its first year, RFS-SSE focused on Result 1 – Developing Arabic Reading Materials for Grades 1 and 2. Based on international best practices and Moroccan expertise, RFS-SSE introduced the phonemic method of teaching, which uses individual syllables and sounds as building blocks for basic reading skills. RFS-SSE engaged with CSOs through small grants to design and implement summer reading activities. The CSOs engaged students from the target provinces and schools in the reading activities. Chemonics tested via EGRA the impact of these summer programs on reducing students’ reading loss over the summer.

Reading for Success - NATIONAL PROGRAM FOR READING (RFS-NPR), implemented by Creative Associates International, Inc (CAII). This is a $19.2 million five-year activity which aims to build on the data, evidence and lessons learned from the experimental reading activity (RFS-SSE) to further improve reading instruction and learning materials; expand the implementation of reading instruction practices to grades 3-4; and support the Ministry in developing a new Arabic language curriculum (grades 1-4) that the Ministry will scale up nationwide. The National Program for Reading will:

1. Improve reading and writing materials used in primary schools: The materials will be aligned to an updated and improved Early Grade Reading and Writing (EGR/W) curriculum with clear standards, benchmarks and appropriately sequenced content. Textbooks will be aligned with the new EGR/W curriculum and will meet expectations for usability, complexity/readability, content and quality. In addition, the program will support the MoE to develop Arabic Language Curriculum and Pedagogical Guide in the Early Grades and establish Arabic language benchmarks with an objective method for monitoring progress of children throughout the year and governments with a method to communicate performance standards.

2. Improve reading and writing instruction in primary schools: NPR is working to develop and deliver an improved training program encompassing extensive preservice and in-service training.

3. Expand Reading and writing enrichment programs: In addition to improved reading and writing materials and effective instruction, students need access to reading and writing enrichment opportunities outside of formal school hours and in the home. The core package will offer enrichment reading clubs that meet multiple times a week after school, tutoring, one-on-one support, a buddy system and more.

4. Improve learning and assessment systems: to support continuous learning and improvements to the curricula, teaching methods, teacher training, reading and writing materials, and enrichment programs requires a cohesive learning and assessment system. The system will provide Creative, USAID, MOE, teachers, administrators, and others to identify blockages, assess effectiveness, and generate evidenced to design, refine and improve EGRA/W package components. The system will include a research agenda, implementation of a series of evaluations, assessments, analytical tools and participatory workshops. This will allow MOE to institute an adaptive management approach to inform program revision, support scale-up, and create a support policy environment and accountability system.

In addition to these goals, and as a cross cutting task, RFS-NPR is building the capacity of the Ministry of Education for sustained implementation. Local capacity building at every level is critical for program implementation, and to sustain the ongoing use of evidence-based approaches after the activity ends. Program implementation is designed to support the role of the Ministry of Education and build its capacity at the central and provincial levels, employing innovative and practical modalities, strategies and activities that will reinforce teacher knowledge and practices that ensure the success of the program beyond the life of activity. Capacity building, which is one of the indispensable elements for ensuring country ownership and responsibility, can be only reached if the activity is carried out hand-in-hand with MOE key officials at all levels. In this sense, RFS-NPR capacity building approach goes beyond simple organization of and participation in capacity building workshops and trainings. RFS-NPR is seeking ways of providing the MOE with capacity building opportunities in real settings and in a practical and comprehensive manner. RFS-NPR will develop a milestone plan with the MOE and use this, alongside regular feedback loops, to ensure the effective delivery of this component and sustainability of results. In addition to hands on learning, RFS-NPR may include partnerships with external entities, such as national and international universities, to provide MOE officials and technical staff with focused and intensive training (including training with certification) in key areas such as early grade reading and child literacy, curriculum development, teacher training, coaching, assessment, and other.

Reading For Success - IMPROVING DEAF CHILDREN’S READING THROUGH TECHNOLOGY (RFS-IDCRT): This activity was awarded to Chemonics in October 2015, with a total estimated cost of $1,563,935 and a revised end date of October 2018. The objective of this activity is to improve the availability and quality of reading materials and instruction available for deaf children and ultimately to improve their literacy through: the development of software that helps teachers and parents easily create and publish Moroccan Sign Language-supported educational materials; training of teachers; and assessment (Early Grade Reading Sign Language Assessment, EGRSLA). A cost extension of the Cooperative Agreement was signed in April 2017 extending the project until October 2018 and increasing the budget to accommodate the added activities. These activities primarily cover another year of EGRASLA testing, more teacher training, and the expansion of MSL Clip and Create to include more functionality and database expansion, including Chemonics’ first and second grade national curriculum words, and regional variations.

Reading for Success - HUMAN AND INSTITUTIONAL CAPACITY DEVELOPMENT (RFS-HICD): This $2,112,724 award to The Kaizen Company aims to ensure that USAID and the Ministry have a full understanding of the human and institutional challenges that may hinder the implementation of the Ministry’s reform efforts, and that the Ministry would receive a package of actionable solutions to face identified challenges. The purpose of the HICD Analysis of Morocco’s MOE is to support the implementation of the 15-year reform entitled “Vision 2015-2030”, especially the components that will be supported by both USAID/Morocco and the Millennium Challenge Corporation (MCC). The activity was procured in June 2016 and an end date of June 30, 2017.

C. Project/Activity Monitoring, Evaluation, and Learning Plan (PMELP/AMELP) Reading for Success - SMALL SCALE EXPERIMENTATION (RFS-SSE): The purpose of this MELP is to collect data, and assess accurately the progress of RFS-SSE towards its goals. The RFS-SSE team uses the collected data to measure the success of project efforts to identify first and second graders’ reading competencies via EGRA and SSME tools in order to provide data that will assist in identifying interventions to improve reading skills as well policy environment to support reading. The MELP describes the RFS-SSE project methodology for activity monitoring and evaluation and is used to manage the collection of performance data. A performance indicator summary table includes performance indicators, data sources and collection methods, timing of data collection, disaggregation, and units of measure. The results are used to regularly update and engage relevant stakeholders, including USAID, to analyze progress and quality.

In addition to the MELP, both DevResults (Chemonics) and TraiNet (USAID) are used to store the collected data. The DevResults Monitoring and Evaluation (M&E) software are used for the project’ M&E data collection, storage, management, and reporting. Its web-based interface can be used from any internet-enabled device (tablet, mobile phone, desktop). Where internet access is unavailable or inconsistent, data can be captured via formatted Excel spreadsheets and other offline tools and later uploaded. Features include Geographic Information System (GIS) functions, data visualizations, results dashboards, and document and photo storage capabilities. All documents will be provided upon award by the Education Office and Chemonics.

Reading for Success - NATIONAL PROGRAM FOR READING: RFS-NPR employs several highly consultative and participatory research and evaluation methods to test activities and support scale-up. The “stepped” approach (1) tests the effectiveness and viability of the EGR/W instructional and enrichment interventions in schools; (2) provides the opportunity to identify and alleviate policy, procedural, and resource blockages in the delivery and support/ supervision systems at local, provincial, and regional levels; and (3) supports the MOE roll-out of its EGR/W program nationally. With the Assessment Quality Assurance Working Group (AQA-WG), led by CNEE, Creative is developing a research agenda, reviewing SSE results, detailing key research questions and identifying the types of studies required. The agenda includes methods, sampling, and periodicity. The research agenda should be updated annually to reflect emerging areas of inquiry. Some of the research questions that RFS-NPR may address include:

· What are the major policy factors that significantly contribute to or inhibit the project effectiveness in the Morocco context?

· What should successful teacher profile for teaching reading look like and how should teachers be supported in terms of training, in-school support and community support?

· Which one of the four enrichment models that significantly contribute to student reading outcomes and to what extent?

· What types of correlations exist between teachers’ classroom practices and student reading outcomes and to what extent?

Reading For Success - IMPROVING DEAF CHILDREN’S READING THROUGH TECHNOLOGY (RFS-IDCRT): The proposal for Improving Deaf Children’s Reading Through Technology (IDCRT) was written using a logic model structure. It contains hypotheses, goals, objectives, tasks, inputs, outputs and outcomes. These are incorporated both into the body of the proposal and into the Life of Activity Performance Plan. The purpose of this structure is to clearly set the course of the project’s implementation, explicitly define those measures and products that provide evidence of completion and quality, and to monitor progress on an ongoing basis so as to provide for timely corrective actions and sustaining efforts that bolster positive outcomes.

Monitoring of project implementation is specifically addressed within Objective 12 of IDRT’s workplan, is highly structured, and has a number of measures incorporated to facilitate month-by-month analysis and assertion of compliance. Each of the 48 tasks is tethered to the timeline, so that at any given moment it can be ascertained whether the project is staying on task and on time.

Reading for Success - HUMAN AND INSTITUTIONAL CAPACITY DEVELOPMENT (RFS-HICD): A monitoring and evaluation plan is not applicable to this activity.

IV. EVALUATION QUESTIONS

The evaluation of RFS and of RFS-NRP will investigate research questions in three broad categories: 1) Implementation progress, 2) Project management, 3) Lesson learned and sustainability. Please note that the questions and sub-questions below are illustrative and will need to be further developed and finalized in collaboration with USAID prior to the finalization of the evaluation design.

RFS Project Draft Evaluation Questions Project Implementation:

1. To what extent did project activities contribute to the reading and literacy goals and objectives of RFS (Why/why not?)

a. How and why did students under the RFS-SSE activity realize the observed learning results, if any?

b. How and why did students under the RFS-IDCRT activity realize the observed learning results, if any?

c. What were the strengths and the gaps of the project activities in contributing to the objectives of RFS?

2. What key lessons and recommendations in reading and literacy can be drawn from the successes and limitations of RFS project activities in order to more effectively achieve the Education Development Objective: “Enhanced Educational Attainment for Children at the Primary Level”?

Project management:

3. Have the reading and literacy project components and their activities been well managed at the national, regional and local levels?

4. Were the roles and responsibilities clear to key stakeholders and where they carried out correctly?

5. How did the MOE’s lead role in the reading and literacy project activities influence the implementation, management, and results of project activities?

Lessons Learned and Sustainability:

6. To what extent did the project activities contribute to establishing a strong and sustainable national system for reading and literacy?

a. To what extent (and how) has the ministry taken over responsibilities of project activities and ownership of the national system for reading and literacy?

b. How did RFS activities intersect with, provide learning about and move the dial on each of the 5Ts of reading instruction?

7. What can be learned from the project activities that can help inform Morocco's new CDCS?

a. Which populations of children currently in (or not in) the educational system are not benefiting from the current RFS project activities and how can they be better served in the next CDCS?

b. How can the Development Objectives in the current CDCS be better linked and integrated in the next CDCS in order to target populations of children not benefiting from the current RFS project activities?

RFS-NPR Midterm Performance Evaluation Questions Implementation progress:

1. To what extent did RFS-NRP build on the data, evidence and lessons learned from RFS-SSE and how was RFS-NRP useful in strengthening the design, implementation and management of RFS-NRP?

2. To what extent is the activity on track to meet its overall objectives by the end of the agreement?

a. How and why are students under the NPR activity realizing the observed learning results, if any?

b. What are the strengths and the gaps of the project activity in reaching its objectives?

c. Have girls and boys benefited from the activity differentially? If so, how?

3. What key lessons and recommendations can be drawn from completed reading and literacy tasks during the past school year that can help inform the rest of the implementation of NPR?

4. To what extent is the activity effectively using activity-level M&E data to make necessary and relevant adjustments to program design, implementation and management?

Project Management:

5. Is the structure and the skill mix of the activity team as it stands adequate to effectively and efficiently fulfill the required tasks?

6. Is the activity team working well with local counterparts and USAID?

7. Are the roles and responsibilities clear to key stakeholders and are they carried out correctly?

Lessons Learned and Sustainability:

8. What are the prospect of sustaining the activity benefits in the long run?

9. What can be learned from the activity that can be useful for the development of future reading and literacy activities?

V. EVALUATION APPROACH

In the subsections below, general guidance applicable to both evaluations is provided on methodology, sampling, data sources and collection, data analysis, and methodological limitations. In their responses, offerors should propose a specific evaluation strategy for each evaluation that specifically addresses the needs and requirements of each evaluation.

Evaluation Designs. For both the RFS project evaluation and the NPR midterm performance evaluation, the Evaluator is expected to employ a rigorous mixed methods evaluation approach to address the research questions outlined above. The overarching analytical strategy should be clearly summarized in an evaluation matrix that includes the following information: evaluation questions, sub-evaluation questions, indicators or assessment criteria, sampling/selection criteria, data sources and collection methods, and data analysis methods. The evaluation team, in collaboration with USAID, will finalize the overall evaluation methodologies for each of the two evaluations before fieldwork begins.

Sampling Strategies/Selection of Respondents. For qualitative samples, the Evaluator should carry out relevant data collection with a representative sample of purposively selected USAID project beneficiaries, partners, and host government counterparts at appropriate levels; and other key program stakeholders as appropriate. The Evaluator should propose a robust strategy for selecting respondents that will enable adequate geographical and language representation; cross-site comparison; and the capture a diversity of perspectives, among other key considerations. Criteria for sample selection should be developed by the Evaluator in consultation with USAID.

For probability samples, the Evaluator should present a strategy that is consistent with the evaluation needs. The description of the sampling strategy should include at least: 1) the anticipated approach to determine the sampling frame, 2) the stratification strategy, and 3) the minimum detectable effect calculations to determine sample size. The final sampling strategy will be determined by the Evaluator in consultation with USAID/Morocco and USAID/Washington.

Data sources and collection methods. The Evaluator should collect both primary data and secondary data. Primary data may include, but is not limited to surveys, key informant interviews, focus group discussions, and observations. For secondary data, the Evaluator should consider complementary sources of secondary data, such as M&E data, activity documents, census survey and/or school administrative data (e.g. test scores). Prior to any field work, the Evaluator is expected to carry out a targeted review and analysis of recent pertinent published literature on educational program evaluations in Morocco and MENA as well as relevant project documents and performance data provided by USAID to: 1) gain a better and more in-depth understanding of the implementation of the RFS project and, 2) assess the quality and relevance of the data to help answer the evaluation questions. The Evaluator is expected to collect at least one round of data collection and should outline a clear plan and specify which data exactly would be collected and why.

The Evaluator should use, whenever possible, pre-existing data collection instruments with documented strong measurement validity and internal reliability and with a suitable plan for cultural adaptation. For new instruments, the Evaluator should propose a plan for the development, cognitive testing, and piloting of the new data collection instruments. It should be noted that:

· Prior to beginning field work (including field testing), the evaluation team is required to share data collection instruments with USAID for review and feedback before they are used in the field.

· Issues of data confidentiality must be addressed as part of the IRB process.

· The use of electronic data collection for quantitative data is required.

· Data collection methods must take into account contextual factors to maximize data reliability and minimize unintended consequences.

The Evaluator is encouraged to propose innovative methods for tracking study participants as relevant and collecting reliable data on outcomes of interest over time. As part of the data collection process, the Evaluator should implement a rigorous participant tracking system that includes a unique ID for each participant for the purposes of tracking the progress/status of each participant during, throughout and after their engagement with the project’s activities.

Data Analysis. The Evaluator should analyze both primary and secondary data. All secondary data should be assessed for quality and relevance in answering the research questions. The Evaluator should provide a data analysis plan for the qualitative and for the quantitative data that directly addresses each evaluation question with specific methods for analyzing the data. As part of the plan, the Evaluator should specifically discussed how the qualitative and quantitative data will be weighted and integrated (whenever relevant) to address the evaluation questions.

Methodological limitations. As part of USAID’s evaluation policy, the Evaluator is required to discuss and document any issues potentially affecting the quality of evaluation data (including the data validity, integrity, timeliness, precision, and reliability) in the evaluation planning stage, and to be assessed on an ongoing basis during implementation. The Evaluation team and USAID should discuss all limitations and measures to address or overcome limitations in the implementation phase and these should be detailed in the reports. The evaluation reports should be clear and transparent about any notable limitations and if and how they may affect the evaluation’s findings, conclusions, and recommendations.

VI. EVALUATION TEAM COMPOSITION

The Evaluator must propose a staffing plan that fully supports the requirements of this SOW. The staffing plan should at the minimum include the three key positions, as described below, and specifically address how the proposed team members comply with the positions’ qualification requirements. As part of this response, the Evaluator must complete the table below and provide CVs in an annex for all team members proposed.

Additionally, the staffing plan must describe what each team members’ primary responsibility will be, how the team members will work together, and their anticipated LOE for the duration of the evaluation activities.

Evaluation Team Experience

(e.g. PI, etc.)

The Offeror is subject to FAR 9.5 Organizational and Consultant Conflicts of Interest. The Offeror must review the regulation and determine if there is a conflict of interest for individuals or the organization. If a conflict of interest exists, the Offeror must submit a Conflict of Interest Migitation Plan for the Contracting Officer to review and analyze.

Illustrative Positions Evaluation Team Leader (Key personnel): Working closely with the Statistician and Principal Investigator, the Evaluation Team Leader will be responsible for overall and day-to-day management of contract activities, involving multiple tasks across multiple locations. S/he is the principal interlocutor between the Contractor team members and USAID, and is responsible for the overall compilation of all deliverables under this contract. S/he is responsible for staffing, project planning, project financials, and staff direction and oversight, and maintains and manages the relationship with USAID. Collaborating closely with the PI and senior statistician, S/he will also lead the quantitative components of the evaluation, the quantitative data analysis, the cost analyses, and the reporting through reports, publications, journal articles, conference presentations, and social media.

· Master’s degree or higher in economics, statistics or related field.

· 3+ years of experience conducting quantitative research.

· Strong background and training in quantitative research methodologies

· Experience and knowledge of data analysis and data management techniques using STATA

· Experience managing and coordinating complex evaluations, including evaluations of international projects.

· Strong organizational, oral and written communication skills.

· Proven ability in writing scientific papers and outcomes research reports, and ability to use research findings to evaluate program implementation and operations.

· Candidates speaking French in addition to English preferred.

Principal Investigator (Key personnel): The Principal Investigator will be responsible for supervising the quality of the evaluation design and execution, including qualitative analyzing qualitative data identifying and disseminating findings, lessons learned, and evidence from evaluations.

· Master’s degree or PhD in sociology, anthropology, education or related field with relevant experience in ethnographic research.

· Minimum 8 years of experience working in development, preferably in education or another social sector; minimum 7 years of experience managing and conducting qualitative and mixed methods evaluations.

· Strong background and training in qualitative and ethnographic research methodologies.

· Experience managing or conducting evaluation of education and early grade reading programs (in the Moroccan context preferred)

· Experience conducting site visits, key informant interviews, and focus group discussions

· Experience analyzing data using qualitative analysis software such as N’Vivo

· Experience in successful dissemination, including but not limited to writing and publishing research reports for scholarly (peer reviewed) and non-scholarly audiences, delivering presentations and providing consultation to a diverse range of stakeholders, and effectively tailoring messages to the audience.

· Candidates speaking French required, and Arabic preferred.

Senior Statistician: Working closely with PI and TL, the statistician will provide technical guidance on the quantitative components of the evaluations, as necessary and relevant, including on: the quantitative components of the evaluation (and sampling frameworks), analysis of quantitative data and associated the cost analyses, and the reporting through reports, publications, journal articles, conference presentations, and social media.

· Master’s Degree, Phd in Economics, Statistics, or educational measurement

· 3-5 years of experience in monitoring and evaluation with experience in international settings required.

· Strong background and training in quantitative research methodologies

· Demonstrated technical competency in designing and implementing primary data collection and analysis.

· Strong analytical skills required. Experience in engaging program staff in interpreting and learning from findings is desirable.

· Ability to effectively coordinate multiple evaluations simultaneously, working well under tight time constraints and meeting deadlines with attention to detail and quality.

· Candidates speaking French in addition to English preferred

Research Assistant: The Research Assistant/Writing and Analysis will work with the Principal Investigator to perform a variety of tasks including, but not limited to cleaning and analyzing data, supporting field-based evaluation work, data analysis and interpretation, and support in writing of analytical reports, journal articles, and other dissemination efforts.

· Bachelor’s degree or higher in sociology, anthropology, education, or related field with relevant experience in ethnographic research, Master’s degree preferred.

· 3 or more years of experience working in development, preferably education or another social sector, preferably conducting qualitative research.

· Experience conducting site visits, key informant interviews, focus group discussions.

· Knowledge of qualitative data analysis software such as NVivo or ATLAS.ti.

· Strong qualitative, analytical, and writing skills.

· Candidates speaking French in addition to English preferred

DRAFT 2

image1.png