Attachment_J.16_Pre-Solicitation_Industry_Feedback_Q&A.pdf

Attachment J.16 – Pre-Solicitation Industry Feedback, Questions & Answers

USAID released an industry notice and request for information through www.fbo.gov on December 14, 2017. Below are responses to questions/suggestions raised.

A. GENERAL QUESTIONS

1. Resource Levels and Assumptions: XXX recommends that USAID provide further information about its budgetary assumptions, and specifically what MOGE resource responsibilities will be for the anticipated program. The draft SOO does not specify what

MOGE will finance, leaving open the question what will be the responsibility of the project to finance. The USAID business forecast indicates that the project value will be between $25 million and $50 million, and the IAP mentions $48 million. Given this funding level and the anticipated scale of the activity, a budget analysis shows a shortfall in funding, unless the MOGE underwrites significant inputs, such as provision of teacher and student materials, including teacher guides, student textbooks and supplementary materials. For example, at $3/student for materials for 1.4 million children, total costs would come to $4.2 million. Should some or all materials need to be replaced during the life of the project, costs will reach $13 million or more. Other significant project expenses will include transport for coaching visits and training.

Under USAID’s Read to Succeed, the MOGE’s inability to handle these expenses put the project outcomes at risk. Based on the information provided in the SOO about the numbers of schools, teachers, etc., the planned coverage of Let’s Read will be four times larger than RTS with only twice the level of resources.

USAID Response: MOGE is not expected to provide any resources. Based on lessons learnt from the previous projects, Let’s Read, Zambia will not depend on Government to provide teaching and learning materials. The contractor is expected to provide a complete package for reading instruction and practice and also ensure impactful reading results. The project cost will depend on the mix of interventions that the offeror proposes.

2. We appreciate USAID’s need to reach out globally for offerors of the LRZA. Is there an opportunity for potential offerors to sub-contract local firms? Such an opportunity might help to fast track mobilization and follow-up on issues of sustainability.

USAID Response: Yes, however USAID does not facilitate the process and each offeror is responsible to create their own team.

B. QUESTIONS RELATED TO THE INDIVIDUAL ACQUISITION PLAN

http://www.fbo.gov/

Attachment J.16

Pre-Solicitation Industry Feedback & Questions

1. The pre-solicitation document states that “Let’s Read, Zambia will promote reading and writing in one of the seven local official languages.” However the goals of the program are only to achieve success in reading and writing skills in the seven languages. Would USAID clarify if there a requirement for the children to be proficient in reading and writing in English as well as the local languages?

USAID Response: The focus of Let’s Read, Zambia will be mainly teaching reading in the local languages.

2. The risks for the LRZA IAP are well identified (section 1.7 of IAP). An important aspect of managing risk is creating space for acknowledging failure early on. Five years is a long time and contractors cannot stop actions, such as policy change, from happening. Besides, contractors have in the past concealed early failures in the hopes of sorting things out without being open with USAID. Certainly, the modest and uneven change in reading results shared in the IAP show that past USAID interventions may not have achieved compelling results. USAID recently revised the ADS 201 (201.3.1.2) Program Cycle Principles to address four key requirements of program design and management imploring its agents to: a) Apply Analytic

Rigor to Support Evidence-based Decision-Making; b) Manage Adaptively through Continuous

Learning; c) Promote Sustainability through Local Ownership; and d) Utilize a Range of

Approaches to Achieve Results. How is USAID envisioning the application of the Program

Cycle Principles?

USAID Response: USAID has and will continue to apply the Program Cycle principles referred in your question within design and implementation.

3. In the Individual Acquisition Plan released with the SOO, USAID notes that Let’s Read, Zambia will build on the work and experience of USAID/Zambia’s previous projects: Time to

Learn, Read to Succeed, STEP-Up, and Zambia EdData. USAID also indicates that its Country

Development Cooperation Strategy retains the same intermediate and sub-intermediate results as during the implementation of those four projects. While sub-intermediate result 3.1.4 –

HIV/AIDS Impact on Education Mitigated – is included in the results framework on page 1 of the SOO, references to this work are not included throughout the rest of the SOO. Can

USAID please confirm that the Let’s Read, Zambia Activity will not include focus on mitigating the impact of HIV/AIDS on education (which we expect would be done with PEPFAR funds rather than Development Assistance funds)?

USAID Response: Yes, Let’s Read, Zambia will not focus on mitigating the impact of

HIV/AIDS on education. Its focus is on improving reading levels among Zambian children.

4. How will the LRZA build on USAID Zambia’s past work with Read to Succeed, Time to

Learn and Step up Zambia (see 2.1.a of IAP)? It is important for USAID Zambia to show clearly what has worked well and vice versa. What are the key lessons learned?

USAID Response: Additional information has been provided in attachments.

5. Section 3.2 on market research indicates that local Zambian firms are incapable of implementing the LRZA. How is USAID envisioning the LRZA with regards the Implementation

& Procurement Reform (IPR) requirements? USAID Zambia must create space for local organizations to prove otherwise. There really is very little that is complicated about the LRZA request. Besides, the global companies are dependent on nationals to do the work. This includes the local CORs who do a lot of hand holding on technical matters and reporting, managing high rotations of underperforming COPs, and running interference with the MOGE.

USAID Response: USAID Zambia does not exclude local organizations from participating in procurements, on the opposite; we encourage it, particularly as subcontractors. Presently, USAID is exploring ways of building the capacity of local organizations so that they can participate and implement USAID procurements. In addition, through the previous USAID

Education activities, a few local organizations partnered with international organizations in the implementation of activities.

C. QUESTIONS RELATED TO THE STATEMENT OF OBJECTIVES

1. Paragraph 2 indicates achievement in math and reading, yet the IR3.1. box indicates reading only. Please confirm the intended focus of the project.

USAID Response: The intended focus of the activity is reading only.

2. The Objectives cited in section C.8 do not appear to reflect the CDSC Development

Objective 3 Results Framework presented in C.3. For example, SIR 3.1.3 (equitable access) and

SIR 3.1.4 (HIV/AIDS impact mitigation) do not figure in the objective descriptions. Please clarify to what extent, where, and how the SIRs are incorporated into the Objectives in C.8.

USAID Response: The Education office no longer has access to HIV/AIDS and Water for the

Poor funds. For that reason SIR 3.1.3 and SIR 3.1.4 are not addressed in Section C.

3. Please clarify how the Let’s Read project will align with the Catch-up program.

Specifically: Is it expected that Catch-up will expand to the same target provinces as Let’s Read?

Who will implement and fund this expansion? Is Let’s Read expected to do this? Or will the alignment be limited to the two provinces in which the Catch-up program currently operates?

Will Catch-Up be required to use Let’s Read materials and methods?

USAID Response: The Catch up program will complement Let’ Read, Zambia activity. Let’s

Read is focusing on pre-primary through grade 3, while the Catch- up program is focusing on grades 3 to 5 (supporting struggling readers). Funding for the Catch-up activity is separate and also the activity is in only two of the Let’s Read, Zambia target provinces, that is, Eastern and

Southern. The Catch-up is a remedial activity. However, with the overlap in grade 3, the Catch-

Up activity presents opportunities to provide a continuum of support to leaners, particularly those in the Let’s Read, Zambia target provinces. The Catch-up will not be required to use

Let’s Read, Zambia materials. But the two activities are expected to collaborate and share information and best practices. In addition, the Catch-Up does not replace the approved literacy curriculum but complements it by providing remedial support to struggling learners.

4. Please clarify how the target of 1.4 million children was calculated. Are they unique children? Does the number represent the current enrollment figure for the targeted grades, representing only one year or does it refer to the number of children that will flow through the grades over the life of the project?

USAID Response: It refers to the number of children that will flow through the grades over the life of the project.

5. Not all schools have Early Childhood Centers, yet the Activity means to ensure that

“pre-primary (kindergarten) students in the targeted provinces will develop oral language skills and emergent literacy skills to prepare them for successful transition to Grade 1.” Could

USAID please clarify if the Activity is to target all pre-primary students, even if there is no Early

Childhood Center connected to the school? What is the expected mechanism for this support, since the Activity is not required to establish Early Childhood Centers?

USAID Response: No, USAID expects the Contractor to reach children enrolled in ECE centers or annexes in the target public and community schools. There will not be any other mechanism to support children who are not captured in the target schools. Please note that there are other stakeholders that support ECE e.g. UNICEF and the partner is expected to work in collaboration with them.

6. The objectives in the draft SOO do not readily align with the project’s development hypothesis, presented on page 2, and do not reflect a project’s results framework. Could

USAID please clarify how the Intermediate Results derived from the development hypothesis

(i.e., teaching-learning materials in local language available; teachers trained, mentored and coached in appropriate instructional techniques; EGR assessment system established for data-driven decision-making; MOGE systems and capacity strengthened; and community and parental support for education increased) should be subsumed/organized and treated under the

Objectives.

USAID Response: The SOO directly addresses SIR 3.1.2 and SIR 3.1.1. The Results framework on page 2 of the SOO is an extract from the current CDCS which will be reviewed in the near future.

7. Performance Tracking System:

a. The purpose of the Performance Tracking System and its use are not clear. The proposition that USAID would review how each individual school is performing and intervene, either to improve performance or to enforce application, does not seem feasible. We suggest that the Performance Tracking System, which produces this level of detail, is best used by the school and district to ensure that schools are receiving the support needed and producing the outcomes expected. Moreover, community involvement often revolves around using local school assessments directly to improve student/school performance, increase transparency with community on these matters, and engage parents and community in enhancing children’s learning as it is relevant to actual assessed performance.

While there is utility in a standardized school-based testing system, the school-based performance tracking system is not a feasible means of obtaining accurate data to track Let’s

Read project progress (especially for Grade 3, as noted under Objective 3 on page 5) or to provide the precision and reliability the MOGE will need for national planning because:

Currently standardized tests do not exist for school use; teachers are charged with developing tests themselves, so results cannot be aggregated. It will take time for standardized tests to be developed, introduced, and used in the schools. It is unlikely that a school-based system can be developed and implemented in time for baseline purposes, especially in every school.

School-based assessments, if administered by school personnel, are subject to bias, either in terms of “upgrading” student scores or selecting better students for testing (if done on sample basis for individual student testing).

Relying on zonal- or district-based coaches to collect and/or upload data is a risky proposition, given the irregularity of visits, and would reduce the time they are able to spend on coaching support (rather than on data collection or entry).

The type of data required—correct words per minute—requires testing students individually, which is time-consuming and probably beyond the time available to teachers without seriously diminishing instructional time. For example, as a conservative estimate, 15 minutes per test for 60 students per class would consume 15 hours per term, which arguably is better spent on instruction. If teachers (or coaches) are expected to randomly select students, it is likely that bias will occur.

As the draft SOO points out under C.9 (Operational Constraints), distances and the lack of appropriate devices and connectivity in many schools will inhibit the timely submission of data.

Using two different systems and possibly types of tests for data collection (national assessment system for Grade 2 and a school-based assessment system for Grade 3) will not produce comparable results and allow for joint analysis.

Given the factors outlined above, sample-based tests, administered by non-school personnel, would be more appropriate for obtaining accurate CWPM and comprehension performance data. XXXX recommends that USAID clarify the purpose and application of the school-based assessment system and address the issues raised above in the RFP.

USAID Response: The purpose of the performance tracking system is to track student progress on a regular basis since the EGRA is conducted every other year. The contractor is expected to collect data as the children are assessed based on the literacy curriculum. The contractor is expected to report progress in reading for target students from Grades 1 through

3 based on the Primary Literacy Program.

b. Does USAID anticipate that the performance tracking system discussed under Objective

1 would apply to pre-primary classes as well as primary school classes?

USAID Response: No, currently schools are not required to complete a School Based

Assessment three times a school term for pre-primary.

c. While XXXX generally endorses the idea of a school-based performance tracking system, we recommend that USAID:

Clarify the purpose and application of the school-based assessment system and address the issues raised below.

Consider sample-based tests, administered by non-school personnel to obtain CWPM and comprehension performance data.

Currently, the discussion around the performance tracking system in the draft SOO seems to conflate the primary use of the system as means of helping schools and districts to directly support problem issues in schools and grades with USAID’s need to report on student outcomes and project progress. Because a school-based assessment system will take time to establish, should not use individually administered tests (required for oral reading fluency measurement), and will always be subject to bias, it is not an appropriate means of collecting data to report results. In addition, real-time reporting is problematic given the distance and paucity of devices (computers, tablets, phones) and the lack of interconnectivity in many areas, as noted under Operational Constraints in the SOO. Supplying schools with such devices and connectivity will be costly.

USAID Response: USAID is interested in tracking reading progress from Grades 1 through 3 using a tracking system. The system or rather school based results will help schools and districts to directly support problem issues in schools and grades.

d. Substantively, the LRZA is conceptualized to work at the school level as the primary driver of reading improvements. Thus in terms of overall approach, the LRZA targets schools and but acknowledges the importance of system and accountability. Are these aspects quiet assumptions for LRZA?

USAID Response: Both systems (as in effective education structure) and accountability are critical to LRZA as they are underlying principles to improved reading levels. By working with

Ministry personal at various levels of the Ministry, LRZA will be affecting systems. USAID’s emphasis on learner performance tracking is one way of ensuring accountability by answering the fundamental question; ‘Are children reading at the correct level?’

e. The data collection points are only focused on learners even though the system and its accountability for learning gets a lot of mention. For the implied system response, does USAID have parameters for how system will be influenced and measured in terms of performance?

USAID Response: EGRA will include a component on Snapshot of School Management

Effectiveness. The external evaluation will also assess the impact on systems.

f. As USAID is aware, under the STEP-Up Zambia project, the Local Learner Performance

Tracker included fields to gather information from a range of subjects and grades (alongside early grade reading) per the ministry’s request. Would USAID be amenable to having Let’s

Read, Zambia create a dashboard that similarly tracked a range of different subjects and grades if such a tool was preferred by the ministry? While Let’s Read Zambia’s focus will, of course, be on early grade reading outcomes, a tool with the potential to be more broadly applied might be more attractive to the ministry (and hence more sustainable).

USAID Response: USAID welcomes ideas or suggestions that help generate timely data in an efficient manner.

8. Reading Intervention Package

a. Please confirm that the MOGE is expected to deliver the interventions to the schools. It is our understanding that the project will work through government systems and personnel.

USAID Response: No, the offeror is expected to deliver the interventions.

b. Please specify the resources that the MOGE will provide, besides normal staff functions.

Is the MOGE expected to finance the printing and provision of teaching-learning materials (i.e., teacher guides/materials, student textbooks/readers, student supplementary reading books/materials, other teaching learning materials)?

USAID Response: No, given the operational constraints of the GRZ.

Is the MOGE expected to finance teacher training and coaching, including face-to-face training and coach-related expenses, such as transport to schools?

USAID Response: No, the contractor will be responsible for such interventions.

Has USAID addressed with the MOGE the need to reduce per diem allowances or to limit the requirements to hold workshops outside Lusaka, to avoid the high cost of per diem payments.

USAID Response: The CO has provided guidance on this matter and the letter is attached.

c. Please take into consideration that the SPRINT model, which the MOGE uses for training delivery, is not feasible to ensure quality training delivery or coaching due to understaffing. The two zonal staff charged with teacher training and support have to support all grades and all subjects (not just reading for KG-3), in addition to performing their school-based duties. We suggest that USAID address in its agreement with MOGE the need to add and/or dedicate MOGE staff for this requirement.

USAID Response: USAID will continue to engage the ministry over this issue. In addition, the Education Cooperating Partners had brought this issue to the attention of the ministry. The ministry organizational structure has been reviewed but not yet approved by cabinet.

d. Does Objective 2 include pre-service teacher training?

USAID Response: No.

9. Meet the Metrics

a. Please clarify that the target scores presented refer to benchmarks for proficient readers.

USAID Response: These are benchmarks for emergent readers.

b. Please clarify that the targets presented are end-of-project status targets.

USAID Response: No, but for the Grade 3 it will be the end of project targets following a cohort of unique beneficiaries.

c. Our understanding is that the targets are based on the 2015 EdDATA standards and benchmarks workshop conducted by RTI with the MOGE. The benchmarks were national and included data on better-off provinces than those targeted by Let’s Read, which likely increased the average. Given this context, please consider lowering the targets for the targeted provinces.

USAID Response: The benchmarks are national targets and therefore apply to the Let’s

Read, Zambia target provinces. The benchmarks setting took into account a number of factors that influence reading in Zambia, hence setting low targets for the country. USAID will not lower the targets because the contractor is only expected to meet the emergent reader targets which are reflected in the SOO.

d. Please specify whether the targets in the table for Grade 2 and 3 are the same for community schools and all languages? Is it an average? Please note that there were performance differences between languages and school types under the RTS and Time to Learn programs.

USAID Response: Yes, USAID is aware that there were performance differences between languages, hence the low targets. We are also aware that though community schools are poorly resourced, their performance is almost the same as public schools. In some cases, community schools perform even better than some public schools.

e. Please clarify whether the Zambia Grade 2 National Assessment Survey will be assisted by the Education Data Activity.

USAID Response: No, USAID will not conduct a national survey; however, the Education

Data activity will conduct EGRA in the Let’s Read, Zambia target provinces.

f. Please clarify whether the Zambia Grade 2 National Assessment Survey uses an EGRA-type test instrument and is administered by non-school affiliated data collectors. If so, then the

Grade 2 data it collects and reports and the Grade 3 data the school-based assessment system collects and reports are not comparable and cannot be jointly analyzed to determine how cohorts are progressing.

USAID Response: It is anticipated that the Education Data activity will employ test instruments and methodologies similar to/or improved versions of those used in the previous

EGRA. EGRA is not a longitudinal study; therefore, it is not comparable or meant to track progress of cohorts’ overtime. USAID will address the issue of grade 3.

g. Please clarify whether “measured through school-based assessment” (column 1, row 2 in the table) refers to the Performance Tracking System described under Objective 1 on page 4. If so, please note our comment under Objective 1 above--that using the Performance Tracking

System is not feasible to measure Let’s Read student performance. If this refers to something else, please provide further description.

USAID Response: USAID will address this issue.

h. Please clarify whether the MOGE or the Education Data Activity will be collecting data on

KG and Grade 1 student performance.

USAID Response: No. The Let’s Read, Zambia project will collect data on reading levels for grades 1 through 3 based on the phonics literacy curriculum.

i. Please clarify whether “proposed percentage change” refers to changes in scores or in the percentage of students achieving the score.

USAID Response: It refers to student scores, a student is expected to achieve at least 40% out of 100% in comprehension.

j. The statement of objectives lists that the project will reach, at a minimum—20,000 teachers, 4,250 head teachers, 4,250 deputy head teachers and 425 Zonal In-service coordinators. Could USAID please clarify if the Contractor would have to recruit these staff directly, or would the staff be provided by the Government of the Republic of Zambia (GRZ) or the Ministry of General Education (MOGE).

USAID Response: No, these are estimated staff within the target provinces currently.

k. Are the targets fixed? Can USAID take into consideration that deeper engagement in fewer schools may produce better results?

USAID Response: Yes, the targets are fixed as they are based on MOGE approved targets and guidelines.

l. The ultimate outcome of the LRZA is well communicated (i.e. desired reading competency). What are the key intermediate outcomes that are expected of the LRZA?

USAID Response: Given the low targets set for reading levels, all learners reached with reading interventions should at least move from non- readers to readers.

10. USAID states that “[t]he contractor’s objective is to identify and create networks with private sector entities that will leverage additional resources. The contractor must be able to quantify the potential leverage from private sector and how it will directly impact this activity.”

Could USAID please clarify if private sector partners are expected to make in-kind contributions to the project?

USAID Response: Yes, private sector partners are expected to make contributions but they not necessarily have to be in-kind. Any type of contributions will have to be within the interest of the project and learner.