B. SCMP Mid-term Evaluation - - BPA Call RFQ Attachment.pdf

SINDH COMMUNITY

MOBILIZATION PROGRAM

MIDTERM PERFORMANCE EVALUATION

DECEMBER 28, 2017

This publication was produced for review by the United States Agency for

International Development. It was prepared by Management Systems International, A Tetra Tech Company.

SINDH COMMUNITY MOBILIZATION

PROGRAM

MIDTERM PERFORMANCE EVALUATION

Contracted under Order No. AID-391-C-15-00004

Performance Management Support Contract

DISCLAIMER

This report is made possible by the support of the American people through the United States Agency for

International Development (USAID). The contents are the sole responsibility of the Management Systems

International and do not necessarily reflect the views of USAID or the United States Government.

Sindh Community Mobilization Program – Midterm Performance Evaluation iii

ACKNOWLEDGMENTS

This report represents a joint effort of a large number of people. Jared Berenter was the primary author. Management Systems International (MSI) staff in Pakistan planned and managed the evaluation, developed the data collection instruments, led the data analysis, and contributed substantially to the writing. Monitoring and Evaluation Specialist Saeed Ahmad served as field team lead. Dr. Hina Hussain

Kazmi, Rehana Paveen Shaikh, Shahzad Ahmed, Waseem Abbas, and Khurshid Khan provided field support, expert advice, and local context in designing and implementing the evaluation and interpreting the data. Babur Wasim, MSI’s statistics and evaluation specialist, contributed to developing the evaluation sample and to analyzing the data.

Finally, the respondents who gave generously of their time and knowledge to participate in surveys and interviews deserve special thanks.

Sindh Community Mobilization Program – Midterm Performance Evaluation iv

CONTENTS

Acronyms ........................................................................................................................... vi

Definitions ......................................................................................................................... vii

Program Summary ...........................................................................................................ix

Executive Summary

Evaluation Purpose and Questions Theory of Change Evaluation Questions

Program Background Sindh Education Sector Reform Sindh Basic Education Program Community Mobilization Program

Evaluation Methods and Limitations Data Sources Sampling for School-Level Primary Data Collection Data Analysis Methods Data Limitations

Findings Question 1: CMP’s Contribution to Access to Schooling Question 2: Effectiveness of CMP Community Engagement Activities Question 3: CMP Support for Education Sector Reforms Question 4: Relevance of CMP Performance Indicators and Targets

Conclusions

Recommendations

Annexes Annex 1: Statement of Work Annex 2: Assignment Work Plan Annex 3: Team Composition and Conflict of Interest Disclosures Annex 4: Interviews Conducted by the Evaluation Team Annex 5: Data Collection Instruments Annex 6: Documents Reviewed Annex 7: Sampling Annex 8: Background on Sindh Education Sector and Sector Reforms Annex 9: Impact of CMP on Enrollment Annex 10: SMC Subcommittee Membership, by Sex

Sindh Community Mobilization Program – Midterm Performance Evaluation v

List of Tables

Table 1: Program Summary ............................................................................................................................. ix Table 2: Key Informant Selection Table 3: Population Frame Table 4: Sample Frame Table 5: Group Interview Sample Table 6: Distribution of Sampled Schools Table 7: Sampled Schools by Location Table 8: ITSA Regression Output Table 9: Girls’ Enrollment in Primary Schools Table 10: Boys’ Enrollment in Primary Schools Table 11: Girls’ Enrollment in Higher than Primary Schools Table 12: Boys’ Enrollment in Higher than Primary Schools Table 13: Girls’ Total Enrollment in all Levels Table 14: Boys’ Total Enrollment in all Levels Table 15: Total Enrollment, Boys and Girls at all Levels Table 16: SMC Subcommittee Membership, by Sex

List of Figures

Figure 1: Map of Program Area ....................................................................................................................... x Figure 2: Top Factors Affecting Parents’ Enrollment Decisions Figure 3: Comparison of Factors Affecting Parents’ Enrollment and Retention

Decisions Figure 4: Summary of CMP’s Contribution to Enrollment Figure 5: Interpretation of Coefficients Figure 6: Summary of CMP Contribution to Enrollment

Sindh Community Mobilization Program – Midterm Performance Evaluation vi

ACRONYMS

CFC Charter for Compassion

CMP Community Mobilization Program

DEO District Education Officer

DST District Support Team

EMO Education Management Organization

GoS Government of Sindh

IBA Institute of Business Administration

IRD International Relief and Development

MSI Management Systems International

PPP Public-Private Partnership

PSLM Pakistan Social and Living Standards Measurement Survey

RFP Request for Proposals

RSU Reform Support Unit (GoS)

SBEP Sindh Basic Education Program

SELD School Education and Literacy Department (GoS)

SEMIS Sindh Education Management Information System

SMC School Management Committee

TEO Taluka Education Officer

USAID United States Agency for International Development

USG United States Government

Sindh Community Mobilization Program – Midterm Performance Evaluation vii

DEFINITIONS

The evaluation team developed the following definitions and measurements:

• Attendance: A student’s physical presence in the classroom.

• Campus schools: Schools into which other schools have been merged or consolidated under the Sindh education reforms. Campus schools serve as the administrative and/or physical hub for the other merged schools.

• Consolidation: Distinct government schools operating on the same campus, or catering to the same child population in a local community, converted into a single functional school.1

• Construction schools: Schools newly constructed under the Sindh Basic Education Program.

• Dropout: Students are considered dropped out after 90 consecutive days of absence from school.

• Education management organization (EMO): Private sector organization tasked by the

Government of Sindh School Education and Literacy Department to manage government-owned schools. EMOs are selected through a competitive bidding process. Under the first request for proposals from EMOs issued by the Government of Sindh, school management contracts were issued to Sukkur Institute of Business Administration and Charter for Compassion. Other EMOs that have won school management contracts include the Indus Resource Centre and

Beaconhouse.

• Education manager: Divisional directors of education, district education officers, and taluka education officers.

• Enrollment: All students on the general register of a school. For purposes of this evaluation, terms relating to enrollment (including new enrollment and re-enrollment) are defined in the

Community Mobilization Program’s Monitoring, Evaluation, and Learning Plan under Indicator

1.B, the second of two goal-level indicators pertaining to enrollment (referred to as “Goal-b”).

• Gross enrollment rate: The Pakistan Social and Living Standards Measurement defines the gross enrollment rate as the number of children attending primary, middle, or matric school, divided by the number of children of respective age for that level.

• Neighboring schools: Schools that received CMP community engagement and mobilization interventions and are eligible for small grants for school improvement but are not newly constructed.

• New enrollment: Students of school age who never previously enrolled to study in a formal education setting but have become enrolled through United States Government assistance.

• Re-enrollment: Students who were previously enrolled in a formal education setting but discontinued study and have since become re-enrolled.

1 School consolidation covers three types of schools: (a) adjoining schools consolidated administratively; (b) embedded schools or schools sharing the same premises; (c) schools that are within a stipulated distance of each other, with one strategic school among them having the horizontal and/or vertical space for expansion to accommodate students from merged schools. See Government of Sindh, School Education and

Literacy Department, http://www.sindheducation.gov.pk/educationReforms/sc.jsp.

http://www.sindheducation.gov.pk/educationReforms/sc.jsp

Sindh Community Mobilization Program – Midterm Performance Evaluation viii

• Retention: Graduation from one grade and enrollment in the next grade. (Pakistan’s

Constitution provides for 10 years of free and compulsory education.)

• School management committee (SMC): The SMC is a participatory body established and notified in every public school to support school management, facilitate school improvements, ensure financial accountability, and monitor education activities. The SMC is comprised of an executive body and a general body. The executive body includes a secretary (school head teacher), elected chairperson (typically a parent or influential member of the community), and three elected members (also typically parents). The broader SMC general body includes all parents or guardians of enrolled students.

• Subcommittee: The Community Mobilization Program has formed SMC subcommittees at the school level to increase community involvement and engage more parents and community members in school management activities. The program initially formed five subcommittees which were later restructured into three, namely (a) social mobilization and school improvement planning committee, (b) local resource and grants management committee, and (c) school health and nutrition committee. Subcommittees generally are comprised of one representative of the SMC executive body and volunteers from the SMC general body.

• Upgradation: Upgradation refers to the process for formally upgrading existing primary level schools to middle-level schools, and middle/elementary-level schools to high schools, to reduce the enrollment gap between primary and secondary education. Upgradation is often accompanied by provision of additional teachers and facilities.

Sindh Community Mobilization Program – Midterm Performance Evaluation ix

PROGRAM SUMMARY

TABLE 1: PROGRAM SUMMARY

Title/Field Program Information

Start date August 7, 2013

Location Sindh Province (Dadu, Jacobabad, Karachi, Kashmore, Khairpur, Larkana, Qambar Shahdadkot, and Sukkur districts)

Implementing partner(s)

International Relief and Development (IRD)

Novated to Blumont Engineering Solutions Inc. on December 29, 20172

USAID/Pakistan Mission Strategic

Framework objectives addressed

USAID/Pakistan’s Education Results Framework Development

Objective 4: Improved opportunities for learning and work

• IR 4.1: Improved educational access

• IR 4.3: Improved accountability in education

2 The evaluation took place before Blumont Engineering Solutions Inc. began implementing CMP in December 2017. This report mentions IRD as the implementing partner since the findings refer to activities conducted under IRD. When relevant, the report directs recommendations to

Blumont Engineering Solutions Inc. as the party responsible for implementing recommendations.

Sindh Community Mobilization Program – Midterm Performance Evaluation x

FIGURE 1: MAP OF PROGRAM AREA

Sindh Community Mobilization Program – Midterm Performance Evaluation 1

EXECUTIVE SUMMARY

Evaluation Purpose and Questions

The purpose of the Community Mobilization Program (CMP) midterm evaluation is to assess CMP’s progress toward achieving program objectives. The evaluation includes a review of key performance indicators and targets and recommends strategies to improve the measurement of program performance. Evaluation findings will inform future USAID programming decisions for community engagement and mobilization in the education sector.

The evaluation addresses the following questions:

1. How and to what extent has CMP contributed to improving access to schooling for boys and girls in target communities?

2. To what extent have CMP processes been effective for engaging target communities?

3. To what extent have CMP’s strategies in support of Government of Sindh (GoS) education reforms, specifically school consolidation and public-private partnerships, been effective for achieving GoS reform objectives?

4. Are CMP indicators appropriately defined, to what extent has CMP achieved its targets, and are these targets relevant to achieving program objectives?

Summary of Findings and Conclusions

CMP provides a valuable contribution to the Sindh Basic Education Program (SBEP), facilitating SBEP’s infrastructure components through community engagement at the school level and advocacy and technical assistance at the policy level. The evaluation finds that CMP’s development hypothesis holds true, namely that activities that build community awareness of education processes and engage parents in those processes have a positive effect on access to schooling for boys and girls.

CMP interventions contend with a multiplicity of factors—internal to both schools and households— that affect parents’ enrollment decisions. The perceived costs and benefits of enrolling one’s child in school vary for boys and girls and evolve as children grow. Physical and affordable access to schooling, with adequate facilities and quality instruction, are key factors driving enrollment decisions. Within this decision-making context (and to the extent that enrollment serves as a meaningful indicator of access), growth in enrollment during CMP’s period of performance demonstrates that CMP engagement strategies have increased access to schooling. Feedback from parents, head teachers, and school management committees (SMCs) suggests that CMP has contributed to the socialization of education— in other words, to a collective recognition of the accessibility and desirability of education.

The SMC is the chief mechanism by which CMP has engaged with target communities. The evaluation found that SMCs are appropriate institutional vehicles for community engagement aimed at increasing enrollment. SMCs are legally notified and participatory in their composition, have authority and responsibility to spend government funds on school improvements, and enjoy a sense of ownership from government institutions in the education sector. The evidence suggests that CMP’s strategies to activate and strengthen SMCs have increased SMCs’ capacity to engage communities and provide support for school management, and thus contributed to enrollment. CMP’s capacity-building efforts

Sindh Community Mobilization Program – Midterm Performance Evaluation 2 have provided SMCs and parents with clarity on SMC roles and responsibilities, strengthened SMC processes for financial management and planning, and instilled SMC executive and general bodies with knowledge and confidence to fulfill their intended purposes. With CMP’s assistance, newly formed SMC subcommittees have further broadened SMCs’ scope and community involvement in SMC activities.

CMP’s efforts to engage and strengthen SMCs have increased interaction between schools and communities, as well as parents’ active involvement in school affairs. CMP strategies have also facilitated interaction between schools and education officials, creating opportunities to mobilize government resources for school improvement and upgradation.

In addition to community mobilization and strengthening of school management processes, CMP supports two Sindh education sector reforms: (1) private sector management of government schools under the Public-Private Partnership (PPP) Act of 2010 and (2) the School Consolidation Policy of 2012.

CMP has supported these reforms at both the provincial and school levels with mixed results. With respect to education reforms under the PPP Act, CMP’s provision of resources and technical assistance to the PPP Node at the Sindh School Education and Literacy Department (SELD) has been integral to establishing standard operating procedures, successfully piloting the education management organization

(EMO) model, and engaging PPP stakeholders at the provincial and district levels. Although post-EMO transition support for school management lies outside CMP’s scope, CMP actively supports EMOs in the conduct of community engagement activities. However, EMO respondents expressed a lack of clarity regarding the roles and responsibilities of CMP, EMOs, and government stakeholders in the implementation of the EMO model, which suggests a need for continued institutional coordination in the post-transition phase to achieve GoS reform objectives.

With respect to the School Consolidation Policy, the GoS’s Reform Support Unit (RSU)3 reported little need for regular CMP technical inputs toward consolidation reforms at the provincial level. At the district and school levels, CMP has eased implementation of consolidation by providing education managers and head teachers with orientation around consolidation processes and procedures.

Sensitization and technical assistance for consolidation at the school level, however, fall within the purview of CMP’s conventional activities. District education officers (DEOs), for example, reported

CMP contributions to consolidated schools in the form of small grants and identification of needs for school improvement.

Finally, the evaluation assesses the suitability of enrollment as an indicator of access to schooling. The evaluation finds enrollment to be an insufficient, and potentially misleading, indicator of access for two reasons. First, enrollment does not capture attendance. External factors such as economic need, household obligations, or lack of transportation curtail attendance even if they do not curtail enrollment.

Second, most sampled CMP schools were fully enrolled and did not have capacity for additional enrollment. In such cases, enrollment does not effectively capture CMP’s performance. Likewise, aggregate enrollment targets established in the CMP Monitoring, Evaluation, and Learning Plan do not distinguish enrollment attributable to CMP, account for transfers of students from one school to another, or account for the longer-term impact of CMP’s community engagement efforts.

Recommendations

Based on these findings, the evaluation team recommends the following:

1. To better capture the effectiveness of education programs for improving access to schooling, USAID should include attendance and retention as indicators for access to schooling in future

3 The RSU is the chief entity responsible for policy formation and oversight of consolidation reforms.

Sindh Community Mobilization Program – Midterm Performance Evaluation 3 programming. USAID may also consider monitoring school capacity and out-of-school children at the community level over time to better understand changes in new enrollment, dropout rates, and retention, and inform realistic enrollment targets based on the local context.

2. CMP should have a strategic plan to mobilize or advocate for stopgap supply-side interventions to address overenrollment or accommodate students displaced by new school construction.

Where appropriate and practical, CMP should shift to attendance and retention activities in overcrowded schools.

3. Activities to strengthen SMC processes for engagement and school improvement should include an explicit knowledge-building component for SMC general bodies concerning roles, responsibilities, and practices of the SMC. Forums for such activities include SMC general body meetings and SMC subcommittee meetings.

4. CMP should establish formal mechanisms to transfer knowledge and best practices for EMO management to the PPP Node and district support teams (DSTs).

5. CMP should implement a plan to institutionalize stakeholder coordination at the district level to support EMOs before and after the transition to EMO school management. This action may require modification to CMP’s current scope.

6. CMP should conduct a needs assessment exercise based on lessons learned from consolidated schools to assist RSU in identifying implementation challenges and improving implementation of the School Consolidation Policy.

Sindh Community Mobilization Program – Midterm Performance Evaluation 4

EVALUATION PURPOSE AND QUESTIONS

The CMP midterm evaluation assesses CMP’s progress toward achieving program objectives, namely to improve access to schooling, particularly for girls, and support GoS education sector reforms. The evaluation identifies lessons learned about whether, how, and under what conditions community engagement has been effective for improving access to schooling. The evaluation includes a review of select CMP performance indicators and targets and recommends strategies to improve the suitability of these indicators and targets for measuring program performance. Evaluation findings will inform future

USAID programming decisions with respect to how community engagement can best support SBEP and other education sector programs in Sindh.

Theory of Change

The CMP theory of change posits that community engagement and mobilization in the education process build community awareness of education processes, enhance coordination between communities and district governments to expand girls’ enrollment, and increase community involvement in school management and GoS education sector reforms, thereby improving access to schools for boys and girls.

In the context of CMP, community engagement and mobilization include activities such as enrollment drives, community events, and other outreach efforts aimed at increasing enrollment and parental participation in school affairs; development and implementation of social mobilization and school improvement plans with communities and school stakeholders; strengthening of SMCs and formation of participatory subcommittees in target schools; and capacity building and engagement of DEOs. Education sector reforms include merging, consolidating, and upgrading schools and piloting a PPP model for school management that outsources school management to EMOs.

Evaluation Questions

Based on the theory of change, the evaluation addresses the following evaluation questions:

1. How and to what extent has CMP contributed to improving access to schooling for boys and girls in target communities?

2. To what extent have CMP processes been effective for engaging target communities?

3. To what extent have CMP’s strategies in support of GoS education reforms, specifically school consolidation and public-private partnerships, been effective for achieving GoS reform objectives?

4. Are CMP indicators appropriately defined, to what extent has CMP achieved its targets, and are these targets relevant to achieving program objectives?

PROGRAM BACKGROUND

The following details education sector reforms in Sindh and the respective roles of SBEP and CMP in supporting these reforms.

Sindh Community Mobilization Program – Midterm Performance Evaluation 5

Sindh Education Sector Reform

Despite constitutionally guaranteed rights to free and compulsory education in Pakistan, the GoS faces significant challenges ensuring universal access to schooling. Gross enrollment rates in Sindh have stagnated or declined in recent years—a trend exacerbated by damages to government schools inflicted by floods in 2010—and lag significantly behind both Punjab and Khyber Pakhtunkhwa.4 Gross enrollment remains significantly lower for girls than for boys and declines precipitously after primary school. The

GoS has undertaken measures to address these trends and other challenges in the education sector. At the policy level, the GoS has implemented two medium-term strategic frameworks for sector reform:

the Sindh Education Reform Plan, implemented from 2006–2012, and the Sindh Education Sector Plan, implemented from 2014–2018. This evaluation considers two specific reforms to improve school management and accountability:

• School consolidation: In 2012, the GoS introduced the School Consolidation Policy to make better use of limited resources, eliminate unviable schools, and increase access to higher-level schooling. School consolidation placed adjoining schools, embedded schools (i.e., schools situated on the same premises), or schools within a stipulated distance of each other under single management structures. Under the School Consolidation Policy, one school may provide physical space to accommodate students from merged schools, or a central “campus” school may oversee the administrative functions of neighboring schools. The consolidation process is managed by the RSU and implemented by the regional directors of education in Sindh.

• Public-private partnerships for school management: Private sector management of public sector schools is a long-standing practice in Sindh. Such arrangements have historically been loosely structured and often philanthropic in nature. The PPP Act, passed in 2010, introduced a structured framework for engaging the private sector in public sector service provision. In the education sector, the act provided legal cover for outsourcing school management and education service delivery to the private sector. To this end, SELD established a PPP Node tasked with developing a regulatory framework for partnering with the private sector to provide education services in public schools. With CMP’s assistance, the PPP Node is piloting a model for outsourcing public sector school management to private sector entities known as EMOs (see

Annex 8).

Sindh Basic Education Program

In 2012, in collaboration with the GoS, USAID launched SBEP to support the GoS Sindh Education

Reform Plan, with a focus on increasing and sustaining enrollment at the primary, middle, and secondary levels in seven districts of northern Sindh and five towns of Karachi. SBEP includes six components:

(1) construction of schools affected by the 2010 floods in the program districts (106 schools constructed under SBEP received CMP support and will transition to EMO management); (2) support for GOS policy reforms to merge, consolidate, and upgrade schools through construction of schools; (3) improvement in early grade reading in primary schools; (4) community mobilization, with a focus on increasing girls’ enrollment and improving the nutritional status of children; (5) technical assistance to

SELD; and (6) architect and engineering support for school construction.

4 Without estimating the overall effect of the 2010 floods on gross enrollment, a preliminary damage and needs assessment conducted by the

World Bank, Asian Development Bank, and Government of Pakistan estimated that 5,655 schools in Sindh were partially damaged or completely destroyed by the floods—more than in any other province. This constitutes approximately 12 percent of the 47,552 schools for which SEMIS data was available for the 2010–2011 school year.

Sindh Community Mobilization Program – Midterm Performance Evaluation 6

Community Mobilization Program

As a cornerstone of SBEP, CMP supports the other SBEP components as well as GoS education reform efforts. CMP aims to sustain community mobilization and school-based management by focusing on four key objectives:

1. Increasing communities’ involvement in GoS education reform of merging, consolidating, and upgrading schools.

2. Improving coordination between the community and district administration for increased girls’ enrollment.

3. Improving child nutrition in selected communities and government schools through research that informs innovation and good practice.

4. Sustaining community mobilization and school-based management through PPPs. Under a newly conceived PPP model, the GoS has initiated a process of outsourcing the management of public sector schools to reputable EMOs.

CMP is implemented by International Relief and Development (IRD)5 in partnership with Health and

Nutrition Development Society, Indus Resource Center, and Agha Khan University – Human

Development Program. CMP is implemented in Sukkur, Khairpur, Larkana, Qambar Shahdadkot, Jacobabad, Kashmore, and Dadu districts in northern Sindh, and in Kemari, Lyari, Orangi, Gadap, and Bin

Qasim towns of Karachi.6

EVALUATION METHODS AND LIMITATIONS

The evaluation team employed a mixed methods approach, relying on both quantitative and qualitative data from multiple sources to triangulate findings and generate credible and robust evidence to answer the evaluation questions. The following sections detail data sources, sampling, and data limitations.7

Data Sources

Quantitative, qualitative, and other secondary data sources included the following:

Quantitative Data

Secondary data from the SEMIS school census provided by the RSU constituted the evaluation’s principal source of quantitative data. The evaluation team received full or partial SEMIS data for 29 districts and approximately 49,700 schools in Sindh for academic years 2008–2009 through 2015–2016. School-level indicators contained in the SEMIS data included boys’ and girls’ enrollment, infrastructure (e.g., number of rooms and number of classrooms), facilities (e.g., boundary walls, water, washrooms, and electricity), and number of teachers, among others. SEMIS served as the primary source of data for generating a comparison group of schools to estimate CMP’s effect on enrollment under Question 1.

5 On December 29, 2017, USAID novated the project from IRD to Blumont Engineering Solutions Inc.

6 For additional background information and resources on the status of the education sector in Sindh, development of school management committees, and recent education sector reforms in Sindh, see Annex 8.

7 For a more detailed explanation of how data sources and analysis methods were used to address each evaluation question, see Table 2, “Summary of Proposed Data Collection and Analysis Methods,” in the assignment work plan in Annex 2.

Sindh Community Mobilization Program – Midterm Performance Evaluation 7

The evaluation drew additional quantitative data from CMP’s performance monitoring reporting.

Specifically, the evaluation team used program monitoring data to assess the extent to which CMP met its enrollment targets and the appropriateness of those targets under Question 4.

Qualitative Data

The evaluation team collected qualitative data through group and key informant interviews at the school, district, and provincial levels (see Annex 4 for a full inventory of interviews conducted). At the school level, the team conducted group interviews with parents and SMCs and key informant interviews with head teachers from selected CMP and non-CMP schools. Where appropriate, the team conducted key informant interviews with EMO school managers. School-level data informed Questions 1, 2, and 3.

At the district and provincial levels, the team conducted key informant interviews with select stakeholders and decision-makers. Stakeholders identified by the team included relevant provincial-level staff within the SELD, including the offices of the RSU, Project Management and Implementation Unit, and PPP Node; DEOs and taluka education officers (TEOs); and district-level EMO staff. District- and provincial-level data informed Question 3.

Key informant interviews with CMP implementing partners informed the assessment of CMP support for policy reforms under Question 3 and the assessment of indicators and targets under Question 4.

Other Secondary Data

To establish CMP’s operating context, scope of CMP activities, and reported progress toward achieving program outcomes, the evaluation team conducted a structured review of program documents and other relevant secondary data, including quarterly and annual reports; meeting minutes of the CMP advisory group, program steering committee, DSTs, SMCs, and other committees; proceedings of social mobilization sessions; DST terms of reference; EMO concession documents; education sector policy notifications; and other baseline studies, assessments, reports, and databases (Annex 6).

Sampling for School-Level Primary Data Collection

CMP community engagement interventions occurred primarily at the school level. Thus schools (or school catchment areas) constituted the evaluation’s primary unit of analysis. For school-level primary data collection, the evaluation team sampled 21 CMP schools and 3 non-CMP schools for comparison.8

The 24 total schools in the sample included 22 schools across 6 districts in northern Sindh and 2 schools from among the 5 towns of Karachi where CMP activities were implemented.9

The 21 CMP schools in the sample were selected according to criteria designed to capture the full range of CMP interventions. The sample included 5 schools awarded EMO contracts and 16 randomly selected construction and neighboring schools. The sample ensured representation of the following:

• Boys’, girls’, and mixed schools;

• Primary, middle, and secondary/high-secondary schools;

• Neighboring and campus (construction) schools; and

8 Comparison schools were randomly selected from a sample of non-CMP schools from CMP districts and with characteristics similar to CMP schools, matched using 2012–2013 SEMIS data. Matching criteria included variables related to infrastructure, school facilities, student/teacher ratios, and location. However, the evaluation was not able to control for other donor activities in non-CMP schools (see Annex 7).

9 For security reasons, one intervention district (Jacobabad) was excluded from the sample. The sample ensured distribution of schools across each of the other six districts and Karachi.

Sindh Community Mobilization Program – Midterm Performance Evaluation 8

• High- and low-performing schools with respect to enrollment growth.10

At each school, the evaluation team interviewed the school head teacher, a group of parents randomly selected based on the enrollment status of their children, and members of the school’s SMC.11 Annex 7 summarizes the distribution of sampled schools by district and intervention type and provides a more detailed sample distribution of schools by sex, grade level, and enrollment growth trends.

Data Analysis Methods

The evaluation team employed quantitative and qualitative analysis methods to triangulate the data sources described above. These methods included the following:

Quantitative Analysis

The quantitative component of the evaluation’s analysis estimates CMP’s effect on regressed enrollment trends for boys and girls in CMP districts. To generate these estimates, the evaluation team used SEMIS data from 2012–2013 to generate a matching sample of non-CMP schools located in CMP districts and with characteristics similar to CMP schools. The matched sample allowed the evaluation team to control for other variables that may affect enrollment, such as school infrastructure and student/teacher ratios.

A comparison of changes in enrollment in CMP schools and their matching schools produced estimates of change in enrollment rates attributable to CMP (Annex 9).

Qualitative Analysis

Beyond quantitative estimates of CMP’s impact on changes in enrollment, the analysis predominantly consists of content analysis of qualitative data. Qualitative data from group and key informant interviews helps to explain quantitative findings in Question 1 and is the primary data source informing Questions 2 and 3. The qualitative analysis distinguishes school- and district-level findings from provincial-level findings. At the school level, the evaluation team identified thematic patterns across 24 sets of school-level group and key informant interviews to explain how and why factors contributed to outcomes of interest. At the provincial level, the evaluation relies on analysis of informed insights from key informants at relevant institutions.

Generally, where there is homogeneity or commonality across respondents (for example, in group interviews with parents or SMCs or in key informant interviews with DEOs), the analysis aggregates and reports the frequency of qualitative responses. The analysis triangulates these frequencies with illustrative responses that clearly explain knowledge, attitudes, or behaviors of interest. The evaluation pays particular attention to how gender affects themes in the qualitative data.

Data Limitations

Key limitations of the evaluation approach, and methods used to attenuate the effects of these limitations on data validity, included the following:

10 For purposes of this study, school performance was defined by change in school enrollment between 2008–2009 and 2015–2016. Schools in the 20th percentile in terms of increase in gross enrollment, calculated as a percentage of 2008–2009 enrollment, were classified as “high-performing.” All other schools were classified as “low-performing.”

11 For group interviews with parents, the evaluation team sought to include parents of newly enrolled children, parents of children with sustained enrollment over time, parents of children who had dropped out of school, and parents of children never enrolled. The team relied on school general registers to select parents of enrolled students and dropouts and used convenience sampling to recruit parents of children who had never enrolled in school.

Sindh Community Mobilization Program – Midterm Performance Evaluation 9

• Availability, quality, and consistency of SEMIS data: The evaluation’s quantitative analysis was constrained by the availability and reliability of SEMIS data. Without class-level data, for example, the evaluation team was unable to measure changes in enrollment by age group.

Changes in reporting procedures and force majeure events (i.e., 2010 floods) further compromised the reliability of SEMIS data over time. To mitigate this limitation, the evaluation team conducted thorough assessments of data quality and incorporated quality deficiencies into its interpretation of results. The evaluation team further supplemented and triangulated estimates of gross enrollment with perception data from qualitative interviews.

• Availability of data for other donor activities: The evaluation team did not have access to data for other donor activities that may have affected study outcomes in CMP or non-CMP schools. The team therefore could not control for non-CMP activities to isolate the effect of

CMP on those outcomes. Where possible, the evaluation team mitigated this limitation in its primary data collection by identifying such activities in sampled schools. This limitation could not be mitigated in the quantitative analysis of SEMIS data.

• Sample size: Given time and budget constraints for primary data collection, the evaluation team selected a number of schools in the evaluation sample (24) to balance breadth (i.e., larger sample, less detailed instrument) with depth (i.e., smaller sample, more detailed instrument).

The sample size necessarily imposed limits on the evaluation team’s confidence in the generalizability of the evaluation findings. To address this limitation, the evaluation team employed a largely qualitative approach to answering “how” and “why” questions to ensure depth of response, while strategically sampling schools to capture variation in type of intervention and type of school. The evaluation team also relied on multiple data sources to triangulate and verify findings.

• SBEP integration: CMP constitutes one of six components of SBEP, each of which may have had a significant effect on CMP outcomes, such as enrollment. The broad scope and degree of integration of SBEP components limited the extent to which the evaluation team could control for the effect of each component, and thus the extent to which the team could attribute outcomes specifically to community mobilization and engagement activities. To mitigate this limitation, the evaluation team’s sampling strategy purposively selected newly constructed schools and employed a qualitative approach to answer “how” and “why” CMP contributed to outcomes of interest.

• Response bias: Key informants and group interview participants were subject to response biases that could influence findings and lead to under- or over-estimations of program impact.

Specifically, parents with time and willingness to participate in group interviews may have been systematically more likely to routinely engage with schools and/or CMP and thus may overstate the value of CMP community engagement activities. Conversely, people whose children were not enrolled in schools may have chosen to disengage from the education setting entirely and may either lack familiarity with or understate the value of community engagement activities. To mitigate this bias, the evaluation team relied on multiple data sources and probed suspected respondent biases.

• Respondent recall: Key informant interviews relied on informants with credible knowledge of a program’s impact on evaluation outcomes. Organizational turnover or other factors impeding respondent recall may have affected the reliability of key informant data. The evaluation team closely consulted with USAID and CMP’s implementing partners to identify appropriate institutions and individuals with relevant knowledge of CMP.

Sindh Community Mobilization Program – Midterm Performance Evaluation 10

FINDINGS

Question 1: CMP’s Contribution to Access to Schooling

How and to what extent has CMP contributed to access to schooling for boys and girls in target communities?

To determine how CMP has contributed to access to schooling for boys and girls in target communities, the evaluation considered the context in which parents make decisions about their children’s schooling.

The effectiveness of CMP activities for increasing enrollment, retention, and attendance hinges on the interplay of community engagement with a multitude of household and school-level factors, including opportunity costs, availability of schooling options close to the home, and supply-side factors such as quality of instruction and school facilities. A consistent theme that emerges here and in the context of all four evaluation questions is the importance of supply-side factors in determining the effectiveness of

CMP strategies for achieving program objectives. Few parents reported community engagement activities as the primary factors motivating their enrollment decisions. Nevertheless, parents frequently referenced determining factors directly affected by CMP activities, namely their own awareness of the value of education, their motivation to educate their children, and their children’s motivation and interest in attending school. Parents and head teachers alike reported that CMP positively influenced enrollment decisions and that outreach activities played a significant role in motivating parents and students to engage with the school community. Enrollment trends in the SEMIS data reinforce this finding, showing a greater increase in enrollment in CMP schools than in comparable non-CMP schools.

Factors Affecting Enrollment and Retention

The decision to enroll and keep one’s child in school was an active one. Parents weighed many factors, both internal to the school and internal to the household. How these factors, particularly those that are economic or that derive from gender norms, influenced decision-making changed as children grew older.

Key household factors affecting parents’ enrollment decisions included awareness of the importance of education and motivation to educate one’s child, perceived economic costs of education, and proximity of available schooling options to the home.

Parents’ awareness of the value of education and their motivation to educate their children emerged as the most commonly reported factors driving enrollment decisions in sampled communities (see Figure

2). Parents’ desire for their children to be educated derived from desires for their children to be good citizens; good mothers, wives, or members of the household in the case of girls; or (as discussed below) good economic earners in the case of boys. In at least three cases, this desire derived from recognition of education as a right or religious obligation.

Other frequently cited household factors included the perceived economic costs or benefits of education and physical proximity between home and school. These factors were often closely interlinked. Parents commonly cited transportation costs, for example, which increased with distance between home and school. In addition to short-term economic costs, parents considered the opportunity costs of not sending their children to work and the potential longer-term economic benefits of education. Parents in 18 of the 24 communities felt that education would create future economic opportunities for their children. Conversely, perceived lack of employment opportunities for educated youth served as a deterrent to schooling—in Qambar Shahdadkot, for example, a parent noted that even “the educated people also become laborers.” Beyond cost, proximity between home and school presented a complex set of other exigencies. In addition to transport costs, parents considered the

Sindh Community Mobilization Program – Midterm Performance Evaluation 11 availability of transport, the safety of children who would need to travel long distances, and, for girls, the availability of male accompaniment.

For boys and girls at an early age, factors affecting enrollment generally did not vary by gender, with one important exception. Cultural norms and attitudes about girls’ education posed an additional impediment to girls’ enrollment (see Figure 2). In 16 of the 24 sampled communities, parents cited social barriers to enrolling their daughters in school, including a need for girls to assist with household chores, fears of permitting greater mobility for girls, general societal opinions of the value of girls’ education, and, according to head teachers from at least 3 communities, parents’ fears of girls intermingling with boys of different castes. Where parents had to choose between a boy’s education and a girl’s, many chose that of the boy. A parent in Sukkur noted, “If I have to choose between educating my daughter or son, then I will choose my son. The reason is that girls’ education is not appreciated in our society.”

Such barriers notwithstanding, parents or head teachers in nine communities noted positive cultural changes encouraging parents to enroll their daughters in school.

Key supply-side factors internal to the school—including quality of instruction, infrastructure, and school facilities—also affected parents’ enrollment decisions.

When deciding whether to enroll their children, parents placed great value on the internal conditions of a school, including both the quality of instruction and the quality of school facilities. Parents from 20 of the 24 schools reported that considerations of teaching quality affected their enrollment decision, including the quality of instruction, number of teachers, teachers’ attendance, and familiarity with teachers as members of their community.

FIGURE 2: TOP FACTORS AFFECTING PARENTS’ ENROLLMENT DECISIONS

Source: Analysis of MSI interview data.

The availability and condition of school facilities were also of primary importance to parents’ enrollment decisions. Parents expressed extreme reticence to send their children to schools that did not have basic amenities such as electricity, running water, furniture, functioning washrooms, or good sanitary

Sindh Community Mobilization Program – Midterm Performance Evaluation 12 conditions. Parents in at least seven schools noted lack of space for their children in the classroom.12 It was often the case, particularly for girls, that a school in poor condition was the only schooling option in a community. If necessary, girls used the washrooms of nearby homes, but for cultural and safety reasons this was generally considered inappropriate after fifth grade. The need for adequate school facilities was thus particularly acute for girls. A parent in Karachi described the challenges that girls faced when enrolled in schools of poor condition:

“Still there are a number of children in the village who are not enrolled in school. Still parents want to send their children to school, and children also want to come to school, but schools don’t have space.

There is no washroom facility in the school … Boys go to mosque washrooms, but girls cannot use the mosque washroom or go to nearby houses. Girl students cannot leave school and return home just to use the washroom. This is a problem for girl students.”

Conversely, parents in 11 communities reported that a new school building, or the promise of a new building, had affected or would affect their enrollment decision. In Khairpur, a parent directly linked the availability of proper facilities with girls’ enrollment, noting, “After the construction of the new school building, the enrollment of female students has increased. Now girls come in the evening shift too, now they know the importance of education.”

As children grew older, the perceived costs and benefits of education changed. Factors affecting girls’ and boys’ retention diverged.

As children grew older, the perceived economic costs of and social barriers to education increased.

Parents’ awareness and motivation to educate their children was less important to their decision to keep their children enrolled in school than to their initial enrollment decision (Figure 3). The reasons differed for boys and girls. As boys grew older, the expectation that they should work or contribute financially to the household began to compete with and often outweighed the perceived need for continued education. As girls grew older, cultural barriers to continuing schooling intensified. For both boys and girls, supply-side factors—fewer middle and secondary schools than primary schools and higher transport costs associated with reaching more distant middle and secondary schools—were key considerations underlying retention decisions after primary school.

For boys, the expectation to work or contribute financially to the household was a key factor affecting retention.

Group interview responses indicated that opportunity costs were likely to…