Annex_I-_Previous_EDR.pdf

Evaluation Design Report Impact of the Agricultural Development Project

Burkina Faso

Draft

Draft: June 2014 Final: November 2014

Authors:

IMPAQ International, LLC

Submitted by:

Project Director: Jacob Benus IMPAQ International, LLC 10420 Little Patuxent Parkway, Suite 300 Columbia, MD 21044

Submitted to:

Project Officer: Jennifer Witriol Millennium Challenge Corporation 875 15th Street, NW Washington, DC 20005

TABLE OF CONTENTS

List of Acronyms ................................................................................................................. iii

1. Introduction

2. Farmer Training and Di Irrigation

Di Irrigation Investment on PAPs and Non-PAPs

2.2. Program Logic

2.2.1. Farmer Training

2.2.2. Di Irrigation

2.3. Economic Rate of Return and Beneficiary Analysis

2.3.1. Economic Rate of Return

2.3.2. Beneficiary Analysis

3. Literature Review

4. Evaluation Design

4.1. Evaluation Questions

4.2. Farmer Training

4.2.1. Sample Size

4.2.2. Data Sources

4.2.3. Data Analysis

4.3. Di Irrigation Investment on Non-PAP

4.3.1. Sample Size

4.3.2. Data Sources

4.3.3. Data Analysis

4.4. Di Irrigation Investment on PAP

4.5. Timeframe

4.6. Monitoring and Evaluating Data Quality

5. Project Requirement and Staffing

5.1. Summary of Institutional Review Board Requirements and Clearances

5.2. Data Access, Privacy and Documentation Plan

5.3. Dissemination Plan

5.4. Evaluation Team

REFERENCES

APPENDIX A: LAND ALLOCATION CRITERIA FOR NON-PAP

IMPAQ International, LLC Page i Report Name Date of Report

APPENDIX B: Example of Table with Sample Characteristics, Baseline

APPENDIX C: ILLUSTRATIVE BASELINE REPORT OUTLINE

APPENDIX D: ILLUSTRATIVE FinaL REPORT OUTLINE

APPENDIX E: Example of Table for Impact Estimates

IMPAQ International, LLC Page ii Report Name

LIST OF ACRONYMS

ADP Agricultural Development Project AMVS Sourou Valley Development Authority ARF Access to Rural Finance BDS Business Development Service BRIGHT II Burkinabé Response to Improve Girls' Chances to Succeed CAT Commission for the Attribution of Land DA Diversified Agriculture DID Difference-in-Difference EDR Evaluation Design Report ERR Economic Rate of Return GDP Gross Domestic Product IHSN International Household Survey Network IRB International Review Board IRIS Institutional Reform and the Informal Sector IWRM Integrated Water Resources Management MCC Millennium Challenge Corporation MCA-BF Millennium Challenge Account-Burkina Faso M&E Monitoring and Evaluation MDE Minimum Detectable Effect MIS Market Information System Non-PAPs Non-Persons Affected by the Project NORC National Opinion Research Center O&M Operation and Maintenance PAPs Person Affected by the Project PNGT2 Deuxième Programme National de Gestion des Terroirs PFIs Participating Financial Institutions RCT Randomized Control Trial RFF Rural Finance Facility RLG Rural Land Governance RRP Roads Rehabilitation Project TA Technical Assistance WUAs Water User Associations

IMPAQ International, LLC Page iii Report Name

1. INTRODUCTION

The agricultural sector in Burkina Faso, as in other low-income countries in the early stages of structural transformation, relies heavily on small family plots and traditional farming systems.

The sector is predominantly rain-fed and subject to climate variability. As a result, agricultural potential is constrained by low land and labor productivity, as well as high production costs.

To promote pro-poor economic growth, MCC signed a USD 480.9 million Compact Agreement with the Government of Burkina Faso in 2008. The Compact entered into force in 2009, initiating the five-year timeline for project implementation.

Under the Compact, MCC has strategically invested funds in the following four projects:

Agriculture Development (ADP) Project (US$ 142 million) provides water management and irrigation technology along with support to on-farm production and related activities in the country’s southwest.

Rural Land Governance (RLG) Project, RLG (US$ 60 million) implemented land policy and regulatory reform, established village level land conflict and administration offices and formalized land rights across 47 Project communes, Ghanzourgou, Sourou and the Di Irrigated Perimeter.

Roads Rehabilitation (Roads) Project (US$ 194 million) develops and rehabilitates road segments in western and southwestern Burkina Faso.

Burkinabé Response to Improve Girls' Chances to Succeed (BRIGHT II) Schools Project (US $ 29 million) extended Burkina Faso’s MCC Threshold Program, including the construction of 132 schools in 10 of the country’s 45 provinces.

Exhibit 1 shows the geographic location of the four Compact projects:

• The ADP Project (Project Dévelopment de l’Agriculture) area is depicted in yellow and is mostly in the western part of the country;

• The RLG Project (Project Sécurisation Finciére) area is depicted in green and is spread throughout the country;

• The Roads Project (Project Désenclavement) is depicted by heavy dark lines in the western part of the country; and

• The BRIGHT II Project is depicted in orange in the northeast part of the country.

IMPAQ International, LLC Page 1 Report Name

Exhibit 1: Geographic Location of the four Compact Projects

IMPAQ International, LLC Page 1 Report Name

The Agricultural Development Project (ADP) seeks to extend the agricultural use of land as a way to increase the quantity and value of agricultural production. This will boost income and employment in rural areas and make the rural economies more competitive. Specifically, the project targets the main constraints in rural Burkina Faso: water management, low capacity, as well as poor access to information, markets, inputs, and credit. The expected results include growth in agricultural production, an increase in irrigated land and improved access to rural credit in the intervention zones.

The ADP project has three main activities:

Water Management and Irrigation – the goal of this activity is to promote improved water management resulting in better access to water through increased water resource protection and more efficient irrigation, including water availability, water delivery, flood control and dam safety. This is expected to help support and protect investments (infrastructure and human/institutional capacity) in the Sourou Valley and Comoé Basin.

Agricultural Diversification (DA) – builds on the delivery of water resources in the project zones by supporting on-farm production and related activities throughout the agricultural value chain.

Access to Rural Finance (ARF) – provides access to term loans by participating financial institutions as a way to fund creditworthy borrowers in the region.

This Evaluation Design Report describes IMPAQ’s proposed evaluation design; specifically our design for two sub-activities, Farmer Training and the Di Irrigation investments, of the Agricultural Diversification (DA) Activity.

The report is organized as follows:

Section 2 provides detailed information about the Farmer Training and Di Irrigation investments interventions, including an in-depth description of the program logic;

Section 3 presents a brief literature review of studies addressing links between agriculture interventions and agricultural productivity and poverty;

Section 4 summarizes the key research questions, presents the evaluation design, the data sources, the econometric methodology, and the proposed timeline to capture short, interim and long-term beneficiary streams; and

Section 5 concludes the report with a description of planned compliance to various administrative requirements.

IMPAQ is also preparing similar evaluation design reports for other components of the MCC Compact, including impact evaluations of the Roads Project and the RLG Project.

IMPAQ International, LLC Page 2 DRAFT Burkina Faso Evaluation Design Report

2. FARMER TRAINING AND DI IRRIGATION

This report presents the evaluation design for the Farmer Training and Di Irrigation sub-activities of the DA activity. These sub-activities were designed to assist beneficiaries to diversify and intensify crop production. Our evaluation will analyze the following interventions:

Technical Assistance Services – a program designed to provide farmers in irrigated and rain-fed areas with extension services and demonstration farms to improve their production, process, and storage techniques. The Technical Assistance Services program also provided starter kits and training to over 7,000 farming households in agriculture, agro-forestry, animal health, and animal husbandry. Starter kits included certified seeds or plants, fertilizers, basic farm tools and post-harvest production bags. Farmer training included training on management of soil fertility and organic fertilization, crop rotation, precautions to take with pesticides, and irrigation management.

Strengthening Animal Health Services – a program designed to help revive the livestock sector by improving animal health. It includes vaccination programs, as well as the introduction and implementation of an artificial insemination campaign. It also equipped public agencies with analysis and medical care kits, and builds the capacity of public agencies to provide technical support and professional training to veterinary practitioners.

Value Chain Development – a program focused on post-harvest aspects of the value chain, including storage, transportation, and transformation for some of the most profitable crops. The program trained producers groups and individuals to add value to agriculture-related commercial activities.

Since these components of the DA activity were implemented in the same areas and often to the same farmers, most households are affected by multiple sub-activities. In addition those affected by the Di Irrigation Investment also received farmer training although to less of an extent as those outside of the Di Irrigated Perimeter. This was largely due to timing of the irrigation system and allocation of land.

Di Irrigation Investment on PAPs and Non-PAPs

In the Di commune in the Sourou Valley in Western Burkina Faso, MCC collaborated with the government of Burkina Faso to build new irrigation infrastructure on a perimeter of 2,240 hectares (ha) of land. The Commission for the Attribution of Land (CAT), a committee of local stakeholders and officials, allocated plots of land in this perimeter to beneficiaries in two phases.

IMPAQ International, LLC Page 3 DRAFT Burkina Faso Evaluation Design Report

Phase 1 - CAT distributed 1,640 ha to people with pre-emptive rights (also called Persons Affected by the Project, or PAPs) and to people living in the five underprivileged villages near the perimeter. These villages included Oué, Bouna, Poro, Niassari and Donon.

Phase 2 - CAT distributed 600 ha by lottery to eligible applicants who did not receive land in Phase 1 and who lived in the six provinces of the region known as the Boucle du Mouhoun. These provinces included Mouhoun, Sourou, Nayala, Kossi, Balé, and Banwa.

CAT, with MCA assistance, a l s o launched an awareness campaign to encourage individuals to apply for land grants.

Additionally, all beneficiaries from the Di new perimeter received starter kits. The starter kits included a mix of seeds, fertilizers, and tools to finish the irrigation work.

IMPAQ will be evaluating the Di Irrigation Investment on Non-PAPs. MCC has also requested that IMPAQ add an evaluation of the Di Irrigation Investment on PAPs.

2.2. Program Logic

The ADP intervention seeks to unlock agricultural and non-agricultural economic capacities and resources, thereby contributing to poverty reduction and economic growth in Burkina Faso. In the following section, we provide the ADP logic and further describe the mechanisms by which we expect results to be realized highlighting the components of the ADP that IMPAQ will evaluate.

Exhibit 2 presents the conceptual model in the form of a program logic diagram for the AD activity. It identifies activities, outputs, and outcomes. The left side shows activities while the right side shows outputs and outcomes. Exhibit 2 also presents the impacts we expect the project to realize upon completion.

2.2.1. Farmer Training

Farmer Training provides farmers with information on best practices through training and technical assistance in agriculture and agro-forestry, as well as in animal health and animal husbandry. As a result, we expect farmers to adopt agricultural technology, practices, and farm-management techniques that will lead to an increase in farm productivity and a reduction in post-harvest losses in the short-term (as soon as one growing season following training).

The establishment of producer associations will strengthen the value chain of agricultural outputs. This sub-activity will enable producers to diversify their sources of income and to capture more of the value of their crops. In the short term, we should observe more market transactions and business linkages.

IMPAQ International, LLC Page 4 DRAFT Burkina Faso Evaluation Design Report

The sub-activities related to animal health services seek to strengthen the livestock sector by improving animal nutrition and health. They are designed to strengthen the sector by training veterinarians and veterinary technicians, providing equipment and care kits, as well as vaccinating cows against bovine peripneumonia and chickens against Newcastle disease. In addition, this sub-activity will train producers on improved livestock production and the safe transportation of animals. This sub-activity should reduce the high animal mortality rate due to poor hygiene, malnutrition, disease and lack of suitable housing. In the short term, we should observe improved livestock productivity and animal health, as well as higher demand and access to veterinary services and medications.

For these reasons, we expect short-term outcomes for the diversified agriculture to increase incomes for producers and other actors along the livestock and agriculture value chains (impacts).

Increased investments and agricultural productivity can lead to enhanced food security and increased incomes (long-term outcomes). In fact, for people who earn a living through agricultural production, growth in productivity means higher and more stable incomes if there is sufficient market access and demand for the product being produced. Thus, they have a greater ability to sustain basic household needs. Growth in agricultural productivity can also promote overall economic growth. Studies suggest that growth in agricultural output and wages can be an effective means of reducing poverty in the poorest countries (Bezemer and Headey, 2006).

IMPAQ International, LLC Page 5 DRAFT Burkina Faso Evaluation Design Report

Exhibit 2: Program Logic for the Diversified Agriculture Project

Activity/Sub-Activity Outputs Short-term Outcomes Long-term Outcomes Impacts

TA for farmers (rain-fed & irrigated production)

TA & incentive kits delivered to farmers Increased diversification of crops

Farmers realize sustainable increases in productivity, yields and profits

Higher incomes for producers as well as other actors in livestock and agriculture value chains

7000 households trained in 2 tracks:

Vegetable: agriculture and agro-forestry

Animal health and animal husbandry (chicken/cows)

Producers adopt practices to increase productivity (animal & agriculture) & reduced post-harvest losses

Incentive kits used by farmers

Value Chain Development

48 producer associations & agribusinesses trained to add value to commercial activities

Producers & agribusinesses increase value added to their commercial services

Sustained efficiency gains & value addition for beneficiaries

Producer Associations Established

More business linkages & market transactions

Animal Health Services

Improved livestock services:

Vets trained

Equipment & meds provided Rehab of vet schools/labs 500,000 chickens and cows vaccinated from PCP and new castle

Improved livestock productivity & animal health

Sustained increases in livestock productivity

Increased demand for vet services

Improved access to veterinary services and meds

IMPAQ International, LLC Page 6 DRAFT Burkina Faso Evaluation Design Report

2.2.2. Di Irrigation

Access to water and irrigation is a major determinant of land productivity and the stability of crop yields. Additionally, irrigated land productivity is significantly larger compared to rain-fed land productivity (WB, 2008a). The Di Irrigation Perimeter activity will provide more irrigated land to farmers and is expected to place additional land area under production. As a result, we should observe an expansion in irrigable land and surface area under production. In the short term, improved access to irrigated land is expected to lead to increases in the intensity and diversity of crop production.

The formalization of land rights of Di beneficiaries of irrigated parcels is expected to contribute to tenure security and willingness to invest in the land. Since the lack of formal land tenure rights is major component of land insecurity in Burkina Faso, formalizing land tenure rights should increase farmers’ sense of security and lead to greater investments. Formal land tenure rights coupled with stronger institutions and land management tools should help reduce the risks to livelihoods (short and long-term outcomes). The provision of titles/leases aids to improve land tenure/security and hence willingness of households to invest in their land and its productivity/value. At the same time, the training of farmers will encourage farmers to adopt agricultural technology, practices, and farm-management techniques that will lead to an increase in farm productivity and a reduction in post-harvest losses in the short-term.

Exhibit 3 presents the conceptual model in the form of a program logic diagram for the Water Management and Irrigation activity. As indicated in the conceptual model, we expect the project’s short-term outcomes to unlock economic opportunities of households and individuals living in areas affected by the projects. In particular, the IWRM activity has the potential to increase productive investments, promote efficient use of land and water resources, increase agricultural production, productivity, and employment opportunities, as well as increase the value of land and assets (impacts).

IMPAQ International, LLC Page 7 DRAFT Burkina Faso Evaluation Design Report

Exhibit 3: Program Logic for the Water Management and Irrigation Project

Activity/Sub-Activity Outputs Short-term Outcomes Long-term Outcomes Impacts

Development of Di Irrigation

Perimeter

Development of 2,240 ha of irrigated land within the Di perimeter:

Establish Di irrigation system Formalization of land rights for beneficiaries

Increase in cropping intensity diversification of crops & higher crop yields

Increased net revenue per hectare per year

Higher farm incomes from increased agricultural productivity

Improved land tenure

IMPAQ International, LLC Page 8 DRAFT Burkina Faso Evaluation Design Report

2.3. Economic Rate of Return and Beneficiary Analysis

2.3.1. Economic Rate of Return

The IMPAQ team will use the results from the impact evaluation to recalculate the Economic Rate of Return (ERR) for comparison to the ERRs that MCC used to inform the Compact investment decisions. The ERR is the discount rate at which the discounted benefits equal the discounted costs. MCC calculated the ex-ante ERR (for Di and Lery Dam) at the start of the project using the model presented in Exhibit 4. We will replicate an estimate of the ERR using ex-post benefits and costs. Specifically, we will estimate benefits from the following impact evaluations: benefits associated with improvements in cultivation, benefits from increased hectares planted, benefits from investments and adoption of training, improvements in irrigation, etc. We will also estimate the costs associated with these benefits and apply the ERR model in Exhibit 4.

ERR Model5

(REVENUE – COSTS)WITH PROJ * (HA CultivatedWITH PROJ) minus (REVENUE – COSTS)WITHOUT PROJ * (HACultivatedWITHOUT PROJ)

We will also conduct a cost-effectiveness analysis to compare the effects per dollar invested with comparable measures of typical irrigation and land tenure investments.

2.3.2. Beneficiary Analysis

In 2010, MCC conducted a Beneficiary Analysis6 which found that the MCC investments are expected to benefit approximately 1.1 million people in Burkina Faso by the completion date of the projects and activities. The results of this Beneficiary Analysis are described below.

5 The ERR formula comes from MCC’s Power Point Presentation Overview of Economic model by Benjamin Bryant, August 2013.

6 Retrospective Beneficiary Analysis: Burkina Faso Compact. Reflecting information available as of: April 2008 Date completed: August 2010. Millennium Challenge Corporation, August 2010.

Exhibit 4: Economic Rate of Return Model

IMPAQ International, LLC Page 9 DRAFT Burkina Faso Evaluation Design Report

Exhibit 5 presents the methodology used to estimate the number of individual beneficiaries and Exhibit 6 shows the resulting beneficiary counts by activity. Beneficiary calculations are based on the number of discrete individuals whose incomes are expected to rise from an investment project, as well as the members of their household who are likely to share benefits.

Exhibit 5: Methodology for Estimating Beneficiaries

Methodology for Estimating Beneficiaries

Project Activity/Investment Estimation Methodology

Agriculture Development

Lery Dam Residents of the Sourou valley whose livelihoods will be protected by the rehabilitation of Lery Dam and avoidance of catastrophic failure of the existing Dam.

Di Irrigation Farmers and farm employees in the Di region, included in the Lery Dam count.

MCC estimates that most of the infrastructure activities wi l l generate benefits throughout their 20–30 year lifespans. During this time, MCC expects the underlying population to grow.

Therefore, beneficiary counts are adjusted by the national population growth rate over the project lifespan. This growth-adjusted figure provides an estimate of cumulative activity beneficiaries.

Inasmuch as MCC’s initial beneficiary counts and estimates were based on limited knowledge and information, the current evaluation may be able to improve on the original analysis with added information. We will gather additional data and attempt to improve the overall counts and assess the overlap in beneficiary counts between the Lery Dam and Di Irrigated Agriculture Activities.

Exhibit 6: Estimated Beneficiaries in Year 20, by ADP Sub- Activities

Estimated Beneficiaries in Year 20 by Project Activity

Project Activity Estimated Beneficiaries Agriculture Development

Lery Dam 65,920 65,920

Di Irrigation 26,577 ADD Farmer training non Di portion?

IMPAQ International, LLC Page 10 DRAFT Burkina Faso Evaluation Design Report

The IMPAQ team will also conduct the ex-post Beneficiary Analysis by examining the distribution of benefits by subgroup, such as gender and income groups. Our Beneficiary Analysis will aim to answer the following questions:

What proportion of the beneficiaries is poor?

How many people will benefit from increased household incomes because of the project?

How much, on average, will each individual beneficiary gain from the project?

How much will beneficiaries of different sub-activities gain from the service provided?

For each dollar invested by MCC, how much will be gained by the poor?

The following section presents the key features of previous research on the impact of irrigation investments and agricultural extension services.

IMPAQ International, LLC Page 11 DRAFT Burkina Faso Evaluation Design Report

3. LITERATURE REVIEW

The following review of the literature on the impact of improvements in irrigation and agricultural training services is not exhaustive. However, the review presents key features of previous research that are relevant to an assessment of the impact of irrigation investments and agricultural training services.

Development of the agricultural sector has been shown to play an important role in reducing poverty (Christiaensen et al., 2011).1 Lipton, Litchfield, and Faurès (2003) provide a conceptual framework linking irrigation and poverty. In short, irrigated land has the potential to reduce poverty by increasing land productivity and crop yields. Several empirical studies conducted in Sub-Saharan Africa have found positive effects of irrigation on poverty alleviation. Minten & Barrett (2006) used a unique, spatially-explicit dataset to study the link between agricultural performance and rural poverty in Madagascar. Results showed that communes with higher rates of adoption of improved agricultural technologies and broader access to improved irrigation infrastructure enjoyed lower real food prices, higher real wages for unskilled workers, greater profitability for farmers, and better welfare indicators. The authors noted that while access to improved irrigation infrastructure leads to higher uptake of improved technologies, the coefficient estimates was small. This indicated that irrigation alone could not stimulate a rapid uptake of improved technologies.

Pender et al. (2002) investigated changes in agriculture and land management practices in the highlands of the Tigray region of Ethiopia between 1991 and 1998. The authors found that irrigation was an important factor underlying different livelihood strategies and favoring the production of perishable cash crops. Furthermore, the authors found that irrigation contributed to intensified land use and changes in crop choice.

A recent study by Dillon (2010), investigated whether differences in the scale of irrigation projects were related to different impacts on poverty and production in Mali. Dillon used propensity score matching to establish a counterfactual comparison group and thus identify the impact of irrigation on poverty and production. The results of the study showed that small-scale irrigation schemes had larger effects on agricultural production and agricultural income than large-scale irrigation schemes. However, the results may be biased by household or individual unobservable characteristics. The advantage of the current evaluation is that we rely on randomized allocation of access to irrigated land to identify the impact of irrigation on poverty.2

Fan et al. (2000) analyzed the impact of six different types of public investments on growth and poverty reduction in rural China. The results of this study showed that government

1 Refer to Dethier and Effenberger (2011) on a review of the literature linking agricultural development to poverty reduction.

2 To our knowledge, this is the first project that allows for the investigation of irrigated land allocations through random allocation.

IMPAQ International, LLC Page 12 DRAFT Burkina Faso Evaluation Design Report investments in the rehabilitation and expansion of irrigation systems in rural China only had a modest impact on agricultural production growth. Furthermore, they had a lower impact on the reduction of rural poverty and income inequality.

Although most studies have found positive effects of irrigation on poverty, many irrigation schemes are usually complemented with other agricultural interventions. Isolating the impact of a single component can be challenging. Extension services complement agricultural investments by providing information and inputs to assist farmers at achieving the highest possible returns on their land. Anderson and Feder (2007) provide a conceptual framework outlining farmers’ demand for extension services. Empirical studies investigating the impact of extension services on productivity find mostly positive effects of extension services.3

Agricultural extension services have been attributed to the Green Revolution in Asia and have motivated the high adoption rates of extension services in Africa (Hazell, 2009). Earlier studies from Kenya and Burkina Faso have found positive and significant impacts of farmer training (Bindlish and Evenson, 1997). However, many of the earlier studies used cross-sectional data limiting the causal interpretation of their results. Krishnan and Patnam (2014) use longitudinal data from rural Ethiopia to investigate the relationship between extension agents and the adoption of fertilizer and improved seeds. The authors found that extension agents had a large, positive impact in the short term. However, the impacts on adoption decreased over time.

While these studies have contributed to our understanding of extension services and investments in irrigation, they do not provide definitive conclusions about the impacts of this type of interventions in developing countries. To overcome the deficits in previous studies, we will develop a counterfactual that is able to isolate the effect of ADP interventions on agricultural and economic outcomes. Thus, the results of this study will add to our knowledge and understanding of the impact of extension services and investments in irrigation.

3 Refer to Oehmke, Anandajayasekeram and Masters (1997) for a thorough review on the returns of extension services.

IMPAQ International, LLC Page 13 DRAFT Burkina Faso Evaluation Design Report

4. EVALUATION DESIGN

As described in the following sections, a rigorous impact evaluation needs to link the observed changes in outcomes to the intervention. An impact evaluation must identify a counterfactual to compare the outcomes of those who receive the program benefits with similar individuals who do not receive them. An impact evaluation requires the use of experimental or quasi-experimental evaluation methods that compare those affected by the intervention and those who are not affected.

4.1. Evaluation Questions

IMPAQ designed an evaluation to address a number of key research questions regarding the outcomes and impacts of each of the program interventions. The impact evaluations will address research questions under the following three outcome categories:

Agriculture production Household income Land tenure

Research Questions Related to Agricultural Production:

Has the land under production and the level of intensity of the land under production increased?

Has the volume of agricultural production and yields increased, including high-value crops (vegetables) or crops that require intensive water use (rice)?

Have farmers adopted improved farming practices, such as the use of chemical fertilizers or improved seeds?

Research Questions Related to Household Income:

Has there been a change in the components of household income?

Has there been a change in the total household income?

For each activity, we will address additional research questions that are specific to the activity.

Additional key research questions for farmer training activity include questions related to livestock practices.

Research Questions Related to Livestock Practices (Farming Training):

Have farmers vaccinated their animals?

Has the number of animal losses fallen?

Has livestock income increased?

Has access to veterinary services increased?

IMPAQ International, LLC Page 14 DRAFT Burkina Faso Evaluation Design Report

Additional key research questions for the Di Irrigation activity include questions related to land tenure.

Research Questions Related to Land Tenure (Di Activity)

Has the perception of land security improved?

Has the enhanced land security affected land investments?

In the following subsections we discuss our approach to conducting the impact evaluation for each activity. Each subsection includes a description of the methodology, sample size requirements, data sources and analysis plan. The last subsection discusses the timeframe of data collection and evaluation activities.

4.2. Farmer Training

To estimate the impact of the farmer training and to answer the key research questions, IMPAQ will use difference-in-difference (DID) with ex-ante matching.4 Matching ex-ante means matching prior to the full survey based on variables collected at the enumeration stage,5 as opposed to matching after the baseline survey data collection (ex-post). This method gives a greater probability of finding a match for each treated household based on the large pool of comparison households obtained at the enumeration stage, and thus reduce sample size requirements.

To implement DID with ex-ante matching, households were matched before the survey. This required a detailed enumeration of the treatment and comparison areas to collect variables for use in matching treatment and comparison households. In 2010, NORC at the University of Chicago and CERFODES in Burkina Faso conducted an extensive enumeration in the treatment and comparison villages selected by IRIS for 23,651 households. This sample included 4,964 households in treatment area and 18,687 in control areas. NORC designed the sample as a one-level stratified matched-pair sample (treatment-comparison).6 Using data from the enumeration survey, each household from the treatment zones was matched to a similar household from the comparison zones. The method used for matching was a form of generalized distance measure based on all these variables. Then, for each treatment unit, the best match was defined as the one closest to the treatment unit in terms of this distance measure (nearest neighbor).7 Next, a random sample of treatment households, together with their matched comparison households, was selected for the household survey sample.

4 The original overall evaluation approach was developed by the previous contractor IRIS. IMPAQ was selected in 2010 to complete and implement the original evaluation design.

5 An enumeration is a listing (census) of households carried out in the areas covered by the project (both treatment and comparison areas). This census is necessary to construct the sampling frame (universe) from which a given number of households will be sampled and included in the survey.

6 Households were randomly selected from treatment and comparison areas.

7 Details of matching process performed by NORC can be found in Caldwell (2011).

IMPAQ International, LLC Page 15 DRAFT Burkina Faso Evaluation Design Report

The variables used for the matching included:

Household’s primary economic activity: market gardening, irrigated agriculture, livestock breeding, fishing, fruit arboriculture, forestry, and crafts.

Household’s experience in agricultural production: work on irrigated perimeter, rainy season crops, and dry season crops.

Socio-economic level: roof and wall material, ownership and use of agricultural equipment, and ownership and use of oxen or donkeys.

Demographics: gender of household head and household size.

4.2.1. Sample Size

The previous evaluation contractor determined that the sample size needed to estimate the impact of the farmer training intervention was 2,164 households: 1,082 from the treatment area and 1,082 from the comparison area.8 However, baseline data was collected before training services became available. Therefore, the 2,164 households were selected and surveyed before knowing which households/individuals would receive training services. As a result, only a subset of the household survey sample actually received training services.

Among the surveyed households, only 624 households had at least one household member who had been trained. Thus, about 62% of the sampled households had received training either in agriculture or in livestock management.

Based on IMPAQ’s recalculation of statistical power, the reduced treatment sample will only be able to detect statistically significant program impacts on income if the program leads to changes greater than 21%.9 IMPAQ also performed additional power computations using other selected variables, in particular average use of chemical fertilizers (Urea) and livestock expenditures. The results of these additional power calculations indicate that, with the available sample, we will be able to detect statistically significant program impacts on chemical fertilizer if it is greater than 34% and on livestock expenditures if it is greater than 52%.

Results of the power calculations are summarized in Exhibit 7:

8 The initial sample size was determined by the National Opinion Research Center (NORC). We reviewed available documentation from NORC power calculations. Our understanding is that the original NORC power computations arrived at a sample size of 1,082 treatment and 1,082 comparison households to detect an MDE of approximately 30% in each of the four sectors initially considered separately for the analysis (agriculture, forestry, livestock and fishery – i.e. approximately 270 households in each sector).

9 Power computations on agricultural income had already been presented in a memo submitted to MCC in September 2013.

IMPAQ International, LLC Page 16 DRAFT Burkina Faso Evaluation Design Report

• In Column (1) we assume that data will be available from the 62% of households in the treatment areas that received training (approximately 620). We also assume an equal size comparison group. In this case, with 620 treated and 620 comparison households we estimate that we could detect approximately a 23% change in agricultural income, a 35% change in urea expenditures and a 52% change in livestock (vaccinations) expenditures.

• In Column (2) we assume that data will be collected from 620 treatment group members and approximately 1000 comparison group HHs. In this case, a 21%, 34% and 48% can be detected on agricultural income, urea and livestock expenditures

In summary, we estimate that the MDE under different sample size scenarios will be 21% to 23% for income, 34% to 35% for urea expenditures and 48% to 52% for livestock expenditures.

That is, we will be able to detect a change in agricultural income if the change is in the 21% - 23% range or larger, and changes in urea (livestock) expenditures if they will be 34% to 35% (48% to 52%) or larger.

Exhibit 7: Power Computations, Selected Variables

Variable (1) (2) Agricultural Income Mean 477,580 SD (treatment) 614,633 SD (comparison) 417,630 N treatment 620 620 N comparison 620 1,000

MDE 108,307 101,600

MDE/mean 22.7% 21.3% Urea Expenditures Mean 27,640 SD (treatment) 61,700 SD (comparison) 23,800 N treatment 620 620 N comparison 620 1,000

MDE 9,639 9,398

MDE/mean 0.349 0.340 Livestock Expenditures Mean 11,300 SD (treatment) 31,400 SD (comparison) 25,900 N treatment 620 620 N comparison 620 1,000

MDE 5,933 5,457

MDE/mean 0.525 0.4829

IMPAQ International, LLC Page 17 DRAFT Burkina Faso Evaluation Design Report

Notes: the following assumptions on the parameters have been used

• Power = 80% ; significance level =5% ; Baseline-follow correlation in income = 0.3 ; Attrition rate between baseline and follow-up = 15%; non-response rate = 10%

• Data on agricultural income come from Burkina Faso PNG2 data (2011) – (FCFA amounts)

SD for treatment uses data from Boucle de Mouhon & Cascades regions; SD for comparison uses data from all regions.

• Data on urea and livestock expenditures come from NORC baseline report

4.2.2. Data Sources

MCA-BF contracted NORC and CERFODES to collect the baseline data for the evaluation using a household survey in the treatment and comparison areas. The first round of the baseline data was collected immediately after the dry season of 2011. The second round was collected immediately after the rainy season of 2011. A separate contractor will be in charge of the follow-up data collection.

The baseline survey collected data on key outcomes such as earnings and agricultural production. It also collected data on household demographics, socioeconomic status, and other important background characteristics that the evaluation will use as control variables in regression models. The global household survey is comprised of three questionnaires:

A household questionnaire to gather information on household composition, as well as the activities of each household member.

An agricultural questionnaire to collect detailed information on crops, irrigation techniques and land use.

A livestock questionnaire to collect information on livestock ownership, income for the sale of livestock products and vaccinations.

In addition to the global household survey, CERFODES carried out an independent crop yield survey on a sample of households and a barymetric survey on a sample of cattle. The crop yield survey replaced the crop yield data obtained from the baseline household survey, as the baseline survey data on crop yields was known to have significant data quality issues. The crop yield data collection used crop yield measurement squares (carrés de rendement) where the harvest is directly measured to come up with an estimate for the entire field. The barymetric survey obtained data on cattle weight and milk production on a total of 704 cattle (494 cattle in the intervention area and 210 in the control area).

Exhibit 8 summarizes the key data elements in the baseline and follow-up global household survey, crop yield survey and the barymetric survey

IMPAQ International, LLC Page 18 DRAFT Burkina Faso Evaluation Design Report

Exhibit 8: Main Variables for Diversified Agriculture Questionnaires

Questionnaire/Description of Variables

HOUSEHOLD QUESTIONNAIRE

Household members' relations to household head: age, gender, education, marital status Activities of the various household members Participation in agricultural organizations

AGRICULTURAL QUESTIONNAIRE

Plot size, agricultural production in the plot by crop Irrigation techniques Labor inputs and other agricultural inputs and equipment

LIVESTOCK QUESTIONNAIRE

Livestock-type and number of animals and value Production and commercialization of animal products Livestock deaths Vaccinations

CROP YIELD QUESTIONNAIRE

Agricultural production in the plot by crop

BARYMETRIC QUESTIONNAIRE

Livestock measurements and weight Quantity of milk produced per cow Livestock-type and number of animal Production and commercialization of animal products Livestock deaths Vaccinations

Following the analysis of the data collected at baseline, we will adjust the follow-up survey to improve data accuracy and collect additional information on agricultural production.

4.2.3. Data Analysis

Prior to assessing the impact of project interventions, IMPAQ will analyze each data file (baseline, midline, and follow-up data) and provide MCC with a Data Quality Assessment (DQA), Data Quality Report (DQR), Baseline Report, Midline Report, and Final Report.

The DQA will assess the quality of data after we receive the data from the data collection firm. The data will be checked for completeness and internal consistency. In particular, IMPAQ will:

• Check for duplication of records;

• Ensure that all variables are consistently coded;

• Ensure that all datasets and datasets from different rounds and be merged; and

• Produce frequencies and cross tabulations to identify missing values, outliers, and other data problems;

IMPAQ International, LLC Page 19 DRAFT Burkina Faso Evaluation Design Report

Any problems identified with the data will be communicated with the data collection firm in order to correct any data issues or problems.

Following the DQA, IMPAQ will prepare a DQR which will report on the quality of the data for the purpose of the impact evaluation. The DQR will involve a review of datasets, data collection documentation, training manuals, baseline reports, sampling documentation, email exchanges between IMPAQ and MCA-BF, and field visit trip notes. The DQR will focus on the key MCC indicators and variables of interest for the impact evaluation. The goal of the DQR is to ensure the integrity and reliability of the data with particular emphasis on MCC indicators and the variables of interest for the impact evaluation.

Additionally, IMPAQ will produce a baseline, midline, and follow-up report. These reports will include a brief description of the project, the evaluation design, and the data collection activities. It will then summarize the data.

Appendix B presents a table shell that will be used to present characteristics of the treatment and comparison groups. The results of the baseline, midline, and final analysis will be shared with stakeholders in Washington D.C. and Burkina Faso. In addition, the report will include descriptive statistics of household characteristics and differences between the treatment and comparison groups of the main variables of interest. We will present the results of the analysis in a variety of formats, including charts and tables. Appendix C shows an illustrative outline of the baseline report.

IMPAQ submitted a DQA for the interim crop yield data on April 14, 2014. IMPAQ submitted the final revised DQR for the baseline data to MCC on July 24, 2014. IMPAQ found that the data is of high quality and can be used to construct an analysis file for impact evaluation.

Impact analysis

To estimate the impact of farmer training, animal health and value chain management, the evaluation uses a Difference in Difference with matching methodology. We will specify econometric models that will compare how household outcomes living in areas affected by the project changed over time with the changes in outcomes of households in the comparison group.

This approach is captured by a multivariate DID regression.

After follow-up data is collected for the treatment and (matched) control group, DIDs will be estimated on the matched sample using the following regression approach:

Outcome = 𝛼𝛼 + 𝛽𝛽𝛽𝛽 + 𝛾𝛾𝛾𝛾 + 𝛿𝛿(𝛽𝛽 ∙ 𝛾𝛾) + 𝜆𝜆𝜆𝜆 + 𝜀𝜀. (1)

The left-hand side of the equation is the outcome variable of interest. The variables on the right-hand side include:

IMPAQ International, LLC Page 20 DRAFT Burkina Faso Evaluation Design Report

A dummy variable 𝛽𝛽 which equals 1 if the observation is in the treatment group and zero otherwise. The estimate of 𝛽𝛽 captures the group effect. In other words, 𝛽𝛽 controls for any differences in the outcome variable that are associated with being in the treatment group.

A dummy variable 𝛾𝛾, which equals 1 in the follow-up year and zero in the baseline year. The estimate of 𝛾𝛾 captures the time effect. In other words, 𝛾𝛾 controls for any changes in the outcome variable that occur over time and are common for treatment and comparison group members.

An interaction term (𝛽𝛽 ∙ 𝛾𝛾), which equals 1 if the observation is in the treatment group and in the follow-up year and zero otherwise (i.e., for comparison group members in both the baseline and follow-up years, and for the treatment group in the baseline year). The estimate of 𝛿𝛿 captures the impact of the project on the outcome variable— this is the parameter of interest.

A vector 𝜆𝜆 of other relevant explanatory variables that may be related to the outcome of interest and will help control for baseline household characteristics. For household models, 𝜆𝜆 will include the education, gender, and age of the household head. Including these explanatory variables will reduce the amount of unexplained variation in the outcome variable, thereby increasing the accuracy of our parameter estimates.

For each regression model, we will estimate: the parameters 𝛼𝛼, 𝛽𝛽, 𝛾𝛾, 𝛿𝛿, and the elements of the vector 𝜆𝜆. All things being equal, positive parameter estimates will indicate that the corresponding right-hand side variable is associated with an increase in the outcome measure.

Likewise, negative parameter estimates will indicate a negative association. We will use t-tests to measure the statistical significance of the parameter estimates. Where we find statistically significant differences, we can be confident that the corresponding right-hand side variable has an effect on the outcome variable.

IMPAQ will estimate two separate estimates. As noted previously, approximately 62 percent of treated households received training. If we restrict the sample to the approximately 620 households that received training and their matched pairs, the estimate of β will capture the average treatment effect. In other words, β will estimate the average effect of receiving training.

The second estimator that IMPAQ will estimate is the intent to treat estimator. This analysis will use the full sample of households (1,082). Using the full sample the estimate of β will capture the Intent to Treat. β will identify the effect of being eligible for farming training.

Sub-group analysis

We will also examine whether the impact of the projects differ by subgroups, such as gender and income. For example, for the DID model, we can expand the basic regression model of equation

(1) to include terms that capture other potential subgroup effects:

Outcome = 𝛼𝛼 + 𝛽𝛽𝛽𝛽 + 𝛾𝛾𝛾𝛾 + 𝜉𝜉𝜉𝜉 + 𝛿𝛿1(𝛽𝛽 ∙ 𝛾𝛾) + 𝛿𝛿2(𝛽𝛽 ∙ 𝜉𝜉) + 𝛿𝛿3(𝛾𝛾 ∙ 𝜉𝜉) + 𝛿𝛿4(𝛽𝛽 ∙ 𝛾𝛾 ∙ 𝜉𝜉) + 𝜆𝜆𝜆𝜆 + 𝜀𝜀(2)

IMPAQ International, LLC Page 21 DRAFT Burkina Faso Evaluation Design Report

As before, the left-hand side variable is the outcome of interest. The right-hand side variables are the same as in the basic regression model, except for the explanatory variables, added for the subgroup models:

A dummy variable S that equals 1 if the observation is in the subgroup and 0 otherwise.

The estimate of ξ accounts for differences in outcomes associated with the subgroup of interest.

An interaction term (T ∙ S) equals 1 if the observation is in the treatment group and the subgroup of interest, and zero otherwise. In equation (2), an interaction term (F ∙ S) that equals 1 if the observation is in the follow-up period and the subgroup of interest, and zero otherwise. The estimate of δ3 captures the incremental time effect for observations in the subgroup.

In equation (2), the incremental treatment effect for the subgroup of interest is measured by the interaction term (T ∙ F ∙ S) that equals 1 if the observation is in the treatment group, in the follow-up period and in the subgroup of interest. The estimate of δ4 captures the potential differential effect of the DA project for the subgroup. This is the parameter of interest.

In equation (2), the incremental treatment effect for the subgroup of interest is measured by the coefficient δ4 on the interaction term (T ∙ S) that equals 1 if the observation is in the treatment group and in subgroup of interest.

Thus, our estimate of 𝛿𝛿4 (𝛿𝛿2) will indicate whether the impact of the project is different for the subgroup of interest.