Annex_IIb-_Farmer_Training_Baseline_DQR.pdf

Millennium Challenge Corporation

MCA-Burkina Faso Agricultural Survey in the Sourou Valley and Comoé Basin: Data Quality Review Report

Contract No:

Task Order No. MCC-10-0111-CON-20 TO02

July 23, 2014

Submitted to:

Jack Molyneaux Project Officer: Jennifer Witriol

Millennium Challenge Corporation 875 15th Street, NW

Washington, DC 20005

Submitted by:

IMPAQ International, LLC 10420 Little Patuxent Parkway

Suite 300

Columbia, MD 21044 www.impaqint.com http://www.impaqint.com/

TABLE OF CONTENTS

1. INTRODUCTION

2. STUDY ZONES

3. HOUSEHOLD SURVEY

3.1 Sampling Frame

3.2 Household Sampling and Data Collection

3.3 Data Collection Instruments

3.4 Key Variable Distributions

3.5 Data Issues Encountered

4. CROP YIELD MEASUREMENT SQUARES SURVEY

4.1 Sampling Frame

4.2 Crops Stratification and Farmer Sampling

4.3 Key Variable Distributions

4.4 Data Issues Encountered

5. FISHING SURVEY

5.1 Sampling Frame

5.2 Sampling

5.3 Data Collection

6. INSTITUTIONAL SURVEY

6.1 Number of Individuals to Interview

6.2 Data Collection

6.3 Data Issues Encountered

7. CONCLUSION

REFERENCES

APPENDIX 1: DIAGRAM OF DATA COLLECTION

APPENDIX 2: LIST OF STUDY VILLAGES AND PROVINCES

APPENDIX 3: FINAL STATUS OF DATA COLLECTION

APPENDIX 4: SUMMARY OF DATA QUALITY REVIEW STEPS

APPENDIX 5: SUMMARY STATISTICS, SELECTED VARIABLES

APPENDIX 6: POWER COMPUTATIONS

IMPAQ International, LLC Page i Data Quality Review Report

TABLE OF EXHIBITS

Exhibit 1: Diagram of Survey Relationships Exhibit 2: Map of Burkina Faso by Provinces Exhibit 3: Enumeration Results Exhibit 4: Initial Sample of the Household Survey Exhibit 5: Survey Modules Exhibit 6: Average Planted Area Per Crop (Hectares) Exhibit 7: Income Per Household (FCFA) From Selling Agricultural Products Exhibit 8: Income Per Household (FCFA) From Selling Forest-Derived Products Exhibit 9: Income Per Household (FCFA) from Income-Generating Activities Exhibit 10: Number of Measurement Squares Expected and Actually Marked in Dry Season Exhibit 11: Number of Measurement Squares Expected and Actually Marked in Rainy Season Exhibit 12: Distribution of Farmers by Age and Gender Exhibit 13: Area and Yield Per Hectare of Irrigated Cereal Crops in Dry Season Exhibit 14: Area and Yield Per Hectare of Irrigated Cereal Crops in Rainy Season Exhibit 15: Distribution of Fishing Sites Listed for Data Collection and Fishing Sites Actually Visited Exhibit 16: Number of Interviews to Conduct per Organization in Each Region Exhibit 17: Number of Perimeter Surveys to be Conducted Exhibit 18: Sample Perimeters for Comoé Basin Exhibit 19: Datasets for NORC Household Survey Exhibit 20: Summary of Missing Values on Selected Variables Exhibit 21: Summary of Some Consistency Checks in the Household and Agriculture Modules in Dry Season Exhibit 22: Demographic Characteristics in Dry Season Exhibit 23: Household Members Activities in Dry season Exhibit 24: Parcel Characteristics in Dry season Exhibit 25: Land Area Under Production (Hectares), Selected Crops in Dry Season Exhibit 26: Use of Agricultural Inputs on the Parcel Exhibit 27: Power Computations, Selected Variables

IMPAQ International, LLC Page ii Data Quality Review Report

1. INTRODUCTION

This report reviews the quality of the baseline (pre-intervention) data collected by the National Opinion Research Center (NORC) and the Centre d'Etudes, de Recherches et de Formation pour le Développement Economique et Social (CERFODES), the local partner in Burkina Faso. The data collection was contracted by the Millennium Challenge Account – Burkina Faso (MCA-BF) to evaluate the Millennium Challenge Corporation’s Agricultural Development Project in the country.

In 2008, the government of Burkina Faso and the Millennium Challenge Corporation (MCC) signed a five-year USD 480.9 million Compact for 2009–2014 to finance a poverty-reduction program. The overall objective of the Compact is to contribute to poverty reduction in the country via economic growth. The Compact consists of four distinct projects:

Agricultural Development Project, Rural Land Governance Project, Roads Project, and

Burkinabé Response to Improve Girls’ Chances to Succeed.

The Agricultural Development Project (ADP) seeks to extend the agricultural use of land as a way to increase the quantity and value of agricultural production. This will boost income and employment in rural areas and make the rural economies more competitive. Specifically, the project targets the main constraints in rural Burkina Faso: water management, low capacity, as well as poor access to information, markets, inputs, and credit. The expected results include growth in agricultural production, an increase in irrigated land and improved access to rural credit in the intervention zones.

The project has three main activities:

Water Management and Irrigation – the goal of this activity is to promote improved water management resulting in better access to water through increased water resource protection and more efficient irrigation, including water availability, water delivery, flood control and dam safety. This was expected to help support and protect investments (infrastructure and human/institutional capacity) in the Sourou Valley and Comoé Basin.

Agricultural Diversification (DA) – builds on the delivery of water resources in the project zones by supporting on-farm production and related activities throughout the agricultural value chain.

Access to Rural Finance (ARF) – provides access to term loans by participating financial institutions as a way to fund creditworthy borrowers in the region.

Each of these 3 main activities is in turn, composed by specific sub-activities as follows:

Water Management and Irrigation consists of the following sub-activities:

Development of Integrated Water Resource Management (IWRM) plans,1

1 This will cover the Sourou Valley and nine villages in the Comoé Basin.

IMPAQ International, LLC Page 1 Data Quality Review Report

Lery Dam Investment, Di Irrigation Investment, including provision of irrigated land to Persons Affected by the Project (PAPs), 5 villages with traditional rights in the area and Non-Persons Affected by the Project (Non-PAPs) and

Institutional Capacity Building for Water User Associations (WUAs) and the Sourou Valley Authority (AMVS) to provide Operation and Maintenance O&M to water infrastructure.

The DA activity consists of the following sub-activities:

Technical Assistance to Farmers, Strengthening Animal Health Services, Value Chain Development, and

Market Information System (MIS) and Rehabilitation of Rural Markets.

The Access to Rural Finance (ARF) comprises the following sub-activities:

USD 10 million loan fund toward the Establishment and Implementation of a Rural Finance Facility (RFF) that provided loans to Participating Financial Institutions (PFIs) for approved loans to rural agricultural end borrowers.

Capacity Building support PFIs to expand rural lending, including training in procedures and policies to evaluate applications and disburse agriculture-related loans.

Business Development Services (BDS) to build business skills among agriculture enterprises and to assist in the development of bankable agriculture projects, reducing risk for the PFIs.

The original intent of the baseline survey data collection was to use the data to cover all activities and sub-activities of the ADP. However, the NORC survey is now considered to cover only selected DA sub-activities. Specifically, the survey covers the Technical Assistance to Farmers, the Strengthening Animal Health Services and the Value Chain Development sub-activities. The Market Information System and Rehabilitation of Rural Markets are not covered by the NORC survey.

Since 2011, a local contractor (AD10) has been implementing the Technical Assistance to Farmers sub-activity. Program services have included the delivery of training sessions and tool kits to beneficiary farmers in the intervention zones. The impact evaluation of this sub-activity has been designed to answer four research questions:

1. Have farmers intensified and diversified their production of crops?

2. Has the volume of agricultural production increased? If so, by how much?

3. Has there been an impact of project activities on the sources and level of household income?

Has livestock income increased?

4. Does the impact vary by gender, age and income group?

IMPAQ International, LLC Page 2 Data Quality Review Report

IMPAQ prepared an Evaluation Design Report for this activity.2

In April 2011–May 2012, NORC and CERFODES conducted the following:

An enumeration survey in treatment and comparison areas, a household surveys covering the dry and rainy seasons, two crop yield measurement surveys (carrés de rendement) covering the dry and rainy seasons, a survey of aquaculture production (fishing), and an institutional community survey during dry season.

Exhibit 1 represents how these surveys relate to each other (See Appendix 1 for more detailed diagram).

The objectives and Terms of Reference (TOR) of the baseline data collection are presented in NORC’s Final Data Collection Report as follows:

MCA-BF contracted NORC and its local partner CERFODES to perform data collection in order to establish a baseline for the intervention for the purpose of performance monitoring and impact evaluation. According to the Terms of Reference in the Request for Proposals, the general objective of the project is to carry out “surveys in the areas of intervention of the agricultural projects to establish a database for the monitoring and evaluation of their implementation.” The specific objective is to carry out “a survey in order to collect information for the establishment of a baseline of selected indicators for the MCA M&E Plan, in order to monitor the implementation of the PDA.”3

As indicated in Exhibit 1, some of the surveys were conducted in both the treatment and comparison areas, while other surveys were conducted in the treatment areas only. In addition, as described in NORC TOR, while some of the surveys had been designed to establish a baseline for impact evaluation, others had been collected for monitoring purposes only.

In particular, the household survey (Box A in Exhibit 1) was designed to obtain descriptive information to satisfy the project monitoring objective, as well as information to establish a baseline to measure the impact of the ADP intervention. This survey was collected from a random sample of households in the treatment and comparison areas. As a result, these data can serve as a basis for measuring program impacts. That is, by comparing baseline data with post-intervention survey data, we can estimate program impacts of the ADP intervention (see more detailed diagram in Appendix 1). This survey contains information on household characteristics, use of agricultural inputs, and agricultural production for various crops, among other variables.

As the information from the household survey is self-reported, this raises the concern that crop yields obtained from household declarations may be subject to measurement error. To mitigate this concern, a crop yield survey (carrés de rendement) was designed to obtain objective measures of crop yields to be compared to the household self-reported measures and correct them using econometric extrapolation

2 Evaluation Design Report, Impact of the Agricultural Development Project (Draft), June 2014 3 NORC, FINAL DATA COLLECTION REPORT, MCA-Burkina Faso Agricultural survey in the Sourou Valley and Comoé Basin (2012), page 21.

IMPAQ International, LLC Page 3 Data Quality Review Report techniques. This survey was collected on a sub-sample of households taken from the overall household survey (Box B in Exhibit 1).

Exhibit 1: Diagram of Survey Relationships

Intervention Zones (Sourou Valley and Comoé Basin)

Treatment Sample (1082)

Comparison Zones (17 provinces and Comoé )

[A] Household Survey Dry Season (October 2010 to May 2011)

Rainy Season (June 2011 to November 2011)

[D] Fishing Survey Cold

[C] Crop Yield Measurement Squares

Survey Rainy & Dry

[E] Institutional Survey Dry

Burkina Faso

Comparison Sample (1082)

[B] Crop Yield Measurement Squares

Survey Rainy

In addition to the crop yield survey on the household survey sub-sample, NORC/CERFODES used the same methodology to implement another crop yield survey on a sample of farmers in the treatment villages only (see Box C in Exhibit 1). This survey was designed to serve the program monitoring objective. Specifically, the objective of this survey was to measure the production of rain fed, market garden, off-season and irrigated crops. As explained in more detail in Chapter 4, this crop yield survey uses a different sampling frame than the household survey.

The fishing survey was drawn from a sample of households in the intervention zones that earned income through fishing. These households, however, were not part of the treatment sample in the

IMPAQ International, LLC Page 4 Data Quality Review Report household survey. Furthermore, the fishing survey was not collected to serve as a baseline for an impact evaluation. Rather, the fishing survey can only provide performance monitoring data. Finally, the institutional survey was carried out in treated areas only and targeted institutions in communes and villages. Data was collected on livestock, fishing, forestry and access to markets and other topics. This is an independent survey data collected for monitoring purposes only and not impact evaluation.

This overview confirms that these surveys had been collected for different purposes, several of the surveys used different sampling frames, and there is little relationship across surveys.

Based on our review of the NORC/CERFODES data, we have found that the household baseline data could serve as a baseline for measuring program impacts. We have, however, identified two significant issues:

1. Reduced sample size: the treatment sample includes fewer households than originally anticipated. That is, the household sample includes only 624 households who received training services as compared with the expected sample of 1082 households in the treatment area,4 and

2. Quality of the crop yield measurement data: Our review as well as NORC’s assessment of the crop yield data suggest serious limitations in data accuracy.

Boxes 1 and 2 summarize these two issues. We provide more detailed information in subsequent sections of this report.

Issue #1: Reduced Sample Size The baseline survey was collected in 2011–2012 from a random sample of households in the treatment area. The data was collected before training became available and before it was possible to identify the beneficiaries of the training program.

The training program carried out by AD10 did not use the same identification code as the baseline survey. As a result, AD10 data could not be merged with the household baseline data. To overcome this problem, two options were considered: (1) match AD10 and baseline data using all the information available (e.g., name, location, gender, etc.), or (2) collect new data.

After reviewing the options, MCC decided to match the records of trainees collected by AD10 with the household baseline survey data. After several AD10 matching attempts, MCA-BF and AD10 proceeded with collecting new data from all households in intervention area. They reported that 62% of the baseline households had been matched.

With only 62% of the expected treatment household sample size, the available sample is substantially smaller than expected in the original design. IMPAQ reestimated the statistical power calculations with this sample and determined that the matched sample size will only be able to detect relatively large program effects (i.e., an impact on income of 21% or larger).

Issue #2: Accuracy of Crop Yields The same methodology to measure crop yields was used to implement both surveys, i.e. the one on the

4 Enquête d’identification des ménages communs AD10 et Enquête globale. Synthèse des résultats by MCA-BF.

IMPAQ International, LLC Page 5 Data Quality Review Report sub-sample of households from the global household survey, and the one on the separate sample of farmers in treatment villages. The surveys, therefore, suffer from the same methodological concerns.

In particular, during the data-cleaning phase, it became clear that some of the rice and maize crop yield values from the measurement squares were inaccurate (e.g., some values were too high to be plausible). Enumerators collected crop yield weights were both before and after crop processing took place. It is difficult to compare these measures as a result of the differential moisture content in individual crops. Ideally, the extra moisture in crop harvest should be removed before reporting crop yields.

To correct the inaccuracies, the data collection firm established an upper bound for rice yields based on agriculture literature. All values that were coded higher than upper bound were reset (i.e. top coded) to the upper bound.5 In addition, maize values which were weighted and recorded before crop processing were excluded from the dataset. We believe these issues compromise the validity of the crop yield data and we believe that using these data could lead to biased impact estimates.

Another data limitation of both crop yield surveys is small sample sizes. In the crop yields survey that was intended to be used to correct households declarations (crop yield survey B in Exhibit 1), data was collected on only approximately 300 farmers and only during the rainy season. Similarly, in the crop yields survey designed to collect information for monitoring purposes (crop yield survey C in Exhibit 1), only 85 farmers were surveys during the dry and 143 during the rainy season. These small sample sizes together with the other data limitations identified create serious data limitations.

In the first part of this report we focus on the review of the household survey data which are the core data to be used for impact evaluation and contain data necessary for the impact analysis. Inasmuch as the crop yield measurement survey was collected on a small sample and the fact that data was collected only in the rainy season, we do not believe that these data can be used to enhance the impact evaluation.

In Chapters 4 through 6, we review the remaining surveys collected by NORC/CERFODES (crop yield measurement, fishing, institutional). These surveys may be used for monitoring key sectors of interest in the intervention areas.

For each survey discussed below, we present the sampling frame, sampling coverage, data collection, descriptive statistics of key variables and the data issues encountered. The rest of the report is organized as follows: 2) Study zones; 3) Household survey; 4) Crop Yield Measurement Squares Survey;

5) Fishing survey; 6) Institutional survey and 7) Conclusion.

5 NORC, FINAL DATA COLLECTION REPORT, MCA-Burkina Faso Agricultural survey in the Sourou Valley and Comoé Basin (2012), page 120

IMPAQ International, LLC Page 6 Data Quality Review Report

2. STUDY ZONES

Burkina Faso is located in a vast plateau south of the bend of the Niger River. Most of the country is flat and has relatively low rainfall. Burkina Faso is home to an important hydrographic network which comprises three basins: the Volta, the Comoé and the Niger. The Agriculture Development Project (ADP) is designed to improve conditions in the Comoé Basin and the Sourou Valley located in the Volta Basin (See Exhibit 2). Thus, the intervention zones cover 30 villages from two provinces, the Comoé and Sourou (green areas in the map). The comparison zones cover 60 villages from provinces chosen to be as similar as possible to the intervention zones (including the Comoé province). These provinces were selected by MCA-BF and IRIS (the previous evaluation contractor).

Exhibit 2: Map of Burkina Faso by Provinces

MCA-BF chose the 30 treatment villages in Sourou and Comoé provinces based on selection criteria set to maximize the possibility of success.6 Consequently, IRIS and NORC selected 60 comparison villages in 17 provinces and Comoé through a purposive sampling rather than a probabilistic procedure. For villages in each of the two intervention provinces, comparison villages were selected to satisfy the following criteria:

6 Leonard, Zalla, Bambio and IRIS, 2010.

IMPAQ International, LLC Page 7 Data Quality Review Report

The comparison provinces and villages match, most importantly, according to average annual rainfall as nearly all farmers in the treatment zone practice rain fed agriculture as a primary occupation.

The villages have similar size of irrigation perimeter.

They have a similar irrigation system.

Their population sizes are of similar levels.

Throughout this report, “treatment” in provided tables refers to 30 villages in the Sourou and Comoé provinces. Appendix 2 lists all villages and provinces which are part of the treatment and comparison areas.

IMPAQ International, LLC Page 8 Data Quality Review Report

3. HOUSEHOLD SURVEY

The household survey provides results for indicators related to agriculture, livestock, forestry, access to markets, access to credit, expenditure and health. The data was collected in both the intervention areas and the comparison zones.

To understand the sampling frame for the surveys, it is important to understand the original evaluation design developed by IRIS. IRIS had planned to implement a double-difference with matching methodology. The evaluation design can be summarized as follows:

1. The treatment group consists of a sample of households located in villages from the intervention zones.

2. The comparison zones were selected to be as similar as possible to Sourou and Comoé in terms of the variables considered having an effect on the outcomes of the evaluation.

3. The data used for matching and household-level stratification was collected through an enumeration survey in every intervention and comparison village. Using data from the enumeration survey, the comparison households were matched to treatment households.

4. The matched pairs were selected using a stratified sample.

Additional details on the household survey design and data collection are presented in the following sections.

3.1 Sampling Frame

To establish a sampling frame for the household survey, NORC and CERFODES produced a full list of the households in both the treatment and comparison villages. They generated an exhaustive list of every household in the 90 villages selected by IRIS and MCA-BF, 30 of which were treatment villages and 60 of which were comparison villages.

In each village, enumerators traveled from compound to compound, or from household to household, depending on the physical configuration of the village. In each compound, they counted the number of economic households and collected data from the heads of each household using a simple questionnaire with questions needed for matching. At the end of the enumeration, a total of 23,651 households had been listed (see Exhibit 3).

The variables collected during the enumeration survey included:

Activity sector of the household: market gardening, irrigated agriculture, livestock breeding, fishing, fruit arboriculture, forestry, crafts;

Agricultural production: work on irrigated perimeter, rainy season crop, dry season crop;

Socio-economic level: roofing and wall material, ownership and use of agricultural equipment, ownership and use of oxen or donkeys;

Demographics: gender of the head of household and size of the household.

IMPAQ International, LLC Page 9 Data Quality Review Report

Exhibit 3: Enumeration Results

Number of households enumerated Number of villages enumerated Treatment 4,964 30 Comparison 18,687 60 Total 23,651 90 Source: NORC report

3.2 Household Sampling and Data Collection

The sample was designed as a one-level stratified matched-pair sample (treatment-comparison). Each pair includes one treatment and one comparison household. Using data from the enumeration survey, each household from the treatment zones was matched to a similar household from the comparison zones. Next, a random sample of treatment households together with their matched comparison households was selected for the household survey sample. As indicated in Exhibit 3, the resulting sample included 1,082 pairs of households (a total of 2,164 households) from 12 regions and 887 villages.

Exhibit 4: Initial Sample of the Household Survey

Comparison zone Treatment zone Total Households 1,082 1,082 2,164 Regions 10 2 12 Provinces 18 2 20 Communes 34 5 39 Villages 58 30 88 Source: NORC report

The data collection was conducted in two phases to accurately reflect the two main agricultural cycles:

Dry season – This data collection used October 2010 to May 2011 as a reference period. The data was collected from May to June 2011.

Rainy season –This data collection used June 2011 to November 2011 as a reference period. The data was collected from December 2011 to January 2012.

During the dry season, a total of 2,172 heads of household were interviewed, from the originally planned sample of 2,164. The target number of households was reached, with a slight positive difference due to replacements and additions of household pairs. The refusal rate was extremely low (only four cases) and 98 households could not be located. The replacement rate was 4.6% (98 households).

7 While there were 90 villages in the enumeration, the final sample of villages for the household survey included only 88. This is due to matching being done at the household level and not the village level. Thus, there were no matched households from 2 of the comparison villages.

IMPAQ International, LLC Page 10 Data Quality Review Report

Replacements and additions had an impact on the progress of the interview teams. These replacements were justified for several reasons:

Incorrect landmarks given during listing, Misspelled household names, Disappearance of some households and

Permanent relocation of some households.

During the rainy season, data was collected at a total of 2,163 households. Eighteen households could not be reached for a variety of reasons. Some heads of household could not be found, others were absent, had emigrated or moved. Some refused to respond altogether. This led to a total loss of 0.82% from the dry season. The coverage rate was 99.17%, which is sufficient for a high-quality analysis.

3.3 Data Collection Instruments

The household survey used a questionnaire composed of seven modules, each focused on a different area of interest. This questionnaire was used for both the dry-season and rainy-season phases (See Exhibit 5).

Exhibit 5: Survey Modules

1. Household

2. Agriculture

3. Livestock

4. Forestry

5. Credit and expenditures

6. Food security

7. Health

Each module comprises several sub-sections which resulted in several datasets. Most data are present in both the dry and humid season. A more detailed description of data modules with selected data quality checks performed on the data is provided in Appendix 4.

3.4 Key Variable Distributions

In this section we provide a description of few variables. A more detailed summary for selected variables is presented in Appendix 5.

Exhibit 6 presents the average cultivated area by type of crop. As indicated in the exhibit, the planted area for maize was higher during the rainy season than the dry season. The area allocated for rice was similar in both seasons. Maize generally occupied the largest planted areas. Other crops included tomatoes, cabbage, peppers, okra, onions, millet, sorghum, beans, black-eyed peas, groundnuts and cotton, among others.

IMPAQ International, LLC Page 11 Data Quality Review Report

Exhibit 6: Average Planted Area Per Crop (Hectares)

Crop Statistic Dry season Rainy season Comparison Treatment Total Comparison Treatment Total

All crops Mean 0.34 0.44 0.40 0.72 0.74 0.73 St. d. 0.52 0.85 0.72 0.93 1.08 1.02 N* 1,702 2,109 3,811 5,985 6,741 12,726

Maize Mean 0.36 0.53 0.43 0.89 0.75 0.80 St. d. 0.40 1.05 0.75 1.16 1.14 1.15 N 168 126 294 792 1,501 2,293

Rice Mean 0.86 0.46 0.71 0.44 0.93 0.70 St. d. 0.96 0.46 0.83 0.54 1.26 1.02 N 215 130 345 802 887 1,689

Source: IMPAQ computations using agriculture module data (PARTEI_E1 and M_AGRICULTURE_PARTIE_E1 for dry and rainy season, respectively. See Appendix 4 for a detailed list of datasets). *N is the number of parcels planted with any given crop. A household can operate multiple plots and in this case each plot is counted differently.

Households earned an average of FCFA 236,500 from selling crops during the rainy season and FCFA 689,810 during the dry season (See Exhibit 7). The averages include only households which sold agricultural products.

Exhibit 7: Income Per Household (FCFA) From Selling Agricultural Products

Statistic Dry season Rainy season Comparison Treatment Total Comparison Treatment Total

Mean 425,879 915,264 689,027 249,925 223,990 236,500 St. d. 775,024 2,574,533 1,974,485 399,555 412,507 406,301 N 570 663 1,233 452 485 937 Source: IMPAQ computations using agriculture module data (PARTEI_E5SUITE and M_AGRICULTURE_PARTIE_E5SUITE for dry and rainy season, respectively. See Appendix 4 for a detailed list of datasets).

During the dry season, the mean revenue from selling forest-derived products for households that sold these products was FCFA 69,855. In the rainy season, the mean revenue was 51,440 FCFA (See Exhibit 8).

Exhibit 8: Income Per Household (FCFA) From Selling Forest-Derived Products

Comparison Treatment Total Comparison Treatment Total

Mean 77,940 62,080 69,855 53,900 48,070 51,440 St. d. 148,749 115,846 133,135 295,975 102,220 234,447 N 328 341 669 304 222 526 Source: NORC report

Average revenue from income-generating activities (such as trade or wage activities) was higher in the dry season by approximately FCFA 30,000 (See Exhibit 9). This difference may be explained by the fact that many households are busy with agricultural activities during the rainy season. Additionally, income from these activities is higher in the treatment areas than in comparisons areas during both seasons.

IMPAQ International, LLC Page 12 Data Quality Review Report

Exhibit 9: Income Per Household (FCFA) from Income-Generating Activities

Comparison Treatment Total Comparison Treatment Total

Mean 259,815 315,980 291,670 194,800 313,100 261,930 St. d. 442,056 693,708 598,390 346,822 1,274,796 988,451 N 635 832 1,467 545 715 1,260 Source: NORC report

3.5 Data Issues Encountered

Since 2011, AD10 has implemented the Agricultural Diversification activity of the ADP by delivering a set of training sessions in agriculture and livestock management to over 7,000 individuals. The major issue encountered with the household data is that IMPAQ cannot identify individual program participants (i.e., training beneficiaries) within the household data sample.

The difficulty identifying program participants within the household data has to do with the fact that the data was collected before AD10 training services became available. When services became available, AD10 provided services to individual farmers in the targeted (i.e., treatment) areas that applied for these services. As a result, only a subset of the household survey sample actually received AD10 treatment.

The main reason that IMPAQ is unable to identify the household sample members who received AD10 training services is the lack of common identifiers in the household data and the AD10 data. As a result, it is difficult to identify individuals in the household data who received AD10 training services. This lack of common identifiers prevents a direct merger of household data with the AD10 database. After discussions with AD10, AD10 suggested that it may be feasible to identify individuals in the household data who received training. The following approach was used by AD10 to match individuals in the two datasets:

First, AD10 attempted direct matching by comparing names in the AD10 training lists with the household survey names. To be able to this, AD10 used village names and names of head of households to merge individuals in both files. This approach resulted in a sample of 213 matched households after correcting for names misspellings. According to AD10, this low match rate was unrealistic because they had covered a large fraction of households from the intervention villages.

Next, to improve the matching, AD10 fielded a brief identification survey among the entire original households sample in treated villages during August 11-15, 2013. AD10 selected enumerators from their own trainers who delivered the services so that they can also recognize in person any training participants among surveyed household members. The identification survey collected data on total household members, total members who received AD10 training, the type of training received, the number of the individuals as marked in the AD10 participation card and whether the individual was recognized by the enumerator. Furthermore, enumerators had to verify if households possessed AD10 training participation cards and note their identification numbers in the survey.

IMPAQ International, LLC Page 13 Data Quality Review Report

Based on AD10 documentation,8 AD10 enumerators visited 1,026 out of 1,068 of the households in the intervention area (95.1%).9 Among the surveyed households, 624 have at least one household member who has been trained by AD10. Thus, as a result of several months of effort, AD10 indicated that about 62% of the sample households were matched. This means that approximately three-fifths of households in the household data sample had received AD10 training either in agriculture or in livestock management.

IMPAQ has not yet received the AD10 enumeration survey data and has not tested the mergeability of the identifiers with the original household survey. If there are errors in the matching of household and AD10 data and/or inconsistencies in the unique IDs collected during the identification survey, the identification of households that participated in AD10 training may remain a challenge.

Finally, the original power calculations indicated the need for a sample of 1082 treatment group households and a sample of 1082 comparison group households. At this point we only have a sample of 624 treatment households (or 62%). As a result of this shortfall, the impact analysis will have a lower statistical power than originally projected to detect meaningful program effects. Based on IMPAQ’s recalculation of statistical power, this reduced treatment sample means that we will only be able to detect statistically significant program impacts on income if the program leads to changes greater than 21%.10 We also performed additional power computations using other selected variables, in particular average use of some chemical fertilizers (Urea) and income from livestock products. Results of these additional power computations are presented in Appendix 6. The results of these additional power calculations indicate that, with the available sample, we will be able to detect statistically significant program impacts on chemical fertilizer Urea if it is greater than 34% and on livestock expenditures if it is greater than 52%.

8 Enquête d’identification des ménages communs AD10 et Enquête globale. Synthèse des résultats by MCA-BF.

9 The less than 100% coverage rate is due to households or individuals leaving villages, death of members, or households were not known in the village. In the case of village Nion, it was not covered by the brief identification survey as AD10 didn’t deliver trainings there.

10 See Appendix 6.

IMPAQ International, LLC Page 14 Data Quality Review Report

4. CROP YIELD MEASUREMENT SQUARES SURVEY

In addition to the household survey, NORC and CERFODES also conducted an independent data collection initiative using crop yield measurement squares (carrés de rendement). This methodology measures the exact production area and product weight to accurately determine crop yield. Crop yield typically encompasses two factors: (1) the amount of a particular crop that a piece of land can grow and

(2) the monetary value of the crop. This particular survey was meant to estimate the production of different crops in selected production areas.

This survey covered only treatment villages and was designed to serve the program monitoring objective. Thus, this survey was intended to be representative of the treatment areas but not to be directly linked to the households from the original household sample. Specifically, the objective of this survey was to measure the production of rain fed, market garden, off-season and irrigated crops. During the rainy and dry seasons, crop yields were collected from crops whose production contributed to the revenue of target farmers.

Crop yield measurement squares were used to isolate plots of land and calculate the crop yields within the squares. The following procedures11 were used in this survey:

Early in the season, a square is marked off with sticks or ropes. At the end of the season, the crop within the marked square is harvested separately from the rest of the field and is weighted by the surveyor. Based on the production from the sample area, total production for the field is calculated.

When crops were ripe for harvest, the number of plants remaining in the square was counted.

The output of maize, pod plants, tubers, and root plants were weighed. Certain crops were directly weighed. Cereals were shelled or milled before weighing. Grain crops were dried to a certain weight to ensure reliable measurement. In these cases, agents weighed the harvest, then deducted and dried a sample of the production and weighed it again. The actual production quantity was calculated as if the entire harvest had been dried.

For crops in which harvest is not done all at once (tomatoes, peppers, peas and leaf vegetables), the production was weighed at each harvest until the end of the harvest period. Production was calculated as the sum of these weights.

To maintain consistency over time, these crop yield measurement squares were marked so that enumerators can return to the same plots for subsequent data collection rounds.

11 The methodology to install measurement squares to measure production is based on the one used by Direction Générale de la Promotion de l’Economie Rurale (DGPER), the national office for agricultural statistics in Burkina Faso. DGPER and MCA-BF approved this methodology and provided quality assurance review and report on the collected data.

IMPAQ International, LLC Page 15 Data Quality Review Report

4.1 Sampling Frame

This independent survey was not designed to be used for the impact evaluation but to collect indicators for monitoring purposes. While it was designed to cover the entire intervention zone, it is a separate survey with different sampling units (production sites and farmers as opposed to households) as described below.

During the global household survey enumeration, an inventory of production sites was generated to serve as a sampling frame for the production measurement during both seasons. A “site” is defined as a concentration of farmers, such as a group of farms which surround a dam, riverbank, borehole, or well.

A village can contain one or several of these sites. Sixty-five sites were selected (40 in Sourou and 25 in Comoé).

The enumeration was done in two phases. The first phase of the sampling frame was the development of an exhaustive list of all sites in the intervention zone. The second phase consisted of the development of an exhaustive list of farmers (secondary units) in the sample sites.

4.2 Crops Stratification and Farmer Sampling

The following selected crops were measured:

Dry season: maize, rice, onions, tomatoes, potatoes, zucchini, cabbage, eggplant, peppers, lettuce, green beans, cucumbers

Rainy season: rice and maize12

The sampling was done in two stages. The treatment area was subdivided into groups which correspond to the intervention villages. Random sampling of farming units was done independently and identically in each village.

Selection of primary units: In each village, the primary units correspond to sites selected according to a systematic draw and weighted proportionally to the number of farmers in the site.

Census of farmers in primary units: In each sample site, a census was made of all farmers.

Selection of secondary units: The secondary units were randomly chosen from this census in each primary unit by simple random sampling. For each site, ten farmers were randomly selected to have their production measured.

In total, 3,308 farmers were listed during the dry season and 3,725 during the rainy season. During the dry season, 85 of these farmers were revisited and 167 crop yield measurement squares were marked on the ground. During the rainy season, a sample of 143 farmers was revisited and 159 crop yield measurement squares were marked.

Dry season: In Comoé, eleven sites were listed in eleven villages (See Exhibit 10). According to the initial sampling plan, this implies a sample of 110 farmers. However, only 24 (21.8%) were selected in nine

12 Market gardening activity is limited in the rainy season due to infestations and floods.

IMPAQ International, LLC Page 16 Data Quality Review Report villages (81.8%). At the end of the measurement period, 52 crop yield measurement squares were marked. Twenty-nine of these were marked on maize plots and 23 on rice.

In Sourou, 21 sites were listed in nine villages. According to the sampling plan, this implies a total sample of 210 farmers. However, only 61 were actually selected in five villages. At the end of the measurement period, 115 crop yield measurement squares were marked: 40 on maize plots and 75 on rice.

Exhibit 10: Number of Measurement Squares Expected and Actually Marked in Dry Season

Expected number Actual number COMOE Villages 11 9 Sites 11 8 Farmers 110 24 Squares marked 110 52 SOUROU Villages 9 5 Sites 21 21 Farmers 210 61 Squares marked 112 115 Source: IMPAQ computation and NORC report

Rainy season: In Comoé, 68 farmers in 14 villages and 22 irrigated sites were selected for measurement (See Exhibit 11). At the end of the measurement period, seven villages (50%) covering eight sites (36.4%) were measured, for a total of 49 farmers (72.1% of the original 68), with 49 crop yield measurement squares marked (43.8% of the original 112). These included 13 on maize (26.5%) and 36 on rice (73.5%).

In Sourou, 92 farmers in seven villages and 30 irrigated sites were selected. At the end of the measurement period, five villages (71.4%) covering 21 sites (70%) were measured, with 110 squares marked (150.8% of the original 61). Ninety-four of the expected 92 (102.2%) observations were made, including 78.7% on maize and 21.3% on rice.

Exhibit 11: Number of Measurement Squares Expected and Actually Marked in Rainy Season

Expected number Actual number COMOE Villages 14 7

Sites 22 8 Farmers 68 49

Squares marked 61 49 SOUROU Villages 7 5

Sites 30 21 Farmers 92 94

Squares marked 112 110

IMPAQ International, LLC Page 17 Data Quality Review Report

4.3 Key Variable Distributions

In both provinces, most farmers in the sample are 30 to 40 years old (See Exhibit 12) among the total sample of listed farmers. The next largest categories are farmers aged 40–50 and 20–30. Women represent less than 4% of all farmers.

Exhibit 12: Distribution of Farmers by Age and Gender

Province Age range Gender

Male Female Total N Percentage N Percentage N Percentage

COMOE Younger than 20 24 4% 0 0% 24 4% 20–30 174 29% 20 27% 194 29% 30–40 208 34% 23 31% 231 34% 40–50 114 19% 10 14% 124 18% Older than 50 85 14% 21 28% 106 16% Total 605 100% 74 100% 679 100%

SOUROU Younger than 20 23 1% 0 0% 23 1% 20–30 444 17% 8 16% 452 17% 30–40 927 36% 20 39% 947 36% 40–50 780 30% 15 29% 795 30% Older than 50 399 16% 8 16% 407 16% Total 2,573 100% 51 100% 2,624 100%

Source: IMPAQ computation and NORC report

The mean area of irrigated cereals in Comoé in the dry season was 0.18 hectares (ha) for maize and 0.28 ha for rice (See Exhibit 13). Yields were 1.55 Tons per hectare (T/ha) for maize and 4.29 T/ha for rice.

Note that all irrigated cereal crops are monoculture. In Sourou, the mean area of irrigated cereals was

0.27 ha for maize and 0.48 ha for rice. Yields in Sourou were higher than in Comoé, with average rice yield of 5.65 T/ha.

Exhibit 13: Area and Yield Per Hectare of Irrigated Cereal Crops in Dry Season

Province Crop Area mean (ha) N Yield Mean (T/ha) N COMOE Maize 0.18±0.25 33 1.55±0.96 30

Rice 0.28±0.14 23 4.29±0.93 23 SOUROU Maize 0.27±0.20 43 2.21±1.50 40

Rice 0.48±0.13 75 5.65±2.09 75 Source: IMPAQ computation using PROD_CER_CORRIGE_170912 and NORC report. Rice yields are in terms of paddy.

During the rainy season, in Comoé, the average area of irrigated cereal crops was 0.46 ha for maize and

0.2 ha for rice. Average yields were 1.76 T/ha for maize and 2.64 T/ha for rice (See Exhibit 14). In Sourou, the average area of irrigated cereal crops was 0.54 ha for maize and 0.53 ha for rice. Average yields were 3.07 T/ha for maize and 6.28 T/ha for rice.

IMPAQ International, LLC Page 18 Data Quality Review Report

Exhibit 14: Area and Yield Per Hectare of Irrigated Cereal Crops in Rainy Season

Province Crop Area mean (ha) N Yield Mean (T/ha) N COMOE Maize 0.46±0.32 25 1.76±0.30 25

Rice 0.2± 0.16 36 2.64±1.21 34 SOUROU Maize 0.54± 0.52 89 3.07±1.10 89

Rice 0.53±0.18 23 6.28±0.94 21 Source: IMPAQ computation using PRODUCTION SITE CORRIGE 170912 and NORC report

4.4 Data Issues Encountered

The two major issues with the crop yield data are the unreliable quality of the measures and the inadequacy of the sampling methodology for impact evaluation.

During the data cleaning, some aberrant values were identified. Crop yield weights were collected both before and after crop processing took place, thus resulting in moisture content differentials in the crops.

Ideally, extra moisture in the crop harvest should be removed before reporting maize and rice crop yields.

To correct the distortion in the data, NORC and CERFODES established an upper bound of 9 T/ha for rice yields based on agriculture literature. All values higher than this upper bound were excluded and replaced by the mean of the series. The NORC team compared the observed yields to those obtained by other studies in Burkina Faso. Ouédraogo S. et al., (2003) found a general trend of increased area, yield, and production of rice between 1981 and 2000, primarily on irrigated land. Among more recent studies reporting average yields in the literature, in 2012, farmers registered average rice yields of 7.64 T/ha as part of a Cornell University study on Systems of Rice Intensification based on agricultural trials in Burkina Faso. These yields reflected an increase of 70% compared with traditional rice culture and is one of the highest average yields reported in the literature. Additionally, compared to the lowland production figures provided by the Permanent Agricultural Survey13 over several seasons (2010–2011), the mean yields in the data were higher than those in lowland areas. Allowing for a reasonable margin, this resulted in an upper bound of plausible rice yields of 9 T/ha.

Moreover, maize values recorded before the crop processing were excluded from the dataset to eliminate inconsistent data. Data collectors applied different methods for calculating maize yields, which resulted in inaccuracies due to moisture content differentials affecting the overall maize weight. These values were replaced by the mean of the series.

Other issues included missing values and the fact that a sample of 10 farmers per site was not always achieved due to the late start of the survey. Thus, although the sampling design was supposed to allow for extrapolation on provincial level for monitoring purposes, the coverage of this study and the accuracy of the data are such that they may not provide the basis needed for an accurate assessment of the impact of the project.

13 Government survey overseen by the Ministry of Agriculture in Burkina Faso which centralizes agricultural statistics.

IMPAQ International, LLC Page 19 Data Quality Review Report

Some of these concerns were clearly pointed in the quality assurance report14 by Direction Générale de la Promotion de l’Economie Rurale (DGPER), the national office for agricultural statistics in Burkina Faso that examined the data. DGPER concludes that the rice and maize crop yields data are of inacceptable quality and should not be used.

14 DGPER. Octobre 2012. ANALYSE DE LA QUALITE DES DONNEES DE L’ENQUETE AGRICOLE DU MCA/BF and DGPER. October 2012. Appréciation globale des bases de données MCA.

IMPAQ International, LLC Page 20 Data Quality Review Report

5. FISHING SURVEY

The survey was carried out from October 2011 to February 2012. Information was collected on the characteristics of people engaged in fishing, their production and income, as well as the use of labor.

The fishing survey seeks to quantify the total production of fish in the intervention area. It includes data on income derived from fishing, as well as its contribution to food security in fishing households. The objective of fielding this survey was for monitoring purposes and not impact evaluation. The data was only collected in the treatment areas, for a brief period, and on a new independent sample.15

5.1 Sampling Frame

Before the survey began, an enumeration was carried out in the villages of the treatment area in tandem with the household survey enumeration. This enumeration collected data on the presence of fishing sites in villages, as well as the number of occasional and full-time fishers. The fishing sites were defined as places where fishing is practiced full-time by at least five people.

5.2 Sampling

Data collection was done in villages that practiced fishing in the intervention areas of Comoé and Sourou. A two-level sampling plan was used. The first level was designated as the “site”, an area including all villages in both provinces listed as practicing some form of fishing activity. The first level of the sample is therefore an exhaustive list covering almost the entire fishing sector in the target areas.

On the second level, five fishermen at each site were selected from…