Attachment 9_AID-696-A-16-00008 Isuku Iwacu MTE Final Evaluation Report.pdf

EVALUATION REPORT

Isuku Iwacu Mid-term Evaluation Final Report

Limited Internal Distribution

September 18, 2019

This publication was produced for review by the United States Agency for International Development (USAID). It was prepared by WASH for life.

TABLE OF CONTENTS

Acronyms………………………………………………………………………… ii Executive Summary…………………………………………………………….. iv Glossary…………………………………………………………………………… ix Introduction……………………………………………………………………… 1 The Intervention………………………………………………………………… 1 Purpose and Scope of the Evaluation………………………………………… 2 Methodology……………………………………………………………………… 2 Findings and Conclusions………………………………………………………. 4

1. Appropriateness of the Activity design………………………………………… 4

2. Effectiveness of the Activity……………………………………………………. 16

3. Performance, Impact and Sustainability Prospects…………………………….. 22

4. Building Local Capacity…………………………………………………………. 25

5. Gender and Equity……………………………………………………………… 29 Learning and Capitalization…………………………………………………… 32 Recommendations………………………………………………………………. 34 Utilization Plan…………………………………………………………………… 37 List of Annexes…………………………………………………………………… 37 ii

Acronyms

ACB Amasezerano Community Bank AMIR Association of Microfinance Institutions in Rwanda AOR Agreement Officer’s Representative BCC Behavior Change Communication CBEHPP Community-based Environmental Health Promotion Programme CHAIN Community Health and Improved Nutrition CHC Community Health Club CSO Civil Society Organization DBC Designing for Behavior Change (Framework) DSC District Sanitation Center EQ Evaluation Question FGD Focus Group Discussion FSM Fecal Sludge Management FY Fiscal Year (October 1 – September 31) GOR Government of Rwanda HQ Headquarters IR Intermediate Result JMP Joint Monitoring Program KII Key Informant Interview LCSO Local Civil Society Organization M&E Monitoring and Evaluation MEL Monitoring, Evaluation and Learning MFI Microfinance Institution MHM Menstrual Hygiene Management MINALOC Ministry of Local Government MININFRA Ministry of Infrastructure MOH Ministry of Health MOU Memorandum of Understanding NGO Non‐governmental Organization NISR National Institute of Statistics Rwanda NSP National Sanitation Policy ODF Open Defecation Free PD Program Description PPP Public-Private Partnership PWD Person / People with Disabilities Q Quarter (i.e. 3 months of Fiscal Year) RWF Rwandan Francs SACCO Savings and Credit Cooperative SDGs Sustainable Development Goals SGF Sanitation Guarantee Fund iii

SILC Saving and Internal Lending Community SO Strategic Objective U Ubudehe (a quintile of social class) USAID United States Agency for International Development VSLAs Village Savings and Loan Associations WASAC Water and Sanitation Corporation WASH Water, Sanitation and Hygiene ZOD Zero Open Defecation iv

Executive Summary Background Isuku Iwacu is a four‐year rural sanitation Activity in Rwanda, funded by the United States Agency for International Development (USAID) and executed by a consortium of international non‐ governmental organizations (NGOs), headed by SNV USA and with World Vision International and Water for People. The Activity’s overall goal is to ‘increase local ownership and capacity to deliver sustainable, high quality sanitation and hygiene services in order to decrease childhood stunting.’

The Activity is being implemented in eight districts in Rwanda, where it will help an estimated 500,000 people gain access to improved household sanitation, as defined by the Joint Monitoring Program (JMP), while ensuring that Rwandans living in 137 targeted villages across the eight districts are using, at a minimum, JMP‐defined improved sanitation and living in open defecation free (ODF) environments (USAID/Rwanda refers to this ODF approach as Zero Open Defecation–ZOD).

Evaluation purpose and scope The four-year Activity started in September 2016 and has passed the mid-term. WASH for life has consequently been contracted to conduct a Mid-term Evaluation of the Activity. This evaluation assessed the appropriateness of the Activity’s design and basic assumptions and its progress and performance to date. WASH for life also identified lessons learned and successes to inform the remainder of this Activity.

The scope of evaluation encompasses the implementation of the Activity from its start date, September 2, 2016, through Fiscal Year 2019, Quarter 1 (FY19Q1), December 31, 2018. The Mid-term Evaluation addresses the following broad evaluation questions (EQs):

1. How appropriate was Isuku Iwacu’s initial activity design of building capacity and applying market‐based approaches to increase demand for sanitation products and access to sanitation services in the targeted districts and villages?

2. How effective are Isuku Iwacu’s current implementation strategies to successfully expand access to sanitation services? What changes could be made to increase effectiveness in the remaining two years?

3. To what extent have the different implementation approaches been successful? Which of them appear to be sustainable and should be prioritized going forward?

4. To what extent has Isuku Iwacu been successful in building capacity to improve sustainable access to sanitation?

5. How has Isuku Iwacu’s approach taken into consideration issues of gender and equity?

Methodology WASH for life undertook this evaluation between April and August 2019, applying a tailored qualitative approach. Data collection was conducted through: comprehensive document review of internal reports (for information and quantitative data) and relevant external literature; 50 key informant interviews and where necessary, group informant interviews; 6 community focus group discussions (FGDs); and 13 site visits for direct observations.

v

Key findings As the first project of its kind in Rwanda, Isuku Iwacu was designed to place SNV (and the consortium members) as facilitators for promoting market-based sanitation – rather than exclusively as implementers. As such, there is a need for learning throughout implementation, and furthermore, a need to document capitalization for internal processing and external sharing.

1. Appropriateness of the Activity design Conceptually, Isuku Iwacu had a good Activity design. It was meant to be complementary to other USAID WASH, Health and Nutrition Activities, although there is also an associated confusion that might cause overlapping efforts by USAID Activities (Isuku Iwacu and Gikuriro), or even worse, gaps in coverage.

In terms of appropriateness to national goals, Isuku Iwacu shares goals with the Government of Rwanda’s 2016 National Sanitation Policy (NSP). Thus, Isuku Iwacu gained local buy-in and ownership rather quickly (despite few challenges), and demonstrated a good working relationship. The Activity’s work to translate and help disseminate the NSP was very appreciated by local stakeholders, although deeper sanitation concepts (i.e. goals beyond building latrines) did not trickle down from the national level. This may impact local stakeholders’ abilities to take over and scale up sanitation outreach and financial approaches.

The Activity’s first intermediate result aims to increase demand for sanitation products and services. Although, given that the vast majority of households (around 96%) had latrines (improved and unimproved), there was very little need to generate demand for basic access to sanitation. Hence, Isuku Iwacu shaped its efforts to its BCC strategy, appropriately developed using the Designing for Behavior Change (DBC) Framework (though the timing was delayed and strategic development less effective than it could have been).

The Activity’s second intermediate result aims to improve the supply and availability of sanitation products and services in the private sector. Through a series of local studies and international workshops, and trial-and-error with the private sector, Isuku Iwacu realized its original design was not relevant to the context. Isuku Iwacu needed to study more and readjust its strategy.

The New Private Sector Implementation Strategy was developed in March 2018 (though well into the implementation phase), which readjusted the Activity’s direction as an ‘anticipated sanitation market facilitator.’ Though even with the expected engagement of local Civil Society Organizations (LCSOs) in FY20 and the pending functionality of all District Sanitation Centers, this evaluation finds that there isn’t enough time left in the Activity to effectively test and sustainably carry out the Strategy’s remaining phases (sanitation loan testing and scale up).

2. Effectiveness of the Activity This ‘learning by doing and adapting’ learning curve explains many of the delays in working towards IR1 and IR2, and the Mid-Term Evaluation recognizes valuable efforts by Isuku Iwacu to improve sanitation by engaging the various actors of the private sector. The evaluation also offers several recommendations for strengthening such engagement. Though generally, the Activity would have been more effective if there had been early installation of the District Sanitation Centers (DSCs), links with LCSOs, and engagement with Microfinance Institutions (MFIs) and local savings groups to promote sanitation loans from the onset of the Activity. Capacity issues of LCSOs in regard to the private sector led to delays; LCSOs’ capacity building measures have been put in place to mitigate shortcomings.

vi

Building on the findings from EQ1, strengthening collaboration between Gikuriro and Isuku Iwacu (especially for BCC) is an identified key room for improvement in order to support the Activity’s overall effectiveness and measurable impact. Implementation delays and inconsistent sequencing of the DBC Framework might deteriorate behavior change effects of the Activity, leading to lower outcomes.

Finally, the evaluation uncovered some inconsistences in the ZOD definition and resulting technical definitions. This is discussed further under EQ2. Inconsistencies identified by this evaluation in the ZOD definition, technical criteria, and monitoring process must be addressed.

Furthermore, as LCSOs will work with and eventually replace SNV in this Activity to continue sanitation collaborations with the districts and communities, there is the need for strengthened monitoring tools and processes to share and hand over.

3. Performance, Impact and Sustainability Prospects Data indicate an estimated overall progress of 20-25% of the outcome indicators by FY19Q2, i.e. after 31 months of project corresponding to 60% of completed project timeline. Considering the delayed implementation, Isuku Iwacu may consider the opportunity of an extension to have more time to work towards achieving its targets with high quality. With an extension, the Activity can also meet the need for further learning throughout implementation, readjustment of financial approaches as needed, and document capitalization.

The two financial approaches, vouchers and loans, are diverging, taking a different form in each district, even in each cell and village, due to factors linked to socio-economic class, distance from the DSC, amount of BCC received, and mobilization generated. Considering that the loan approach encourages community members to self-finance their own improved latrines, without Isuku Iwacu subsidies, the preceding voucher approach risked creating an aura of assistance.

With two distinct financial approaches, the Activity cannot expect the results of these diverging approaches to be uniform nationwide. The Activity did not pause long enough to learn from the voucher pilot phase and readapt the approach, and it could have benefited from doing so. The challenges and successes vary incredibly by district – even by cell and village. Considering this, the Activity may prefer to adjust slightly the sanitation loan approach to the specific differences and needs of the sector and cell levels – as opposed to carrying out a nationwide approach.

4. Building Local Capacity As part of IR3, Isuku Iwacu provided hands on support to the government. The Activity steered away from traditional capacity building and focused on facilitating sustained access to sanitation products and services by partnering with and strengthening local stakeholders. These stakeholders include local governments, LCSOs, construction companies, and financial institutions.

The Mid-term Evaluation finds that deeper sanitation concepts like adjusting/scaling up financing approaches, ensuring coverage (vs. access), inclusion (EQ5), and verifiable monitoring haven’t transferred down from the national level. These concepts are currently lost at the sector, cell and village levels. There, just the goal of building latrines remains – which is partly effective – however, building latrines and counting numbers doesn’t take into consideration how to reach the most the vulnerable, monitor accurately, and promote coverage. Therefore, as Isuku Iwacu adapts and tests new approaches – such as the forthcoming loan approach – it must continue to engage with national and district-level officials to build their capacity in durable approaches. This is in fact incorporated in the Activity’s FY19 workplan and should be added to the FY20 workplan.

vii

Furthermore, with CHCs being used as primary focal points for community health, this evaluation highlights the importance of strengthening CHC capacity. On the same note, Isuku Iwacu should strengthen the capacity of local leaders, savings groups (i.e. VSLAs) and Amasibo to encourage sanitation coverage.

5. Gender and Equity The Activity conformed to USAID’s Policy on Gender Equality and Women’s Empowerment and was in line with the Government of Rwanda’s National Gender Policy. Women and girls have been targeted because the Activity targeted all households in the selected villages (thus, females were not excluded) – though Isuku Iwacu did not take additional steps to design programming around the specific needs of women and girls.

In terms of equity, the Activity offers no distinct definition; still, it took steps to address the specific sanitation needs of people with disabilities, the elderly and children under five. While relevant and inclusive, these efforts have only recently been launched. Other equity efforts targeted the poorest households (those from Ubudehe 1 category, considered the most vulnerable). However, concerns about the beneficiary selection process and social class categorization highlight a risk for Isuku Iwacu in achieving sanitation access for the most vulnerable. In all, Isuku Iwacu needs to modify or expand its approach to be more inclusive and attentive to the specific needs of all population groups.

Learning and Capitalization Isuku Iwacu encountered many challenges and delays, but it also took informed steps to mitigate them, learn from them and adapt. This evaluation was able to identify several key areas from which the Activity can further learn:

Lessons learned to date

1. It is relevant and critical to take time to pilot the various financial approaches on a smaller scale prior to scaling up nationwide.

2. Due to consequential delays largely from undertaking an innovative sanitation approach in Rwanda, the Activity is not on track to meet its ambitious targets, which could have been reduced. Simply evaluating Isuku Iwacu by its quantitative indicators does not reflect its learning curve.

3. This evaluation finds various evidences that highlight the low level of consistency within the WASH sector in Rwanda, suggesting the need for Isuku Iwacu to take part in learning within the WASH sector for improved program delivery nationwide.

Topics to assess for further learning and advocacy

4. Isuku Iwacu may consider conducting an evidence-based survey during the last months of implementation to collect primary data on health outcomes to show the relationship between the Activity and its nutritional Strategic Objective.

5. USAID/Rwanda can explore the possibility of an Ex-Post Impact Evaluation 1-3 years after implementation ends in order to strengthen learning for future programming and advocacy.

viii

Recommendations Based on the key findings and upon further discussion with SNV and USAID/Rwanda, this Mid-term Evaluation offers several recommendations to help improve the final year and a half of the Isuku Iwacu Activity as well as few recommendations for future WASH programming:

Quality completion of the Isuku Iwacu Activity

1. Continue BCC/outreach activities, including further training and encouraging CHCs, local leaders and Amasibo to mobilize their communities and follow up.

2. Explore the relevance of other business model options available in the country (such as cell-level traders) to expand local access to sanitation products and services at sector/cell/village levels via deeper supply chain mapping.

3. Reaffirm and strengthen the process to reach and pre-declare ZOD status in the targeted villages, and strengthen local/district monitoring and verification capacity.

4. Consider strengthening monitoring and conducting more monitoring activities, given the few but serious discrepancies and issues reported.

5. Act quickly to pilot the sanitation loan approach via AMIR and SACCOs.

6. Carry out more capacity building efforts with LCSOs, MFIs, local leaders and community groups, service providers, and local authorities in order to leave a solid framework in place after the Activity ends.

7. Encourage districts to fully operationalize District Sanitation Centers as soon as possible to increase access to sanitation products and overall Activity effectiveness.

8. Further review the effectiveness and impact of the Isuku Iwacu innovative approaches and develop best practices.

9. Consider requesting a project extension from USAID.

Future WASH programming

10. Address water supply if not secured to gain health and nutrition-related outcomes.

11. As a nexus of SNV programming, promote safe practices for emptying latrine pits in villages and explore the potential for integrated Sanitation and Livelihoods programs.

ix

Glossary The following definitions have been provided by WASH for life for the sake of clarity in this Mid- Term Evaluation. They are either based on globally recognized definitions that have been adapted for this evaluation, or are quoted/adapted from relevant Activity documents. Sources are specified.

Access to sanitation: Ability to acquire affordable services and improved (private) sanitation facilities, as well as safe on- or off-site treatment and disposal of wastewater and sludge. [NSP, 2016]

Coverage: The use of improved (private) sanitation facilities [NSP, 2016] – Including operation and maintenance services, shifting the focus from ‘having latrine facilities’ to ‘safely and properly using one in a way that promotes good health.’

Equity: Consideration of and actions to address different needs of different population groups, including vulnerable people, like the elderly, poorest, and people with disabilities who have special needs – rather than applying a blanket approach to assistance and development programming that doesn’t consider vulnerabilities and specific needs. [Adapted from UNICEF]

Gender: Consideration of and actions to address contextual differences in gender roles, responsibilities, associated barriers, and specific needs – not just of women and adolescent girls, but also men, boys and young girls (also known as gender equity). [European Institute for Gender Equality]

Sanitation marketing: Sharing information with current and potential consumers about the availability of sanitation products and services, including shop locations, costs and varieties available as well as options for sanitation financing. [As understood by the Isuku Iwacu Activity implementation and goals]

Sanitation products: Products and materials needed in sanitation, such as construction materials for improved latrines, cleaning products, sanitary pads (for women), etc. [As understood by the Isuku Iwacu Activity implementation and goals]

Sanitation services: Services related to sanitation, such as the construction, service for emptying pits, cleaning, availability of information at sanitation businesses, etc. [As understood by the Isuku Iwacu Activity implementation and goals]

Zero Open Defecation Pre-Declaration*: Completed by Isuku Iwacu. Term used when all houses in a village have at least an on-site toilet which separates human excreta from human contact, no matter the standard/ quality.

Zero Open Defecation Declaration*: Completed by and at the discretion of each district and all the local authorities, including MININFRA, MINALOC, and MOH. Although not yet known, likely to follow the SDG definition, focusing on getting all households to “basic sanitation services.”

*This evaluation found that a common definition in Rwanda is disputed; thus, an analysis of ZOD is presented in the report body.

1. Introduction

Isuku Iwacu is a four‐year rural sanitation Activity in Rwanda with a budget of nearly USD$10.7 million dollars, funded by the United States Agency for International Development (USAID) and executed by a consortium of international non‐governmental organizations (NGOs). The Activity’s overall goal is to ‘increase local ownership and capacity to deliver sustainable, high quality sanitation and hygiene services in order to decrease childhood stunting.’ The Activity is being implemented in eight districts in Rwanda, where it will help an estimated 500,000 people gain access to improved household sanitation.

The four-year Activity started in September 2016 and has passed the mid-term. WASH for life has consequently been contracted to conduct a mid-term evaluation of the Activity. This evaluation assessed the appropriateness of the Activity’s design and basic assumptions and its progress and performance to date. It also identified lessons learned and successes to inform the remainder of this Activity. The Mission will use this evaluation to improve quality implementation of the Activity and inform future Water, Sanitation and Hygiene (WASH) programming in Rwanda.

2. The Intervention

Isuku Iwacu is a four‐year rural sanitation Activity, funded by USAID, awarded on September 2, 2016, and estimated to be completed by November 9, 2020. The Activity is being executed by a consortium of international NGOs headed by SNV USA with World Vision International and Water for People. Through Isuku Iwacu, USAID/Rwanda will contribute to improvements in sustainable access to WASH services, with an emphasis on the lowest poverty quintile. Isuku Iwacu is being implemented in eight districts: Kayonza, Kicukiro, Ngoma, Nyabihu, Nyanza, Nyarugenge, Ruhango, and Rwamagana. The Activity will help an estimated 500,000 people gain access to improved household sanitation, as defined by the Joint Monitoring Program (JMP) for the Millennium Development Goals, while ensuring that Rwandans living in 137 targeted villages across the eight districts are using, at a minimum, JMP‐defined improved sanitation and living in open defecation free (ODF) environments (USAID/Rwanda refers to this ODF approach as Zero Open Defecation).

The overall goal of Isuku Iwacu is to increase local ownership and capacity to deliver sustainable, high quality sanitation and hygiene services in order to decrease childhood stunting. Over the life of the Activity, to foster local ownership and strengthen capacity building, the Isuku Iwacu Activity has defined the following Strategic Objective (SO):

• Improve access to and encourage correct, consistent use of household sanitation and hygiene facilities in order to decrease childhood stunting.

The Activity will achieve this SO by:

• Directly supporting and contributing to the Government of Rwanda (GOR)-led efforts to improve access to sanitation in Rwanda; and

• Promoting districtwide, private sector‐driven household sanitation and hygiene interventions and district‐ and national‐level capacity development.

Three intermediate results (IRs) will contribute directly to achieving this SO:

• IR1: Increased demand for sanitation products and services;

• IR2: Improved supply and availability of sanitation products and services in the private sector;

• IR3: Improved governance for sustained access to sanitation and hygiene products and services.

3. Purpose and Scope of the Evaluation

The purpose of this mid‐term evaluation is to:

1. Assess the appropriateness of the Activity’s design and basic assumptions;

2. Assess the progress and performance to date; and

3. Inform the remainder of this Activity and future Mission WASH programming.

The scope of evaluation encompasses the implementation of the Activity from its start date, September 2, 2016, through Fiscal Year 2019, Quarter 1 (FY19Q1), December 31, 2018, thus slightly over mid-term. Due to delays in the field visit for data collection, this evaluation also considered new activities, villages and approaches from FY19Q2 for a more robust analysis.

Evaluation Questions The evaluation addresses the following broad evaluation questions (EQs) for the aforementioned time period:

1. How appropriate was Isuku Iwacu’s initial activity design of building capacity and applying market‐based approaches to increase demand for sanitation products and access to sanitation services in the targeted districts and villages?

2. How effective are Isuku Iwacu’s current implementation strategies to successfully expand access to sanitation services? What changes could be made to increase effectiveness in the remaining two years?1

3. To what extent have the different implementation approaches been successful? Which of them appear to be sustainable and should be prioritized going forward?

4. To what extent has Isuku Iwacu been successful in building capacity to improve sustainable access to sanitation?

5. How has Isuku Iwacu’s approach taken into consideration issues of gender and equity?

The detailed sub-questions and applied approach to answering each EQ, including data sources and analysis plan, are elaborated in the Evaluation Design Matrix (Annex A).

4. Methodology

To answer each of the evaluation questions described above, WASH for life applied a tailored qualitative approach. Data collection was conducted through the following methods:

• Comprehensive document review of internal reports (for information and quantitative data) and relevant external literature

• 50 key informant interviews (KIIs) and where necessary, group informant interviews

• 6 community focus group discussions (FGDs)

• 13 site visits for direct observations

1 Considering the delay in implementing the Mid-Term Evaluation, the remaining period is accordingly reduced to 15 months.

On an ongoing basis, primary and secondary data were entered into the evaluation team’s data collection tool (Excel matrix), coded and analyzed, with preliminary findings drawn out early on as well as key queries highlighted for further investigation. The Evaluation Design Matrix (Annex A) expands on the EQs and informs the data collection tools, sources and analysis process by illustrating in detail how the different data collection methods will feed into the analysis plan to answer each evaluation question.

The full methodology detailing the above data collection methods, evaluation approach, site selection, inception and analysis phases, and preliminary findings workshop is presented in Annex B:

MTE Methodology.

Limitations and Assumptions The evaluation team noted that there are few differences in the Activity’s outputs in Kigali area and in the districts within Eastern and Southern provinces. Based on Activity reports, available data and inputs from key SNV staff to highlight differences, site selection was made to be representative for qualitative analysis. All quantitative data come from Activity planning and reporting documents.

A key part of this evaluation (EQ5) investigates aspects related to equity, which is understood as the inclusion of vulnerable groups in the Activity, including people with disabilities (PWD) and the elderly (refer to Glossary for definition). While the evaluation team managed to speak with key stakeholders on this topic, they were unable to identify PWD in the villages to visit, and further, no FGD contained PWD, though efforts were made to invite them. The evaluation team did manage to have formal and informal talks with elderly people in the villages, who are also considered vulnerable when it comes to constructing and accessing household latrines. Through these efforts, the evaluation team has worked to circumnavigate this limitation.

All key stakeholders have been interviewed, except the SNV WASH Sector Lead for Rwanda. The position is crucial for providing with adequate technical support to Isuku Iwacu partners. The person was out of country during the Mid-Term Evaluation (in-country investigations were postponed due to delayed Government approval). This is important for the External Evaluation to indicate the limitations in cross-checking data collection. As a matter of fact, direct evidence of proper in-house technical support could not be verified as expected (no WASH Sector Lead Skype Interview, no access to all technical support notes/reports), but from confirmation of key informants interviews only. As a mitigation measure, it was suggested to proceed to a Skype interview, but it was not feasible for some reasons. Further, two field visit reports from SNV USA/HQ were also collected.

As a consequence, the Mid-Term Evaluation could not include a detailed review of in-house technical support and associated support. Nonetheless, the evaluation received confirmation from key informants that in-house technical support had been provided via meetings with the SNV WASH Sector Lead for Rwanda and other stakeholders, and analysis is based on this information.

5. Findings and Conclusions

The main goal of the four-year Isuku Iwacu rural sanitation Activity transcends sanitation as a standalone topic, stating an objective “to improve access to and encourage correct, consistent use of household sanitation and hygiene facilities in order to decrease childhood stunting and support the Government of Rwanda’s (GOR) work in sanitation.”2

As the first project of its kind in Rwanda, Isuku Iwacu was designed to place SNV (and the consortium members) as facilitators for promoting market-based sanitation – rather than exclusively as implementers. Hence, this Activity was designed, implemented and adapted using innovative approaches. Considering this, some may view Isuku Iwacu as an experimental Activity – a program not intended to meet the quantitative targets outlined, but rather a program that is testing market-based theories in Rwanda, trying to find the most appropriate, effective, efficient, and inclusive approach to ensuring Sanitation for All.3

Altogether, there is a need for learning throughout implementation, and furthermore, a need to document capitalization for internal processing and external sharing. Key informants highlighted that, internally, SNV has taken many steps towards learning, such as by commissioning several consultants to lead research in order to adapt Activity implementation. There are also plans to document capitalization in 2020 during the Activity’s last year of implementation – a beneficial step.

Also supporting learning and quality programmatic adjustments is this Mid-Term Evaluation, which has interrogated the Activity’s three intermediate results through the five evaluation questions.

Significant findings and conclusions of this analysis are presented in five related sub-chapters that refer to the Evaluation Questions.

1. Appropriateness of the Activity design

Relation to other USAID Activities The Isuku Iwacu Activity was designed to be complementary to other USAID WASH, Health and Nutrition Activities (ongoing and planned at the time). While conceptually, this complementary function seemed relevant and with a good design, its implementation is less clear. The Mid-Term Evaluation finds an associated confusion that might cause overlapping efforts by USAID Activities (namely Isuku Iwacu and Gikuriro), or even worse, gaps in coverage on the ground.

CHAIN and Gikuriro Given that this Activity ultimately has a health and nutrition-related objective, it falls under the overall umbrella of USAID/Rwanda’s Community Health and Improved Nutrition (CHAIN) Project Activities. One of those – Gikuriro, the Integrated Nutrition and Water, Sanitation, and Hygiene Activity – is “designed to operationalize USAID/Rwanda’s strategic goal of ending childhood stunting.”4

2 Cooperative Agreement: AID-696-A-16-00008/SNV, Isuku Iwacu Activity, Attachment B – Program Description, p.25 of 95 3 Rwanda’s Economic Development and Poverty Reduction Strategy II (EDPRS II, 2013-2018) includes the goal to ensure universal access to water and sanitation by 2018 (http://sanitationandwaterforall.org/) 4 Isuku Iwacu Program Description, p.30 of 95

EQ1: How appropriate was Isuku Iwacu’s initial activity design of building capacity and applying market‐based approaches to increase demand for sanitation products and access to sanitation services in the targeted districts and villages?

http://sanitationandwaterforall.org/)

Therefore, “Gikuriro serves as the coordinator of all USAID nutrition and WASH investments authorized under the CHAIN project in its eight focus districts; as a result, the implementer of the Isuku Iwacu [SNV and the NGO consortium] should expect to work in close collaboration with Gikuriro to ensure that WASH results are achieved in the target districts via complementary and closely-coordinated interventions.”5

The Isuku Iwacu Activity determined its eight districts for intervention by picking from areas where Gikuriro was also working, allowing Isuku Iwacu – on paper – to build on Gikuriro’s work and objectives, essentially to promote an integrated, cost-effective, and more impactful rural sanitation Activity. Isuku Iwacu’s interactions with Gikuriro during the design phase were more plentiful;

however, on the ground, it is unclear how well this collaboration worked over time (refer to EQ2).

Three key components coming out of this integration are Gender, Behavior Change Communication (BCC) and Grants/Group savings synergy.

1. Gender

Practically, this means that part of the initial design of the Isuku Iwacu Activity built on existing CHAIN and Gikuriro actions, such as the CHAIN Gender Analysis (2014) that provided the first basis of gender attention in the Activity. The Gender Analysis informed some of the gender and equity-related assumptions feeding into the Activity (see further EQ5).

2. Behavior Change Communication (BCC)

It is clear that Isuku Iwacu, like Gikuriro, is following the GOR’s National Social Behavior Change Communication Strategy that focuses on a Community-based Environmental Health Promotion Programme (CBEHPP) approach. Gikuriro promotes CBEHPP through BCC outreach to raise awareness in the communities including through mass media campaigns, such as radio and theater – and Isuku Iwacu builds its activities on that existing momentum. Isuku Iwacu also works with Gikuriro-supported Community Health Clubs (CHCs) to channel messages of sanitation and hygiene into the communities and motivate members to improve their sanitation facilities. This collaboration demonstrates a good design approach to integrate Isuku Iwacu’s objectives with Gikuriro’s objectives, yet community members, village leaders and local authorities are unclear about who is doing BCC outreach (see further EQ2).

3. Grants/Group savings synergy

The two USAID Activities are meant to work together on improving the sanitation supply chain, specifically through the “set-up of supply chain distribution systems (graduated [village savings and loan associations (VSLAs) / saving and internal lending communities (SILCs)] to acquire sanitation products from rural service centers for distribution in villages).”6

According to the Isuku Iwacu Program Description (PD), Isuku Iwacu meant to partner with Gikuriro “to strengthen financial links between graduated CHCs and SILCs/VSLAs to increase community-based lending for local service providers.”7 They have not been targeted as intended so far (see further

EQ2).

Relevance to national policies and goals Isuku Iwacu is an innovative sanitation Activity that includes new market-based approaches and scaling up perspectives to reach total sanitation in rural areas of Rwanda. It started around the time that the Government of Rwanda released its National Sanitation Policy (NSP) in 2016, and due to shared goals, Isuku Iwacu gained local buy-in and ownership rather quickly (despite few challenges).

5 Ibid, p.31 of 95 6 Isuku Iwacu Activity, Collaboration Plan with Gikuriro Program, p.4 7 Isuku Iwacu Program Description, p.43 of 95

8 Reported in the FY17 Annual Report 9 According to the NISR Census 2012 and the 5th Rwanda Demographic and Health survey (DHS–2014/15), as cited in the National Sanitation Policy, p.9-10

The Activity’s work to translate and help disseminate the NSP was a good output, very much appreciated by local stakeholders. Nonetheless, the Mid-Term Evaluation finds that key sanitation concepts did not trickle down from the national level. The revised Ministry of Infrastructure’s (MININFRA’s) 2016 NSP reflects the nuances associated with measuring and achieving Sanitation for All. It clearly explains the differences between sanitation ‘access’ and ‘coverage’ (refer to Glossary) and their link to affordability, market services, knowledge, and behavior change.

However, while the NSP clearly expresses a solid understanding of sanitation access and coverage, in general, those concepts are lost further down at the sector, cell and village levels. There, just the goal of building latrines remains, although the NSP articulates the difference between building and having a latrine facility, and safely and properly using one in a way that promotes good health (i.e.

coverage). In any case, the Isuku Iwacu Activity design and purpose relies heavily on the ownership and engagement of local stakeholders, including national and district governments and CHCs. The implications of their interest, awareness and capacity on sustainability are further discussed in EQ4.

The Isuku Iwacu Activity defines itself, in part, as helping to push the NSP into action. As the largest sanitation program in the country, with ambitious objectives, Isuku Iwacu is therefore expected to bring a significant contribution to the GOR-led efforts to improve access to sanitation in rural areas of Rwanda. The Activity’s goals are in line with the NSP, and therefore, there was quick buy-in from the GOR. Despite few bumps with MININFRA in the planning process,8 national and district level authorities were on board from the beginning. In addition, performance contracts between the national, district, sector, and cell levels promote achievement of these national goals. All levels of the concerned local authorities/bodies interviewed perceive the Activity as a great supportive initiative to achieve sanitation progress, i.e. closing the sanitation gap.

In all, this leads to a good working relationship with the national government, ministries and district officials, as they too have a vested interest in the Activity. Their buy-in (and the push from the GOR performance contracts) means they will most likely continue working to close the sanitation gap even after Isuku Iwacu ends their intervention.

Finally, while the Isuku Iwacu Activity follows the revised JMP Sanitation Ladder (see Annex I), there are consequential discrepancies when comparing JMP to the NSP’s definitions of ‘improved’ and ‘unimproved’ latrines. These discrepancies might be due to the shift in the sanitation ladder that occurred when the MDGs transitioned and updated to the SDGs. The JMP definition follows the MDGs, whereas the NSP’s definition follows the UN Agenda 2030 SDGs. The implications of this on village level mapping and ZOD declarations are presented in EQ2.

Activity design and assumptions This sub-section refers to analysis of the IR1 and IR2 Activity design components, while IR3 (Governance) is further reviewed in its own section, ‘Building Local Capacity.’

Relevance in increasing demand for sanitation products and services The Activity’s first intermediate result aims to increase demand for sanitation products and services.

However, given that the vast majority of households (around 96%9) had latrines (improved and unimproved) prior to the Isuku Iwacu Activity – and the Rwandan government and other actors are actively working towards closing the sanitation gap (latrine access and coverage) – there was very little need to generate demand of basic access to sanitation. As a result, the Activity developed its action by generating increased willingness and ability to access to improved sanitation and to maintain satisfactory standards (proper quality of construction, usage and excreta management). In that sense, the selection of the Designing for Behavior Change (DBC) Framework tool – that is, identifying the priority and effective determinants for behavior change – is very appropriate for the Activity BCC strategy development and implementation.

Improved sanitation coverage as reported by the JMP was at about 75% in 2015 (rural: 71%, urban:

83%), but including shared toilets.10 Per the NSP:

“Open defecation has practically been eradicated and most of Rwandan households have already financed and built their on-site private sanitation premises, albeit only about two thirds comply with the international standard definitions of an improved sanitation facility.

Very few Rwandan households have installed flush toilettes. The prevailing practice remains that water is used for cooking and washing (grey water, discharged mostly on surface) while excreta are disposed with waterless latrines, which is a rational solution considering the scarcity of the average water supply and financial constraints.”11

Rather, based on the context – and as either uncovered or verified by this evaluation – there is need for Isuku Iwacu to promote (1) latrine cleaning with very little water, (2) improved hygiene habits, namely handwashing with soap, and (3) overall safe latrine usage (leading to proper excreta management). These topics fall under BCC and directly work towards Isuku Iwacu’s Strategic Objective of supporting “correct, consistent use of household sanitation and hygiene facilities” – that is, the safe, equitable and proper use of latrines for improved health and nutrition outcomes.

Below, each of the three aforementioned BCC needs and focus areas are analyzed:

1. Latrine cleaning

Many FGD and KII respondents raised concerns about the best methods to clean their improved latrines with very little or even no water (i.e. using tree leaves or branches). There were several reasons for raising such concerns: (a) little or no water supply in the community; (b) fear of filling the pits too fast, given the cost of building a pit latrine and in some areas, the difficulty in digging;

and (c) little or no access to cleaning products in the village.

While the Activity tried to engage communities to have clean latrines, the fact that respondents requested additional support from Isuku Iwacu may suggest that the Activity has not successfully supported the communities to solve such challenges, especially regarding nearby access to cleaning products and desludging services. While Isuku Iwacu-supported District Sanitation Center (DSC) have not helped much so far to create access to cleaning and other sanitation products for households, this might change after DSCs become fully operational (extent of change to be measured by the end of Activity accordingly).

2. Improved hygiene habits Handwashing with soap at the key handwashing times is a major indicator and contributing factor when it comes to measuring sanitation and hygiene and their associated health and nutrition outcomes. The Activity’s BCC strategy focused on handwashing, even promoting the installation and use of Tippy-taps, which was successful to a limited extent. During site visits, the evaluation noted few handwashing stations (with or without soap). Field visit pictures of hygiene promotion materials and obvious outcomes are provided in Annex J: Supporting photographs from site visits, under the header ‘Tippy-taps.’

10 As cited in the National Sanitation Policy, p.9 11 Ibid, p.9

Being that handwashing with soap is as important as having access to an improved, clean latrine, any issues with water supply would therefore jeopardize sanitation efforts. Two villages visited during this evaluation had no nearby water source (Rugarama in Nyanza and Batikoti in Nyabihu). Where there is no water for handwashing and cleaning the latrines, the sanitation approach cannot be considered complete and little health impact should be expected. It is worthwhile to remember the well-informed and acknowledged links between Water, Sanitation and Hygiene (all together) and nutritional status, especially that the Activity’s Strategic Objective aims at decreasing childhood stunting.

While the scope of USAID funding for this Activity did not include water supply, the absence of permanent access to water in some sites can be considered as a critical barrier for improved sanitation, since handwashing needs a stable water supply. In those villages, the relevance of Activity is reduced, as it should have taken water access into consideration or intervened elsewhere for an increased effect. USAID should consider this for future programming and program design.

3. Safe latrine usage

Overall, this means that households in villages where Isuku Iwacu works should understand how to secure and ensure the proper use of improved latrines for all household members. Properly using an improved latrine means keeping it clean and handwashing with soap afterwards. These steps contribute to improved health, as overall, the sanitation facility and the environment remains free from contaminants (i.e. safe).

All in all, during site visits, this evaluation noted many unimproved latrines and, as mentioned, few handwashing stations (with or without soap) near the vicinity of these (both improved and unimproved) latrines. USAID’s Multi-Sectoral Nutrition Strategy (2014-2025), as cited in the Isuku Iwacu PD, notes that “WASH addresses both the direct and underlying causes of stunting” and that “essential WASH interventions include handwashing with soap, treatment and safe storage of drinking water, and sanitary disposal of human feces.”12 Hence, gaps in hygiene behaviors and overall safe latrine usage will hinder the Activity’s outcomes.

In addition to promoting safe latrine usage, Isuku Iwacu’s BCC messaging should continue to encourage the population to install or improve (as relevant) sanitation facilities. This messaging is in line with the Activity’s goals, and it means promoting different methods for sanitation financing, considering that the 2018 Formative Research found that lack of financial capacity was the biggest barrier to having an improved household latrine.

Relevance in creating access to sanitation products and services The Activity’s second intermediate result aims to improve the supply and availability of sanitation products and services (refer to Glossary) in the private sector. It stresses on a critical shift from a traditional, heavily subsidized latrine construction approach to the enhanced ability of low-income families to build and maintain proper sanitation standards. This IR relevantly considers the business side (private sector knowledge of sanitation, existing supply, geographic location of shops) and consumer side (accessibility to the sites, affordability, interest in products and services). To do so, “Isuku Iwacu’s activities will include commercial approaches to understanding local demand throughout the value chain, financial flows in the sector, new product development, and supply chain analysis. Isuku Iwacu

12 Isuku Iwacu Program Description, p.30 of 95 will identify and bring in relevant examples from the region focusing on aspects that can be adapted for Rwanda’s primarily rural, low-income population.”13

Through a series of local studies, research and workshops, and trial-and-error with the private sector (both new and existing sanitation service providers as well as financial institutions), the Activity realized the original PD was not relevant to the context. Isuku Iwacu needed to study more, learn and adjust its strategy and approach. The following pages break down the different components of this approach, their purpose, progress to date, and how and why they have changed over time.

This learning curve explains many of the delays in working towards IR1 and IR2, and the Mid-Term Evaluation recognizes valuable efforts by Isuku Iwacu to improve sanitation by engaging the private sector. The Activity furthermore coordinated many national and regional learning workshops to discuss ideas and issues and set workplans, which helped in getting stakeholders interested and promoted knowledge sharing.

1. Voucher approach

A primary component of Isuku Iwacu’s intervention was the development and application of a latrine voucher, given to households selected by local authorities, and ‘redeemed’ for the construction of an improved household latrine. Voucher beneficiaries needed to provide certain in-kind contributions, both helping to keep the cost of the vouchers down and promoting ownership of the works. In most cases, in-kind contributions included making mud bricks, collecting water for construction, and some other labor such as digging pits.

This approach works towards sanitation access – that is, the availability of a private latrine at the household level, which is a primary goal of the NSP. Thus, the voucher approach supported national goals and district-level performance indicators, so district-level buy-in was smooth.