1 SAEP Programmatic Evaluation.pdf

SOUTHERN AFRICA ENERGY PROGRAM

EVALUATION REPORT

October 19, 2022

DISCLAIMER: This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Social Impact, Inc.. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

i | USAID SAEP EVALUATION REPORT USAID.GOV

CONTENTS

Acronyms ....................................................................................................................................................... iii

Executive Summary ...................................................................................................................................... iv

Introduction

Design and Methodology

Evaluation Questions

Methodology

Data Analysis

Ethical Considerations

Gender and Social Inclusion Plan

Limitations and Biases

Findings and Conclusions

EQ1: To what extent was SAEP able to meet its purpose and related outputs and outcomes (MWs, Connections, Enabling Environment)?

EQ2: What enabled and/or hindered achievements, and how were any hindrances addressed?

EQ3: How did the program design, target setting, and operational decision-making shape and contribute to the achievements of the Program?

EQ4: What were the benefits and drawbacks of Power Africa setting the connections and MW targets for the program?

EQ5: In which countries did SAEP have the most success versus the least success, and why?

EQ6: How is the SAEP perceived and valued by the stakeholders of the Program?

Recommendations

Annex 1: Statement of Work

Annex 2: Bibliography

Annex 3: Informed Consent Statements

Annex 4: Data Collection Instruments

Annex 5: Conflict of Interest Disclosures

Annex 6: Evaluation Design Report

USAID.GOV USAID SAEP EVALUATION DESIGN REPORT | ii

TABLES

Table 1: Evaluation Design Matrix

Table 2: Survey Sample Size by Respondent Type

Table 3: Relevant EQs for Each Respondent Type

Table 4: SAEP Contributions to the Sector

FIGURES

Figure 1: SAEP Actuals vs. Targets for Key Indicators ............................................................................................................................ v

Figure 2: Countries Where SAEP Found the Most and Least Success ...................................................................................... vii

Figure 3: Survey Respondent Perceptions of SAEP’s Impact ........................................................................................................... vii

Figure 4: Original SAEP Design Summary

Figure 5: Total Respondents by Respondent Type

Figure 6: Total Respondents by Gender

Figure 7: Total Respondents by Country

Figure 8: Potential Partner Respondents and Organizations by Country

Figure 9: Actuals vs. Targets for Key SAEP Indicators

Figure 10: Seamstress using SAEP-supported off-grid solar lights

Figure 11: Wind turbines from an SAEP partner

Figure 12: SAEP-supported Solar Home System Kick-Starter Program in Malawi

Figure 13: Satisfaction with SAEP by Respondent Type https://socialimpact.sharepoint.com/sites/equiproj382/Shared%20Documents/05.%20Evaluation%20Report/SAEP%20Final%20Evaluation%20Report_2022.10.07_CLEAN.docx#_Toc116060928 iii | USAID SAEP EVALUATION REPORT USAID.GOV

ACRONYMS

COVID-19 Novel Coronavirus of 2019

ET Evaluation Team

EQ Evaluation Questions

GESI Gender Equality and Social Inclusion

IP Implementing Partner

KII Key Informant Interview

MEL Monitoring, Evaluation, and Learning

MW Megawatt

PAIS Power Africa Information System

PATT Power Africa Tracing Tool

PMEP Performance Management & Evaluation Plan

POC Point of Contact

RFP Request for Proposals

SADC Southern Africa Development Community

SAEP Southern Africa Energy Program

SAPP Southern Africa Power Pool

SOO Statement of Objectives

SI Social Impact

USAID United States Agency for International Development

USG United States Government

USAID.GOV SAEP EVALUATION REPORT | iv

EXECUTIVE SUMMARY

CONTEXT

Power Africa is a United States government (USG)-led partnership that launched in 2013. By 2030, Power Africa aims to achieve 30,000 megawatts (MW) of new generation capacity, 60 million new power connections for homes and businesses, and the unlocking of energy sector potential by improving the enabling environment.1 The Southern African Energy Program (SAEP), a flagship Power Africa initiative funded by the United States Agency for International Development (USAID), aims to increase investment in electricity supply and access in Southern Africa by strengthening regulation; improving planning, procurement, and management of electricity trade; demonstrating and scaling energy efficiency technologies and practices;

and providing capacity building to institutions and human resources.

PURPOSE

This evaluation provides USAID/Southern Africa and the USAID Power Africa team with feedback regarding SAEP successes, challenges, and areas for improvement, which could inform future programming in the energy sector and support programmatic adjustments. Other potential evaluation users include implementing partners (IPs), regional organizations, national ministries, private sector partners, and regulators, who can learn from SAEP’s work.

EVALUATION METHODOLOGY

The SAEP evaluation addressed six core evaluation questions (EQs):

1. To what extent was SAEP able to meet its purpose, related outputs, and outcomes?

2. What enabled and/or hindered achievements, and how were any hindrances addressed (including but not limited to COVID-19 [novel coronavirus of 2019])?

3. How did the program design, target setting, and operational decision-making shape and contribute to the achievements of the program?

4. What were the benefits and drawbacks of Power Africa setting the connections and MW targets for the program?

5. In which countries did SAEP have the most success versus the least success, and why?

6. How is the SAEP perceived and valued by the stakeholders of the program?

To answer these EQs, the evaluation team (ET) employed a mixed-method design across SAEP’s ten countries. On the qualitative side, the ET conducted a desk review of relevant documents and key informant interviews (KIIs). On the quantitative side, the ET administered a short, online survey and reviewed the Power Africa Information System (PAIS), Power Africa Tracking Tool (PATT), and program monitoring data.

The Social Impact (SI) team conducted remote data collection to reduce potential risks related to COVID-19 and mitigate logistical constraints. As a result of small sample sizes, the ET reports findings for all SAEP partner respondents together rather than disaggregating the findings by type of partner (government, utility, etc.). Across the EQs, all respondent groups tended to provide similar feedback. Thus, unless otherwise noted, each finding is consistent across respondent groups (IPs, USAID, and partner organizations).

1 USAID. 2022. Power Africa Website. https://www.usaid.gov/powerafrica https://www.usaid.gov/powerafrica v | USAID SAEP EVALUATION REPORT USAID.GOV

FINDINGS AND CONCLUSIONS

EQ1: TO WHAT EXTENT WAS SAEP ABLE TO MEET ITS PURPOSE, RELATED OUTPUTS, AND OUTCOMES?

SAEP met most of its performance targets, including for new generation and transmission. However, SAEP has not yet met its target of 3 million new electricity connections. Given the extension of the period of performance to June 2023, achieving this target remains possible. In interviews, respondents shared largely positive feedback regarding the results, feedback that aligned with the quantitative results tracked in the performance indicators. Despite SAEP’s focus on new generation, transmission, and connections, however, more respondents noted SAEP’s success in building capacity (mentioned in 27 interviews) than they did the generation, transmission, or connection outcomes (17, 11, and 13 interviews, respectively), suggesting stakeholders viewed SAEP’s capacity building work as particularly beneficial.

Figure 1: SAEP Actuals vs. Targets for Key Indicators (as of June 2023)

EQ2: WHAT ENABLED AND/OR HINDERED ACHIEVEMENTS, AND HOW WERE ANY HINDRANCES

ADDRESSED?

Respondents shared numerous factors enabling and/or hindering SAEP achievements. In addition to the overarching impact of COVID-19, these factors fell into three main categories: SAEP-level factors, partner-level factors, and context-level factors (in decreasing level of importance).

COVID-19 posed a challenge to SAEP implementation through delays and slower operations (21 interviews) and negative effects from travel restrictions (19 interviews). SAEP adapted using virtual approaches (19 interviews), though many stakeholders (22 interviews) felt that in-person activities were more effective.

SAEP-level factors centered on SAEP’s technical capabilities across outcome areas (25 interviews) and the strength of their relationships with partner organizations (23 interviews), both of which most respondents viewed as SAEP strengths that enabled program achievements.

Partner-level factors, primarily noted by IP and USAID staff, centered on several interrelated factors: partner organizations’ willingness to engage with SAEP (19 interviews), their buy-in to the program’s activities (21 interviews), and the extent to which SAEP’s priorities aligned with those of its partners (14 interviews).

Where SAEP and stakeholder priorities align, the ET found partners more likely to engage and more likely to have strong buy-in, all of which support strong success. Insufficient alignment and buy-in is a barrier to success, demonstrating the importance of meeting local needs to improve success. Given the large number of

USAID.GOV SAEP EVALUATION REPORT | vi

SAEP countries and partners, these partner-level factors acted equally as often as drivers of success as hindrances, based on respondent feedback.

Context-level factors centered on political influences (12 interviews) and cultural and linguistic differences (ten and nine interviews, respectively). The former included large political events such as elections, political unrest, and other political changes, which fell outside of SAEP’s control. Respondents noted cases where SAEP’s response to cultural and linguistic differences bolstered success and where it remained a barrier, suggesting room for improvement.

EQ3: HOW DID THE PROGRAM DESIGN, TARGET SETTING, AND OPERATIONAL DECISION-MAKING SHAPE

AND CONTRIBUTE TO THE ACHIEVEMENTS OF THE PROGRAM?

The ET examined several components of program design and implementation to answer this EQ, including SAEP’s strengths and weaknesses, the effectiveness of decision making, SAEP’s approach to gender equality and social inclusion (GESI), and respondents’ recommendations for future programs. Overall, the SAEP design and implementation approach contributed positively to program achievements. SAEP strengths included strong staff (24 interviews), effective communication (23 interviews), and a flexible and adaptable approach to implementation (11 interviews). Weaknesses included staffing concerns (14 interviews), such as issues with specific staff, staff turnover, and staff continuity with partners, and insufficient localization and local input into design (12 interviews).

Overall, respondents applauded the SAEP team for its effective communication and decision-making processes (13 interviews). On the other hand, respondents noted that USAID’s decision-making processes often acted as a hindrance to effective implementation (11 interviews); in particular, respondents highlighted negative impacts from frequent requests from USAID Offices and Missions to add activities or partners into SAEP’s scope, which pulled SAEP staff in multiple, sometimes competing, directions; in some cases, these requests did not align well with SAEP’s core objectives.

SAEP’s GESI approach focused primarily on gender and not on other social groups, according to respondents (20 interviews). IP respondents noted GESI-related benefits related to the involvement of women in the sector (eight interviews) and stakeholder engagement (six interviews). Respondents felt that the effects were largely superficial (seven interviews), in part due to pervasive challenges within the male-dominated sector (12 interviews).

When asked for future programming recommendations, respondents shared a variety of technical and design suggestions. On the technical side, respondents often suggested that future programs continue to focus on the same core areas as SAEP such as new connections (29 interviews), capacity building (21 interviews), generation (19 interviews), transmission (16 interviews), and institutional strengthening (11 interviews). In terms of design, respondents recommended that future programming include more, early input on program design and planned activities from local stakeholders (24 interviews), increase the utilization of local experts (12 interviews), and address the funding/financing gap in the power sector (19 interviews).

EQ4: WHAT WERE THE BENEFITS AND DRAWBACKS OF POWER AFRICA SETTING THE CONNECTIONS

AND MW TARGETS FOR THE PROGRAM?

Only four IP or USAID staff recalled how SAEP’s original performance indicators, in particular the fee indicators, were selected or how the targets were set. Thus, the ET relied heavily on document review to answer this question, including the original SAEP request for proposals (RFP) and technical portions of the SAEP contract. The RFP shows that USAID set minimum requirements for bidders of 3 million new connections and 1,000MW of new, renewable energy generation capacity. In its proposal, Deloitte both

3.0 3.0

The Power Sector Generation Capacity Transmission Capacity Power Access USAID/USG (N=6) IP (N=13) Partners (N=23) Total (N=42)

3.0 2.52.6

1.9

2.3 2.5

1.9

1.4 1.3

2.32.3

1.7 1.8

2.4

1.0

2.0

3.0 vii | USAID SAEP EVALUATION REPORT increased and modified the latter to reflect 4,000MW of generation and transmission capacity. Per the RFP requirements, Deloitte proposed the list of fee indicators. The fee indicators ultimately drove SAEP’s focus given the financial incentive to reach the targets.

Overall, both USAID and IP respondents regarded USAID’s use of fee indicators positively. The fee indicators helped the SAEP team remain focused (five interviews) and pushed them toward ambitious goals (five interviews). On the downside, however, they voiced several concerns with the indicators and targets, including the targets’ unrealistic nature (particularly the new connections target) (11 interviews). Concerns included how attributable the MWs and new connections were to the SAEP-provided technical assistance (seven interviews), the timelines required to move from the direct benefits of SAEP’s technical assistance to longer-term outcomes like generation, transmission, and connections (five interviews), insufficient local input and/or calibration of indicator selection and target setting to local needs (four interviews), and how the regional targets led SAEP to focus on the largest projects (and therefore the largest markets) even when smaller, beneficial projects existed (four interviews).

EQ5: IN WHICH COUNTRIES DID SAEP HAVE THE MOST SUCCESS VERSUS THE LEAST SUCCESS, AND WHY?

According to respondents, SAEP achieved the most success, as defined by the respondents, in Malawi and Mozambique (15 and 12 interviews, respectively) and the least success in Madagascar (six interviews), Lesotho (four interviews), Namibia, Botswana, and Eswatini (three interviews each). In addition, Zambia posed an interesting case as respondents included it on both the most (nine interviews) and least (three interviews) successful lists, due to SAEP’s early, successful work in the country followed by significant political changes that hindered SAEP’s later work in Figure 2: Countries Where SAEP Found the Most and Least Success the country. The factors driving success mirrored those outlined in EQ2 such as alignment of priorities between SAEP and its partners, stakeholder buy-in, and the strength of relationships. For lack of successes in Lesotho, Botswana, Eswatini, and Namibia, respondents noted that the small size of the market in these countries acted as a barrier, in line with the EQ4 finding that larger projects received more attention.

EQ6: HOW IS SAEP PERCEIVED AND VALUED BY THE STAKEHOLDERS OF THE PROGRAM?

Overall, stakeholders provided positive feedback regarding SAEP. They were satisfied with SAEP’s services and believed that SAEP positively affected the power sector. Across stakeholder groups, USAID respondents provided the most positive feedback whereas SAEP partners provided the least positive feedback.

Figure 3: Survey Respondent Perceptions of SAEP’s Impact

USAID.GOV

https://USAID.GOV

USAID.GOV SAEP EVALUATION REPORT | viii

Interestingly, even though SAEP has yet to achieve its performance targets for new connections, stakeholders valued SAEP’s impact on power access more highly than any other area. Meanwhile, feedback regarding SAEP’s impact on generation and transmission received the lowest scores, even though SAEP met those targets. This dichotomy likely relates to stakeholder feedback under EQ4 that the new connection target was unrealistic and challenges linking SAEP’s technical assistance with generation and transmission targets.

RECOMMENDATIONS

Based on the findings and conclusions of this evaluation, the ET recommends the following for USAID as it proceeds with designing the SAEP follow-on program:

1. USAID should ensure the localization of future programs. Localization involves active engagement with local stakeholders throughout the project life cycle so that the activities meet the needs of those it is intended to help. Though both USAID and Deloitte played parts in how localization occurred under SAEP, for future programming, USAID should begin the local engagement process early to ensure that local priorities and needs are taken into account even at the proposal stage. If engagement could not begin before the solicitation phase, then USAID should maintain sufficient flexibility to allow for anticipated partners to influence the design and implementation approach after award. Additionally, USAID should ensure that future programs are staffed in a way that addresses the different cultural and linguistic needs across the region, such as improved engagement of local staff and experts. (The ET developed this recommendation based on localization findings from EQ2, EQ3, and EQ4.)

2. USAID should ensure that future indicator selection and target setting are based on clear criteria. Given that the fee indicators drive program focus, USAID should ensure that the indicators represent the intended focus of future programs and that they link clearly with anticipated interventions. Additionally, the targets should have clear justifications based on realistic criteria that incorporate local input, needs, and priorities. This would ensure ambitious, yet realistic, targets given regional realities. USAID should consider ways to balance regional flexibility with ensuring greater distribution of benefits across countries, regardless of the size of the power market.

(The ET developed this recommendation based on indicator and target-related findings from EQ4.)

3. USAID should ensure that future programming improve on the GESI approach. To ensure a cohesive GESI approach, USAID should first define its GESI goals for a power activity such as SAEP. They can then hold IPs accountable to that vision. (The ET developed this recommendation based on GESI findings from EQ3.)

4. USAID should ensure that future programming addresses the financing gap in the power sector. Though USAID could consider directly funding power investments, future programs could also address the financing gap via other means. Such programming should be based on a thorough study of the primary constraints of power sector financing in Southern Africa that identifies specialized programming requirements, such as credit guarantees or other lending de-risking mechanisms, to meet the needs of both credit seekers and financiers. (The ET developed this recommendation based on design findings from EQ3.)

1 | SAEP EVALUATION REPORT USAID.GOV

INTRODUCTION

CONTEXT

Power Africa is a United States government (USG)-led partnership that launched in 2013. By 2030, Power Africa aims to add 30,000 megawatts (MW) of new generation capacity, 60 million new power connections for homes and businesses, and the unlocking of energy sector potential by improving the enabling environment.2

A lack of adequate and reliable power in the Southern Africa region limits competitiveness and affects the livelihoods and health of the population, particularly those living outside urban areas. The five-year Southern African Energy Program (SAEP), a flagship program funded through a partnership with Power Africa and USAID/Southern Africa, aims to increase investment in electricity supply and access in Southern Africa by strengthening the regional enabling environment and facilitating power transactions. SAEP, implemented by Deloitte, launched in 2017 with activities in ten countries: Angola, Botswana, Eswatini, Lesotho, Madagascar, Malawi, Mozambique, Namibia, South Africa, and Zambia.3 Working across all of these countries enabled SAEP to take a regional approach to addressing sector needs. In 2022, SAEP received a one-year extension, making it a six-year program expected to end in June 2023.

Figure 4: Original SAEP Design Summary (SAEP Fact Sheet)

Though SAEP aims to increase generation and access to electricity, SAEP does not directly fund major infrastructure investments. Rather, SAEP works to incentivize, facilitate, and enable investments through technical assistance, policy reform, capacity development, and transaction facilitation. Across the ten partner countries, SAEP’s work ebbed and flowed, with some countries receiving more support in some years than in others.

To identify interventions during the annual work planning process, the SAEP team meets with key stakeholders in each partner country to discuss potential interventions and ways in which SAEP could

2 USAID. 2022. Power Africa Website. https://www.usaid.gov/powerafrica.

3 SAEP originally targeted Zimbabwe as well, but ultimately did not implement activities in the country.

https://www.usaid.gov/powerafrica

USAID.GOV SAEP EVALUATION REPORT | 2

support their electrification, generation, and other sectoral improvement goals. SAEP’s flexible and adaptable approach to implementation involves iterative changes and adaptations to the work plan.

SAEP’s theory of change states that by strengthening regulation; improving planning, procurement, and management of electricity trade; demonstrating and scaling energy efficiency technologies and practices; and providing capacity building to institutions and human resources for energy sector management, SAEP aims to increase investment in electricity supply and access in Southern Africa. SAEP’s support works to address reliance on electricity/energy imports. Long-term program targets include 3,000MW of new power generation, 1,000MW of new transmission capacity, and 3 million new connections (on- and off-grid).

EVALUATION PURPOSE AND AUDIENCE

The purpose of this evaluation is to provide USAID/Southern Africa and the USAID Power Africa team with feedback and recommendations for improving the SAEP activity and informing future programming in the energy sector. This evaluation equips USAID teams with evidence to support programmatic adjustments, inform future design efforts and strategies, and make informed decisions. Other potential users of the evaluation include implementers, regional organizations, national ministries, private sector partners, and regulators, who could adapt lessons learned from SAEP.

Commissioned by Power Africa and contracted via USAID/Southern Africa, Social Impact (SI) was selected to conduct a utilization-focused, mixed methods evaluation with an emphasis on broad stakeholder engagement to promote its use and evidence-based decision making. SI’s technical design kept quality, utilization, and feasibility firmly in focus while ensuring broad coverage of all ten SAEP countries.

DESIGN AND METHODOLOGY

EVALUATION QUESTIONS

The Evaluation Team (ET) worked closely with USAID to refine and finalize the Evaluation Questions (EQs) in order to ensure mutual understanding and maximize utility and learning:

1. To what extent was SAEP able to meet its purpose, related outputs, and outcomes (MWs, Connections, Enabling Environment)?

2. What enabled and/or hindered achievements, and how were any hindrances addressed (including but not limited to COVID-19)? Provide specific examples.

3. How did the program design, target setting, and operational decision-making shape and contribute to the achievements of the program?

4. What were the benefits and drawbacks of Power Africa setting the connections and MW targets for the program?

5. In which countries did SAEP have the most success versus the least success, and why? (focus on off-grid interventions; unlocking financing for private sector companies; improving the enabling environment)

6. How is the SAEP perceived and valued by the stakeholders of the program?

METHODOLOGY

The SAEP evaluation relied on multiple methods to obtain data across the ten active SAEP countries. On the qualitative side, the ET conducted both a desk review of relevant documents and literature and key informant interviews (KIIs). On the quantitative side, the ET carried out a systematic review of the Power

3 | SAEP EVALUATION REPORT USAID.GOV

Africa Information System (PAIS) and Power Africa Tracking Tool (PATT) as well as program monitoring data and administered a short, online survey that allowed key stakeholders across the region to have their perspectives heard. To reduce potential risks related to COVID-19, the ET only conducted remote data collection. The following sections describe SI’s mixed methods approach to the evaluation, as outlined in Table 1. Annex 6: Evaluation Design Report provides the ET’s full design report and methodology.

Table 1: Evaluation Design Matrix

EVALUATION QUESTIONS

DOCUMENT/

DESK REVIEW

SECONDARY

DATA

QUANTITATIVE

SURVEY

KIIS

Question 1: To what extent was SAEP able to meet its purpose and related outputs and outcomes (MWs, Connections, Enabling Environment)?

X X X X

Question 2: What enabled and/or hindered achievements, and how were any hindrances addressed?

X X X

Question 3: How did the program design, target setting, and operational decision-making shape and contribute to the achievements of the Program?

X X X X

Question 4: What were the benefits and drawbacks of Power Africa setting the connections and MW targets for the program?

X X X

Question 5: In which countries did SAEP have the most success versus the least success, and why?

X X X X

Question 6: How is the SAEP perceived and valued by the stakeholders of the Program?

X X

Document/Desk Review. The ET conducted a systematic document/desk review to understand the availability and quality of secondary data in each of the ten SAEP countries and the relationship of that data to the evaluation questions, as well as to identify key issues and constraints for further examination in primary data collection. The desk review included SAEP background documents, including quarterly and annual reports, programmatic documents, and relevant stakeholder documentation (see Annex 2: Bibliography for a full list of reviewed documents).

Secondary Data. The ET reviewed PAIS, PATT, and SAEP monitoring data. The ET used data from the PAIS to understand SAEP partners, commitments (and progress toward them), private sector engagement, and a broad range of other work completed through Power Africa. Data from the PATT provided insights on power transactions, the energy mix, and generation capacity. As in the document/desk review, the ET examined data for trends over time and to better understand both progress and challenges. The ET found these databases particularly useful in answering EQ1 on SAEP’s ability to meet its purpose and objectives.

USAID.GOV SAEP EVALUATION REPORT | 4

Quantitative Survey. The SI team implemented a short, online quantitative survey to assess achievements, benefits, successes, and challenges during implementation. This survey enabled rapid collection of key perspectives from a broad set of stakeholders (across the same stakeholder groups noted below for the KIIs) in the ten active SAEP countries that may not have been otherwise reached through interviews. The survey tool contained questions to better understand perceptions of SAEP implementation, achievements, and challenges, with a particular focus on EQs 1, 2, 5, and 6. The data complemented the secondary data and document/desk review, and the ET triangulated it with the more in-depth interview data. Table 2 outlines the final sample size.

Table 2: Survey Sample Size by Respondent Type

RESPONDENT TYPE NUMBER OF RESPONDENTS

USAID/United States Government Staff 6

Implementing Partner Staff 13

Utility Representatives 4

Partner Government Representatives 3

Regional Entities 1

Private Sector Actors 8

Other Donors 7

Total 42

Key Informant Interviews. The SI team held KIIs with a sample of stakeholders to provide context and document stakeholder experiences related to SAEP’s interventions. Employing a semi-structured interview approach, KIIs included questions regarding stakeholder experiences with program implementation and perceived program effects, including respective contributions of different activities to USAID’s results. The ET tailored interview protocols to the different stakeholder types to fit their roles. KIIs lasted approximately 60 minutes for all partner organizations and up to 120 minutes with USAID and IP respondents. Targeted stakeholder groups for KIIs included USAID and embassy staff, IP staff, utilities, regulators, government representatives, representatives of other donors, and private sector actors. The ET selected respondents purposively across countries and stakeholder types to include as broad a spectrum of experiences as possible.

Table 3 summarizes how the ET used feedback from each type of respondent to answer each of the EQs, based on each groups’ relevant knowledge and familiarity with SAEP programming:

Table 3: Relevant EQs for Each Respondent Type

RESPONDENT TYPE EQ1 EQ2 EQ3 EQ4 EQ5 EQ6

USAID/USG Staff X X X X X X

IP Staff X X X X X X

Utility Representatives X X X

Regulator Representatives X X X

Partner Government Representatives X X X

Regional Entities X X X X

Private Sector Actors X X X

Other Donors X X X

Regional 24

Malawi 17

Zambia 9

South Africa 7

Madagascar 6

Eswatini 5

Angola 5

Namibia 4

Mozambique 2

Lesotho 0

Botswana 0

The figures below outline the final respondent sample sizes, distributed across the ten active SAEP countries and stakeholder types (please refer to the Limitations and Biases section for additional information regarding sampling challenges).

Figure 5: Total Respondents by Respondent Type (n=79) Figure 7: Total Respondents by Country (n=79)

IP Staff Private Sector Actors

Government… Utilities Providers

USAID and Mission… Regulators

Other Donors Regional Entities

Figure 6: Total Respondents by Gender (n=79)

Female, 23, 29%

Male, 56, 71%

DATA ANALYSIS

The ET analyzed data through an iterative process during and after the data collection. Triangulation of the desk review documents, quantitative survey, and the KII data allowed the ET to assess trends, consistency, affinity, and contradiction between quantitative and qualitative data, in order to better answer the EQs, generate more reliable findings, and reach well-founded conclusions. The ET applied a gender equality and social inclusion (GESI) lens to understand how findings differed by gender.

Quantitative Data Analysis. The ET performed rigorous data quality checks and reviewed the data using SI’s high frequency check protocols. These checks allowed the ET to identify survey logic, data range, missing values, implausible values, digit preference (for numeric responses), and outlier errors in the data.

After the survey closed, the ET executed a final cleaning process before analysis. The ET utilized Excel software to explore trends and disaggregate by stakeholder type, country, and gender.

Qualitative Data Analysis. During data collection, the ET took detailed notes and recorded (with consent) all discussions. After data collection, the ET utilized a structured coding approach to conduct content and thematic analysis using Dedoose software for identification of emergent themes and contextual factors upon which to draw findings and conclusions and to develop evidence-based recommendations. The ET utilized a Findings, Conclusions, and Recommendations Matrix to triangulate data across data sources and to ensure an evidence-based, triangulated, and systematic analysis process. As a result of small sample sizes, the ET reports findings for all SAEP partner respondents together, rather than disaggregating the findings by type of partner (government, utility, etc.). Across the EQs, all respondent groups tended to provide similar feedback. Thus, unless otherwise noted, each finding is consistent across respondent groups (IPs, USAID, and partner organizations).

5 | SAEP EVALUATION REPORT USAID.GOV

https://USAID.GOV

USAID.GOV SAEP EVALUATION REPORT | 6

ETHICAL CONSIDERATIONS

SI’s in-house institutional review board reviewed the evaluation tools, data collection instruments, and protocols prior to the start of data collection. All members of the ET and management team received ethical research training to ensure honesty and integrity, protection of respondents, and data security. All respondents provided verbal or written informed consent prior to their participation in the evaluation. The evaluation did not attribute any information collected through KIIs to individual participants.

GENDER AND SOCIAL INCLUSION PLAN

As part of the evaluation, the ET assessed the extent to which SAEP implemented and/or adopted planned activities, such as gender-specific training, guidelines, indicators, policies, and strategies. With a GESI-sensitive approach, the ET reviewed secondary data and aimed to maintain gender balance in the KIIs to the extent feasible. As is consistent with more males working across the power sector, the ET interviewed more male than female respondents. The ET used a GESI scorecard to maintain USAID’s Automated Directives System 205 standards. This scorecard allowed the ET to assess gender sensitivity throughout the evaluation process, including within the submitted data collection instruments. The ET disaggregated its analysis in the following sections by gender and stakeholder type as necessary and possible to discern differences and similarities in experiences, perceptions, needs, barriers, and other findings.

LIMITATIONS AND BIASES

The potential limitations as well as mitigation measures for the SAEP evaluation include:

Effects of Short Evaluation Timelines. To meet the timeline of the evaluation USAID established, the ET combined methods and used a remote approach to mitigate this challenge and quickly collect data from a broad range of stakeholders.

Effects of Large Geographic Dispersion. SAEP’s regional focus introduced a unique challenge for ensuring that the ET included the views and perspectives of all stakeholder groups in the evaluation. The dispersion created both logistical challenges for reaching key stakeholders via traditional, in-person data collection in each country and challenges for aggregating findings across widely varying contexts. The evaluation design and remote approach helped mitigate the logistical challenges while ensuring efficient data collection. The SAEP team assisted the ET in encouraging stakeholder participation in the survey and interviews. Regarding aggregation of findings, the ET spoke to a variety of stakeholders and examined data for any potential inter-country differences. To further enable participation, the ET conducted both the survey and interviews in English, French, or Portuguese (based on respondent preference).

Response Bias. The ET recognized that SAEP stakeholders may carry unobserved biases specific to their role, cultural considerations, and/or other factors. This bias could influence their responses, such as being overly positive about achievements or saying what they think the ET or the Power Africa Program Office wants to hear. The SI team designed the instruments carefully and triangulated evidence across methods to minimize potential bias and ensure validity and reliability.

Selection Bias. Neither the interviews nor survey captured the views of all potential respondents, as some potential respondents did not reply to ET outreach and others chose not to participate in the evaluation.

Thus, differences between who participated and who did not could introduce bias into the evaluation. To limit potential selection bias, the ET invited a wide range of stakeholders across all ten countries to participate and sought the support of SAEP staff to encourage participation. The ET invited all stakeholders with email addresses to participate in the survey and contacted more than 200 potential respondents for an interview

7 | SAEP EVALUATION REPORT USAID.GOV

(first via email, with follow-ups by phone if the ET did not receive an email response or if email addresses were unavailable). Despite the ET’s efforts, however, final sample sizes fell short of expectations for several reasons:

● Fewer Potential Partner Respondents Than Expected: Overall, the ET received names and contact information (phone and/or email) for 352 stakeholders involved to some extent with the program (including IP and USAID staff). Among SAEP partners, this included 296 individuals across 140 organizations. However, the number of potential partner respondents and organizations varied widely across countries, as seen in Figure 8.

Figure 8: Potential Partner Respondents and Organizations by Country

Zambia

Malawi

Mozambique

Namibia

South Africa

Eswatini

Angola

Madagascar

Botswana

Regional

Lesotho Respondents Organizations

While the ET anticipated that levels of SAEP activity differed across the ten countries, it had not anticipated the extent to which this was the case. Since the number of potential respondents reflected the extent to which SAEP was active in each country, the varying distribution, itself, was not a problem for the validity of the evaluation’s findings. However, it did limit the ET’s ability to achieve the intended sample size because the original sampling plan assumed that the distribution between countries was more even than it was in reality. For example, the ET had anticipated interviewing respondents from more than five organizations in every country. However, in four countries, SAEP did not even work with that many organizations; thus, the originally planned sample (which the ET designed before receiving detailed contact information) was not possible. Though the ET reached out to more potential respondents in countries with greater SAEP activity, the small potential pool in many countries limited the ET’s opportunities to reach the anticipated sample without overly biasing the sample toward perspectives from a few select countries.

● High Turnover: Many of the potential partner respondents the IP shared either had incorrect contact information or were no longer with their organizations, resulting in numerous undeliverable emails, a challenge that was consistent across different types of partners (e.g., government staff, utilities, etc.). For the survey, of the 333 emails the ET sent (not all potential respondents on the ET’s list included email addresses), 73 (22 percent) bounced back as undeliverable. Similar issues occurred with KII invitations. Often, the ET reached out to multiple people at the same organization. In many

USAID.GOV SAEP EVALUATION REPORT | 8

cases, the ET heard from other respondents at the organization that our intended recipients no longer held the same positions or worked for the same organization as when they engaged with SAEP (see EQ2 for related findings on partner staff turnover). This suggests that, after a particular activity or intervention ended, SAEP may not have maintained contact with these organizations.

● Low Levels of SAEP Awareness: Even among potential respondents that the ET contacted successfully, a surprising number expressed that they lacked sufficient familiarity with SAEP’s work to respond to the ET’s questions, which was particularly common among other donor and government stakeholder respondents.

● Fewer Group Interviews: When reaching out to partner organizations, the ET anticipated conducting one KII with two to three individuals within each organization. However, in the end, partners often preferred an individual KII, identifying a single individual to speak with the ET rather than a small group. This was not an inherent issue for the evaluation, as the ET was still able to achieve its goal of understanding perspectives from that organization. However, the ET’s sampling plan assumed there would be more small group interviews than occurred in practice, resulting in a smaller total sample size. Of the 56 interviews conducted, only 17 included more than one person.

Thus, though the ET collected views from that organization as intended, the total number of individual respondents was lower than expected.

Due to the above issues, the ET received feedback from fewer respondents than anticipated. On the survey, the ET received 42 responses, a response rate of 16 percent of successfully delivered emails. Participation rates of 20-30 percent in online surveys are more common. This resulted in a slightly higher than desired margin of error of 13.8 percent. For KIIs, the ET originally anticipated speaking with at least 136 respondents;

ultimately 79 respondents participated in interviews. The lower-than-expected response rate mostly arose among partner staff, as there were no significant issues in contacting IP or USAID staff. There were no notable differences in response rates across different types of partners (utilities, regulators, government entities, etc.). Across countries, unfortunately, the ET did not find any respondents willing to participate in Botswana or Lesotho (which only had six and four potential respondents to begin with, respectively, none of whom agreed to an interview). Largely due to the unequal gender distribution within the sector (and therefore within those with whom SAEP engaged), the ET received feedback from more male than female respondents.

The reduced sample sizes had two main implications for this report. First, since the number of respondents in each country was quite low in some cases, the ET could not disaggregate findings by country. Regarding Botswana and Lesotho in particular, though the ET did not speak with any local partner respondents in these countries, the evaluation still touches on SAEP’s work there through respondents (such as IP and USAID staff) who were knowledgeable about SAEP’s work across countries. Rather, the ET was only unable to capture the perspectives of local partners in these countries. Overall, the final number of respondents for some types of partner organization was small. Thus, the ET aggregated all partner respondents together, disaggregating findings by three types: USAID staff, IP staff, and partners.

9 | SAEP EVALUATION REPORT USAID.GOV

FINDINGS AND CONCLUSIONS

EQ1: TO WHAT EXTENT WAS SAEP ABLE TO MEET ITS PURPOSE AND RELATED OUTPUTS

AND OUTCOMES (MWS, CONNECTIONS, ENABLING ENVIRONMENT)?

FINDINGS

Multiple data collection sources informed this EQ, including document/desk review, secondary data, a quantitative survey, and KIIs. In particular, program monitoring data helped to provide insights on performance against the targets while KIIs provided stakeholder perceptions of success. The following sections begin with a review of the relevant performance indicators, followed by a discussion of stakeholder perceptions of SAEP success.

PERFORMANCE INDICATORS

As of June 2022, SAEP achieved its targets for new generation and new transmission (5,771MWs achieved out of 4,000MWs targeted) as well as new policies (33 out of 31) and people trained (2,033 out of 1,331) but had not yet achieved its goal for new electricity connections (2,138,029 out of 3 million) according to the most recent monitoring data.

Figure 9: Actuals vs. Targets for Key SAEP Indicators (as of June 2022)

STAKEHOLDER PERCEPTIONS

Definition of Success

• Stakeholders generally believed success aligns with SAEP’s indicators

• Impacts on people’s lives were missing from the indicators, however

Unintended Outcomes

• Few respondents cited unintended outcomes

• Those that did noted some expanded/enhanced activities

Top Successes

• Capacity development

• New generation

• Regulatory support

• New connections

• Utility support

• New transmission

Unachieved Outcomes

• Many respondents felt SAEP achieved its goals and had no unachieved outcomes

• Those that did noted new connections

USAID.GOV SAEP EVALUATION REPORT | 10

Definition of Success. The ET asked all respondents about their perceptions of SAEP success, including how they would define success for the program. In most KIIs (33 interviews), respondents believed that their definition of SAEP success aligned with the program’s performance indicators, namely MWs of generation and transmission, new power connections, and capacity building. However, a number of respondents went further to note that their definition of success included improving people’s lives (11 interviews). The quantitative indicators do not really capture impacts on people’s lives, though it is possible that such impacts were captured via other means such as success stories. These respondents indicated that, while it is important to measure performance in terms of the indicators, there was often more to the story. For example, respondents noted how additional generation capacity could facilitate productive uses or how having a transmission or distribution line could bring power to people without it. These findings were consistent across stakeholder groups.4

One IP respondent spoke to their definition of success:

“From the program perspective, when you achieve your key performance indicators, you are successful, of which I believe SAEP has achieved. But I base my version of success on that in any project I’ve got inputs, outputs, outcomes and impact… But if you go to a rural village and you find the people, now they’re using that power to charge their phones, they are using that power to light their homes at night, during the day they are using electricity for various livelihoods, that would be the impact of SAEP that I will evaluate.”

Top SAEP Successes. When asked about SAEP’s biggest successes, respondents most frequently noted capacity development (27 interviews), new generation (17 interviews), regulatory support (14 interviews), new connections (13 interviews), utility support (13 interviews), and new transmission (11 interviews). SAEP provided extensive training and capacity development services covering varied topics to a wide variety of stakeholder organizations, including utilities, regulators, government entities, and others. In discussing this capacity development work, respondents spoke positively not only about the technical aspects of training, but also about SAEP’s efforts to ensure gender balance, to the extent feasible, among trainees.

Regarding new power generation and transmission capacity, the qualitative data aligned with performance indicator data in showing SAEP’s achievement of the intended results. Interestingly, however, though the program had not yet met the target for new connections, many respondents applauded SAEP’s success in this area. This view that SAEP achieved positive new connection outcomes may be due to the common belief that new connections had unrealistic targets (discussed in more depth under EQ4). Respondents provided a variety of perspectives regarding success in new connections. When asked in which areas SAEP achieved the most success, some referred generally to new connections, such as one IP respondent who simply noted, “So new connections—that has been a great success.” Meanwhile, several others called out SAEP’s off-grid work in particular, such as a USAID respondent who noted, “I would say the off-grid sector is probably the place where it’s been truly transformational in the sense that… there was no off-grid sector in a lot of these countries.” Thus, even though SAEP may not have achieved its quantitative target, qualitatively, many respondents felt that SAEP made a substantial impact nonetheless.

Unachieved Outcomes. When the ET asked about outcomes that SAEP was unable to achieve, a large number believed that SAEP had achieved all its goals (16 interviews). The ET found this view particularly common among partner organizations (13 of the 16 interviews), potentially because many partner

4 Most findings in the evaluation are consistent across stakeholder groups. Thus, unless otherwise noted, findings do not vary notably between stakeholder types (IP staff, USAID staff, and partners).

11 | SAEP EVALUATION REPORT USAID.GOV

respondents equated success with achieving the goals of the specific interventions in which they engaged with SAEP (rather than on the broader SAEP results framework).

Where respondents noted unachieved outcomes, respondents most frequently cited new connections (23 interviews), a finding aligned with the performance indicator data noted above. In most of the other cases, when respondents talked about what SAEP did not achieve, they spoke about failures in specific interventions. For instance, several respondents in Madagascar discussed the delays and difficulties grant winners experienced in utilizing their grant funds for the intended purposes. One IP respondent noted, “In Madagascar, I think the work that we have been trying to do has been catastrophic.