1. SAEP Evaluation Report_2022.09.16.pdf

SOUTHERN AFRICA ENERGY

PROGRAM EVALUATION REPORT

September 16, 2022 i | USAID SAEP EVALUATION DESIGN REPORT USAID.GOV

CONTENTS

Contents ........................................................................................................................................................ i

Tables ........................................................................................................................................................... ii

Figures ......................................................................................................................................................... ii

Acronyms

Executive Summary ................................................................................................................................... iv Context ..................................................................................................................................................... iv Purpose .................................................................................................................................................... iv Evaluation Methodology ........................................................................................................................... iv Findings and Conclusions ......................................................................................................................... v Recommendations .................................................................................................................................. viii

Introduction Context Evaluation Purpose and Audience

Design and Methodology Evaluation Questions Methodology Data Analysis Ethical Considerations Gender and Social Inclusion Plan Limitations and Biases

Findings and Conclusions EQ1: To what extent was SAEP able to meet its purpose and related outputs and outcomes (MWs, Connections, Enabling Environment)?

EQ2: What enabled and/or hindered achievements and how were any hindrances addressed?

EQ3: How did the program design, target setting and operational decision-making shape and contribute to the achievements of the Program?

EQ4: What were the benefits and drawbacks of Power Africa setting the connections and MW targets for the program?

EQ5: In which countries did SAEP have the most success versus the least success, and why?

EQ6: How is the SAEP perceived and valued by the stakeholders of the Program?

Recommendations

Annex 1: Statement of Work

Annex 2: Bibliography

USAID.GOV USAID SAEP EVALUATION DESIGN REPORT | ii

Annex 3: Informed Consent Statements

Annex 4: Data Collection Instruments Key Informant Interview Guides

USAID/Embassy Staff (KIIs) IP Staff and Consultants (KIIs) Utility and Regulator Representatives (KIIs) Partner Government Representatives (KIIs) Regional Power Entities (Southern African Power Pool, SADC, etc) (KIIs) Private Sector Actors (KIIs) Other Donors (KIIs)

All Stakeholder Groups (Survey)

TABLES

Table 1: Evaluation Design Matrix

Table 2: Survey Sample Size by Respondent Type

Table 3: Relevant EQs for Each Respondent Type

Table 4: SAEP Contributions to the Sector (Scale of 1-3; 3=Significantly)

FIGURES

Figure 1: SAEP Actuals vs Targets for Key Indicators ................................................................................. v

Figure 2: Countries Where SAEP Found the Most and Least Success ...................................................... vii

Figure 3: Survey Respondent Perceptions of SAEP’s Impact ..................................................................... vii

Figure 4. Original SAEP Design Summary (SAEP Fact Sheet)

Figure 5: Total Respondents by Respondent Type (n=79)

Figure 6: Total Respondents by Country (n=79)

Figure 7: Total Respondents by Gender (n=79)

Figure 8: Potential Partner Respondents and Organizations by Country

Figure 9: Actuals vs Targets for Key SAEP Indicators

Figure 10: Satisfaction with SAEP by Respondent Type (Scale of 1-5; 5=Very Satisfied) https://socialimpact.sharepoint.com/sites/equiproj382/Shared%20Documents/05.%20Evaluation%20Report/SAEP%20Draft%20Evaluation%20Report_2022.09.15.docx#_Toc114242062 iii | USAID SAEP EVALUATION DESIGN REPORT USAID.GOV

ACRONYMS

COVID-19 Novel Coronavirus of 2019 ET Evaluation Team EQ Evaluation Questions GESI Gender Equality and Social Inclusion IP Implementing Partner KII Key Informant Interview MEL Monitoring, Evaluation, and Learning MW Megawatt PAIS Power Africa Information System PATT Power Africa Tracing Tool PMEP Performance Management & Evaluation Plan RFP Request for Proposals SADC Southern Africa Development Community SAEP Southern Africa Energy Program SOO Statement of Objectives SI Social Impact USAID United States Agency for International Development USG United States Government

USAID.GOV SAEP EVALUATION REPORT | iv

EXECUTIVE SUMMARY

CONTEXT

Power Africa is a United States government (USG)-led partnership that launched in 2013.

Power Africa has three core pillars guiding its work through 2030: adding 30,000 megawatts (MW) of new generation capacity, connecting 60 million new homes and businesses, and unlocking the potential of the energy sector by improving the enabling environment.1 The Southern African Energy Program (SAEP), a flagship Power Africa initiative funded by the United States Agency for International Development (USAID), aims to increase investment in electricity supply and access in Southern Africa by strengthening regulation, improving planning, procurement, and management of electricity trade, demonstrating and scaling energy efficiency technologies and practices, and providing capacity building to institutions and human resources in the sector.

PURPOSE

This evaluation provides USAID/Southern Africa and the USAID Power Africa team with feedback regarding SAEP successes, challenges, and areas for improvement which could inform future programming in the energy sector and support programmatic adjustments. Other potential evaluation users include implementers, regional organizations, national ministries, private sector partners, and regulators, who can learn from SAEP’s work.

EVALUATION METHODOLOGY

The SAEP evaluation addressed six core evaluation questions (EQs):

1. To what extent was SAEP able to meet its purpose, related outputs, and outcomes?

2. What enabled and/or hindered achievements and how were any hindrances addressed

(including but not limited to COVID-19 [novel coronavirus of 2019])?

3. How did the program design, target setting and operational decision-making shape and contribute to the achievements of the program?

4. What were the benefits and drawbacks of Power Africa setting the connections and MW targets for the program?

5. In which countries did SAEP have the most success versus the least success, and why?

6. How is the SAEP perceived and valued by the stakeholders of the program?

To answer these EQs, the evaluation team (ET) employed a mixed-method design across SAEP’s ten countries. On the qualitative side, the ET conducted a desk review of relevant documents and key informant interviews (KIIs). On the quantitative side, the ET administered a short, online survey for key stakeholders and reviewed the Power Africa Information System (PAIS), Power Africa Tracing Tool (PATT), and program monitoring data. Social Impact (SI) conducted remote data collection to reduce potential risks related to COVID-19 and mitigate logistical constraints.

1 USAID. 2016. The Roadmap: A Guide to Reaching 30,000 Megawatts and 60 Million Connections.

v | USAID SAEP EVALUATION DESIGN REPORT USAID.GOV

FINDINGS AND CONCLUSIONS

EQ1: TO WHAT EXTENT WAS SAEP ABLE TO MEET ITS PURPOSE, RELATED OUTPUTS AND

OUTCOMES?

SAEP met most of its performance targets, including for new generation and transmission.

However, SAEP has not yet met its target of 3,000,000 new electricity connections. Given the extension of the period of performance to June 2023, achieving this target remains possible. In interviews, respondents shared largely positive feedback regarding the results, feedback that aligned with the quantitative results tracked in the performance indicators. Despite SAEP’s focus on new generation, transmission, and connections, more respondents noted SAEP’s success in building capacity than they did the generation, transmission, or connection outcomes, suggesting an outsized impact on capacity building compared with the other outcomes.

Figure 1: SAEP Actuals vs Targets for Key Indicators

EQ2: WHAT ENABLED AND/OR HINDERED ACHIEVEMENTS AND HOW WERE ANY HINDRANCES

ADDRESSED?

Respondents shared numerous factors enabling and/or hindering SAEP achievements. Under different circumstances, each of these factors could act as an enabler or as a hinderance. The factors fell into three main categories: SAEP-level factors, partner-level factors, and context-level factors (in decreasing level of importance).

SAEP-level factors centered on SAEP’s technical capabilities and the strength of their relationships with partner organizations. In both cases, respondents typically saw these as strengths of the SAEP program, enabling program achievements. Nonetheless, some respondents noted these factors as hinderances, such as cases in which SAEP was unable to meet partners’ niche technical requirements or build strong relationships.

Partner-level factors centered on partner organizations’ willingness to engage with SAEP, their buy-in to the program’s activities, and the extent to which SAEP’s priorities aligned with those of its partners. The ET found that these interrelated factors demonstrate the importance of meeting

USAID.GOV SAEP EVALUATION REPORT | vi local needs as a means of enabling success. Where the services offered by SAEP aligned with stakeholder priorities, the ET found partners more likely to engage and more likely to have strong buy-in, all of which support strong success. When SAEP and partner priorities are not aligned, organizations are less willing to engage and buy-in is typically lower, acting as a barrier to success. Given the large number of SAEP countries and partners, these partner-level factors acted equally as often as drivers of success as hinderances, depending on the specific context.

Context-level factors centered on political influences and cultural and linguistic differences. The former included large political events like political changeovers due to elections, political unrest, and other large political events. These political factors fell outside of SAEP’s control. Cultural and linguistic differences affected achievements, though SAEP could adapt their approach to the context, such as by hiring local experts in each country. Respondents noted both positive and negative cases related to context-level factors, suggesting room for improvement in SAEP’s handling of cultural and linguistic differences.

EQ3: HOW DID THE PROGRAM DESIGN, TARGET SETTING AND OPERATIONAL DECISION-

MAKING SHAPE AND CONTRIBUTE TO THE ACHIEVEMENTS OF THE PROGRAM?

The ET examined several components of program design and implementation to answer this EQ, including SAEP’s strengths and weaknesses, the effectiveness of decision making, SAEP’s approach to gender equality and social inclusion (GESI), and respondents’ recommendations for future programs. Overall, the SAEP design and implementation approach contributed positively to program achievements. SAEP strengths include strong staff, effective communication, and a flexible and adaptable approach to implementation. Weaknesses included staffing concerns, such as issues with specific staff, staff turnover, and staff continuity with partners, and insufficient localization and local input into program design.

Overall, respondents applauded the SAEP team for its effective communication and decision-making processes. On the other hand, respondents noted that USAID’s decision-making processes often acted as a hinderance to effective implementation; in particular, USAID’s frequent requests for additional activities to be added to the SAEP scope.

SAEP’s GESI approach focused primarily on gender and not on other social groups, according to respondents. Though most activities included a gender focus, respondents felt that the effects were largely superficial, in part due to pervasive challenges within the male-dominated sector.

When asked for recommendations for future programming, respondents shared a variety of technical and design suggestions. On the technical side, respondents often suggested that future programs continue to focus on the same core areas as SAEP, such as new connections, generation, transmission, capacity building, and institutional strengthening. In terms of design, respondents recommended that future programming include more input from local stakeholders, incorporating local input earlier into the design process, increasing the utilization of local experts, and addressing the financing gap in the power sector.

EQ4: WHAT WERE THE BENEFITS AND DRAWBACKS OF POWER AFRICA SETTING THE

CONNECTIONS AND MW TARGETS FOR THE PROGRAM?

Very few implementing partner (IP) or USAID staff knew how SAEP’s original performance indicators, in particular the fee indicators, were selected or how the targets for these indicators were set. Thus, the ET relied heavily on document review to answer this question, including the vii | USAID SAEP EVALUATION DESIGN REPORT USAID.GOV original SAEP request for proposals (RFP) and technical portions of the SAEP contract. Despite the EQ wording suggesting that Power Africa set the targets for MWs and new connections, the document review revealed that USAID set minimum requirements of 3,000,000 new connections and 1,000MW of new, renewable energy generation. However, Deloitte increased the latter to 4,000MW and modified the indicator to include both generation and transmission.

Deloitte identified which indicators would be fee indicators. The fee indicators ultimately drove SAEP’s focus given the financial incentive to reach the targets.

Overall, both USAID and IP respondents regarded USAID’s use of fee indicators positively. The fee indicators helped the SAEP team remain focused and pushed them towards ambitious goals. On the downside, however, they voiced several concerns with the indicators and targets, including the targets’ unrealistic nature (particularly the new connections target), concerns with how attributable the MWs and new connections were to the SAEP-provided technical assistance, the timelines required to move from the direct benefits of SAEP’s technical assistance to longer-term outcomes like generation, transmission, and connections, insufficient localization in indicator selection and target setting, and how the regional targets led SAEP to focus on the largest projects (and therefore the largest markets) even when smaller, beneficial projects existed.

EQ5: IN WHICH COUNTRIES DID SAEP HAVE THE MOST SUCCESS VERSUS THE LEAST

SUCCESS, AND WHY?

According to respondents, SAEP achieved the most success in Malawi and Mozambique and the least success in Madagascar, Lesotho, Namibia, Botswana, and Eswatini. In addition, Zambia posed an interesting case as respondents included it on both the most and least successful lists, due to SAEP’s early, successful work in the country followed by significant political changes that significantly hindered SAEP’s later work in the country. As for what drove and/or hindered success in these countries, the factors mirrored the same factors outlined in EQ2. Respondents cited alignment of priorities between SAEP and its partners, stakeholder buy-in, and the strength of relationships as common factors. Beyond the EQ2 factors, in relation to Lesotho, Botswana, Swaziland, and Namibia, respondents also noted that the small size of the power sector in these countries acted as a barrier, in line with the finding under EQ4 that larger projects received more attention.

EQ6: HOW IS SAEP PERCEIVED AND VALUED BY THE STAKEHOLDERS OF THE PROGRAM?

Overall, stakeholders provided positive feedback regarding SAEP. Overall, they were satisfied with SAEP’s services and believed that SAEP positively impacted the power sector.

Nonetheless, differences existed between stakeholder groups. USAID respondents provided the most positive feedback whereas SAEP partners provided the least positive feedback.

Figure 2: Countries Where SAEP Found the Most and Least Success

USAID.GOV SAEP EVALUATION REPORT | viii

Figure 3: Survey Respondent Perceptions of SAEP’s Impact

Interestingly, even though SAEP has yet to achieve its performance targets for new connections, stakeholders valued SAEP’s impact on power access more highly than any other specific area of the sector. Meanwhile, feedback regarding SAEP’s impact on generation and transmission received the lowest scores, even though SAEP met those targets. This dichotomy likely relates to stakeholder feedback under EQ4 that the new connection target was unrealistic and challenges linking SAEP’s technical assistance with generation and transmission targets.

RECOMMENDATIONS

Based on the findings and conclusions of this evaluation, the ET recommends the following for USAID as it proceeds with designing the SAEP follow-on program:

1. USAID should ensure the localization of future programs. Though both USAID and Deloitte played parts in how localization occurred under SAEP, for future programming, USAID should begin the local engagement process early to ensure that local priorities and needs are taken into account even at the proposal stage. If engagement could not begin before the solicitation phase, then USAID should maintain sufficient flexibility to allow for local stakeholder influence on the design and implementation approach after award. Additionally, USAID should ensure that future programs are staffed in a way that addresses the different cultural and linguistic needs across the region, such as improved engagement of local staff and experts.

2. USAID should ensure that future indicator selection and target setting are based on clear criteria. Given that the fee indicators drive program focus, USAID should ensure that the fee indicators represent the intended focus of future programs and that the indicators link clearly with anticipated interventions. Additionally, the targets should be set using clear justifications and realistic criteria that are based on local input, needs, and priorities. This would ensure ambitious, yet realistic, targets given regional realities.

USAID should consider ways to balance regional flexibility with ensuring adequate attention to all countries, regardless of the size of their power market.

3. USAID should ensure that future programming improve on the GESI approach.

USAID does not yet have a clear definition for what GESI means for power programs.

Thus, to ensure a cohesive GESI approach, USAID should first define its GESI goals for a power activity such as SAEP. They can then hold IPs accountable to that vision.

3.0 3.0 3.0

2.52.6

1.9 2.3

2.5

1.9

1.4 1.3

2.32.3

1.7 1.8

2.4

1.0

1.5

2.0

2.5

3.0

The Power Sector Generation Capacity Transmission Capacity Power Access

USAID/USG IP Partners Total ix | USAID SAEP EVALUATION DESIGN REPORT USAID.GOV

4. USAID should ensure that future programming address the financing gap in the power sector. Though USAID could consider directly funding power investments, future programs could also address the financing gap via other means. To explore these potential options and prepare for program design, USAID should commission a study to examine the primary constraints of power sector financing in Southern Africa and identify any specialized programming requirements, such as credit guarantees or other lending de-risking mechanisms, to meet the needs of both credit seekers and financiers.

1 | SAEP EVALUATION REPORT USAID.GOV

INTRODUCTION

CONTEXT

Power Africa is a United States government (USG)-led partnership that launched in 2013. The Power Africa Road Map outlines three core pillars to guide its work through 2030: adding 30,000 megawatts (MW) of new generation capacity, connecting 60 million new homes and businesses, and unlocking the potential of the energy sector by improving the enabling environment.2

A lack of adequate and reliable power in the Southern Africa region limits competitiveness and affects the livelihoods and health of the population, particularly those living outside urban areas.

The five-year Southern African Energy Program (SAEP), a flagship initiative of this partnership, aims to increase investment in electricity supply and access in Southern Africa by strengthening the regional enabling environment and facilitating power transactions. SAEP, implemented by Deloitte, launched in 2017 with activities in ten countries3: Angola, Botswana, Eswatini, Lesotho, Madagascar, Malawi, Mozambique, Namibia, South Africa, and Zambia. Power Africa selected these countries to enable SAEP to take a regional approach to addressing sector needs. SAEP received a one-year extension, making it a six-year program expected to end in June 2023.

Figure 4. Original SAEP Design Summary (SAEP Fact Sheet)

Though SAEP aims to increase generation and access to electricity, SAEP does not directly fund major infrastructure investments. Rather, SAEP works to incentivize, facilitate, and enable investments through technical assistance, policy reform, capacity development, and transaction facilitation. Across the ten partner countries, SAEP’s work ebbed and flowed, with some countries receiving more support in some years than in others.

To identify interventions during the annual work planning process, the SAEP team meets with key stakeholders in each partner country to discuss potential interventions and ways in which SAEP could support their electrification, generation, and other sectoral improvement goals.

2 USAID. 2016. The Roadmap: A Guide to Reaching 30,000 Megawatts and 60 Million Connections.

3 SAEP originally targeted Zimbabwe as well, but ultimately did not implement activities in the country.

USAID.GOV SAEP EVALUATION REPORT | 2

SAEP’s flexible and adaptable approach to implementation involves iterative changes and adaptations to the work plan.

SAEP’s theory of change states that by strengthening regulation, improving planning and procurement, and management of electricity trade, demonstrating and scaling energy efficiency technologies and practices, and providing capacity building to institutions and human resources for energy sector management, SAEP aims to increase investment in electricity supply and access in Southern Africa. SAEP’s support works to address reliance on electricity/energy imports. Long-term program targets include 3,000MW of new power generation, 1,000MW of new transmission capacity, and 3,000,000 new connections (on- and off-grid).

EVALUATION PURPOSE AND AUDIENCE

The purpose of this evaluation is to provide USAID/Southern Africa and the USAID Power Africa team with feedback and recommendations for improving the SAEP activity and informing future programming in the energy sector. This evaluation equips USAID teams with evidence to support programmatic adjustments, inform future design efforts and strategies, and make informed decisions. Other potential users of the evaluation include implementers, regional organizations, national ministries, private sector partners, and regulators, who could adapt lessons learned from SAEP.

USAID/Southern Africa contracted Social Impact (SI) to conduct a utilization-focused, mixed methods evaluation with an emphasis on broad stakeholder engagement to promote its use and evidence-based decision making. SI’s technical design kept quality, utilization, and feasibility firmly in focus while ensuring broad coverage of all ten SAEP countries.

DESIGN AND METHODOLOGY

EVALUATION QUESTIONS

The Evaluation Team (ET) worked closely with USAID to refine and finalize the Evaluation Questions (EQs) to ensure mutual understanding and maximize utility and learning:

1. To what extent was SAEP able to meet its purpose, related outputs and outcomes (MWs, Connections, Enabling Environment)?

2. What enabled and/or hindered achievements and how were any hindrances addressed (including but not limited to COVID-19 [novel coronavirus of 2019])? Provide specific examples.

3. How did the program design, target setting and operational decision-making shape and contribute to the achievements of the program?

4. What were the benefits and drawbacks of Power Africa setting the connections and MW targets for the program?

5. In which countries did SAEP have the most success versus the least success, and why?

(focus on off-grid interventions; unlocking financing for private sector companies;

improving the enabling environment)

6. How is the SAEP perceived and valued by the stakeholders of the program?

METHODOLOGY

The SAEP evaluation relied on multiple methods to obtain data across the ten active SAEP countries. On the qualitative side, the ET conducted a desk review of relevant documents and

3 | SAEP EVALUATION REPORT USAID.GOV

literature and key informant interviews (KIIs). On the quantitative side, the ET carried out a systematic review of the Power Africa Information System (PAIS), Power Africa Tracing Tool (PATT), as well as program monitoring data and designed a short, online survey to allow key stakeholders across the region to have their perspectives heard. To reduce potential risks related to COVID-19, the ET only conducted remote data collection. The following sections describe SI’s mixed methods approach to the evaluation, as outlined in Table 1.

Table 1: Evaluation Design Matrix

EVALUATION QUESTIONS

DOCUMENT/

DESK

REVIEW

SECONDARY

DATA

QUANTITA-

TIVE SURVEY KIIS

Question 1: To what extent was SAEP able to meet its purpose and related outputs and outcomes (MWs, Connections, Enabling Environment)?

X X X X

Question 2: What enabled and/or hindered achievements and how were any hindrances addressed?

X X X

Question 3: How did the program design, target setting and operational decision-making shape and contribute to the achievements of the Program?

X X X X

Question 4: What were the benefits and drawbacks of Power Africa by setting the connections and MW targets for the program?

X X X

Question 5: In which countries did SAEP have the most success versus the least success, and why?

X X X X

Question 6: How is the SAEP perceived and valued by the stakeholders of the Program?

X X

Document/Desk Review. The ET conducted a systematic document/desk review to understand the availability and quality of secondary data in each of the ten SAEP countries, and the relationship of that data to the evaluation questions, and to identify key issues and constraints for further examination in primary data collection. The desk review included SAEP background documents, including quarterly and annual reports, programmatic documents, and relevant stakeholder documentation (see Annex 2: Bibliography for a full list of reviewed documents).

The desk review included SAEP background documents, including quarterly and annual reports, programmatic documents, and relevant stakeholder documentation (see Annex 2: Bibliography for a full list of reviewed documents).

Secondary Data. The ET reviewed PAIS, PATT, and SAEP monitoring data. The ET used data from the PAIS to understand SAEP partners, commitments (and progress towards them), USAID.GOV SAEP EVALUATION REPORT | 4 private sector engagement, and a broad range of other work completed through Power Africa.

Data from the PATT provided insights on power transactions, the energy mix, and generation capacity. As in the document/desk review, the ET examined data for trends over time and to better understand both progress and challenges. The ET found these databases particularly useful in answering EQ1 on SAEP’s ability to meet its purpose and objectives.

Quantitative Survey. The SI team implemented a short, online quantitative survey to assess achievements, benefits, successes, and challenges during implementation. This short survey enabled rapid collection of key perspectives from a broad number of stakeholders (across the same stakeholder groups noted below for the KIIs) in the ten active SAEP countries that may not have been otherwise reached through interviews. The survey tool contained questions to better understand perceptions of SAEP implementation, achievements, and challenges, with a particular focus on EQs one, two, five, and six. The data complemented the secondary data and document/desk review and the ET triangulated it with the more in-depth interview data. Table 2 outlines the final sample size:

Table 2: Survey Sample Size by Respondent Type

RESPONDENT TYPE NUMBER OF RESPONDENTS

USAID/United States Government Staff 6 Implementing Partner (IP) Staff 13

Utility Representatives 4

Partner Government Representatives 3

Regional Entities 1

Private Sector Actors 8

Other Donors 7 Total 42

Key Informant Interviews. The SI team held KIIs with a sample of stakeholders to provide context and document stakeholder experiences related to SAEP’s interventions. Employing a semi-structured interview approach, KIIs included questions regarding stakeholder experiences with program implementation and perceived program effects, including respective contributions of different activities to USAID’s results. The ET tailored interview protocols to the different stakeholder types to fit their roles. KIIs lasted approximately 60 minutes for all partner organizations and up to 120 minutes with USAID and IP respondents. Targeted stakeholder groups for KIIs included USAID and embassy staff, IP staff, utilities, regulators, government representatives, representatives of other donors, and private sector actors. The ET selected respondents purposively to include as broad a spectrum of experiences as possible.

Table 3 summarizes how the ET used feedback from each type of respondent to answer each of the EQs:

Table 3: Relevant EQs for Each Respondent Type

RESPONDENT TYPE EQ1 EQ2 EQ3 EQ4 EQ5 EQ6

USAID/USG Staff X X X X X X

IP Staff X X X X X X

Utility Representatives X X X

5 | SAEP EVALUATION REPORT USAID.GOV

Regulator Representatives X X X

Partner Government Representatives X X X

Regional Entities X X X X

Private Sector Actors X X X

Other Donors X X X

The figures below outline the final respondent sample sizes, distributed across the ten active SAEP countries and stakeholder types:

Figure 5: Total Respondents by Respondent Type (n=79)

Figure 6: Total Respondents by Gender

(n=79)

Figure 7: Total Respondents by Country (n=79)

DATA ANALYSIS

The ET analyzed data through an iterative process during and after the data collection.

Triangulation of the desk review documents, quantitative survey, and the KII data allowed the ET to assess trends, consistency, affinity, and contradiction between quantitative and qualitative data, aiming to better answer the EQs, generate more reliable findings, and reach well-founded conclusions. The ET applied a gender equality and social inclusion (GESI) lens to understand how findings differed by gender.

Quantitative Data Analysis. The ET performed rigorous data quality checks and reviewed the data using SI’s high frequency check protocols. These checks allowed the ET to identify survey logic, data range, missing values, implausible values, digit preference (for numeric responses), and outlier errors in the data. After the survey closed, the ET executed a final cleaning process before analysis. The ET utilized Excel software to explore trends and disaggregate by stakeholder type, country, and gender.

Qualitative Data Analysis. During data collection, the ET took detailed notes, and record (with consent) all discussions. After data collection, the ET utilized a structured coding approach to conduct content and thematic analysis using Dedoose software for identification of emergent

IP Staff Private Sector Actors

Government Stakeholders Utilities Providers

USAID and Mission Staff Regulators

Other Donors Regional Entities

Botswana

Lesotho

Mozambique

Namibia

Angola

Eswatini

Madagascar

South Africa

Zambia

Malawi

Regional

Male, 56, 71%

Female, 23, 29%

USAID.GOV SAEP EVALUATION REPORT | 6

themes and contextual factors upon which to draw findings and conclusions and to develop evidence-based recommendations. The ET utilized a Findings, Conclusions, and Recommendations Matrix to triangulate data across data sources and to ensure an evidence-based, triangulated, and systematic analysis process.

ETHICAL CONSIDERATIONS

SI’s in-house institutional review board reviewed the evaluation tools, data collection instruments, and protocols prior to the start of data collection. All members of the ET and management team received ethical research training to ensure honesty and integrity, protection of respondents, and data security. All respondents provided verbal or written informed consent prior to their participation in the evaluation. The evaluation did not attribute any information collected through KIIs to individual participants.

GENDER AND SOCIAL INCLUSION PLAN

As part of the evaluation, the ET assessed the extent to which SAEP implemented and/or adopted planned activities, such as gender-specific trainings, guidelines, indicators, policies, and strategies. With a GESI-sensitive approach, the ET reviewed secondary data and aimed to maintain gender balance in the KIIs. The ET ultimately interviewed more male than female respondents, however, mostly due to the power sector’s male-dominated nature. The ET used a GESI scorecard to maintain USAID’s Automated Directives System 205 standards. This scorecard allowed the ET to assess gender sensitivity throughout the evaluation process, including within the submitted data collection instruments. The ET disaggregated its analysis in the following sections by gender, socio-economic status, age, geographic region, and other characteristics as necessary and possible to discern differences and similarities in experiences, perceptions, needs, barriers, and other findings.

LIMITATIONS AND BIASES

The potential limitations as well as mitigation measures for the SAEP evaluation include:

Effects of Short Evaluation Timelines. To meet the timeline of the evaluation established by USAID, the ET combined methods and used a remote approach to quickly collect data from a broad range of stakeholders.

Effects of Large Geographic Dispersion. SAEP’s regional focus introduced a unique challenge for ensuring that the ET included the views and perspectives of all stakeholder groups in the evaluation. The dispersion created both logistical challenges for reaching key stakeholders in each country and challenges for aggregating findings across widely varying contexts. The evaluation design helped mitigate the logistical challenges while ensuring efficient data collection. Regarding aggregating findings, the ET spoke to a variety of stakeholders in each country and examined data for any potential inter-country differences.

Response Bias. The ET recognized that SAEP stakeholders may carry systematic but possibly unobserved biases specific to their role, cultural considerations, and/or other factors. This bias could influence their responses, such as being overly positive about achievements or saying what they think the ET or the Power Africa Program Office wants to hear. The SI team designed the instruments carefully and triangulated evidence across methods to minimize potential bias and ensure validity and reliability.

7 | SAEP EVALUATION REPORT USAID.GOV

Selection Bias. Neither the interviews nor survey reached every potential respondent, and some potential respondents chose not to participate in the evaluation. Thus, differences between who participated and who did not could introduce bias into the evaluation. The ET mitigated this limitation by inviting a wide range of stakeholders across all ten countries to participate and sought the support of SAEP staff to encourage participation.

In the end, the ET engaged more male than female respondents, mostly due to the unequal gender distribution within the sector. The evaluation had lower participation rates than expected among partners (no significant issues occurred with USAID or IP staff participation).

Participation rates of 20-30 percent in online surveys are common; the SAEP evaluation received a 16 percent response rate among successfully delivered invitations. This resulted in a slightly higher than desired margin of error of 13.8 percent. For KIIs, the ET originally anticipated speaking with at least 136 respondents; ultimately 79 respondents participated in interviews. As a result of the below challenges, the ET did not find any respondents willing to participate in Botswana or Lesotho. Additionally, the number of respondents in each country was quite low, as were the final number of respondents by specific respondent type. Thus, the ET could not disaggregate findings by country. To mitigate the small numbers of respondents from most partner types, the ET aggregated all partner respondents together; the ET therefore disaggregated results by three types: USAID staff, IP staff, and partners. Final sample sizes fell short of expectations for several reasons:

• Fewer Potential Partner Respondents Than Expected: Overall, the ET received names and contact information (phone or email) for 352 stakeholders involved to some extent with the program (including IP and USAID staff). Among SAEP partners, this included 296 individuals across 140 organizations. However, the number of potential respondents and organizations varied widely across countries, as seen in Figure 8, below.

Figure 8: Potential Partner Respondents and Organizations by Country

While the ET anticipated that levels of SAEP activity differed across the ten countries, the ET had not anticipated the extent to which this was the case. Though the ET reached out to more potential respondents in countries with greater SAEP activity, the

Angola

Botswana

Eswatini

Lesotho

Madagascar

Malawi

Mozambique

Namibia

South Africa

Zambia

Regional

Organizations Respondents

USAID.GOV SAEP EVALUATION REPORT | 8

small potential pool in many countries limited the ET’s opportunities to reach the anticipated sample.

• High Turnover: Numerous outdated contact details received by the ET suggested that SAEP had not maintained contact with individuals who previously engaged with the program. For the survey, of the 333 emails sent by the ET (not all potential respondents on the ET’s list included email addresses), 73 (22 percent) bounced back as undeliverable. Similar issues occurred with KII invitations. After additional inquiry, the ET found that the majority of the people whose emails bounced back no longer held the same positions or worked for the same organizations when they engaged with SAEP (see EQ2 for related findings on partner staff turnover).

• Low Levels of SAEP Awareness: Even among potential respondents that the ET contacted successfully, a surprising number expressed that they lacked sufficient familiarity with SAEP’s work to respond to the ET’s questions, and particularly common among other donor and government stakeholder respondents.

• Fewer Group Interviews: When reaching out to partner organizations, the ET anticipated conducting one KII with two to three individuals within each organization.

However, in the end, partners often preferred an individual KII, identifying a single individual to speak with the ET rather than a small group. As a result of more frequent individual KIIs, the total number of respondents was lower than expected due to fewer small group interviews. Of the 56 interviews conducted, only 17 included more than one person. Thus, though the ET collected views from that organization as intended, the total number of individual respondents was lower than expected.

FINDINGS AND CONCLUSIONS

EQ1: TO WHAT EXTENT WAS SAEP ABLE TO MEET ITS PURPOSE AND RELATED OUTPUTS AND

OUTCOMES (MWS, CONNECTIONS, ENABLING ENVIRONMENT)?

FINDINGS

Multiple data collection sources informed this EQ, including document/desk review, secondary data, a quantitative survey, and KIIs. In particular, program monitoring data helped to provide insights on performance against the targets while KIIs provided stakeholder perceptions of success. The following sections begin with a review of the relevant performance indicators, followed by a discussion of stakeholder perceptions of SAEP success.

PERFORMANCE INDICATORS

As of June 2022, SAEP achieved its targets for new generation and new transmission (5,771MWs out of 4,000MWs targeted) as well as new policies (33 out of 31) and people trained (2033 out of 1331) but had not yet achieved its goal for new electricity connections (2,138,029 out of 3,000,000) according to the most recent monitoring data.

9 | SAEP EVALUATION REPORT USAID.GOV

Figure 9: Actuals vs Targets for Key SAEP Indicators

STAKEHOLDER PERCEPTIONS

Definition of Success. The ET asked all respondents about their perceptions of SAEP success, including how they would define success for the program. In most KIIs (33 interviews), respondents believed that their definition of SAEP success aligned with the program’s performance indicators, namely MWs of generation and transmission, new power connections, and capacity building. However, a number of respondents went further to note that their definition of success also included improving people’s lives (11 interviews); something not adequately captured in the indicators. These respondents indicated that, while it is important to measure performance in terms of the indicators, there was often more to the story. For example, respondents noted how additional generation capacity could facilitate productive uses or having a transmission or distribution line could bring power to people without it.

One IP respondent noted, “From the program perspective, when you achieve your key performance indicators, you are successful; of which I believe SAEP has achieved. But I base my version of success on that in any project I've got inputs, outputs, outcomes and impact… But if you go to a rural village and you find the people, now they're using that power to charge their phones, they are using that power to light their homes at night, during the day they are using electricity for various livelihoods.

That would be the impact of SAEP that I will evaluate.”

Top SAEP Successes. When asked about SAEP’s biggest successes, respondents most frequently noted capacity development (27 interviews), new generation (17 interviews), regulatory support (14 interviews), new connections (13 interviews), utility support (13 interviews), and new transmission (11 interviews). In discussing SAEP’s capacity development work, respondents spoke not only about the technical aspects of training, but also about SAEP’s efforts to ensure gender balance, to the extent feasible, among trainees.

Regarding new power generation and transmission capacity, the qualitative data aligned with the performance indicator data in showing SAEP’s achievement of the intended results.

Interestingly, however, though the program had not yet met the target for new connections, USAID.GOV SAEP EVALUATION REPORT | 10 many respondents applauded SAEP’s success in this area. This may be due to the common belief that new connection had unrealistic targets (discussed in more depth under EQ4).

Unachieved Outcomes. When the ET asked about outcomes that SAEP was unable to achieve, a large number believed that SAEP had achieved all its goals (16 interviews). The ET found this view particularly common among partner organizations (13 of the 16 interviews).

Where respondents noted unachieved outcomes, respondents most frequently cited new connections (23 interviews); a finding aligned with the performance indicator data noted above.

In most of the other cases, when respondents talked about what SAEP did not achieve, they often spoke about failures in specific interventions. For instance, several respondents in Madagascar discussed the delays and difficulties in delivering solar materials to grant winners.

One IP respondent noted, “In Madagascar, I think the work that we have been trying to do has been catastrophic. There've just been enormous challenges related to things that are out of [our] control… like mini-grid companies not being able to pass their equipment through ports of entry because of changes and taxation laws.”

Unintended Outcomes. In addition to asking about SAEP’s intended outcomes, the ET asked about unintended outcomes, which can reveal unanticipated benefits and/or consequences.

When asked, respondents noted few unintended outcomes, with a notable number (11 interviews) responding directly that there were no unintended outcomes to the program.

Respondents who noted unintended outcomes most frequently highlighted SAEP’s efforts to further aid partners by adding additional interventions (18 interviews). For example, several partner respondents noted appreciation for how, after they started working with SAEP, they realized additional weaknesses in their operations, which SAEP helped them to address.

CONCLUSIONS

Stakeholders’ vision of success for SAEP largely aligned with SAEP’s established results framework, within which SAEP met most of its targets. The program had few unintended or unanticipated outcomes. Though SAEP had not yet met its target for new connections, stakeholders nonetheless believed it belonged among SAEP’s top successes.

EQ2: WHAT ENABLED AND/OR HINDERED ACHIEVEMENTS AND HOW WERE ANY HINDRANCES

ADDRESSED?

FINDINGS

SAEP’s work across its ten partner countries and myriad of stakeholders and partner organizations meant that the program encountered widely varying circumstances and factors that could potentially enable and/or hinder achievements. To assess these factors, the ET asked all interviewees a series of questions regarding SAEP’s work and the factors influencing its success (or lack thereof). In particular, USAID expressed interest in learning about the impacts of the COVID-19 pandemic, and thus, in addition to asking about general factors influencing results, the ET also asked specifically about the effects of COVID-19.

Aside from COVID-19, respondents discussed numerous factors that impacted SAEP’s achievements. These factors fell into three main categories, in descending order of noted frequency: SAEP-level factors, partner-level factors, and context-level factors. For all factors, respondents noted a variety of cases in which the factor could act either as an enabler to SAEP success or as a hinderance. For example, partners’ willingness to engage with the SAEP team

11 | SAEP EVALUATION REPORT USAID.GOV

could be an enabler in cases where the organization wanted to engage but a barrier in cases where an organization resisted SAEP’s efforts.

The following sections describe the implications of the COVID-19 pandemic followed by an exploration of the three factor categories that most significantly impacted SAEP success.

COVID-19

Though respondents in a few interviews (nine interviews) thought that COVID-19 did not have a substantial impact on implementation, the majority (37 interviews) talked about how COVID-19 resulted in challenges to SAEP implementation. The most notable challenges included delays in implementation (21 interviews) and the inability to travel (19 interviews). In regard to delays, the specific reasons for the delays differed. In some cases, respondents said that the shutdowns at the start of the pandemic and then in early 2022 when Southern Africa was hit by a particularly strong wave of infections resulted in delays. In other cases, the general slowdowns in operations across the world as people learned to transition their work online caused delays. The inability to travel caused challenges for a regional program like SAEP. Prior to the pandemic, SAEP staff typically spent significant amounts of time traveling to the ten partner countries, meeting with partners and stakeholders, and working to advance SAEP objectives. This travel largely came to a standstill with the onset of the pandemic, according to interviewees. The COVID-19-related delays and the inability to travel often went hand-in-hand. As one IP respondent noted, “I would have travelled to Angola without COVID, to be honest, because that’s the most effective way… we need relationships, like you need to sit together. We need to eat together, go to project sites. That was hard. Internet connections were actually good, but we couldn’t form those bonds or relationships in the same way.” Thus, the inability to travel impacted relationship building and the team’s ability to move activities forward as swiftly as they might have been able to do otherwise.

After the pandemic started, SAEP quickly moved to a virtual approach – both for stakeholder engagement as well as for training and capacity building. Interviewees across stakeholder groups felt that SAEP made a positive and timely transition (19 interviews). Nonetheless, a significant number of respondents (particularly among IPs and partners) noted that in-person approaches to program implementation were significantly better than virtual approaches (22 interviews). In one interview, a utility representative talked about a particular training they received for how to use a new program. They appreciated the support and virtual training received from SAEP. However, they lamented that SAEP staff did not travel to provide direct support, sitting at the same computer, to work through questions and issues as they went through the process. Overall, respondents appreciated the support; they just noted that being able to work together in-person could have made it even better.

SAEP-LEVEL FACTORS

Aside from COVID-19, respondents most frequently highlighted SAEP-level factors as influencing success. Among these SAEP-level factors, respondents noted the technical capabilities and expertise of the SAEP team (25 interviews) and the strength of the relationships that SAEP had with its partner organizations (23 interviews) most frequently. In terms of technical capabilities, respondents felt that SAEP’s ability to meet partners’ specific needs, even for niche technical expertise, was a key factor for success. In most cases, respondents viewed SAEP’s technical expertise favorably and they cited them more frequently as enablers of success.