J.1.19 External Evaluation Report for Nigeria Power Sector Program.pdf

PHOTO SOURCE: POWER AFRICA, 2023

June 2023

This publication was produced with the support of the United States Agency for International Development (USAID) Power Africa initiative under the terms of the Power Africa Contract Evaluation Services Task Order No. 720-674-23-F-00001.

EVALUATION

External Performance Evaluation of the Nigeria Power Sector Program (NPSP)

Executive Summary

EXTERNAL PERFORMANCE EVALUATION OF NPSP | JUNE 2023 | i

Report Title: External Performance Evaluation of the Nigeria Power Sector Program (NPSP)

Lead Author: Julia Larkin

Supporting Authors: Karin Sosis, Michel Ouedraogo, Sujata Ram, Nicole Economu, and Isabel Morford-Cheibub

Submitted by: ICF

Activity Title: Power Africa Contract Evaluation Services

USAID Task Order Number: 720-674-23-F-00001

Completion Date: June 9, 2023

Photo Caption: Two staff members with Abuja Electricity Distribution Company installing a meter.

The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

EXTERNAL PERFORMANCE EVALUATION OF NPSP | JUNE 2023 | ii

ACKNOWLEDGEMENTS

On behalf of the ICF Evaluation Team, I, Mark Wagner (Evaluation Oversight and Quality Assurance Officer), would like to extend my sincere appreciation to the many USAID staff, Implementing Partners, host country government representatives, development partners, utility and energy companies, and other key informants who contributed their valuable time and insights to support the evaluation process. I am particularly grateful to the USAID staff and Implementing Partners who connected our team to program stakeholders, provided key documentation, and improved our understanding of Power Africa’s experiences, successes, and challenges, as well as to the Contracting Officer’s Representative, Yugi Nair, and the Alternate Contracting Officer’s Representative, Ngonidzashe Makaza, for their continued support and the trust they placed in us throughout the process.

EXTERNAL PERFORMANCE EVALUATION OF NPSP | JUNE 2023 | 1

EXECUTIVE SUMMARY

EVALUATION OVERVIEW

The United States Agency for International Development (USAID) Power Africa (PA) Program Office contracted ICF to conduct external performance evaluations of four PA programs, including the Nigeria Power Sector Program (NPSP), to improve program performance management and efficiency. The lessons drawn from these evaluations are directed toward technical teams, activity managers, and senior management.

ICF developed an overarching evaluation design framework for all four PA programs that included five focus areas: (1) relevance and coherence of the theory of change and program design; (2) efficiency and effectiveness of program delivery; (3) outcomes, impact, and sustainability of program delivery; (4) key barriers to the delivery of results; and (5) lessons learned and recommendations.

The evaluation team employed a combination of data collection and analysis methods. The team reviewed relevant program and external documents and triangulated findings from the desk review with data collected through 55 semi-structured key informant interviews. The evaluation team also distributed a survey to nine private energy companies and received five responses.

SUMMARY OF NPSP

Deloitte Consulting LLP was awarded NPSP under Task Order No. 720-674-18-F-00003 of the Power Africa Indefinite Delivery, Indefinite Quantity (IDIQ) Contract No. 720-674-18-D-00003. NPSP is a five-year program extended to six-and-a-half years with a Total Estimated Cost of $109,258,794 and a period of performance from April 2, 2018, to September 30, 2024. NPSP focuses exclusively on Nigeria.

NPSP, implemented in coordination with USAID/Nigeria, is designed to support comprehensive reform within Nigeria’s power sector. The program has a broad, cross-subsector mandate, aiming to accelerate energy transactions that lead to increased power generation and increased connections to end-users by working with government partners to create the policy, legal, and regulatory frameworks necessary for private sector investment and financial sustainability within the power sector. NPSP has built upon progress made in the prior program, Power Africa Transaction and Reforms Program (PATRP), as well as expanded the focus beyond transaction advisory support to further targeting upstream activities.

Through a range of technical assistance activities, it seeks to address specific constraints in the energy sector across four outcome areas: (1) increased generation capacity; (2) increased off-grid connections facilitated; (3) improved enabling environment; and (4) improved liquidity of the distribution sector.

KEY TAKEAWAYS

Overall, NPSP activities are appropriate and responsive to the Nigerian context. The implementation team is well regarded and is seen as competent, effective, and reliable, producing quality outputs. The challenges are well understood. A core element of NPSP’s approach is the use of embedded advisors, who provide substantial value within the government entities where they are placed. Informants value the high-quality technical reports and other outputs produced by NPSP and NPSP’s hosting and convening activities.

EXTERNAL PERFORMANCE EVALUATION OF NPSP | JUNE 2023 | 2

NPSP effectively engages and coordinates with a wide range of public and private partners and other stakeholders, though there are some concerns. Overall, NPSP is seen as an effective convener and as liaising with the right stakeholders, though the depth and frequency of engagement varies by stakeholder and opportunity. There is a lack of clarity among several development partners as to what exactly NPSP can offer and by extension what the United States Government (USG) priorities and boundaries in Nigeria are. Some development partners have the perception that NPSP tends to take credit for successful outcomes primarily developed by others.

Even with budget cuts, NPSP is progressing well on indicators for which it has the most direct influence. The progress to date varies significantly across categories, with some targets already exceeded with 18 months remaining, such as the number of people trained and new connections. NPSP is highly unlikely to meet its targets relating to generation capacity, but these were based on unrealistic assumptions. Progress on transmission-related indicators is moderate to date.

LESSONS

Personal relationships are highly important in Nigeria. NPSP is building long-term relationships and gaining respect from counterparts and development partners as a neutral source of unbiased technical information. NPSP appears to have struck the balance between relationship development while avoiding any appearance of excessive favoritism or undue influence. The program is seen as having a good local presence overall, which helps in building relationships, as does having embedded advisors.

The program approach must proactively evolve as the context evolves. The context will evolve as new international players engage or disengage, as government priorities change, and as the target audience gains capacity and moves farther along the capacity chain. NPSP’s breadth and country-level approach help to take advantage of synergies across the value chain and adapt to new opportunities and challenges, though the substantial needs over such a broad scope and program length require careful attention to avoid missing synergies or opportunities. The implementation team seeks to adapt and evolve in terms of resource allocation as well as approach as the situation changes. For example, NPSP shifted resource allocations between Outcome Areas over time based on strategic assessments of potential impact in a resource-constrained context.

NPSP’s activities are complex and typically take years to fully mature, so promising activities and transactions may ultimately lose momentum or stall as the context shifts. A program like NPSP needs to be given sufficient time and flexibility to support a variety of activities, only a subset of which will ultimately mature into meaningful outcomes. Appropriate planning must be conducted during the project’s early years to not only accurately assess barriers and opportunities, as was accomplished in NPSP’s End-to-End Assessment, but also to target sufficient investment in the early stages of implementation to ensure success. For example, NPSP built on interventions started under the Power Africa Transaction and Reforms Program (PATRP).

In conclusion, while some activities or transactions are by their nature transformative or lasting, many activities rely on the ongoing political will, capacities, or multi-agency alignment to ultimately achieve the desired outcomes and are not necessarily within NPSP’s control. In this context, focusing on specific transactions helps individual projects or activities evolve even when broader systemic issues remain.

EXTERNAL PERFORMANCE EVALUATION OF NPSP | JUNE 2023 | 3

The USAID reporting structure can miss important nuance and incremental progress. NPSP complies with USAID reporting requirements, which are activity, indicator, and target focused. Yet NPSP’s specific contributions can be unclear at a given time as there are limited indicators tracking quality over quantity, interim achievements, or soft activities such as relationship development. To illustrate this point, consider three of the most notable program successes to date: (1) support to off-grid developers, (2) progress being made in the Federal Government of Nigeria’s (FGN) enabling environment, and (3) initiatives to promote women’s empowerment. The depth or breadth of these successes is not readily discernible from the indicators.

• For example, tracking the number of policies approved does not address the quality or potentially transformative nature of the policies (or lack thereof). The tier of electricity connection—another dimension of quality—also is not tracked.

• Another example is outcome indicators that only reflect successes, which are intermittent and may be weighted toward later stages. This includes annual results for NPSP’s “number of megawatts (MW) from transactions that have achieved financial close” indicator, which range from 13 MW to 990 MW from 2019 to 2022 in the following pattern: 2019: 13 MW; 2020: 990 MW;

2021: 40 MW; and 2022: 45 MW. NPSP’s Fiscal Year (FY) 2023 target is 1,925 MW.

• An additional example includes the importance of soft contributions of embedded advisors (or lack thereof) that do not necessarily come across in reporting, though NPSP has made some adjustments to the narratives to help flesh out the story.

There continues to be insufficient transmission capacity and commercial liquidity to handle significant new on-grid power generation, despite the need. NPSP will not come close to meeting the MW generation or related on-grid targets largely due to systemic issues that have not evolved substantively, including lack of evacuation capacity and collections shortfalls from the Distribution Companies (DISCOs) and government policy decisions. Mitigating complex Aggregated Technical, Commercial, and Collection (ATC&C) losses and continuing to boost off-grid and grid-supportive options remain major priorities for local stakeholders alongside efforts to add generation capacity.

RECOMMENDATIONS

The following recommendations are designed to highlight both existing implementation strategies that are important to continue as well as proposed adjustments for future programs. There will be a great deal of change over the next 12 to 18 months with the new administration in May 2023. The contract extension through September 2024 already helps with continuity and provides NPSP the opportunity to adjust its approach and build new relationships with the incoming administration.

RECOMMENDATION 1: USAID PA should continue its overall approach in Nigeria, but further refine how it communicates with stakeholders. Overall, the current approach and implementation strategy is reasonable for the complex context. Yet there are opportunities for refinement, particularly by more clearly articulating priorities and boundaries to all parties. First, USAID PA should communicate with the implementation team to ensure activities map to overall USAID priorities. Second, USAID PA should clarify what it will and will not do with development partners and beneficiaries, especially in the private sector, to facilitate synergies and interest in further engagement.

Third, USAID PA should communicate joint successes in ways that acknowledge key contributions of others, to deepen trust.

EXTERNAL PERFORMANCE EVALUATION OF NPSP | JUNE 2023 | 4

• ACTION 1. Keep the country-level, sector-wide approach, but ensure relative priorities are clear and reasonable for the implementation team. On balance there are more strengths than weaknesses with a country-level, sector-wide approach. With such a broad scope, there are ongoing risks that any particular topic will receive insufficient attention to efficiently respond to opportunities, as the context continuously evolves. Therefore, USAID PA should regularly articulate any minimum levels of effort or focus and conversely provide the implementation team room for flexibility to allocate resources between Outcome Areas and/or specific activities.

• ACTION 2. Continue the partnership model with regular coordination meetings, but increase the clarity of USAID PA roles, priorities, and boundaries in Nigeria. While development partners are aware of USAID, there is widespread uncertainty on what exactly USAID PA offers. This issue is broader than NPSP but does also affect how NPSP is perceived. USAID PA should take steps to clarify overall USAID priorities on the website, factsheets, and other public materials, in addition to the talking points used by USAID and its representatives, so that development partners as well as potential beneficiaries can more quickly and independently gain a better sense of the basic offering and types of support and engagement possible with USAID PA, including through NPSP. This will help build synergies with development partners as well as increase trust and interest from potential beneficiaries, especially in the private sector.

• ACTION 3. Continue PA’s role in amplifying overall sectoral progress in ways that appropriately acknowledge the role other key development partners play in achieving outcomes. USAID PA should pay careful attention to how progress and successes are framed in program messaging to avoid appearing to take credit for contributions of others, especially those whose role may be even more significant than USAID PA’s role. As above, this issue is broader than NPSP, but does also affect how NPSP is perceived.

RECOMMENDATION 2: The implementation team should continuously ensure relevance through agile program management with strong feedback loops. The implementation team should continue to drive priority interventions, while realistically appraising its sphere of influence. There is an ongoing need to balance support of high-profile initiatives that may not have sufficient lasting support with lower profile but practical initiatives that will outlast political changes. In practice, this means continuously adjusting efforts toward ongoing relationship building with awareness of which activities are likely to achieve significant traction and which are not.

• ACTION 1. Continue to focus on relationship building. The importance of personal relationships and trust building in the Nigerian context cannot be overstated. Continuity and reliability are important to increase trust, as is a permanent local presence. These relationships are important for building lasting successes even when the efforts are not reflected in performance targets in the near term. The implementation team should continue its multi-faceted efforts in these areas to further build on existing gains.

• ACTION 2. Build on successes and current momentum. Notable areas of success with ongoing momentum that should continue include the contributions to the development of Lagos State Integrated Resource Plan (IRP), support to off-grid energy companies, and NPSP’s efforts to promote gender awareness and women’s empowerment issues that continue to evolve.

• ACTION 3. Increase focus on mitigating transmission and distribution issues. Increase focus on mitigating ATC&C losses—as well as continuing to boost off-grid and grid-supportive options—

EXTERNAL PERFORMANCE EVALUATION OF NPSP | JUNE 2023 | 5

to take the next step in addressing the key priorities for local stakeholders and increase Nigeria’s ability to benefit from new generation capacity.

• ACTION 4. Continuously watch for opportunities to expand reach beyond Abuja and Lagos.

Given ongoing security concerns, there are private companies in northern and southern parts of Nigeria, for example, that do not have the resources to attend current events.

RECOMMENDATION 3: In future contracting, USAID PA should ensure a holistic approach to programming and a more nuanced suite of indicators. Many of NPSP’s goals are long term, meaning sufficient investment and targeting need to be made in early years to have a possibility of reaching fruition during program time frames. Therefore, the contracting structure and selection of indicators and performance targets should be considered carefully, to ensure appropriate messaging and incentives are provided to the implementation team to take a holistic approach with a long-term view while still making progress on measurable indicators. The contracting structures also should be sufficiently flexible to accommodate the inevitable shifts as the context changes.

• ACTION 1. Review how contracts are structured and set realistic context-aware and evidence-based performance metrics to promote a holistic approach and avoid distorted incentives.

Appropriate metrics and associated targets reduce the tendency to distort the focus to only “meet the numbers” rather than a more holistic approach that balances short-term opportunities with long-term systemic gains. In fact, wildly unrealistic targets may discourage rather encourage the desired activities. Ultimately, USAID benefits when the implementation team believes it has sufficient room to explore some new ideas or go into more challenging areas if that is where the opportunities are, and not only focus on reliable successes.

• ACTION 2. Include output or leading indicators to better reflect incremental and soft progress.

The next iteration of indicators should include more “leading indicators” or metrics to assess incremental progress and outputs of soft actions directly attributable to NPSP actions. What these are will depend on USAID PA’s priorities and the focus of the activities. Yet the next iteration should also include metrics that address the quality of outputs and outcomes. There are broader efforts within PA to better align indicators to activities and outcomes that should flow down to the next generation of contracts. This will increase USAID understanding of the implementation team’s relative progress and provide more opportunities to credit the implementation team for successes within its control even where longer term and more visible outcomes have not (yet) been achieved.

• ACTION 3. Fully integrate gender into the monitoring, evaluation, and learning (MEL) plan. The next iteration of the MEL plan should more comprehensively address gender in all relevant activities. At a minimum, it should include sex-disaggregated data across relevant indicators. As above, it will be valuable to include metrics of qualitative or downstream progress beyond the number of men or women trained so that key activities, such as those from the successful Energy Sector Women’s Leadership Initiative (ESWLI) program, are documented more comprehensively.

The specific metrics will depend on the focus of the activities and the program logic articulated.

• ACTION 4. Work with the implementation team to tailor reporting to increase transparency of key points of progress and NPSP’s relative contribution to broader outcomes without increasing the burden on or intrusion into internal or commercial decision-making. What specifically will be needed in this area will depend on the context (e.g., the specific contracting structure, indicators tracked, and USAID contract officer’s representative (COR) priorities).

1.2 Organization of the Report

2.3 Evaluation Limitations

3.5 Key Barriers to Results Delivery

4.2 Recommendations