End_Term_Evaluation.pdf

March 2018

This publication was produced by Nathan Associates Inc. for review by the United States Agency for International Development.

US-APEC TECHNICAL

ASSISTANCE TO ADVANCE

REGIONAL INTEGRATION

END-TERM EVALUATION

US-APEC Technical Assistance to Advance Regional Integration (US-ATAARI)

DISCLAIMER

This document is made possible by the support of the American people through the United States Agency for International Development (USAID). Its contents are the sole responsibility of the author or authors and do not necessarily reflect the views of USAID or the United States government.

US-APEC TECHNICAL

ASSISTANCE TO ADVANCE

REGIONAL INTEGRATION

END-TERM EVALUATION

US-APEC Technical Assistance to Advance Regional Integration (US-ATAARI)

TABLE OF CONTENTS

Table of Contents 2

Acknowledgements 3

Acronyms 4

Introduction 5

Methodology 8

Limitations and Challenges Encountered During the Research 8

General Findings 10

Findings By Workstream 14

A. Trade and Investment Facilitation 14

1. Supply Chain Connectivity 14

2. Digital Economy 18

B. Regulatory Reform 22

1. Ease of Doing Business (EODB) 23

2. Structural Reform, including Renewed APEC Agenda on Structural Reform (RAASR) and Good Regulatory Practices (GRP) 27

C. Women and the Economy 31

D. APEC Secretariat Support 35

Case Study at the Economy-Level 39

Workstream: Inefficient Fossil Fuel Subsidies 39

Workstream: Improving Supply Chain Performance 41

Workstream: Agricultural Biotechnology 42

Workstream: Host Year Preparation 43

Conclusion and Reccomendations 44

Annex 1. Logical Flowchart and Indicators 47

Annex 2. List of Interviews 49

ACKNOWLEDGEMENTS

The evaluators wish to thank the respondents who generously gave of their time for interviews conducted during this evaluation. Special thanks are also due to Joshua Templeton, Asia Regional Economist, and Lori Rakoczy, Economist USAID Bureau for Asia, for their assistance in conducting interviews in the Philippines.

ACRONYMS

ANSSR APEC New Strategy for Structural Reform

APEC Asia Pacific Economic Cooperation

AR Advance Rulings

A2C2 APEC Alliance for Supply Chain Connectivity

BMC APEC Budget and Management Committee

DB Doing Business

DOL U.S. Department of Labor

DOS U.S. Department of State

E3/TRR Economic Growth, Education, and Environment’s Office of Trade and Regulatory

Reform

EoDB Ease of Doing Business

GETR Global Enabling Trade Report

HWHE Healthy Women, Healthy Economies

IEA International Energy Agency

LAISR Leaders’ Agenda to Implement Structural Reform

LPI Logistics Performance Index

M&E Monitoring & Evaluation

PAP Pre-Arrival Processing

PCCI Philippine Chamber of Commerce and Industry

PDOE Philippine Department of Energy

PPWE Policy Partnership on Women and the Economy

RAASR Renewed APEC Agenda on Structural Reform

RDMA Regional Development Mission for Asia

STEM Women in Science, Technology Engineering and Mathematics

USAID U.S. Agency for International Development

US-ATAARI US-APEC Technical Assistance to Advance Regional Integration

USG U.S. government

USTR Office of the United States Trade Representative

WE-APEC Women’s Entrepreneurship in APEC

WiT Women in Transportation

INTRODUCTION

The US-APEC Technical Assistance to Advance Regional Integration (US-ATAARI) is a five-year project that aims to strengthen the capacity of the APEC Secretariat and APEC’s foreign-assistance eligible developing member economies to advance regional economic integration and achieve the Bogor Goals of free and open trade and investment in the region.1 The project implements activities that address selected elements of each of APEC's three “pillars” (Trade and Investment Liberalization; Business Facilitation; and Economic and Technical Cooperation), and also strengthen the APEC Secretariat's institutional capacity and strategic focus. US-ATAARI pursues its program objectives by delivering targeted technical assistance, ranging from policy studies and assessments to capacity building, training, and expert advice on integration, trade liberalization and institutional management.

The Implementing Partner for this project is Nathan Associates, which maintains US-ATAARI teams in Washington D.C. and in Singapore at the APEC Secretariat. Per contractual arrangements, US-ATAARI is managed by USAID Asia Bureau’s Technical Services Office (ASIA/TS/EG), with funding and strategic direction from the U.S. Department of State (DOS). US-ATAARI builds upon past assistance provided to APEC by the U.S. Government to assist APEC in achieving its policy objectives by delivering targeted technical assistance, ranging from policy studies and assessments to capacity building, training, and expert advice on integration, trade liberalization and institutional management. The project design is flexible and demand-driven, with Nathan Associates working closely with USAID and State EAP/EP to implement activities that involve a range of USG partners, and reflect USG priorities during any given work plan year.

USG partners develop annual workplans that are submitted to State and USAID and, following approval of the proposed plan, work closely with the US-ATAARI team to implement those activities.

The annual workplans have consisted of activities requested by these partners, which are then funneled through the relevant APEC fora for endorsement. This happens both before and during a given work plan year, and impacts how activities are budgeted and implemented. US-ATAARI interventions are conducted at the regional level, many times on the sidelines of regular APEC meetings, and at the economy-level.

Work at the economy-level is at the request of the individual member economy, always on a volunteer basis, and usually in response to offers of assistance on the workplan/strategic plan of the relevant Fora.

1 The 8 APEC member economies that are eligible for travel and technical assistance support are Chile, Indonesia, Malaysia, Mexico, Papua New Guinea, Peru, Philippines, and Viet Nam. In addition, within the APEC context, China, Thailand, and Russia are officially eligible for travel support, but these three are not eligible for overseas development assistance (ODA), and so US- ATAARI cannot provide funding to these three economies.

OBJECTIVE

As described in project documents, “US-ATAARI’s goal is to strengthen the capacity of the APEC Secretariat and APEC’s foreign-assistance eligible developing member economies to advance regional economic integration (REI) and achieve the Bogor Goals of free and open trade and investment in the region.”2 Within this overarching objective, US-ATAARI has implemented activities, with varying levels of scope and intensity, within the six (6) workstreams as described in the M&E Plan, namely: (i) Green Growth; (ii) Trade and Investment Facilitation; (iii) Regulatory Reform; (iv) Human Security; (v) Women’s Economic Empowerment; and (vi) APEC Secretariat & Host Capacity Building.3 Each of these workstreams has its own specific objectives, defined by the various APEC Working Groups on the one hand, and USG partners on the other.

As the project is now in its final year of implementation, USAID commissioned this end-term evaluation to assess whether, and to what extent, the project has been successful in achieving its intended aims.

Given the wide range of interventions areas, the end-term evaluation was tasked with focusing on a narrower sub-set of activities that received consistent support over the life-of-project and/or were of special interest to the funders. Because of the intentionally narrowed focus, the findings here cannot be interpreted as applicable for other program areas that were not covered in the evaluation. The selected intervention areas covered by this evaluation are:

Under Trade and Investment Facilitation:

1. Supply Chain Connectivity

2. Digital Economy

Under Regulatory Reform:

3. Ease of Doing Business (EoDB)

4. Renewed APEC Agenda on Structural Reform (RAASR)

Women in the Economy:

5. This pillar encompasses a number of sub-categories, including: work supporting the Policy Partnership on Women and the Economy (PPWE); and work within Healthy Women, Healthy Economies (HWHE); Women in Science, Technology, Engineering, and Mathematics (STEM);

Women’s Entrepreneurship in APEC (WE-APEC); and Women in Transportation (WiT). As these sub-categories often involve the same working groups and counterparts, this pillar of activity is evaluated as a whole.

2 USAID Scope of Work from contract, pg. 5.

3 The Results Framework for the M&E Plan is provided in Annex 1.

APEC Secretariat Support:

6. This pillar is intended to strengthen the effectiveness and efficiency of the APEC Secretariat, based in Singapore, and provide assistance to Host Economies who Chair APEC on a rotating, voluntary basis. The evaluation focuses on direct assistance to the Secretariat.

This end-term evaluation thus seeks to examine whether, and to what extent, US-ATAARI met the project objectives for these six activity areas during the life of the project. Their specific objectives are described in the relevant sections below. In addition, the examination of economy-level changes affected through US-ATAARI work – using the Philippines as a case study – includes a discussion of several additional workstreams (namely inefficient fossil fuel subsidy reductions, host-year preparation, and agricultural biotechnology).

Activities covered by the evaluation are best considered within the context of the broader US-ATAARI portfolio. Across all workstreams, US-ATAARI has implemented a total of 133 events (including 109 workshops, with the remainder being policy forums and policy dialogues), in the process training 5,483 participants. The project also supported the production of 46 reports. The workstreams covered in this evaluation account for 27% of all events, 24% of all people trained, and 37% of all reports.4 Gender-related programs account for 42% of events and 52% of training participants in activities covered by this evaluation, but only 11% of all US-ATAARI events and 13% of participants. The contribution of each workstream to this US-ATAARI total is shown below:

Total US-

ATAARI

Trade and Investment Facilitation

Regulatory Reform

Women in the Economy

% of Total Covered by Evaluation

Events 133 12 9 15 27.1%

People Trained

5,483 238 400 697 24.3%

Reports 46 9 4 4 37.0%

Note: Trade and Investment Facilitation only includes Supply Chain Connectivity and Digital Economy; Regulatory reform includes only Ease of Doing Business and RAASR; Women in the Economy all sub-workstreams mentioned above.

4 The figures given in this section and the table exclude the work to support the APEC Secretariat, which is not disaggregated from host year activities. Outputs for that workstream are relatively small and thus the evaluation covers slightly more of the overall outputs then the percentages shown below.

METHODOLOGY

The evaluation was structured around four standard evaluation criteria:

● Relevance – the degree to which specific activities are consistent with the overall goals of USG and APEC; the stated strategy of the APEC Working Group or Sub-Fora; and the degree to which specific activities are consistent with the programming context;

● Effectiveness – the degree to which activities achieve their intended results;

● Efficiency – the use of resources, including financial, institutional and professional, in the conduct of activities to achieve specific results; and

● Sustainability – the extent to which program achievements will be taken up and practiced by member economies in the future without further APEC resources after the project is completed.

The evaluation used three data sources in its analysis:

● Desk review of available documents. These included annual work plans, the Activity M&E Plan, annual and quarterly progress reports, activity completion reports, and other reports drafted or commissioned by US-ATAARI.

● Analysis of M&E data. M&E Data from US-ATAARI’s M&E database includes data on the output and outcome indicators included in the Activity’s M&E Plan. This data was compiled to provide a life-of-project snapshot for the specific activity areas assessed in the evaluation and inform the assessment.

● Key informant interviews of stakeholders. Semi-structured interviews were conducted by telephone with 19 respondents from APEC member economies and U.S. government stakeholders (see Annex 2 for the list of respondents interviewed). In addition, 6 interviews were conducted in the Philippines involving 11 individuals with a view to exploring the range of US- ATAARI’s interventions in a single economy.

LIMITATIONS AND CHALLENGES ENCOUNTERED DURING THE

RESEARCH

Given the complexity of multilateral trade reforms and the nature of the engagement with APEC member economies through this project, it was not possible to conduct an impact evaluation of US-ATAARI that would identify the institutional, policy, and regulatory changes that were made that could be directly attributable to project activities and assess their value in economic terms. Indeed, these kinds of activities are rarely subjected to the most rigorous forms of evaluation exactly because of the technical challenges and high costs associated with policy reform interventions. These limitations are most constraining in our attempt to assess effectiveness and efficiency.

● Assessing effectiveness requires establishing the specific impacts that can be attributed to the program, and in most cases, the evidence for attributable impact is limited. This evaluation draws upon subjective assessments of participants and the formal reports produced by those involved in the project to characterize the magnitude and value of the contributions made by US-ATAARI, and these sources often create a compelling perception of what was achieved, but in every case these characterizations fall short of directly and objectively determining the program’s effect.

● Assessing efficiency requires combining effectiveness assessments with the level of cost and effort required to achieve those impacts. In the absence of quantified impact assessments, the assessment of efficiency is limited to subjective judgments of value-for-money made by the USG participants.

An additional limitation of the evaluation is the scope of the in-person interviews. Originally, the intention was to include a diverse set of stakeholders representing USG agencies, member economies, the APEC Secretariat, and US businesses. In the end, the list of interviewees was heavily weighted toward USG officials; even though all of these stakeholders were represented in at least one or more interviews, the survey was not sufficiently broad that the several private sector representatives or member economy representatives, for example, could be considered to reflect anything beyond their own individual perceptions. As a result, even when the perceptions across all surveyed individuals were consistent, which they usually were, the evaluation is limited in drawing conclusions from these individual interviews.

GENERAL FINDINGS

The general findings fall into two categories: Broad Observations about the performance of the US-ATAARI team and aggregations of workstream-specific assessments.

1. Despite a challenging management structure and broad ambit, the US-ATAARI team is almost universally seen as a professional, high-performing project. The US-ATAARI project is unusual in a number of ways, but perhaps most striking is the diversity of clients it serves within the USG. The team directly responds to managers both in USAID and the Department of State, but also works very closely with senior officials within the Office of the USTR and Department of Commerce (and other agencies in other workstreams). Further complexity in implementation comes from the clear expectation that the project will also work with multiple non-USG stakeholders in ways that almost make them clients, as well. Such non-USG clients include the APEC Secretariat and APEC host economy teams, as well as member economies that are voluntarily engaged in programs focused on their own reform agendas.5 During interviews that spanned several months, respondents might occasionally question the narrow specifics related to strategy or program design, but only very rarely faulted the team assembled by Nathan Associates.

Indeed, they were regularly lauded for the technical expertise both within the team and brought in by the team from outside, and for their flexibility and responsiveness to changing conditions.

The fact that these perceptions were shared across so many partners who might even, at times, have differing institutional interests, is impressive and noteworthy.

2. Relevance to USG. In general, almost all program activities appear to be highly relevant from the perspective of the USG client agencies. At some level, this finding is not surprising, as US-ATAARI’s implementation plan itself reflects the interests and priorities of those same USG agencies.

Programs implemented by US-ATAARI at the direction, and usually with the participation, of officials from the Department of State or the Office of the USTR, for example, would not have been initiated, developed and implemented unless they reflected agency-defined USG objectives.

3. Relevance to Regional Economic Integration. Most of the workstreams (and the activities implemented within them) have obvious and compelling connections to regional economic integration and US policy and private sector priorities. Both Trade and Investment and Regulatory Reform workstreams encompass sets of activities that reflect serious bottlenecks that not only impede regional integration but also disproportionately restrict US investment and trade.

5 APEC Chair rotates annually with that year’s chair referred to as the “Host.”

4. Relevance of Women in the Economy Programming. The relevance of gender-specific programs to APEC emanates from two sources. First, in 2011, APEC Leaders committed to enhancing collaborative work on women’s economic issues, reflected in the creation of the Policy Partnership on Women and the Economy. Second, Department of State officials with APEC-related responsibilities identified this workstream as a prominent US policy concern within APEC.

In this context, the commitment of resources to Women in the Economy program is highly relevant. In some cases, the actual program activities have an indirect connection, at best, to regional economic integration, but this is true of other inside-the-border economic issues. Those engaged in this workstream, both USG officials and private sector participants, assert that severe limitations on women’s participation in other member economies, in particular issues that reduce worker productivity and that restrict opportunities for women entrepreneurs, rise to the level of barriers for normal forms of investment and commerce for US firms. The evidence for this is unclear, but these program areas, which in the past have been controversial at times for challenging on local values, customs and practices, have clearly emerged as important trust-building work programs, as more and more member economies recognize the need to address the role of women within their societies.

5. Difficulty assessing Effectiveness. This evaluation has identified limited evidence that directly addresses the question of whether US-ATAARI activities were effective in accomplishing specific policy reform objectives, and as a result, in most cases the evaluative outcome is “uncertain.” The perception of effectiveness is fairly strong across all programs by those involved in the activities.6 To cite just two recent relevant statistics, 74% of workshop or training participants in 2017 reported that their capacity was enhanced as a result of attending a US-ATAARI event, and follow-up surveys found that 93% of participants in capacity-building activities last year reported applying trade and investment liberalization practices from the program in their work.7 Even at the APEC-wide level, when officials are brought together from multiple economies, survey data reflect not just the appreciation of the participants but also their perception that the substance can be put to use in their official capacity. But more compelling evidence of impact comes from the economy-specific programs, where follow-on events cater specifically to locally-relevant priorities.

6. Compounded difficulty assessing Efficiency. Survey respondents regularly praised the efficiency of program activities, providing subjective judgments as to the scope and the level of cost and effort required being highly consistent with the specific objectives. Unfortunately, independent objective evidence related to efficiency, linking attributable impacts to the efforts and funds expended is generally not available. Moreover, this evaluation was not tasked with undertaking a thorough review of the budget and a comparison of program scope and costs to those implemented by other firms in other contexts and possibly with other funders. In most situations, 6 Evidence of this subjective perception was found consistently both in the interviews conducted for this evaluation and in the M&E data collected after US-ATAARI events.

7 Annual Performance Management Progress Report, FY17 (October 1, 2016 - September 30, 2017), ATAARI-17-084, p. IX.

subjective perceptions such as those referenced here would have highly limited credibility, as often those engaging with the implementing partners see themselves as partly responsible for efficiency, as well (thus reducing the likelihood of critical and self-critical assessments). However, US- ATAARI is unusual because of the multiplicity of clients, often across multiple workstreams. This situation affords opportunity for agencies to observe not only their programs but also those undertaken for other agencies. Indeed, this situation creates a context in which some respondents might actually see a reason to criticize the design and implementation of programs in other workstreams.

7. Sustainability. The evaluation sought to assess the durability of program impacts, an important consideration as the current contract is approaching completion. Again, evidence is limited because even within a five-year program, many of the most important accomplishments have happened only recently and, consequently, there is no lengthy track record for anything to demonstrate achieved sustainability. Moreover, many of the workstreams include both short- and long-term objectives, so while the short-term accomplishments do not yet have a long track record, the long-term objectives may not even have been accomplished yet. Instead, in some cases, US-ATAARI programs aim to sow seeds for reform that may not emerge right away (and indeed may not ever). The issue of sustainability is dealt with under each workstream, but it appears that where economy-specific programs were implemented in support of local priorities, those successes are likely to survive. At the APEC-wide level, some broad and sustained efforts look more likely to survive and continue, exactly because of the lengthy engagement and broad support; such APEC-wide accomplishments that appear more sustainable include the Ease of Doing Business work associated with the World Bank’s Doing Business Indicators, and the Women’s Economic Empowerment work connected to the Women and the Economy Dashboard.

8. The APEC Secretariat and Host-Year Support. One of the pillars of the US-ATAARI project design included the goal of strengthening the capacity of the APEC Secretariat to fulfill its core responsibilities. According to USG officials interviewed for this evaluation, this objective was always a secondary consideration, and one that they viewed having declined over time. Secretariat officials interviewed for this evaluation expressed some frustration that the objective of Secretariat strengthening was not clearly communicated to them and noted that alignment between their priorities and the activities undertaken by US-ATAARI could have been stronger.

However, assistance that was provided is considered anecdotally to have been highly effective (with some work ongoing and too early to evaluate results).

Partly in the place of organizational development support to the Secretariat, US-ATAARI’s institutional engagement has evolved into demand-driven support to member economies in the lead-up to their host year responsibilities, which has actually become an important part of its portfolio. The perception within APEC of US-ATAARI as a professional and responsive technical support team has led to numerous requests for help from member economies as they approach their year to host APEC.8 This demand-driven workplan has, in turn, provided opportunities for US-ATAARI to ensure that issues that are relevant to USG and private sector interests are considered for the host economy agendas year after year. A number of USG officials highlighted the important work US-ATAARI has done in support of host economies and the development of their agendas, noting the increased attention to issues that figure prominently in USG priorities, as well.

8 Several interviews included comments that suggested that host economies might not even see the US-ATAARI team as connected in any formal way with the USG, making requests for help during host years more palatable. This stands in contrast to efforts by other economies to provide similar support. An online article posted at the Nikkei Asian Review on January 25, 2018, described the “competition” around providing security and logistical support and how it touches on the interests of assistance-providing economies: “There are concerns that PNG could seek Chinese help to run the summit and that Beijing would come to dominate the meetings. This has forced Australia, New Zealand and Indonesia to pour in aid to avoid having to deal with China's overbearing presence.”

FINDINGS BY WORKSTREAM

A. TRADE AND INVESTMENT FACILITATION

The Trade and Investment Facilitation (TIF) workstream has a number of different sub-workstreams, but this evaluation will focus specifically on two: Supply Chain Connectivity and the Digital Economy. These two are not unrelated to each other (or to other program areas in the TIF workstream), and many of the people interviewed who were active in one sub-workstream were also engaged or at least informed of activities in the other.

1. Supply Chain Connectivity

APEC Committee: Trade and Investment

Objectives for this Activity Area: This workstream focuses on eliminating impediments within supply chains linking production and trade, on the expectation that increased efficiency of commercial networks will drive economic growth, strengthen and deepen regional economic integration, and increase trade and investment. In 2010, APEC Leaders committed to an APEC-wide goal of improving supply chain performance (i.e., reduction of time, cost, and uncertainty of moving goods) by 10% by 2015 throughout the region. Starting in 2014, US-ATAARI has undertaken a significant work program in support of this objective, and APEC is now in the process of finalizing the second phase of its supply chain work, which will focus on reducing trade costs across supply chains and improving supply chain reliability in supporting the competitiveness of business in the Asia Pacific region.

US-ATAARI Inputs:

Number of Events9 (Workshops and Policy Forums/Dialogues Supported): 8

Number of Workshops: 2

9 The number of “events” reported here includes workshops. The term “workshop” is used to describe an event that has specific capacity-building objectives. US-ATAARI also supports and/or implements official APEC events, such as dialogues and meetings, but such events, unlike workshops, are not considered to have specific training objectives.

US-ATAARI Outputs:

Number of Reports Produced: 8

Number of People Trained: 182

US-ATAARI Program and Outcomes

US-ATAARI has supported the Office of the United States Trade Representative (USTR) since 2014 in the development and implementation of five APEC projects under the Supply Chain Connectivity Sub- Fund: (1) Pre-Arrival Processing (PAP); (2) Expedited Shipments (ES); (3) Advance Rulings (AR); (4) Electronic Payments (EP); and (5) Release of Goods (RG).

Activities in each of these areas have targeted bottlenecks common in the region or in specific economies that impede investment and trade. The multi-year effort on supply chain connectivity began with the launch of the APEC Alliance for Supply Chain Connectivity (A2C2) in 2014 in Beijing coupled with the finalization of a sub-fund in APEC focused on supply chain capacity building efforts related to the WTO Trade Facilitation Agreement. US-ATAARI implemented activities at two levels , supporting regional dialogue between public and privates sector actors while also leading capacity building programs in several APEC economies.

Important accomplishments supported by US-ATAARI activities include:

● The initial workshop and the priority action plan in 2014;

● The delivery of technical reports describing priority challenges, including a series of technical diagnostic reports of “chokepoints” in the global supply chain and APEC Concept Notes;

● The establishment and support of the APEC Alliance for Supply Chain Connectivity (A2C2), a forum for representatives from the public and private sector to review and contribute to APEC initiatives;

● A series of economy-specific activities with:

○ Chile (AR, EP, RG)

○ Malaysia (PAP, AR)

○ Peru (PAP)

○ Philippines (ES)

○ Vietnam (PAP, ES, AR)

Interviews with USG officials involved in these programs indicated that this work with specific APEC economies might be one of the more effective activities undertaken by US-ATAARI in this workstream, and indeed the M&E report for 2017 reported that the Government of Malaysia had undertaken five of the ten reforms in supply chain connectivity recommended in a technical report from US-ATAARI.10

10 Annual Performance Management Progress Report, FY17 (October 1, 2016 - September 30, 2017), ATAARI-17-084, p. X.

But overall, the picture is mixed. The target set by APEC Leaders of a 10% improvement in supply chain connectivity was not met. According to the most recent APEC assessment, released in November 2016, but covering data through 2015 only, “The performance assessment using external indicators makes use of relevant benchmarks from the 2016 Logistics Performance Index (LPI), the 2015 Doing Business (DB) report and the 2014 Global Enabling Trade Report (GETR). The overall LPI score shows a small improvement, still far from the 10 percent target. DB scores show very strong progress in terms of time, exceeding the pro-rata target for both imports and exports: trade transactions were completed 10 percent faster for imports and 12 percent for exports. DB cost figures (adjusted for inflation) for imports and exports also show good progress, achieving the pro-rata target in 2014. It should be noted that nominal costs still show an increase of 7 percent for imports and 2 percent for exports.11

Of course, US-ATAARI activities alone were only part of a broader APEC set of efforts, and so are not directly responsible for the record against this specific target. But there is some evidence that the issues addressed through US-ATAARI activities may have performed better than others. After CTI identified eight potential chokepoints impeding supply chain connectivity, US-ATAARI worked closely with APEC lead economies to produce 3 of 8 diagnostic reports addressing Chokepoint 4 (Inefficient Clearance of Goods at Customs, Lack of Coordination among Border Agencies), Chokepoint 6 (Multimodal Transport Capabilities and Connectivity); and Chokepoint 7 (Variations in Cross-Border Regulations and Standards for Movements of Goods, Services, and Business Travelers). One of those three (inefficient cross-border clearance procedures) received the greatest programming attention, accounting for almost one-fourth of all APEC activities in this workstream. Of the three chokepoints receiving the least attention, only one (regulations and standards) was within US-ATAARI’s work plan.12

Relevance – High

The program has had a substantial and sustained level of activity since 2014, and the overall strategy within the workstream has been generated through the identification of important issues by the Office of the USTR, the discussion of these issues at the APEC-wide level, and economy-specific follow up on a voluntary basis with interested member economies. This approach has delivered a highly relevant program that both reflects US priorities and responds to expressed interests among member economies.

At the APEC-wide level, leadership by the Office of the USTR has focused attention on key obstacles to regional economic integration that, if eliminated, would open significant new opportunities to US businesses. Unclear and inconsistent regulations and practices obviously impede investment and trade for all member economies, but for many US firms, these disadvantages are exacerbated in parts of the APEC region by distance, language, and inexperience; consequently, it seems plausible that efforts to enhance

11 APEC Supply Chain Connectivity Framework Action Plan 2010–2015: Final Assessment, APEC Policy Support Unit, November 2016, p. iii.

12 Ibid.

consistency and transparency could generate new commercial networks. Multiple interviews with USG officials highlighted the strong and consistent relevance of US-ATAARI programs to US policy objectives.

At the economy-specific level, voluntary opt-in by five member economies reflect the relevance of this work to their priorities. In each case, the workshops and trainings implemented by US-ATAARI reflect the issues within the workstream that have been identified for action. For example, support for enhancing Pre-Arrival Processing was provided to officials in Malaysia, Peru and Viet Nam, but not for the member economies (including Chile and the Philippines) that had not signaled a desire to work on these issues.

Effectiveness - Uncertain magnitude, but positive contribution

The fact that APEC has not achieved its overall objective of a 10% improvement in supply chain efficiency is not, in itself, evidence of ineffective programming undertaken by US-ATAARI, but this statement reflects the challenge of assessing effectiveness in such contexts. Interviews with USG officials indicated that both within the State Department and the Office of the USTR, the work on supply chain connectivity in the Philippines and Viet Nam were amongst the most successful efforts implemented by US-ATAARI.

Another USG official pointed to the recent work aimed at promoting full implementation of the WTO Trade Facilitation agreement by APEC member economies. US-ATAARI efforts targeted capacity-building to raise their implementation level. One example cited was the work with Peru in the area of pre-arrival processing, and the idea that the effects of this work could be seen in third-party data sources, such as the World Bank Logistics Performance Index (LPI). This official noted that the work may be too recent to see in the data, and indeed the most recent results in the LPI are from 2016, and so would be unlikely to reflect changes made based on programs in 2016-2017 – Peru’s LPI ranking was slightly better 2016

(69) was actually worse than it was in 2014 (71), but it would be unwise to attribute the overall improvement to US-ATAARI activities (especially given that the score includes modest declines in Peru’s performance on three sub-indicators, including International Shipments and Timeliness).13

The overall goals of a workstream far exceed the ambit of one project, and ultimate progress depends on the actions of governments to implement reforms. The absence of a counterfactual outcome (i.e., what would have happened without the US-ATAARI project) leaves open the possibility that actual supply chain efficiency might have worsened (or improved by more) had this project not been implemented.

That said, a review of evidence available suggests that the program delivered positive results, even if those results cannot be quantitatively measured or valued. The early APEC-level work in this workstream identified priority problems and provided technical information that was available to economies actively engaging in reforms and that will remain a resource as additional economies decide to engage; the economy-specific work clearly supported reform efforts prioritized by those economies. Data from US-

13 https://lpi.worldbank.org/international/global

ATAARI M&E follow-up surveys report that technical support provided in local economy settings was then applied by large majorities of respondents.14

Efficiency – Uncertain magnitude, but subjectively positive

Given the challenge in assessing effectiveness, the complementary assessment of efficiency raises additional challenges in determining a value-for-money proposition. Interviews with related officials from USTR and the Department of State provide consistent positive subjective assessments of the design and implementation of activities in this workstream. Additional interviews with representatives of businesses engaged in these activities were also consistently positive. This evaluation has found no evidence of weak or inconsistent efficiency, such as badly-designed or -implemented programs, in this workstream.

Sustainability – High

Although there is no formal way of guaranteeing sustainability beyond the life of a project, there are several notable characteristics of this workstream that suggest that progress to date will be sustained and that investments towards long-term goals will eventually deliver additional impacts. The workstream itself is a central part of APEC’s core mission and its embrace of independent and transparent indicators establishes a higher level of commitment and accountability than seen in some other workstreams. Further, the technical work generated by US-ATAARI is universally seen as high-quality and relevant to the specific challenges faced by member economies. The fact that reforms undertaken by member economies to date have been voluntary and focused on their own priorities suggests that the risk of reversal, while not zero, is considerably less than would be expected where reforms have been undertaken in contexts of external pressure or funding.

2. Digital Economy

APEC Working Groups/Committees: CTI, the Electronic Commerce Steering Group (ECSG), the Small and Medium Enterprises Working Group (SMEWG) and the Telecommunications and Infrastructure Working Group (TELWG).

Objectives for this Activity Area: The Digital Economy workstream addresses efforts to enhance the efficiency of digital trade and to expand access to the digital economy within and across APEC economies.

This work includes the regulations and institutions related to the digital trade, and access to digital

14 Due to the volume of activities and M&E data, results from specific activities cannot be reported in detail here, but data generated by surveys of program participants can be found in the regular M&E reports generated by US-ATAARI. These surveys include only those who choose to respond and so may include selection bias (i.e., those who respond may not be representative of all participants) and data from self-reporting of post-program activities is considerably less compelling than would be independent observation of actual actions undertaken by agencies.

commerce, as well as the impact of the growth of the digital economy on various sectors of the regional economy, especially on SMEs.

US-ATAARI Inputs:

Number of Events (Workshops and Policy Forums/Dialogues Supported): 4

Number of Workshops: 1

US-ATAARI Outputs:

Number of Reports: 1

Number of People Trained: 56

US-ATAARI Program and Outcomes

US-ATAARI has been active in this workstream since 2014, supporting priorities identified by the Department of State, the USTR and USAID at various times and in ways that have evolved somewhat over time. The multi-year strategy began with an initial effort focused on intellectual property rights (IPR) aimed at protecting creative content, an issue traditionally seen as a priority for the USG given the prominent role of the American private sector in generating new content deserving IPR protections.15

In 2016, the focus of the workstream shifted and broadened somewhat, reflected in the adoption of language focused on the “digital economy.” The USG remained the primary initial driver in this sphere, but the shift from IPR to digital trade undoubtedly increased interest among other member economies and incorporated issues that did not necessarily pit the interests of one economy against others. The term “digital trade” refers to domestic and regional commerce in which the internet and internet-based technologies are involved, a segment of all economies that has experienced rapid growth in recent years.

Increasingly, the internet plays a particularly significant role in ordering, producing, or delivering goods and services, and the role of policies and regulation that facilitate or impede the adoption of related technologies and practices are rising as prominent determinants of the efficiency of local and regional markets. US-ATAARI activities in this workstream have been important for identifying new challenges that may impede regional trade and bringing together member economies to discuss new policy and regulatory practices that might enhance cross-border e-commerce. To date, US-ATAAR activities have focused on APEC-level meetings and workshops, laying the groundwork for economy-specific programs in the future.

Important program activities in this workstream supported by US-ATAARI include:

● Engagement in an initial workshop focused on IPR in 2014;

15 The first Digital Economy activities fell under the rubric “Protecting Creative Content in the Digital Environment” and were described in the “Second Work Plan, January 1 – December 31, 2014, US-ATAARI, pp. 7-8.

● In 2016, in collaboration with a USAID-funded project working with ASEAN, produced a regional study (“Key Issues: Promoting the Digital Economy and Digital Trade in APEC and ASEAN”) that focused on relevant capacity-building opportunities within APEC;

● Further work with State and the USTR led to two trade policy dialogues that discussed digital trade facilitation and proposed a new agenda for action on digital trade;

● In 2017, supported both an additional trade policy dialogue (APEC Trade Policy Dialogue on Identifying Building Blocks for Facilitating Digital Trade) and a subsequent workshop focused on the relevance of regulatory barriers affecting digital trade for small and medium enterprises (SMEs).

Relevance – High

Activities in this workstream started in 2014, and the relevance of IPR work was very high from a USG perspective. Arguably, the issue had considerably less relevance for many other member economies and indeed might be seen as controversial in some economies where IPR standards were low and interest in reform to protect local content was also low. Arguably, IPR protections should be important in other advanced economies and in large and rapidly growing economies whose potential for creative content is high, but in reality conflicting local interests and trade-offs between short- and long-term economic gains have made IPR a sensitive subject in multilateral fora.

The shift towards a new focus on the digital economy, which can include IPR protections, effectively broadened the relevance among other APEC member economies even while maintaining a focus on issues of considerable and disproportionate concern to US firms and the US economy. The pattern of US-ATAARI work has always had high relevance for US priorities and, since 2016, has increasingly encompassed concerns that are relevant to other member economies.

To date, all activities in this workstream have taken place at the APEC-wide level. Different activities have supported priorities seen by various USG clients, including, the Department of State, the Office of the USTR, and USAID. In each case, the specific activities have clear relevance to the respective agency.

These activities have also included participation by US private sector representatives, who characterize the programs as directed at issues of concern. Of course, the relevance of the programs to participating firms also reflects a process of self-selection (i.e., those firms that find the program valuable and relevant consider participating, while those that find activities less compelling do not) and so cannot be interpreted as evidence of relevance to the many firms that do not participate. It is possible that programs have broader relevance to non-participating firms within an industry or perhaps even a sector, but this would depend on the nature of the programs.

This workstream has not yet advanced to economy-specific workshops or trainings. Based on evidence from participants in the trade dialogues, this lack of direct follow-up reflects more the recentness of the workstream than the relevance (or lack thereof) perceived by other member economies. That said, in contrast to the pattern of work in the supply chain connectivity workstream, in which economy-level activities started almost immediately upon conclusion of the first APEC-wide events, the lack of voluntary requests for economy-level support may be an early sign that the perception of relevance among other economies has not yet been demonstrated.

Effectiveness – Uncertain

Interested parties engaged in this workstream, including both USG officials and representatives of interested private sector firms, acknowledge that it is too early to see concrete evidence of effectiveness from this workstream. The newness of the issue area in general and in many APEC economies specifically has led to a workplan that has focused on information generation and problem identification. These same participants emphasize the importance of investing time and effort and program resources into this early-stage work, not out of any certainty of future impact, but rather on the basis of the importance of the issue to the US and the regional economies and the unusual opportunity to help shape the discussion in its early stages.

As one USG official explained, US-ATAARI activities on the Digital Economy need to be seen as contributing to the long-term process of educating policy-makers about the importance of digital trade to the development process today and the need to have policies in place that foster growth in and provide access for SMEs to the digital economy. For now, the activities need to focus on collecting and disseminating information on best practices in the hope of avoiding new impediments to digital trade.

One private sector representative active in this workstream suggested that the impacts were real and tangible, even if they could not be measured or directly attributed to the US-ATAARI workstream. The Philippines became the first economy to adopt a “cloud-first policy,” an action that would not have happened without US-ATAARI activities in this space that raised awareness of the importance of the issue and provided technical training that increased the capacity of the government there to institute the change.

Malaysia has since adopted a similar framework, and Thailand and Indonesia are moving in this direction, according to the interview respondent. Even as the lack of attribution is clear, the respondent suggested that people focusing on impact could easily overlook the value of conferences bringing people together and talking about important issues.

Feedback from the recent workshop focusing on SMEs and the Digital Economy found that almost one in five participants found little to no connection to their work.16 While this might reflect a problem of relevance, a more likely interpretation is that the workstream is so new that activities may be limited in their effectiveness through the challenge of identifying the appropriate participants. Moreover, if the large majority of participants find some relevance, there remains some possibility that specific economies may request follow on work, even if that has not yet happened.

16 ATAARI-17-084, op cit, p. 23.

Efficiency – Uncertain, but room for improvement

This workstream appears to have room to improve in its identification of issues and participants. This finding is not surprising, given that the current focus on the digital economy is new for APEC, new for US- ATAARI, and particularly new for many other member economies. Early investments at the APEC-wide level may have been successful on their own merits if the focus is sustained and regional economy-specific reforms take place, either within APEC and ASEAN or independently. However, at this point, such efficiency is unclear and can only be enhanced if future work in this issue area takes on a better focus as a result of these early investments.

Sustainability – Unclear

Many of the elements that suggest high sustainability in the supply chain connectivity workstream are not present here. The APEC-wide level efforts remain fairly broad and reflect a solid starting point but not concrete results whose sustainability could be measured. The initial trade policy dialogues have added valuable information into the regional discourse on the digital economy, and that is a permanent contribution, but of course its value may diminish fairly quickly over time as technologies and their role in local and regional economies evolve. The absence of evidence of concrete economy-specific policy changes leave open the question of what will be sustained, if anything.

The challenges described above in terms of effectiveness, efficiency and sustainability need not be interpreted as a harsh criticism of early efforts in this workstream - indeed, the early efforts appear to be consistent with a well-conceived program in its early stages.