3._Sustainability_Guide_docx.pdf

PROGRAM MANAGEMENT

GUIDES

SUSTAINABILITY GUIDE

The views expressed in this publication do not necessarily reflect those of the United States Department of State Bureau of International Narcotics and Law Enforcement Affairs or the United States Government.

Table of Contents I. Introduction II. Background III. Sustainability IV. Sustainability Planning V. Written Sustainability Plan VI. Implementation VII. Monitoring and Evaluation VIII. Closeout APPENDIX A – Sustainability Analysis Questions ................................................................... A-1 APPENDIX B – Sustainability Plan Checklist ........................................................................... B-1 APPENDIX C – Program Performance Metric Examples ......................................................... C-1 APPENDIX D – INL Strategic Pillars ........................................................................................ D-1 APPENDIX E – INL Quarterly Progress Report Template......................................................... E-1 APPENDIX F – INL Functional Bureau Strategy & Program Management Guides .................. F-1

SUSTAINABILITY GUIDE | 1

I. INTRODUCTION

The mission of the Bureau of International Narcotics and Law Enforcement Affairs (INL) is to minimize the impact of transnational crime and illegal drugs on the United States, its citizens, and partner nations by providing effective foreign assistance and fostering greater bilateral and multilateral law enforcement and rule of law cooperation. INL leads the Department of State (DoS) anticrime and counternarcotics efforts, including through assistance programs which aid partner nations in building their capacities to extend the reach of justice under the rule of law, considering respect for human rights and gender equality, and to deny safe haven to criminals who would otherwise operate with near impunity. INL programs fight illicit drug production and trafficking, corruption, cybercrime, intellectual property rights fraud, and activities funding crime and using the proceeds of crime.

This Sustainability Guidance seeks to ensure that gains in anticrime/drug measures and partner nation capacities remain intact at the conclusion of INL programs and U.S. government funding. Furthermore, this guidance reinforces the importance of improving the criminal justice and rule of law capabilities of our partner nations.

By taking sound project management measures, which include an emphasis on sustainability, staff domestically and abroad can implement programs with established benchmarks and finite end states which result in project close-out and the transfer of ownership to the host government. The guide is a concise, yet flexible blueprint for considering and incorporating sustainability throughout a project lifecycle. Through enhanced sustainability, INL will ultimately accomplish its vision of advancing peace, security, and justice through international cooperation and assistance.

II. BACKGROUND

Sustainability is one of the five “foreign assistance effectiveness principles” outlined in the Quadrennial Diplomacy and Development Review. “Effective” foreign assistance produces lasting, positive outcomes and contributes to achieving U.S. national security and foreign policy goals. In INL’s case, those goals focus on combatting international crime and/or drugs.

However, program sustainability has proven to be a major challenge in many developing nations for a variety of reasons, some of which can be addressed at the program design stage, while others may be addressed during implementation

2 SUSTAINABILITY GUIDE

through monitoring. Therefore, it is important that factors which affect sustainability are articulated and incorporated as far in advance as possible and evaluated across the program life cycle. This guide demonstrates how to do so.

III. SUSTAINABILITY DEFINED

Sustainability, as defined within this guidance, is the ability of INL’s host country partners and beneficiaries to take complete responsibility for the foreign assistance programming administered by INL, and maintain or improve program outcomes and impacts beyond the life of the program and U.S. government funding. In a broader context, several terms are used when referring to sustainability, which include: continuation, maintenance, integration, and incorporation, all of which refer to the ability of program results to be perpetuated.

While synonymous with a program’s end state, sustainability is one component of a project life cycle that includes planning, implementation, monitoring and evaluation. A key element of program sustainability is institutionalization or routinization (i.e., the process by which projects are incorporated into existing structured processes and systems.). When activities within existing INL programs and projects are sustained by the host government, (i.e., courses emphasizing lawful, effective investigative techniques, etc.) they are no longer INL activities or projects and are instead part of the host nation’s standard practice.

IV. SUSTAINABILITY PLANNING

The most substantial impact on sustainability can be made during the program design phase. Letters of Agreement (LOAs), and their subsequent amendments, with host nations and international organizations are paramount during this phase and all those which ensue. LOAs are instruments which document the assistance partnership between the U.S. Government and the government of the partner nation or international organization. Further, LOAs serve as agreements that establish and support projects designed to enhance the government of the partner nation’s capability to provide more professional, accountable, and responsive criminal justice and rule of law institutions and services. LOAs articulate shared performance goals which support INL’s vision of: Advancing Peace, Security, Governance, and Justice through International Cooperation, Partnerships, and Assistance. They also identify performance measures which allow both parties to

SUSTAINABILITY GUIDE | 3

see progress towards achievement of the goal. LOAs not only serve as a framework for INL programming, but more importantly, they document commitment between the U.S. Government and partner nations, which includes program goals and monitoring and evaluation processes that all aid in building host nation sustainment.

Program inputs, outputs, stakeholders, resources and outcomes should be developed for LOAs and all successive programming mechanisms with the mindset of fostering sustainable benefits as early in the process as possible to increase the likelihood of achieving sustainability of the program.

INL program offices must work with host country partners and interlocutors to identify an appropriate level of sustainability which is considered essential to the staying power of the program and a plan which indicates how they intend to meet these goals and objectives. Sustainability planning for INL programs requires flexibility given the breadth of INL programs and the varying factors that determine which benefits and results will continue beyond the life of the program.

However, the following general steps should be utilized during program planning to the greatest extent possible:

1. Specify which results should be sustained after the life of the program.

Begin by clarifying the program goals, objectives, and desired outcomes.

Next, program offices should determine which benefits do not need to continue (e.g., funding), which benefits are critical and will have the support to continue, and which may not be sustainable by the end of the program.

As part of the program identification and design process, INL program officers can group program components into one of the following categories:

Fully Sustainable All results may be sustainable due to significant buy-in from the partner country, minimal recurrent costs, and other factors.

Partially Sustainable Certain results may be sustainable and others may not.

Not Sustainable None of the program’s results may be sustainable.1

1 In cases where there are concerns about the program’s sustainability, the program officers are expected to detail those concerns as well as the reason(s) why programming is nonetheless recommended (if applicable), via a decision memo to the responsible Deputy Assistant Secretary before any program activity is initiated.

4 SUSTAINABILITY GUIDE

2. Choose an appropriate implementing partner, given the type of foreign assistance being provided. Ensure implementing partners have the subject matter expertise and capabilities for the training or benefit being provided.

3. Limit recurring costs. Providing high-tech equipment that is expensive to maintain or building expansive facilities with high operational and overhead costs increases the risks to the sustainability of a program. Consider a range of alternatives to meet the need and factor the longer-term ownership cost into decision making.

4. Provide tailored assistance to the country. It is important to study the feasibility of particular programs in the host country before undertaking them. What worked well in one country may not work as well in another for reasons of culture, security, infrastructure, literacy rates, or other factors.

5. Be inclusive. In addition to furthering stated U.S. policy goals (such as those articulated in the National Action Plan on Women, Peace and security), inclusiveness has other benefits tied to sustainability. For example, the greater the range of stakeholders engaged in the program activity (e.g., ensuring women and minorities are participants, promoting government-community dialogue on criminal justice issues, improving the public’s access to information), the greater the community of advocates for the permanent, positive change the program activity is designed to facilitate.

6. Conduct a sustainability analysis. Perform an analysis2 to identify and analyze the presence or absence of factors that are likely to impact, either positively or negatively, the benefits INL wishes to sustain. A recent study3 identified the following five factors which consistently promote and lead to program sustainability:

I. The program or projects could be modified over time;

II. A “champion” is present within the host country government who pushes hard for institutionalization;

III. The program or projects are a good fit with the organization’s (i.e., the beneficiary’s) mission and vision;

IV. The program provides clear benefits to host nation partners; and

2 See Appendix A for examples of questions to consider when conducting a sustainability analysis.

3 RAND Corporation Assessment of Program Sustainability in Three U.S. Department of Justice, Bureau of Justice Assistance Criminal Justice Domains - 2014

SUSTAINABILITY GUIDE | 5

V. Stakeholders in other organizations provide support.

Sustainability analyses should be documented in program planning documents, performance management plans, and program progress reviews.

7. Organize the program to be implemented, monitored, and evaluated for sustainability and allocate resources for these purposes. Staff should work with INL/RM/PA&E and INL implementing partners to craft indicators and performance measures to gauge the status or likelihood of sustainability at regular intervals throughout the life of the project(s). Project and sustainability goals and objectives must be monitored and quantifiably measured to identify any variances so that progressive corrective action can be taken. INL/RM/PA&E can also aid INL program offices in conducting evaluations which can determine the sustainability and effectiveness of INL programs. See Appendix C for examples of INL program performance measures.

8. Document the sustainability plan. Develop a written document that includes each of the elements of sustainability, if practical, and establishes an action plan for transitioning the program to the host nation.

V. WRITTEN SUSTAINABILITY PLAN

Having a documented sustainability plan intact prior to the start of a program is vital in attaining and maintaining sustainable outcomes. Just as planning for sustainability is not something that should be left until program implementation or closeout, the written sustainability plan should also be completed during the planning phase. The written plan helps INL staff work through the key steps in planning, implementing, and monitoring for the sustainability of INL programs.

The sustainability plan is important as it helps in sustainability planning on an ongoing basis, and should be referenced and revised as necessary throughout the program lifecycle.

Following the completion of the sustainability planning phase, creating a written sustainability plan for all programs is strongly encouraged. A documented sustainability plan is vital in ensuring the continuity of care for programs within INL and the ability to withstand staff turnover and resource reallocation over time.

While there is no single method of anticipating and addressing all sustainability challenges, a written sustainability plan will provide a road map to guide INL, host

6 SUSTAINABILITY GUIDE

nation governments, and experts during work on sustainability efforts. A written plan can also strengthen host nation and interlocutor buy-in and understanding of the efforts needed to keep the program results intact and it can serve as a guide to support the ongoing management of the program throughout the implementation and close-out phases. A sound sustainability plan can help identify what resources are necessary to sustain the program, encourage the development of partnerships and support collaboration, and help define progress and the necessary steps needed to ensure long-term success after U.S. government funding ends.

To help ensure the impacts of our programs are sustainable beyond their periods of performance, the written sustainability plan4 should include the following elements, at a minimum:

1. Program goals and outcomes;

2. Stakeholders;

3. The specific impacts or results that should be sustained;

4. The resources needed to sustain the results;

5. Sustainability indicators and performance measures as they relate to the program and project(s); and

6. The capacity needed for host nation ownership and the conditions that are necessary for a transition to the host nation.

Sustainability plans should be valid for no more than two years5. For programs slightly longer than two years the plan should be revised accordingly to reflect current conditions, dependencies, and resources as they relate to sustainability. For long-term programs expected to last five years or more, sustainability plans should be written for each phase of the program. Program phases can be identified based on several factors, which include, but is not limited to time period, deliverable(s), or INL Strategic Pillar6.

4See Appendix B for examples of items to consider and actions to take in documenting a sustainability plan.

5 Similar to strategies, sustainability plans must remain current to be valid given the fluid fiscal climate and global landscape.

6 INL’s Strategic Pillars as of November 2012 are: Anticorruption; Counternarcotics; Corrections; Justice;

International Cooperation; Law Enforcement; and Organized Crime/Terrorism. See Appendix D for pillar definitions.

SUSTAINABILITY GUIDE | 7

VI. IMPLEMENTATION

Program implementation, i.e., the period of performance, typically presents the greatest challenges to sustainability. During this phase, implementers execute activities and objectives, realizing that when put into practice, they are often faced with difficulties. Implementers and project managers must make tough delegation decisions, determining when the host nation is prepared to assume responsibility for program activities. Further, while true success is evidenced far beyond the end of the period after U.S. government funding and presence have subsided, implementers and project managers must implement for sustainability.

Implementation that produces sustainable results requires increased attention to local capacity building and financial planning, which includes monitoring recurrent costs and introducing mechanisms to augment and eventually replace U.S.

government funding.

To strengthen the focus on local capacity building and financial planning during the implementation phase, the following actions are recommended:

1. Build and maintain consensus among the implementation team (INL, interlocutors, etc.) and key stakeholders (host nation government) for the sustainability plan. Gain agreement on goals, objectives, and strategies by developing the sustainability plan with participation from other agencies and organizations. Spell out clearly to the host government (e.g., through a letter of agreement or memorandum of understanding) the host nation’s required contributions to program implementation, the expected outputs and outcomes, how/when progress toward achieving these outputs/outcomes will be measured, and INL expectations following program implementation.

2. Create incentives and disincentives that compel key parties to take actions needed for sustainability. Sustainability requires a forward-looking perspective, which often conflicts with daily management. Appropriate incentives may persuade others to focus on sustainability.

3. Develop the capacity to implement the sustainability plan. Responsibility must be transferred from INL and their implementers to the host nation government. Capacity-building should be incorporated into all activities.

4. Revisit, monitor and report on sustainability indicators and performance measures. Ensure progress is made on the sustainability plan through quarterly and/or semi-annual progress reviews that revisit sustainability assumptions and indicators and review the external environment.

8 SUSTAINABILITY GUIDE

Progress reviews should assess: the status of achieving agreed performance outputs and outcomes; obstacles thereto and how to overcome them; the financial picture, including obligation and liquidation rates, and the status of future appropriations (as appropriate); any changes to the host nation environment, specifically those related to leadership of relevant host nation institutions, political will to support the program, and financial ability to support current and future resource requirements related to the program. If/as necessary, program officers should adjust program implementation to reflect concerns that arise in the course of progress reviews. Where these concerns cannot be sufficiently addressed through adjustments, program officers/post should consider program termination.

VII. MONITORING AND EVALUATION

Monitoring and evaluation activities track progress towards program and project goals and objectives. If the program simply declares outputs or an end of project status as its goals and objectives, monitoring and evaluation will simply be designed to measure progress towards those objectives. If sustainability is a stated program objective, monitoring and reporting on progress towards sustainability is more likely to occur.

Program managers should document monitoring efforts along with performance reporting to include record meeting notes with host nations that discuss performance of LOA projects. International Organization’s performance reporting should be reviewed for progress and sustainability. Use of the established Interagency Agreement (IAA) quarterly reporting template (see Appendix E) and indicators and performance measures established in IAA work plans ensures interagency implementers analyze and report on sustainability on a quarterly basis.

Besides reporting progress towards outcomes, the quarterly report should state if the project is on track to have demonstrated continued impact beyond the life of the program. Also, a mid-term evaluation should be conducted for large programs to analyze program progress to date, including sustainability, and recommend changes or redirection of the program based on current progress. Contracts and grants contain their own reporting requirements that enable INL to track progress towards the goals and objectives as stated in those awards.

SUSTAINABILITY GUIDE | 9

VIII. CLOSEOUT

During program, project, or phase closeout, final progress reports should be reviewed and analyzed in relation to the final iteration of the sustainability checklist or written sustainability plan, if applicable, and note any deviations from the intended sustainability goals or objectives. If there is no checklist or written sustainability plan, INL staff should work with their implementers to determine and document if there is or will be continued impact beyond the life of the program. If transition to the host nation or sustained results is inopportune, staff should decide if a foundation for sustainability is in place and what will be needed to ultimately achieve sustainability.

INL staff should also document lessons learned to log valuable information to use for future programs and facilitate information sharing within the bureau. Further, a summative evaluation can be conducted at the end of the program to determine if the intended outcomes and impacts, sustainability or otherwise, were achieved.

SUSTAINABILITY GUIDE | A-1

APPENDIX A – SUSTAINABILITY ANALYSIS QUESTIONS

This appendix provides examples of questions7 for INL program offices to consider when conducting sustainability analyses. The questions are grouped into a number of project design dimensions, with the understanding that not all questions are relevant and others can be added.

1. Is there a demonstrable local demand and ownership, where a broad segment of the community has a stake currently or potentially in ensuring that the activity or service continues after the project ends?

2. What host nation capability gap(s) exist; why; what is the impact of these gaps on broader USG national security and policy goals in the country/region, particularly those articulated in INL’s Functional Bureau Strategy and the Integrated Country Strategy; what is the impact of the gaps on the host nation’s goals and priorities?

3. How will INL work to build up the skills and capacity of local stakeholders whose involvement will be critical for maintaining development gains after the project ends?

4. How can INL nurture effective institutions – governmental, civil society, and private sector – to analyze, implement, and evaluate activities in relevant criminal justice and rule of law areas?

5. How will INL ensure that relevant activities or services are gradually tied to sustainable financing models, either through private-sector participation or through sustainable, publicly-managed arrangements?

6. What are the financial costs of the project, its recurrent costs, and its maintenance capability and costs (if applicable) and will future revenues be adequate?

7. How can INL Functional Bureau Strategy objectives related to local capacity development and use of host country systems help to achieve sustainability?

8. If the project is in support of the host country government, has the project determined if this activity is represented in that government’s short term or long term planning process? Does the project support or contribute to the achievement of a partner government’s strategy? If it is a local civil society or private sector organization, has the activity been reflected in the organization’s planning documents and proposals to other donors?

7 Source: USAID Project Design Sustainability Analysis Tool

SUSTAINABILITY GUIDE | B-1

APPENDIX B - Sustainability Plan Checklist The following steps are actions one should conduct in their sustainability efforts -from planning, to the documentation of a written sustainability plan, and monitoring and evaluation. The items are grouped into several phases, with the understanding that every item will not be relevant to each program. Given this fact, no single standard of program sustainability is universally appropriate.

Rather, this checklist is intended to encourage mindfulness of sustainability and present considerations of sustainability during the design process.

• Establish an informal sustainability group comprised of 3-4 staff with project management experience, knowledge of INL key issues, and knowledge of or experience working with the host nation government.

Bringing in other people, both internal and external, provides knowledge and experience and benefits sustainability planning in several ways which include: technical, political, country, and stakeholder perspectives.

• Draft the program goals and objectives. This process helps clarify the desired changes the program aims to achieve.

• Draft the results (outcomes) the program aims to achieve. These are the key benefits that should continue after funding ends.

• Identify the types of activities that will lead to the desired results once the program comes to an end. The identification of specific activities will help refine objectives and determine what is feasible given program resources.

• Identify the benefits of your program and the stakeholders who benefit the most from it. The institutional capacities INL seeks to instill and those who will directly benefit should be clarified and engaged early in the process.

• Work with the sustainability group to set short- and long-term goals for sustainability. Intermediate and long-range goals aimed at expanding the institutional capacity of partner nations are paramount in measuring the program results for sustainability.

• Assess which activities are most likely to support both short- and long-term goals. All activities may not be sustainable; however, those that are needed to achieve the program’s desired outcomes should be.

• Assess the partner nation’s law enforcement and rule of law capabilities as measured against each of INL’s 7 Strategic Pillars. A review which examines INL funds invested and the host government’s standing in each

B-2 SUSTAINABILITY GUIDE

pillar will aid in helping INL identify when and where to reallocate funds or cease support. While conducted during the pre-program phase, the pillar assessment can also be used after implementation to help determine if results are or have been sustained. Such reviews will not only answer the question of “Are these results sustainable?”, they can also aid the INL Front Office and program offices identify which programs have yielded the greatest return on investment from a sustainability perspective.

• Determine and prioritize which activities should be continued based on INL’s needs and desired results. Activities which have a direct impact on sustaining the program’s desired results should be prioritized in the event of resource constraints which may hinder the implementation of all planned activities.

• Consult with post and partner agencies to gather their input on program priorities. The interests and concerns of key stakeholders should be taken into consideration.

• Identify the key partners who would be instrumental in sustaining program success (host Government agencies, non-governmental organizations, etc.), especially those with whom INL will collaborate long-term. The presence of “program champions” who push for institutionalization are key in sustaining results.

• Identify others from whom you would like buy-in for the successful implementation of the program’s sustainability plan.

• Identify roles and responsibilities for each key stakeholder in sustaining the program’s results. Role differentiation and its understanding by each member of the team is critical to the team's working relationships and to the successful fulfillment of the program.

• Identify the resources needed to sustain the program’s results.

Consider roles and resources that can be shared and whether more or fewer resources will be needed as activities become institutionalized.

Although the program is intended to strengthen and reform criminal justice systems, the necessity to build up support beyond INL funding is often neglected. Considering strategies which will reduce partner nations’ dependency on INL funding and resources will enhance their ability to sustain program results.

• Identify resource challenges that need to be addressed to achieve the sustainability goals and objectives. Because sustainability (and its

SUSTAINABILITY GUIDE | B-3

predecessors, local financial planning and capacity-building) is not as pressing when allocative decisions are made, implementers find that funds for later activities geared to promoting sustainability are not available because these activities were not foreseen in year one. Accordingly, consider and plan to fund activities throughout the lifecycle of the program, specifically for sustainability strengthening activities.

• Determine how to adapt to changes in the needs and preferences of the host nation, management, and other stakeholders. Needs and stakeholders may change over time. A concise plan of action for reviewing and addressing changes will increase the likelihood of sustainment.

• Develop a rough budget for the program based on current experience with the activities and results that should be sustained. Benefits will not be produced, let alone sustained, without adequate resources — financial, human, and technical — to sustain them. Although budgets are difficult to forecast in today’s unpredictable environment, benefits cannot continue post program unless ample resources are in place at the start.

• Identify and establish a system for ongoing communication with partners and key stakeholders about program progress and sustainability activities. Reporting documents, meetings, and regular communication will keep all appropriate stakeholders apprised of anything that may hinder program success or sustainability. The institution of regular quarterly and/or annual review and planning workshops, sustainability assumptions and indicator discussions, and additional communication methods will aid in building a sustainability constituency among key stakeholders.

• Work toward institutionalizing program activities and results into regular operations. Responsibility must be transferred from program staff and implementers to the host government. Capacity-building should be incorporated into all activities. The implementing organization must respond to a dynamic environment and be accountable for sustained impact.

Host nation resources (qualified human capital, supporting organizational structures, and funding) required to implement the program and sustain the outcome(s) achieved must be made clear.

• Develop systems for collecting and analyzing data on sustainability.

Ensuring a system is in place that captures sustainability indicator data is important in determining whether sustainability goals are within reach or require revision.

B-4 SUSTAINABILITY GUIDE

• Define what sustainability success looks like and establish benchmarks against which to measure progress. Both INL and the host nation need a clear understanding of what the program is meant to do, the timeline for performance, and how/when progress will be assessed. Indicators and performance measures can inform INL and its implementers if the program achieved its sustainability goals and objectives.

• Document the written sustainability plan. Taking the actions done during the previous steps, a written sustainability plan will provide a roadmap to guide stakeholders during work on sustainability efforts. A written plan also strengthens stakeholder buy-in and understanding of the effort needed to ensuring the program results are sustained.

• Revisit the sustainability plan at key points during the program life cycle and update it to reflect changes as you learn about the effectiveness of the program. Ensure all changes are documented as appropriate. Change is inevitable; therefore, the sustainability plan should be reviewed for progress, accuracy, and revision, if necessary, at regular intervals during the life cycle. Regularly reviewing progress, impediments, and changes (e.g., to assumptions, political leadership, resources, security) enables us to implement necessary course corrections (which could include termination in some circumstances).

SUSTAINABILITY GUIDE | C-1

APPENDIX C – INL PROGRAM PERFORMANCE METRIC

EXAMPLES

The purpose of this document is to assist Program Officers and Implementers with developing performance measurements that:

1. Are quantifiable

2. Consistent with the project scope, schedule and funding

3. Meet SMART (specific, measurable, attainable, relevant and time bound) criteria

4. Can be combined with other metrics to determine program impact

5. Must link to the INL Functional Bureau Strategy (Goals and Objectives)

INL Performance Metrics can often be placed within a variety of categories, including:

1. Training Courses provided

2. Workshops conducted

3. Train the Trainer programs

4. Mentoring/Advising

5. Measuring Complex Changes over Time

6. Development and Delivery to Design, Build, and/or Equip

Within each category, performance levels can be measured in 3 ways:

1. Reaction: measures how participants reacted to the intervention

2. Learning: measures how much knowledge increased as a result of the intervention (requires baseline data)

3. Behavior: measures how far participants have changed their behavior or applied what they have learned based on the intervention (requires baseline data)

PERFORMANCE METRICS ASSOCIATED WITH TRAINING COURSES

PROVIDED

Training programs are formalized learning experiences that use a prescribed curriculum to transfer practical knowledge, skills, and competencies that relate to specific useful skills to an audience of learners. Performance measures must include both Level 1 and Level 2 as described below. Level 3 is optional, but recommended, if possible.

http://en.wikipedia.org/wiki/Knowledge http://en.wikipedia.org/wiki/Skill http://en.wikipedia.org/wiki/Competence_(human_resources)

C-2 SUSTAINABILITY GUIDE

Learning (Level 1): Performance metrics could focus on an expected level of increased knowledge of the participants using pre-test scores to determine the baseline, followed by post-test scores to measure what was learned. The quantitative metric is based on the average of the two scores. A qualitative analysis should then discuss if the increase in knowledge met the anticipated results. If not, then the analysis should also include discussion on changes that can be made to the training course to secure the desired results.

*PA&E recommended standard

Learning (Level 2): Performance metrics could also measure the degree to which participants are applying what they have learned in the training to their workplace activities and/or behavior. A common means for applying a measurement is through observation, whereby participants are observed by the course designer (or supervisor) in a practical real-world situation and are evaluated and scored based on a standard checklist. This can be done from a random sampling of training participants.

PERFORMANCE METRICS ASSOCIATED WITH WORKSHOPS

CONDUCTED

Workshops are less formal educational modules than training courses. They can be in the form of a seminar, lecture, demonstration, simulation or the like, that emphasizes the exchange of ideas and the demonstration and application of techniques, skills, etc. It emphasizes problem-solving, hands-on training, and requires the involvement of the participants.

Example for Learning (Level 1) Performance Metric Participants trained in the INL-funded __ (title/type of course) ______ will have an average increase in knowledge of at least 10%* between pre- and post-test scores.

Example for Learning (Level 2 ) Performance Metric One Month following INL-funded training, a sample of _20_%* of participants who underwent the _(title/type of course) will undergo a retention test (i.e., learning objectives checklist) and earn a score of at least 50%.*

SUSTAINABILITY GUIDE | C-3

Performance metrics can measure the satisfaction rates over the period of performance for participants undergoing like workshops. This can be accomplished through surveys or questionnaires preferably using a Likert type rating scale. Measuring satisfaction rates can give additional insight into how effective the workshop content was and if any modifications may be needed.

* PA&E recommended standard

PERFORMANCE METRICS ASSOCIATED WITH TRAIN THE TRAINER

PROGRAMS

A train-the-trainer model enables experienced personnel to show a less-experienced instructor how to deliver courses, workshops and seminars. A train-the-trainer model can build a pool of competent instructors who can then teach the material to other people.

Learning (Level 1): This follows the same methodology as explained in Level 1 of the “Performance Metrics Associated with Training Programs.” Metrics could also measure the ability for the new instructors to present information effectively, respond to participant questions and lead activities that reinforce learning. This can be measured through pre- and post- tests scores and/or observation using a course designer or subject-matter expert standardized checklist.

Learning (Level 2): This follows the same methodology as explained in Level 2 of the “Performance Metrics Associated with Training Programs.” Performance can be evaluated through a retention test or observation in a training session.

Example for Reaction Performance Metric Satisfaction surveys will be distributed to participants of all INL-funded workshops conducted and results will indicate at least a 70%* overall satisfaction rate per workshop.

Standard Example for Learning (Level 1) Performance Metric Participants trained in the INL-funded __ (title/type of course) ______ will have an average increase in knowledge of at least 10% between pre- and post-test scores.

C-4 SUSTAINABILITY GUIDE

PERFORMANCE METRICS ASSOCIATED WITH

ADVISING/MENTORING

Mentoring/advising is a dynamic, collaborative, reciprocal and sustained relationship between an experienced professional and a lesser one focused on the transfer of knowledge, skills and behaviors to a prescribed level. Performance metrics could measure the increase in knowledge, skills and behavior at various intervals during the mentoring/advising period using a prescribed survey or checklist.

PERFORMANCE METRICS ASSOCIATED WITH AND MEASURING

COMPLEX CHANGES OVER TIME

INL interventions may include measuring complex changes by the host country over time, such as systems reform, meeting specific international standards and measuring increased coordination and information sharing among host country agencies.

Performance metrics could compare the acceleration of participants’ ability to produce specific outputs within a particular organization’s structure. This evidence of change can be demonstrated, for example, through an increase or decrease in tangible outputs, such as cases prosecuted. This metric requires a quantitative baseline data against which any change is measured.

Standard Examples for Learning (Level 2) Performance Metric 20%* of train-the-trainer students who underwent the__ (title/type of course) ___ and were later observed received a retention score of at least 50% based on a Train the Trainer retention scorecard (i.e., teaching objectives checklist).

Standard Examples for Performance Metric For all mentoring relationships, the mentee will establish specific objectives that are agreed upon with the mentor prior to the second meeting. The mentor will administer, at a minimum, bi-annual (twice a year) standardized checklists to gauge the number of objectives achieved with a minimum result of 50%* or higher.

SUSTAINABILITY GUIDE | C-5

1. Increase in tangible outputs include: Cases prosecuted, calls into tip lines, land patrols, asset forfeitures, convictions, number of investigations, public perception, implementation of new management tools

2. Decrease in tangible outputs include: Backlogged cases, eradication of illicit materials, processing time, response time

INL Program Officers and Implementing Partners can monitor a host of these performance metrics, but would need to directly link these metrics to INL-funded training, workshops and mentoring/advising initiatives to dollars spent by Department of State (DOS) to determine return on investment. Another area to examine would be testing criminological theories for causative factors of change over time.

PERFORMANCE METRICS ASSOCIATED WITH DEVELOPMENT AND

DELIVERY TO DESIGN, BUILD, AND/OR EQUIP

Performance metrics that involve drafting legislation, implementing a system, procuring equipment or other start-up activity are typically measured using a “Yes/No” metric. These types of activities can be broken down into smaller, measurable milestones and assigned a numerical percentage value. Quarterly reporting would indicate the percentage complete and the final metric would be “Yes/No.”

Examples for Behavior Performance Metric _(Description of activity)______ will (increase/decrease) by _(xx)__% from the baseline period of _ (specified baseline period)___ by _(defined end time).

(and/or)

_ (Agency or office within an Agency/Organization)____ will demonstrate their new skills and knowledge through a _(xx)__% increase in __(type/ description of activity)__ as measured by __(defined baseline data).

Investigators from __(specific office, agency or other defined entity) will engage and investigate _(xx)__% more criminal cases in CY 2015 than from the previous benchmark of 2013.

C-6 SUSTAINABILITY GUIDE

Standard Examples Performance Metric The _ (system or type of infrastructure____ will be installed/built and operational with an expected outcome of (description of benefit). It will be 50% complete by (anticipated midpoint date) and 100% complete by (anticipated completion date).

_ (Governmental Agency or Entity)_____ will draft a new __(description of document)___ that achieves (description of benefit). It will be 50% complete by (anticipated midpoint date) and 100% complete by (anticipated completion date).

SUSTAINABILITY GUIDE | D-1

APPENDIX D – INL STRATEGIC PILLARS - 2013

Pillar Pillar Definition

Anticorruption This includes preventive or enforcement activities directed at combating corruption, as a standalone activity or incorporated into other programming.

Counter Narcotics

This includes the activities leading to eradication, interdiction, law enforcement, and demand reduction (designed to prevent the cultivation, use or distribution) of illegal narcotic drugs.

Corrections Sector

This includes agencies, institutions and civilian and sworn personnel responsible for managing individuals suspected or convicted of violating the law. Institutions include detention centers, jails, correction centers, prisons, penitentiaries, probation and parole. Depending on the system, it may also include those agencies, institutions and personnel responsible for managing juvenile offenders and other vulnerable populations in similar settings.

Justice Sector

This includes judiciary and courts, the prosecution service, related Ministry, and the defense bar (including both private attorneys and public defenders), formal and informal institutions providing oversight, guidance and support to these areas, and the legal framework (codes, statutes and law) under which a host nation operates.

International Cooperation

This includes all activities that promote the cooperation between nations relevant to criminal justice in which a nation is a signatory and has membership/participation in conventions, agreements and international organizations.

Law Enforcement

This includes policing, criminal investigations, public order, internal security, evidentiary forensic analysis, customs and border security and formal and informal oversight such as the Ministry of Interior or Ministry of Internal Affairs.

Organized Crime/Terrorism

This includes efforts to combat illicit financial hubs related to crime-terror pipelines and illicit financial flows as well as cross-border capacity building to strengthen governments’ abilities to disrupt and dismantle threat networks.

SUSTAINABILITY GUIDE | E-1

APPENDIX E – INL IAA QUARTERLY PROGRESS REPORT

TEMPLATE

Guidance: This sample Quarterly Progress Report is meant to serve as a guide and example for how the report should look. INL allows implementers the flexibility to organize their reports as they wish as long as all of the relevant information about the quarterly activities and analysis regarding the progress of project activity are included.

Report due dates to the INL Program Office are listed below. Timely and complete reports improve processing of future interagency implementing agreements.

Q#1: Oct 1 - Dec 31 Due: February 15

Q#2: Jan 1 - Mar 31 Due: May 15

Q#3: Apr 1 - Jun 30 Due: August 15

Q#4: Jul 1 - Sep 30 Due: November 15

For more guidance or feedback about your Progress Report, please discuss with

INL/RM/PA&E.

E-2 SUSTAINABILITY GUIDE

Project Information In the final form, the following fields will be presented with the type of data input listed.

Country:

Implementer:

Project Title:

INL Pillar:

IAA SINLEC Number:

Funding Obligation:

Funding Amount:

Project Dates:

Quarter (Dates) being discussed:

Date Progress Report is submitted:

Primary Point of Contact/Title:

Phone Number:

Email:

Additional POCs:

Summary This section is meant to provide highlights from the reporting quarter, primarily key activities accomplished or challenges faced that impeded progress towards project objectives.

Contextual Information/Background This section should be no longer than a few paragraphs and is meant to briefly explain the current operating environment where the program is taking place and how that is impacting (positively or negatively) the implementer’s ability to carry out the program activities.

Project Progress by Project Component/Objectives This section comprises the main portion of the progress report. Describe by each project objective the significant activities that took place during the reporting period. The section should not just be a laundry list of activities that occurred during the reporting period, but should include an analytical evaluation of how the program activities are helping to achieve the project objectives. When possible, detail any tangible impact or success from the project activities. The objectives and

SUSTAINABILITY GUIDE | E-3

measures should match the work plan appendix from the implementing agreement.

(Repeat for all Project Objectives, activities and performance measures.)

Project Component #1:

Title:

Objective:

Input:

Outputs:

Outcomes:

• Performance Measurement #1.1:

• Progress toward measurement:

• Links to INL Functional Bureau Strategy FY 2014-2016:

Problems or Challenges and Corrective Action Plan/Risk Identification and Mitigation Detail any problems or challenges in implementing any of the program components and activities. INL understands that many of its implementers work in difficult operating environments and does not fault them when unforeseen problems arise. But it is important to document these difficulties in the progress reports (and also report these challenges to their Program Officer as they occur).

If the challenges will seriously impede the progress of the project, provide a corrective action plan for the next quarter with an updated timeline of program activities. This should be organized by project component.

Project Component Title:

• Challenge:

• Corrective Action Plan or explanation why no plan is necessary:

• Corrective Action Plan Progress:

Proposed Activities for Next Quarter Provide a list of the activities plans for the following quarter.

Project Component Title:

• Proposed Activities:

E-4 SUSTAINABILITY GUIDE

Sustainability and Progress Towards Exit Strategy This section should clearly delineate how elements of the project are having a multiplier effect and are sustainable beyond the life of the project. What is the impact beyond the direct beneficiaries of the project (e.g. participants trained go on to train other people, workshop participants use skills from a workshop to enhance a national level election that affects the entire populace)? Is there demonstrated, continuing impact beyond the life of the project or support that will occur after INL funding ceases?

Project Component Title:

• Description of project sustainability and progress towards exit strategy:

Additional Information Include any information that doesn’t fit in the other categories in this section.

SUSTAINABILITY GUIDE | F-1

APPENDIX F – INL FUNCTIONAL BUREAU STRATEGY &

PROGRAM MANAGEMENT GUIDES

The Presidential Policy Directives on Global Development and Security Sector Assistance, and the Quadrennial Diplomacy and Development Review all stress the critical importance of emphasizing sustainability of foreign assistance investments to the greatest extent possible and practicable. INL has committed to highlighting how sustainability is essential to the success of the program by referencing and reinforcing this approach to successful project management in the INL Functional Bureau Strategy and the program management guide series which includes the Project Design Guide;

Guide to Developing a Performance Management Plan; and Guide to Results Framework. The Sustainability Guide is the latest addition to these core documents.

Collectively, this program management series aids INL in the proper design, performance, monitoring, and sustainment of INL programs, ultimately resulting in the transition of responsibility to host country partners.

INL Functional Bureau Strategy – The Functional Bureau Strategy describes INL’s strategy for accomplishing its vision of accomplishing peace, security, and justice through international cooperation and assistance and performing its mission to pursue DoS policy changes and foreign assistance programming that prioritize: improving citizen safety; securing borders and ports; enabling capable and accountable governments; assisting communities at risk, and; facilitating information-sharing and regional cooperation.

• A key principle of INL’s strategy is sustainability. INL’s foreign assistance programs need to be rooted in host-country ownership to be sustainable.