ATTACHMENT_11_-_72051418R00006_(01).pdf

USAID/COLOMBIA EVAL

QUARTERLY REPORT #1:

29 MAY – 30 JUNE 2013

JULY 2013

This publication was produced for review by the United States Agency for International Development. It was prepared by Mark Renzi, Management Systems International.

USAID Colombia EVAL Quarterly Report #1

29 MAY – 30 JUNE 2013

Contracted under AID-514-C-13-00003

Evaluation and Analysis for Learning (EVAL) Project

DISCLAIMER

The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

DRAFT: Evaluation and Analysis for Learning (EVAL) Quarterly Report #1: 29 May to 30 June 2013 i

CONTENTS

USAID/Colombia EVAL Quarterly Report # 1: 28 May to 30 June 2013

1. Summary

2. Introduction

2.1 Activity Background

2.3 Quarterly Report Format

3. Progress against work plan and results Achieved

4. Challenges encountered and proposed solutions

6. Lessons learned

7. Planned activities for the next quarter

8. Factors influencing (positively or negatively) prospects for achievement of results 4

9. Required or recommended changes to the work plan

DRAFT: Evaluation and Analysis for Learning (EVAL) Quarterly Report #1: 29 May to 30 June 2013 1

Team EVAL Management Systems International (MSI) Centro Nacional de Consultoría (CNC) dTS Sabre Systems Centro de Investigaciones para el Desarrollo de la

Universidad Nacional (CID)

CINDE

Economia Urbana Fundción Evaluar Gexponcial Qualitas Rosario University Selva

1. SUMMARY

This short period was dominated by activity start-up.

Logistically, the COP relocated to Bogotá, placed adverts for Colombian staff, and identified possible sites for the project office and for expat staff accommodation. La Macarena was chosen as the likely office site since it is walking distance to EVAL’s main subcontractor, Centro Nacional de Consutlaría de Colombia (CNC), is in the middle of the Bogota’s university sector, is closer to USAID and the GOC than the area where most implementing partners are located, and yet is close to the IPs for when collaboration is needed. It is also a very cost-effective option as cost/square foot is relatively low.

Technically, the COP engaged in a series of meetings with USAID and CNC to prepare for the upcoming products required by USAID. Early on it became clear that USAID had reconsidered the portfolio of products that it would require from EVAL. Some impact evaluations would be converted to performance evaluations, some products would be eliminated, others added. Accordingly, USAID and MSI agreed to embark on a planning process that would result in a contract modification while responding to USAID’s urgent time requests for certain products. We will plan in detail those products for which information is available and provide rough estimates for the others as part of this process in the coming quarter. The first annual work plan was also submitted, reflecting these dynamics.

2. INTRODUCTION

2.1 Activity Background

Management Systems International (MSI) is the Prime Contractor for the Evaluation and Analysis for Learning (EVAL) Project (AID-514-C-13-00003). MSI will work with many of the potential subcontractors listed in the accompanying text box to access the broad range of skills required to meet USAID/Colombia’s needs in providing the following products:

Impact evaluations;

Performance evaluations; and

Assessments.

The five-year activity—running until May 2018—is based in Bogotá, with access to field offices throughout the country. It is a demand-driven contracting vehicle that enables USAID/Colombia’s Program Office to obtain evaluations, assessments, and related planning and analytic services for programmatic learning and to support Mission accountability and the needs of the Mission’s Technical Offices.

2.3 Quarterly Report Format

This Quarterly report is organized by the management issues most important to USAID, as detailed in Section F.3.A of the EVAL contract.

DRAFT: Evaluation and Analysis for Learning (EVAL) Quarterly Report #1: 29 May to 30 June 2013 2

3. PROGRESS AGAINST WORK PLAN AND RESULTS

ACHIEVED

Since this period only covers one month, during which we developed a new work plan, there is no significant progress against the work plan to report. Future Quarterly Reports will have more to discuss.

4. CHALLENGES ENCOUNTERED AND PROPOSED

SOLUTIONS

During this period the Mission determined that it would likely eliminate some of the evaluations described in the existing EVAL contract, shift some from impact to performance evaluations, and add other products, particularly a large number of institutional assessments. In fact, as displayed in Table 2.1, below, well over half of the current product portfolio would change, requiring considerable effort to integrate into the current contract and budget.

TABLE 4.1: USAID’S CONTEMPLATED CHANGES TO PRODUCTS UNDER THE

EVAL CONTRACT

Product Product Type

Nature of Change IE PE Assess.

Products that would be Changed

AJP X Change from IE to PE

PROTIERRA X X Change activity evaluation from IE to PE

Green House Gas O Eliminate

ADAM X Include evaluation of MIDAS activity, as well

Mental Health O Eliminate

Land Mines O Eliminate

Products that would be Added

PROTIERRA –

Gender Component

X X Analyze project to develop scaling-up approach for innovative gender component Focus IE on gender component

PLANEDH X Add new post-completion PE

GOC Assessments X Add new institutional assessments (possibly as many as six)

Products with No Changes

CDSI X

ACIP X

BIO-REDD X

Human Rights III X

Victims Program X

Land + Livelihoods X

USAID Forward X

DRAFT: Evaluation and Analysis for Learning (EVAL) Quarterly Report #1: 29 May to 30 June 2013 3

Tropical Forestry X

Implementing the proposed changes will require a contract modification and budget revision, based on the best available information available on the desired tasks. This is dependent upon USAID providing at least detailed oral descriptions of what the tasks on the immediate horizon would entail. Hopefully, these would include those indicated red italics in Table 4.1, above.

With this information, we do not anticipate any problems.

5. PLANNED ACTIONS NOT COMPLETED

None.

6. LESSONS LEARNED

In discussions with Mission staff, it seemed that there may be several themes that are critical to the CDCS and which could be addressed in all products. Initial ideas for themes include: the peace process, disadvantaged persons, and USAID Forward (partly in preparation for the upcoming assessment on that theme.) If this is the case, some of these topics may form part of a backbone for the Mission’s learning agenda for the CDCS period.

7. PLANNED ACTIVITIES FOR THE NEXT QUARTER

In the coming quarter we will finalize a contract modification and begin work immediately on an ambitious work program for urgent products—even as a contract modification is being finalized—focusing on:

Develop understanding of approach to mid-term evaluations of CDSI (Colombia Strategic Development Initiative) and ACIP (Afro-Colombian and Indigenous Program);

Begin an assessment of scaling-up and designing an IE methodology for the gender component of PROTIERRA (People-to-People Resolution of Land and Natural Resources Conflicts in Colombia);

Begin a post-activity performance/sustainability evaluation examining together both the ADAM (Areas for Municipal-Level Development) and the MIDAS (More Investment for Sustainable Development Programs) activities;

Begin a post-activity performance evaluation of PLANEDH (National Human Rights System and National Plan for Human Rights Education);

Begin three assessments of GOC (Government of Colombia) institutions; and

Gather data and begin planning for the other products that have later start dates.

We will also establish our office and have the full team employed.

DRAFT: Evaluation and Analysis for Learning (EVAL) Quarterly Report #1: 29 May to 30 June 2013 4

8. FACTORS INFLUENCING (POSITIVELY OR

NEGATIVELY) PROSPECTS FOR ACHIEVEMENT OF

RESULTS

No negative factors were noted.

Positive factors, at this early date appear to be:

Talented and motivated M&E Team at USAID; and

Commitment by USAID staff in the Post-Award Meeting to encouraging the evaluations to find truth and tell it.

9. REQUIRED OR RECOMMENDED CHANGES TO THE

WORK PLAN

The changes noted in Section 4 will require revisions to the work plan. Those are now being considered.

QUARTERLY REPORT #2:

1 JULY – 30 SEPTEMBER 2013

OCTOBER 2013

QUARTERLY REPORT #2

1 JULY – 30 SEPTEMBER 2013

The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency

DRAFT: Evaluation and Analysis for Learning (EVAL) Quarterly Report #2: 1 July – 20 September 2013 i

USAID/Colombia EVAL Quarterly Report # 2: 1 July 2013 – 30 September 2013

1. Summary

2. Introduction

2.1 Activity Background

2.3 Quarterly Report Format

3. Progress against work plan and results achieved

4. Challenges encountered and proposed solutions

6. Lessons learned

7. Planned activities for the next quarter

8. Factors influencing (positively or negatively) prospects for achieveming results 5

9. Required or recommended changes to the work plan

DRAFT: Evaluation and Analysis for Learning (EVAL) Quarterly Report #2: 1 July – 20 September 2013 1

1. SUMMARY

This period was occupied with administrative start up, technical scoping, and shifting contractual gears to provide USAID greater flexibility to pursue its analytic agenda.

Administratively, we hired all staff consistent with the original operational plan, developed our contracting arrangements with our principal Colombian subcontractor, and released a Request for Proposals for additional Colombian subcontractors to assist with future products. We also reached general agreement with the COR over the dimensions of an EVAL contract modification. A draft contract modification budget was being reviewed by the Contracts Office at the end of the period. Technically, we invested considerable effort discussing potential products and working with USAID to develop specific scopes of work (SOWs) for some of them. A SOW was finalized for the Land Mines performance evaluation and considerable progress was made towards completing several others.

2. INTRODUCTION

2.1 Activity Background

Management Systems International (MSI) is the Prime Contractor for the Evaluation and Analysis for Learning (EVAL) Project (AID-514-C-13-00003). MSI will work with Colombian subcontractors to access the broad range of skills required to meet USAID/Colombia’s needs in providing the following products:

Impact evaluations;

Performance evaluations; and

Assessments.

The five-year activity—running until May 2018—is based in Bogotá, with access to field offices throughout the country via our main subcontractor, Centro Nacional de Consultoría (CNC). It is a demand-driven contracting vehicle that enables USAID/Colombia’s Program Office to obtain evaluations, assessments, and related planning and analytic services for programmatic learning and to support Mission accountability and the needs of the Mission’s Technical Offices.

2.3 Quarterly Report Format

This Quarterly report is organized by the management issues most important to USAID, as detailed in Section F.3.A of the EVAL contract.

1. PROGRESS AGAINST WORK PLAN AND RESULTS

ACHIEVED

All appropriate significant administrative mobilization actions were completed, as indicated by Table 3.1, which copies planned administrative activities from Work Plan #1 in the left column and reports on progress for each task in the right column.

Table 3.1: Progress against planned administrative tasks in period

DRAFT: Evaluation and Analysis for Learning (EVAL) Quarterly Report #2: 1 July – 20 September 2013 2

Project Mobilization and Start-up Tasks from Work Plan #1

Status as of 30 September 2013

Staffing Mark Renzi (from MSI staff) Completed Keri Culver (from MSI staff) Completed

Ana María Rivera (from CNC staff) Completed

Local Fin/Admin office staff Completed

Planning Post-Award meeting Completed

Team planning meetings Completed

Scope Evaluation Requirements Completed

Interim Funding for CNC Completed

Subcontract for CNC Completed

Facilities/Resources Establish Office Pending: planned office was changed to accommodate increased staff and scope via anticipated contract modification

Procure additional office equipment Pending, as immediately above

Develop IT and information management systems

Completed

Systems Obtain EVAL bank accounts Completed

Adapt accounting systems Completed

Sign subcontract with CNC Completed

Set up project management system Completed

Set up reporting system and produce quarterly reports

Completed

Project Design Submit draft of Year One work plan Completed

Discuss product revisions with USAID Completed

Modify EVAL Contract with CO In process, draft modification being reviewed by CO

Review Reg. 216 compliance requirements

Pending access to MONITOR, which had not yet been granted by the end of the period.

Technically, progress was slow, but steady, as EVAL was chiefly occupied with helping the Mission develop SOWs for products and considering how to adapt the EVAL contract to increase flexibility. But, as indicated in the summaries below (where we have copied the planned actions from Work Plan #1, and aligned them with actual actions during the period), we have accomplished most of the actions that were under our control.

DRAFT: Evaluation and Analysis for Learning (EVAL) Quarterly Report #2: 1 July – 20 September 2013 3

CSDI IE: PLANNED VS. ACTUAL ACTIONS DURING PERIOD

Anticipated Actions Anticipated Accomplishments

Due Date Status as of 30-Sep-

Communicate with IPs and DevTech/Econometría on baseline process, challenges, lessons learned, cost issues, data quality, etc.

Determination of the utility of current data sets and approaches and any modifications needed for baseline and subsequent data collection waves

Q 4 Done. Feasibility determined.

ACIP IE: PLANNED VS. ACTUAL ACTIONS DURING PERIOD

Anticipated Actions Anticipated Accomplishments

Due Date Status as of 30-Sep-

Communicate with ACDI/VOCA and DevTech/Econometría on baseline

Determination of the utility of current data sets and approaches and any modifications needed for baseline and subsequent data collection waves

Q-4 Done. Feasibility determined.

ADAM/MIDAS PE: PLANNED VS. ACTUAL ACTIONS DURING PERIOD

Accomplishments

Due Date Status as of 30-Sep-

Discuss evaluation design with USAID Feasible evaluation approach determined

Q-4 Done. SOWs drafted; in discussion

Gather data for evaluation Q-4 Done, pending final

SOW

PROTIERRA PE: PLANNED VS. ACTUAL ACTIONS DURING PERIOD

Due Date Status as of 30-Sep-

Work with USAID and MC to understand the project and EVAL’s role

EVAL can anticipate USAID’s requirements

Q4 Done. SOWs drafted; in discussion

GOC ASSESSMENTS: PLANNED VS. ACTUAL ACTIONS DURING PERIOD

Accomplishments

Due Date Status as of 30-Sep-

Discuss scope of products with USAID Clear understanding of product

Q-4 Ongoing

QUARTERLY REPORT #13:

1 APRIL – 30 JUNE 2016

JULY 2016

Development. It was prepared Roger Rasnake, Management Systems International.

QUARTERLY REPORT #13

1 APRIL – 30 JUNE 2016

Management Systems International Corporate Offices 200 12th Street South Suite 1200 Arlington, VA 22202, USA\ Tel: + 1 (703) 979-7100

The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency

USAID/Colombia EVAL Quarterly Report #13 i

USAID/Colombia EVAL Quarterly Report #13

1 April – 30 June 2016

CONTENTS

1. Summary

2. Introduction

2.1 Activity Background

2.2 Quarterly Report Format

3. Progress Against Work Plan and Results Achieved IE 1: CELI – Consolidation and Enhanced Livelihoods Initiative IE 2: ACIP: Afro-Colombian and Indigenous Program

PE1: ADAM/MIDAS

PE2: PROTIERRA – People-to-People Resolution of Land and Natural Resources Conflicts in Colombia – Gender Component PE 4: Public Policy Program PE 5: Victims Unit Evaluation Completed PE 6: Access to Justice Completed

PE 7: BIOREDD+

A 2: Public/Private Partnerships A 3: Institutional Assessment of the National Center for Historical Memory

(CNMH)

A 4: Transition Criteria for USAID Assistance to GOC Institutions A 5: Case Studies of Sur de Tolima and Bajo Cauca A 6: Case Study of Tumaco A 7: Case Study of La Macarena A 8: Case Study of Magdalena Medio A 9: Directory of Potential “Col-Col” TA Providers A 10: Tertiary Roads Assessment

4. Challenges Encountered and Proposed Solutions

5. Planned Actions Not Completed

6. Lessons Learned

7. Planned Activities for the Next Quarter

8. Factors Influencing (Negatively or Positively) Prospects for Achieving Results

9. Required or Recommended Changes to the Work Plan

USAID Colombia EVAL Quarterly Report #13 1

Implementation is on track.

Administratively, USAID approved a realignment request submitted by EVAL to reinstate the COP position and to add support for a Technical Director for the life of project. After an earlier decision not to seek an increase in the EVAL contract technical ceiling amount, USAID and EVAL will monitor the situation moving forward to determine if one might be needed in the future.

Technically, impact evaluations continued to absorb much of our labor during the period, while the assessment CLIN continued apace. No performance evaluation work was carried out. In the next period the CELI impact evaluation will be fully concluded, with considerable assessment work and no performance evaluations planned. The portfolio is summarized below:

• Impact Evaluations: In concluding the ACIP mid-term impact evaluation report, we conducted debriefing workshops with the implementer and USAID, and made final revisions in the report. We subsequently undertook intensive data analysis and report writing, and delivered a draft report for the CELI mid-term impact evaluation. In the coming period we will complete the incorporation of stakeholder feedback into the main CELI evaluation report and provide five regional sub-report drafts.

• Assessments: We began work on a comprehensive case study of the Magdalena Medio region, and delivered an inventory of Colombian grass-roots organizations potentially capable of providing technical assistance (TA). We also undertook an assessment on tertiary roads intended to contribute to further defining USAID’s future activities in this arena.

• Performance Evaluations: No performance evaluations were identified for the reporting period.

2. INTRODUCTION

2.1 Activity Background

Management Systems International (MSI) is the prime contractor for the Evaluation and Analysis for Learning (EVAL) Project (AID-514-C-13-00003). MSI works with Colombian subcontractors to access the broad range of skills required to meet USAID/Colombia’s needs in providing products under the following CLINs:

• Impact Evaluations;

• Performance Evaluations; and

• Assessments.

USAID Colombia EVAL Quarterly Report #13 2

The five-year activity—running until May 2018—is based in Bogotá, with access to field offices throughout the country via our main subcontractor, Centro Nacional de Consultoría (CNC). It is a demand-driven contracting vehicle that enables USAID/Colombia’s Program Office to obtain evaluations, assessments, and related planning and analytic services for programmatic learning, and to support Mission accountability and the needs of the Mission’s Technical Offices.

2.2 Quarterly Report Format

This Quarterly Report is organized by the management issues most important to USAID, as detailed in Section F.3.A of the EVAL contract.

3. PROGRESS AGAINST WORK PLAN AND RESULTS

ACHIEVED

EVAL continued to work on the ACIP and CELI impact evaluations under the impact evaluation CLIN during the period. Under the Assessment CLIN, we completed all work on the La Macarena Case Study, and began work on the Magdalena Medio Case Study, the Inventory of Potential TA Provider Organizations and a tertiary roads assessment.

The following sections describe activities for each product during the period relative to the current Work Plan #7. The tables included below are extracted and updated from that document:

IE 1: CELI – Consolidation and Enhanced Livelihoods Initiative

Anticipated Actions Anticipated Accomplishments

Due Date Comments, if any

Report Production All data analyzed and draft report produced

June 2016 Draft Report Completed

During the period, the evaluation team worked intensely to analyze and interpret survey results, and to compile and analyze activity performance and contextual information to produce a report draft.

IE 2: ACIP: Afro-Colombian and Indigenous Program

Anticipated Actions Anticipated Accomplishments

Due Date Comments, if any

Implementer and USAID discussion

Comments provided April 2016 Completed

Incorporate comments into final report

Final report May 2016 Completed

After a Mission presentation, EVAL worked with USAID and the ACIP implementer to resolve questions and concerns to produce the final report.

USAID Colombia EVAL Quarterly Report #13 3

PE1: ADAM/MIDAS

Completed.

PE2: PROTIERRA – People-to-People Resolution of Land and Natural Resources Conflicts in Colombia – Gender Component

PE 4: Public Policy Program

Completed.

PE 5: Victims Unit Evaluation

Completed.

PE 6: Access to Justice

Completed.

PE 7: BIOREDD+

Completed.

A 2: Public/Private Partnerships

Completed.

A 3: Institutional Assessment of the National Center for Historical Memory (CNMH)

Completed.

A 4: Transition Criteria for USAID Assistance to GOC Institutions

Anticipated Actions Anticipated Accomplishments Due Date Comments, if any

No further actions anticipated.

USAID received a final draft of the Feasibility Analysis from EVAL. Subsequent internal discussions in USAID determined not to proceed.

TBD Removed from consideration

USAID decided not to proceed with this activity.

USAID Colombia EVAL Quarterly Report #13 4

A 5: Case Studies of Sur de Tolima and Bajo Cauca

A 6: Case Study of Tumaco

Completed.

A 7: Case Study of La Macarena

A 8: Case Study of Magdalena Medio

Anticipated Actions

Anticipated Accomplishments

Due Date Comments, if any

Work plan developed Team collecting secondary and field data

April 2016 Completed

Field work conducted Data ready for analysis July 2016 On target

Write draft report Draft report submitted to

USAID

August 2016 On target

Incorporate comments into final report

Final report received August 2016 On target

This case study will be the last in the series, with a strong comparative focus. The bulk of the work was carried out in this reporting period, which will provide the basis for final reporting leading up to the end of the upcoming quarter.

A 9: Directory of Potential “Col-Col” TA Providers

This activity was ongoing throughout the reporting period, with a draft report delivered at the end of the quarter.

A 10: Tertiary Roads Assessment

The first period of work in Colombia was carried out by the consultant and the EVAL team through meetings with USAID, implementers, government entities, and a field visit. Toward the end of the reporting period further discussions and meetings took place, and the resulting assessment will be finalized in the next reporting period.

4. CHALLENGES ENCOUNTERED AND PROPOSED

SOLUTIONS

No significant challenges were encountered.

USAID Colombia EVAL Quarterly Report #13 5

6. LESSONS LEARNED

None.

7. PLANNED ACTIVITIES FOR THE NEXT QUARTER

Administratively, we hope to work with USAID to finalize a realignment of the EVAL contract budget.

Technically, we will finalize the CELI mid-term impact evaluation reports, the Magdalena Medio Case Study, and the tertiary roads assessment. Discussions within USAID point to upcoming assessment work relating to biodiversity and value chains, as well as evaluation design and initial research relating to the rural finances portfolio. Ongoing discussions with the Departamento de Planeación Nacional (DNP) concerning EVAL support in several areas of mutual interest in evaluation between USAID and the DNP will hopefully lead to a Memorandum of Understanding authorizing EVAL support to this entity. No performance evaluations are currently planned for the balance of the contract.

8. FACTORS INFLUENCING (NEGATIVELY OR

POSITIVELY) PROSPECTS FOR ACHIEVING RESULTS

None.

9. REQUIRED OR RECOMMENDED CHANGES TO THE

None required.

DRAFT: Evaluation and Analysis for Learning (EVAL) Quarterly Report #2: 1 July – 20 September 2013 4

Accomplishments

Due Date Status as of 30-Sep-

Develop SOW; begin recruiting Prepared to conduct work

Q-4 Delayed. USAID has been shifting its focus

Commence work Q-4 Delayed, per above

We had not anticipated working on the Land Mines evaluation or on Section 118-119 Biodiversity and Forestry assessment during the period. But, we made rapid progress on both products, completing the Land Mines SOW and submitting a draft final SOW for Section 118/119. We accelerated our review of the data availability for the two impact evaluations: ACIP and CSDI, completing the review for CSDI.

2. CHALLENGES ENCOUNTERED AND PROPOSED

No significant challenges.

3. PLANNED ACTIONS NOT COMPLETED

None, although the process of finalizing SOWs has taken longer than expected, resulting in delays from when we had hoped to begin work on actual product production. This was particularly true with ADAM/MIDAS, PROTIERRA, and GOC institutional assessments.

4. LESSONS LEARNED

In finalizing one SOW, and making considerable progress on several others, it became clear that having EVAL participate actively alongside Mission technical and program staff in SOW development has added value to the process, both in developing technically sound SOWs and in gaining stakeholder buy-in to the technical approach. In future, we need to extend the length of time we plan for developing SOWs. An iterative approach to developing SOWs for more the complex products is essential; but our experience to date indicates that the process requires months, not weeks, to complete.

5. PLANNED ACTIVITIES FOR THE NEXT QUARTER

Administratively, in the coming quarter we will move into and equip our new office, add staff, and finalize a contract modification. Technically we will:

Begin work on performance evaluations of:

o Land Mines;

o PROTIERRA;

o ADAM/MIDAS;

Begin work on the following assessments:

o Forestry and biodiversity pursuant to Sections 118 and 119;

DRAFT: Evaluation and Analysis for Learning (EVAL) Quarterly Report #2: 1 July – 20 September 2013 5 o Public/Private Partnerships; and

Develop SOWs for other products that USAID considers important.

6. FACTORS INFLUENCING (NEGATIVELY OR

POSITIVELY) PROSPECTS FOR ACHIEVEING RESULTS

No negative factors were noted. Positive factors include:

A results-oriented USAID Contracting Office, facilitating the accomplishment of USAID programmatic priorities; and

A Front Office interested in using EVAL to support strategy.

7. REQUIRED OR RECOMMENDED CHANGES TO THE

Beginning with this document EVAL will produce work plans every six months, rather than annually. This change will be reflected in the upcoming EVAL contract modification.

EVAL is a demand-driven project: it exists to produce specific products that can meet the evolving learning priorities of USAID. Some of these – such as evaluations of major projects – can be anticipated, and are unlikely to change, except for timing. Other products – such as rapid studies to inform USAID’s work associated strategy formulation and with respect to the peace process – can emerge as priorities suddenly.

USAID has requested the flexibility to be able to meet both demands through the EVAL contract mechanism.

To accommodate this, USAID and MSI are in the process of modifying the contract. No changes, beyond those that will be reflected in the upcoming contract modification, are required or recommended.

We hope that halving the work planning cycle will enable us to limit the number of future changes to the work plan while providing USAID with flexibility and enabling it to track EVAL progress.

QUARTERLY REPORT #3:

1 OCTOBER – 31 DECEMBER 2013

JANUARY 2014

QUARTERLY REPORT #3

1 OCTOBER – 31 DECEMBER 2013

The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency

Evaluation and Analysis for Learning (EVAL) Quarterly Report #3: 1 October – 31 December 2013

USAID/Colombia EVAL Quarterly Report # 3: 1 October 2013 – 31 December 2013

1. Summary

2. Introduction

2.1 Activity Background

2.3 Quarterly Report Format

3. Progress against work plan and results achieved

4. Challenges encountered and proposed solutions

6. Lessons learned

7. Planned activities for the next quarter

8. Factors influencing (negatively or Positively) prospects for achieveing results

9. Required or recommended changes to the work plan

Administratively, during this period USAID and MSI managed to complete a contract modification to accelerate EVAL production, pursuant to USAID demands. Technically, EVAL hit its stride towards the end of calendar year, working simultaneously on several performance evaluations and fielding multiple teams throughout Colombia while developing SOWs for a number of assessments and commencing pre-SOW efforts on its first institutional assessment. All tasks were on schedule, except for the work on the 118/119 Assessment, which has fallen behind due to EVAL’s difficulties recruiting a team; and the Public/Private Partnership study, which was held up due to delays within USAID approving the draft SOW.

2. INTRODUCTION

2.1 Activity Background

Management Systems International (MSI) is the prime contractor for the Evaluation and Analysis for Learning (EVAL) Project (AID-514-C-13-00003). MSI works with Colombian subcontractors to access the broad range of skills required to meet USAID/Colombia’s needs in providing the following products:

Impact evaluations;

Performance evaluations; and

Assessments.

The five-year activity—running until May 2018—is based in Bogotá, with access to field offices throughout the country via our main subcontractor, Centro Nacional de Consultoría (CNC). It is a demand-driven contracting vehicle that enables USAID/Colombia’s Program Office to obtain evaluations, assessments, and related planning and analytic services for programmatic learning and to support Mission accountability and the needs of the Mission’s Technical Offices.

2.3 Quarterly Report Format

This Quarterly report is organized by the management issues most important to USAID, as detailed in Section F.3.A of the EVAL contract.

3. PROGRESS AGAINST WORK PLAN AND RESULTS

ACHIEVED

The main administrative task for the period was successfully completing the first EVAL contract modification, which called for EVAL to accelerate completion of products and allowed for hiring of two additional staff (a Contracts Manager and a Manger of Institutional Strengthening) to make this possible. As part of the contract modification process, USAID and MSI agreed to shift to six-month work plan cycles. Accordingly, during this period we submitted Work Plan #2 to USAID, covering the period October 2013 to 31 March 2014. Efforts in this quarterly report to track progress against EVAL’s work plan will be based on that document, superseding Work Plan # 1 (that had covered the period 1 October 2013 to 30 September 2014).

Technically, production accelerated dramatically during the period, with EVAL at one point having seven evaluation teams in the field simultaneously. The preparatory work in developing internal systems and working with CNC paid off, as all trips – many to complicated areas – were completed without a significant hitch.

IE 1: CSDI – Colombia Strategic Development Initiative

Due Date Status: 31-Dec-13

Verify data availability for mid-term data collection

Data inventoried and checked against needs

October

Completed

IE 2: ACIP: Afro-Colombian and Indigenous Program

Verify data availability for mid-term data

Data inventoried and checked against needs

Nov-2013 Completed

PE 1: ADAM – Areas for Municipal-level Development – Sustainability

Assessment (plus MIDAS)

SOW approval USAID approves Oct-13 Completed

Collect data Data ready for analysis Dec-13 Completed

Analyze data Analysis completed for

Dec-13 Ongoing

Schedule shifted to permit more rapid overall completion.

PE 2: PROTIERRA – People-to-People Resolution of Land and Natural Resources Conflicts in Colombia – gender component

Anticipated Actions Anticipated Due Date Status: 31-Dec-13

Complete the SOW, conduct Interim products completed Nov-13 Completed outcome mapping and begin data

PE 3: Performance Evaluation: Attention to Land Mine Victims in Colombia

Collect data Data ready for analysis Nov-13 Completed

Analyze data Analysis completed for

Draft report and prepare debriefing Draft final report submitted; debriefing held for USAID

Nov-13 Completed

Dec-13 Report completed

Debriefing is awaiting USAID scheduling.

A 1: Tropical Forestry and Biodiversity (FAA 118 and 119) Analyses

Due Date Status 31-Dec-13

SOW approved USAID approves SOW Oct-13 Completed

Interviews; map collection; data

All data collected for analysis

Dec-13 Not begun

EVAL has experienced problems fielding a team for this task. The start date for the TMP is now scheduled for January 2014.

A 2: Analysis of USAID Experience with Public/Private Partnerships (PPPs)

Due Date Comments, if Any

SOW approved USAID approves SOW Oct-13 Not completed

Interviews; map collection; focus groups;

case study data collection.

Report drafted submitted;

Debriefing prepared

All data collected for analysis

Draft Final report submitted; debriefing for USAID

Nov-13 Not completed

Dec-13 Not completed

USAID has experienced difficulty providing approval for the SOW. Accordingly, work will not begin until the next quarter.

A 3: Institutional Assessment of Center for Historical Memory

(CHM)

EVAL hired EVAL’s Institutional Strengthening Manager (Francisco Osuna), as well as STTA Catalina Morales to work on the task. Mr. Osuna led EVAL’s participation in a preliminary USAID/CHM meeting, which resulted in a decision to go forward with the task. EVAL submitted a draft SOW for the task to USAID, but had not received approval during the period. Nevertheless, the team held a couple meetings with CHM to lay the groundwork for engagement once the SOW is approved.

A 4: Analysis Graduation/Transition Criteria for USAID Assistance to GOC Institutions USAID did not develop the SOW for this assignment during the period. Accordingly, work will not begin until next quarter.

4. CHALLENGES ENCOUNTERED AND PROPOSED

Efforts to begin the 118/119 assessments are behind schedule due to difficulties recruiting a team, and the Public-Private Partnership assessment is behind schedule due to delay in USAID approval for the SOW.

6. LESSONS LEARNED

EVAL’s design relies heavily on the use of Colombian consultants. During the first six months we have learned the following about how the consultant market functions:

The community of proven short-term consultants and technical experts in Colombia is relatively small;

Experts frequently move in and out of organizations that they might be asked to evaluate, potentially making them reluctant to alienate the subjects of their analysis;

To survive, consultants must juggle many assignments at once, making it difficult for them to commit to blocks of time working exclusively on a particular task;

Colombian labor law does not favor working extended days in the office of a firm for which a consultant works;

More “senior” consultants are often reluctant to go to the field for evaluations;

Most experts are unaccustomed to contact with clients during the work period, or receiving technical direction over the nature of the work – preferring to return with a completed product without client interference;

Deadlines are often viewed as being flexible; and Report, language, and presentation standards differ from what USAID anticipates.

Fortunately, we have been able to access talented consultants through CNC’s excellent networks and have been successful in helping them accept EVAL product standards, even thriving under our very hands-on approach to quality management. However, we have learned that EVAL must bend more with respect to timing. We must be patient with respect to deadlines and understand the need of consultants to juggle multiple tasks simultaneously. In their capacity as consultants, Colombian law does not allow any one employer to dictate their time with as much precision or with as many as full-time hours as we would prefer.

Accordingly, in future, we will extend the overall duration of our work cycle, while demanding products from consultants more in advance of USAID due dates than we have to date. We are still exploring how to ensure fully unbiased analyses, even if the results might alienate organizations and individuals being evaluated.

7. PLANNED ACTIVITIES FOR THE NEXT QUARTER

No major administrative tasks are anticipated in the next quarter. Technically, we will:

Finish work on performance evaluations of:

o Land Mines; and o ADAM/MIDAS;

Continue work on the o Performance evaluation of PROTIERRA by conducting the second round of field work; and o Institutional assessment of the Center for Historical Memory by submitting the pre-diagnostic study and beginning the full analysis;

Begin work on the following assessments:

o Tropical forestry and biodiversity pursuant to Sections 118 and 119; and o Public/Private Partnerships;

Develop final SOWs and most likely begin work on:

o Graduation/Transition criteria for USAID assistance to GOC institutions; and o Performance evaluation of the Public Policy Project; and

Pending USAID approval, provide technical assistance to CNC to increase its survey capability.

8. FACTORS INFLUENCING (NEGATIVELY OR

POSITIVELY) PROSPECTS FOR ACHIEVEING RESULTS

No negative factors were noted.

On the positive side, MSI’s main subcontractor, CNC, demonstrated a remarkable ability to field evaluation teams, at one point simultaneously fielding seven teams on three different evaluations to a wide range of challenging locations – all without a significant complication.

9. REQUIRED OR RECOMMENDED CHANGES TO THE

QUARTERLY REPORT #4:

1 JANUARY – 31 MARCH 2014

APRIL 2014

QUARTERLY REPORT #4

1 JANUARY– 31 MARCH 2014

200 12th Street South Suite 1200 Arlington, VA 22202, USA

The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency

USAID/Colombia EVAL Quarterly Report #4 i

USAID/Colombia EVAL Quarterly Report #4:

1 January– 31 March 2014

USAID/Colombia EVAL Quarterly Report #4: 1 January 2014 – 31 March 2014

1. Summary

2. Introduction

2.1 Activity Background

2.2 Quarterly Report Format

3. Progress Against Work Plan and Results Achieved

4. Challenges Encountered and Proposed Solutions

6. Lessons Learned

7. Planned Activities for the Next Quarter

8. Factors Influencing (Negatively or Positively) Prospects for Achieving Results

9. Required or Recommended Changes to the Work Plan

USAID/Colombia EVAL Quarterly Report #4 1

Work proceeded basically according to schedule, apart from delays required to accommodate USAID’s planning processes and due to delays associated with EVAL’s procuring services for the performance evaluation of the Public Policy Project. There were no significant obstacles to successful implementation.

Administratively, during this period MSI completed a competitive bidding process to add additional subcontractors to a short list for future work and hired its new Contracts Manager, Liliana Botero.

Technically, EVAL’s portfolio shifted more towards assessments, as it began work on the Public Private Partnerships Assessment and the Institutional Analysis of the National Center for Historical Memory and continued work on the 118/119 Assessment while producing several draft SOWs for the Transition Criteria Assessment. On the performance evaluation side, EVAL handed in the final evaluation report for ADAM/MIDAS, continued working on PROTIERRA, completed the SOW for the Public Policy Program, and began work on an SOW for the rapid evaluation of the institutional strengthening component of ACIP as an interim effort to provide data in advance of the mid-term ACIP impact evaluation. Dates for the ACIP and CSDI impact evaluations were rescheduled to 2015 to improve the utility of the mid-term and final data collection waves.

Management Systems International (MSI) is the prime contractor for the Evaluation and Analysis for Learning (EVAL) Project (AID-514-C-13-00003). MSI works with Colombian subcontractors to access the broad range of skills required to meet USAID/Colombia’s needs in providing the following products:

Impact evaluations;

Performance evaluations; and Assessments.

The five-year activity—running until May 2018—is based in Bogotá, with access to field offices throughout the country via our main subcontractor, Centro Nacional de Consultoría (CNC). It is a demand-driven contracting vehicle that enables USAID/Colombia’s Program Office to obtain evaluations, assessments, and related planning and analytic services for programmatic learning, and to support Mission accountability and the needs of the Mission’s Technical Offices.

2.2 Quarterly Report Format

This Quarterly report is organized by the management issues most important to USAID, as detailed in Section F.3.A of the EVAL contract.

USAID/Colombia EVAL Quarterly Report #4 2

ACHIEVED

The main administrative task for this quarter was completing a competitive process to enable EVAL to access a broader range of Colombian firms to work on products. The following firms were added to our roster:

Economía Urbana Fedesarrollo G Exponencial Fundación Evaluar Oportunidad Estratégica.

Technically, EVAL’s portfolio is shifting to having a greater emphasis on assessments, as contemplated in last year’s contract modification, and as detailed below. EVAL worked on two assessments (118/119 and Public Private Partnerships) in the period, and worked with USAID to plan two more. During this quarter, EVAL completed one performance evaluation (ADAM/MIDAS), continued working on another (PROTIERRA), and began work on a third (Public Private Partnerships). USAID agreed to postpone the dates of the mid-term data collection waves for the impact evaluations (ACIP and CSDI) to enhance the utility of the data gathered. As part of this change, USAID wanted some relatively quick analyses performed on focused aspects of those programs in 2014. Early in the next period, EVAL will present a draft SOW to examine the policy coordination component of ACIP. After some discussion, USAID opted to use another contracting mechanism to conduct analysis of dynamics in the CSDI areas.

The following sections describe activities for each product during the period, relative to the second work plan.

IE1: CSDI–Colombia Strategic Development Initiative

Anticipated Actions Anticipated

Accomplishments Due Date Status: 31 Mar 2014

Discuss new dates for evaluation

USAID considers revising dates Feb 2014 Completed (dates shifted to 2015)

IE2: ACIP: Afro-Colombian and Indigenous Program

Due Date Status: 31 Mar 2014

Discuss new dates for evaluation

USAID considers revising dates Feb 2014 Completed (dates shifted to 2015)

USAID/Colombia EVAL Quarterly Report #4 3

PE1: ADAM/MIDAS

Accomplishments Due Date Status: 31 Mar 2014

SOW approval USAID approves Oct-13 Completed

Collect data Data ready for analysis Dec-13 Completed

Analyze data Analysis completed for report Dec-13 Completed

Draft report and prepare debriefing

Draft final report submitted;

debriefing held for USAID

Jan-14 Completed

Final report drafting Final report submitted to

USAID

Feb-14 Completed

PE 2: PROTIERRA – People-to-People Resolution of Land and Natural Resources Conflicts in Colombia – gender component

Anticipated Actions

Accomplishments

Due Date Status: 31 Mar 2014

During this period EVAL will complete the SOW, conduct outcome mapping and begin data collection

Interim products completed Nov 13 Completed, as well as additional deliverable on case study methodology.

PE 3: Public Policy Project

During this period, EVAL drafted the SOW for this evaluation, which was approved by USAID.

EVAL solicited proposals from its subcontractors to complete the product.

A1: Tropical Forestry and Biodiversity (FAA118and119) Analyses

Due Date Status: 31 Mar 2014

Report drafting Draft report submitted Mar-14 Submitted 1 April

A2: Analysis of USAID Experience with Public/Private Partnerships

Due Date Status: 31 Mar 2014

SOW finalized USAID approves SOW Feb-14 Completed

Interviews, map collection;

focus groups; case study data

All data collected for analysis May-14 Ongoing*

USAID/Colombia EVAL Quarterly Report #4 4

Draft report written;

debriefing prepared

Draft final report submitted;

debriefing for USAID

Jun-14 Not completed

*Note: Initially, USAID required more time than anticipated to complete the SOW.

Subsequently, gaining USAID approval for the Public Private Partnership sites to be visited has proved time consuming. Thus, the delivery date for the draft report has had to be adjusted to June, 2014.

A3: Institutional Assessment of National Center for Historical Memory (CNMH) – not included in relevant work plan

Production was delayed significantly as USAID has had to align its interests in an assessment with CNMH’s interest in receiving services to prepare them to make a proposal to the GOC for institutional restructuring. We expect that this will likely be resolved in the next quarter.

Nevertheless, during this period EVAL:

Submitted and revised the Pre-Diagnostic Report Identified and recruited technical team to conduct the work Gathered additional data, including from OIM, to support the analysis Conducted a Team Planning Meeting to begin work and drafted a detailed work plan In early March, all work was suspended, pending agreement between USAID and CNMH on the scope of the task Accordingly, all deliverables had to be rescheduled As of the end of the period, details were still being worked out between USAID and CNMH over the scope of the work.

A4: Analysis Graduation/Transition Criteria for USAID Assistance to GOC Institutions – not included in the relevant work plan

USAID and EVAL had several meetings and produced a series of three drafts of the SOW, which was not finalized during the period. It has proven to be a time consuming – but productive – design process.

Other: Certificate in Evaluation Course Keri Culver prepared, and is now delivering, MSI’s Certificate Course in Evaluation.

Classes run for five weeks through Semana Santa, followed by a five-week practicum.

In this period, teams of three to four students are performing supervised evaluations for local NGOs. Students return at the end of May for five more weeks of classes on advanced topics, meeting every Wednesday morning at a neighboring university.

Attendees are from EVAL, its subcontractors, GOC (including DNP, Regalias), universities, and frequently used consultants.

USAID/Colombia EVAL Quarterly Report #4 5

4. CHALLENGES ENCOUNTERED AND PROPOSED

USAID is embarking on an innovative use of institutional assessments to inform Mission decision-making regarding reducing USAID presence in Colombia. Working with CNMH we learned that GOC institutional priorities sometimes vary from those of USAID. Achieving alignment, in this case, required a series of preliminary meetings, drafting a full Pre-Diagnostic report, fielding concerns from CNMH, and an additional meeting between USAID and CNMH leadership to fully understand the intersection of interest. In this case, it was for USAID to combine an institutional assessment with conducting analyses to assist CNMH prepare for a restructuring bid. USAID and EVAL should bear this experience in mind when planning additional GOC institutional assessment work and the broader Transition Criteria development.

Work with GOC institutions must incorporate their corporate objectives while meeting the requirements included in USAID product specifications.

6. LESSONS LEARNED

None.

7. PLANNED ACTIVITIES FOR THE NEXT QUARTER

No major administrative tasks are anticipated in the next quarter. Technically, we will:

Finish work on the:

- Assessment of Section 118/119 Assessment of Tropical Forests and Biodiversity; and

- Assessment of Public/Private Partnerships.

Continue work on the:

- Performance evaluation of PROTIERRA by conducting the second round of field work; and

- Institutional assessment of the Center for Historical Memory by resuming work and conducting analysis.

Begin work on:

- Transition Criteria Assessment (including finalizing the SOW); and

- Performance evaluation of the Public Policy Project.

Develop final SOW and likely begin work on the rapid performance evaluation of the institutional strengthening component of ACIP.

USAID/Colombia EVAL Quarterly Report #4 6

POSITIVELY) PROSPECTS FOR ACHIEVING RESULTS

No negative factors were noted.

9. REQUIRED OR RECOMMENDED CHANGES TO THE

WORK PLAN

The upcoming work plan (#3: 1 April to 30 September 2014) will include products related to Transition Criteria and work with CNMH.

QUARTERLY REPORT #5:

1 APRIL – 3O JUNE 2014

JULY 2014

QUARTERLY REPORT #5

1 APRIL– 30 JUNE 2014

200 12th Street South Suite 1200

The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency

USAID/Colombia EVAL Quarterly Report #5 i

USAID/Colombia EVAL Quarterly Report #5

1 April– 30 June 2014

1. Summary

2. Introduction

2.1 Activity Background

2.2 Quarterly Report Format

3. Progress Against Work Plan and Results Achieved

4. Challenges Encountered and Proposed Solutions

6. Lessons Learned

7. Planned Activities for the Next Quarter

8. Factors Influencing (Negatively or Positively) Prospects for Achieving Results

9. Required or Recommended Changes to the Work Plan

USAID Colombia EVAL Project Semiannual Work Plan #5 1

Work proceeded basically according to schedule, with the level of current and anticipated work ramping up dramatically by the end of the quarter. There were no significant obstacles to implementation.

Administratively, MSI is directly contracting for experts more than originally contemplated, while continuing to contract for most work through its main subcontractor. Relatively soon, a contract modification will be required if USAID agrees to support this moderate shift in implementation approach.

Technically, EVAL’s portfolio continued to be weighted towards the Assessment CLIN, as:

EVAL submitted the draft Public Private Partnerships Assessment Report and began a second draft based on partial USAID feedback; revised the draft 118/119 Assessment report; delved more deeply into the institutional analysis…