Attach_23-Measuring_Change_in_Market_Systems.pdf

Monitoring and Measuring Change in Market Systems:

The Systemic M&E Principles in the Context of the Kenya Market Assistance Program

By Luis E. Osorio-Cortes, Marcus Jenal and Margie Brand ca se s tu dy

Copyright © 2013 The SEEP Network

Sections of this publication may be copied or adapted to meet local needs without the permission of The SEEP Network, provided that the parts copied are distributed for free or at cost—not for profit. Please credit The SEEP Network and “Monitoring and Measuring Change in Market Systems: The Systemic M&E Principles in the context of the Kenya Market

Assistance Program.”

This report was produced for review by the U.S. Agency for International Development (USAID) under the Knowledge-Driven Microenterprise Development (KDMD) Project, implemented by the QED Group LLC (Contract #EEM-C00-08-00004-00). The views expressed are those of the authors and do not represent the views of the U.S. Agency for

International Development of the United States Government.

For any commercial reproduction, please obtain permission from:

The SEEP Network 1611 N. Kent St., Suite 610

Arlington, VA 22209 (tel): 1 202 534 1400 (fax): 1 703 276 1433

Email: seep@seepnetwork.org

Printed in the United States of America.

To access this publication online, visit www.seepnetwork.org.

Acknowledgements vi

Executive Summary vii

Introduction 1

Background 2

Theoretical Background 2

Overview of the Principles 2

Case Overview: The Kenya Market Assistance Program 3

Objectives 5

Methodology 6

Process 6

Structure of the Report 6

Discussion of the Methodology 6

Part 1: Application of the Principles on the Ground 7

Principle 1: Indirectness of Impact 7

Principle 2: Depth of Impact 9

Principle 3: Network-driven Change 11

Principle 4: Unpredictability 14

Principle 5: Sensitivity to External Signals 16

Principle 6: Information Deficit 17

Principle 7: Sustainability as Adaptability 20

Overarching Aspects 22

Part 2: Discussion of the Principles in Light of the Findings 24

Principle 1: Indirectness of Impact 24

Principle 2: Depth of Impact 26

Principle 3: Network-driven Change 27

Principle 4: Unpredictability 27

Principle 5: Sensitivity to External Signals 28

Principle 6: Information Deficit 28

TABLE OF CONTENTS

Principle 7: Sustainability as Adaptability 29

Part 3: Conclusion 30

Usefulness of the Principles as an Analytical Framework 30

Suggestions for Future Work 30

Final remarks 30

Annexes 32

Annex 1: Overview of Indicators for Depth of Impact 32

Annex 2: Example of a Results Chain Indicating Indirectness of Impact, 34 Depth of Change, and Network-driven Change

Annex 3: Benchmarking Tool 35 v

ACRONYMS

ASI Adam Smith International

CODA Cotton Development Association

DFID UK Department for International Development

FHI 360 Family Health International 360

FIT FIT Resources Ltd.

ICT Information Communication Technologies

KGT Kenya Gatsby Trust

KMT Kenya Markets Trust

KRT Knowledge and Results Team

M4P Making Markets Work for the Poor

MAP Market Assistance Program

MaFI Market Facilitation Initiative

M&E Monitoring and Evaluation

NGOs Non-Governmental Organizations

R&D Research and Development

SEEP Small Enterprise Education and Promotion Network

SNV Kenya Netherlands Development Organization Kenya

USAID United States Agency for International Development vi

ACKNOWLEDGEMENTS

We would like to express our deep gratitude to the organizations and people who made this case study possible with their trust, support, experience, time, patience and openness.In particular, to Jeanne Downing, Kristin O’Planick, Jennefer Sebstad and Rachel Gomes from USAID , and Kathryn Kelly from QED for their technical inputs and patience.

We would also like to thank USAID for it’s generous financial support which made this study possible. We thank SEEP for their ongoing support to MaFI and the Systemic M&E initiative . Last but not least, we would like to thank all the great people we interviewed and who, despite their extremely busy schedules, gave us all the time we needed to obtain the information that constitutes the essence of this study, namely:

Name Function

Mike Field Portfolio Adviser

Richard Waddington Portfolio Director

Emmah Odundo Regional Manager - Kisumu

Chiranjibi Tiwari Portfolio Manager - Water Access

Francis Muema Portfolio Manager - Cotton

Violet Omenyo Intern - Cotton

Susan Maina Portfolio Manager - Inputs and Aquaculture

Clement Tulezi Portfolio Manager - Media

Irene Angwenyi Media Officer

Wanjiku Kimamo Operations Director

Judy Odongo Knowledge and Results Manager

Peter Mbiyu Knowledge and Results Manager

Hannington Isiaho Intern - Knowledge and Results

Julio Disouza (not MAP staff) Managing Director – Farmshop (agro-input dealer)

Rachel Brooks (not MAP staff) Managing Director-Echo Mobile vii

A basic premise of systemic M&E is that markets are complex adaptive systems that continually evolve and adapt.

To be effective, M&E frameworks for inclusive market development must embrace systemic thinking and complexity science as well as cultivate flexibility and organizational learning. Following this new premise, this case study presents arguments for using new basic principles to design and manage appropriate M&E frameworks.1

This paper evaluates the validity and usefulness of seven principles for appropriate design and management of systemic M&E frameworks. It puts them to the test in the context of the Market Assistance Program (MAP) in Kenya and its institutional host, the Kenya Markets Trust (KMT). MAP’s main objective is to improve the income growth of poor people using a facilitation approach and catalytic interventions that target the underlying systemic constraints that hinder their participation in selected markets, such as cotton, dairy and aquaculture.

The following list describes the seven principles that were exposed to the context of MAP and some of the main lessons learned and insights gained from each principle:

Principle 1: Indirectness of Impact. This principle forms one of the basic premises of MAP—that systemic change cannot be directly delivered by a development initiative from the outside but needs to emerge from within the system. All MAP staff members agree with this principle and are implicitly using it to inform their strategies. However, three important issues remain unresolved:

• The difference between direct and indirect beneficiaries becomes practically useless in the context of facilitation of systemic change;2

• Despite this, donors are still very interested in reporting on direct and indirect beneficiaries;

• However, counting indirect beneficiaries is still a technical challenge.

This creates a challenge for MAP’s M&E framework, as the information needed for management and learning is different from the information required by donors. MAP has created a “twin-track M&E system” in order to resolve this tension.

Principle 2: Depth of Impact. The concept of depth of impact was widely accepted by the program staff and is also reflected in the program’s theory of change and M&E framework. The program is giving strong emphasis to behavioral aspects of the market actors but also looks beyond the individual market actors and their business models to find patterns that are emerging in the wider system. The importance of depth of impact manifests in the numerous and varied ways that MAP is monitoring and measuring results. The teams are looking for a whole range of indicators and signals that tell them that change is occurring at deeper, more structural levels of the system. However, there is still pressure from donors to focus on reporting impacts in the form of superficial changes, such as increased income or numbers of jobs.

Principle 3: Network-driven Change. The program works with collaborators who have potential to produce changes that “reverberate” throughout the system. However, change is driven by the collaborators’ networks (for example local input retailers), not by them as individuals. MAP staff members are learning about the importance of understanding

1. An M&E framework is the set of approaches, processes, structures, procedures, methods, and tools that sustains the operation of monitoring and evalua-tion in a given program. In this paper, the term M&E framework is used as an equivalent of M&E system to avoid confusion with the word system, which is used in reference to markets.

2. There are two reasons for this: First, facilitators engage key actors who can drive change within the system. These actors can benefit from this interaction (e.g., through a subsidy to pilot a new business model or technology), but their function is not to receive a benefit from the facilitators but to collaborate with them to transform the system (this is why the term collaborator is proposed). Second, there are multiple feedback loops and synergies whereby the so-called beneficiaries benefit from an improved system, but they also benefit the system through their everyday decisions and attitudes (the typical dynamic agent-structure). Even the word beneficiary suffers in the context of facilitation of systemic change.

EXECUTIVE SUMMARY

viii the structures and dynamics of networks and realizing that market actors are not just “dots on a market map.” As a consequence, the selection of collaborators who can mobilize their own networks and align them with the objectives of the program is crucial. MAP staff members create conditions for self-selection of key collaborators and pay attention to their influence on their own networks. However, MAP’s M&E framework does not yet have procedures or tools to measure network structures and dynamics. A lot remains in the subjective or intuitive domain, linked to the people who interact with the networks on a regular basis.

Principle 4: Unpredictability. Practically all MAP staff members recognize this principle and are trying to overcome the challenges posed by it through an organizational culture that rewards faster learning, course correction, and adaptation to unexpected circumstances. MAP is moving away from long-term strategic analysis and planning of fixed indicators and goals toward more flexible strategies, plans, and measurement. It is also putting in place one of the cornerstones of iterative learning and effective response: the creation of spaces for the teams to discuss and re-evaluate interventions, exchange lessons quickly and reflect about progress and challenges. They are also working together to make sense of emergent patterns through an organizational culture that promotes trust and values mistakes and problems as opportunities for success.

Principle 5 - Sensitivity to external signals: this principle has shown to be an important determinant of MAP’s facilitation activities. When MAP staff members approach market actors, these react in different ways depending on their perceptions, preconceptions and interests. How a program uses its own identity is, thus, very important and needs to be tailored to the specific situation. This finding challenged the original hypothesis that minimizing the possibility of being perceived as an influential subsidy-giver or helper was the only recommended strategy for facilitators. The key lesson is that successful facilitators can manage the signals they send to promote self-selection and determine the honesty and commitment of strategic actors who are willing to collaborate. MAP teams are learning that such honesty and commitment are indicators of positive trends and have even stopped collaborating with markets actors when these traits are lacking. However, the identification of such trends still depends on staff member’s “gut feelings”.

Principle 6: Information Deficit. MAP teams strive for continuous learning through observation, discussion, and analysis of the information produced by the M&E framework. At the same time, the framework is updated whenever necessary to reflect the needs of the staff and is, hence, co-evolving with the staff’s knowledge and understanding of the system. MAP staff members are learning the importance of doing fewer detailed, hands-off analyses (“cold” analyses) and more “full immersions” in the system, which entail closer and more frequent interactions with stakeholders to actually experiment with new ideas. Consequently, they attach a high value to information from field observations. However, once a MAP team agrees that a given type of new behavior change is important, it is added to the formal M&E framework to be measured regularly.

Principle 7: Sustainability as Adaptability. The adaptability of the market system and the adaptability of the market actors are at the center of the program’s aims. This is evident in their efforts to increase the capacity of market actors to continue changing their own systems in the eventuality of shocks or new trends. The operationalization of this principle and particularly the measurement of adaptability seem to be in a rather early stage and are not formally articulated in the program documents or in the functions or roles of the different units.

An aspect that relates to more than one principle is the evolutionary nature of the M&E framework. MAP has learned the importance of starting with a framework that is, on the one hand, manageable and simple and that evolves organically according to the needs and skills of the teams. On the other hand, the framework must stay focused on relevant aspects of the dynamics of the market system. There are two processes of co-evolution taking place in parallel. In one process, the M&E framework evolves as team members gain a better understanding of their own work and the market system; in the other, the M&E framework evolves with changes in the market system itself.

ix

One of MAP’s most interesting M&E strategies is to build the capacity of key market actors to become effective sources of relevant information both for themselves and for the program (instead of MAP team members collecting all the monitoring data themselves). A collaborator of the program can be not only an effective driver of change, but also an effective provider of useful information about changes in the system.

The case study shows high applicability and relevance of the principles in the context of MAP and provides concrete examples of how an innovative program and staff committed to facilitation, systemic change, and organizational learning are applying the principles in their own organization and with market actors.

This report will contribute to how the principles are defined, explained, and communicated and, ultimately, to how they are applied by other practitioners, donors, and policymakers in inclusive market development programs in different contexts. Consequently, it makes sense to find ways to document more cases that can lead to the identification of patterns in the application and benefits of the principles in different contexts.

Introduction

MaFI and the MaFI-festo

The Market Facilitation Initiative (MaFI) is a working group of the SEEP Network that focuses on the question of “how to become effective facilitators of inclusive market development initiatives.” In other words, the group’s focus is on how to create appropriate conditions—at the lowest costs and effort—for public and private market actors to make their systems work better to reduce poverty on a larger scale and more sustainably.

One of the main lessons emerging from several MaFI discussions is that becoming an effective facilitator is not just a technical challenge. It is also an organizational and political challenge, in the sense that project staff members are influenced by rules, procedures, and incentives created and enforced by their own organizations and donors, which hamper the ability to facilitate effectively.

In response to this insight, MaFI started a series of conversations that led to a set of proposals called “The MaFI-festo,” 3 to change the rules of the game to make international development cooperation more “facilitation-friendly.” The MaFI-festo focuses on four critical issues: (1) how facilitators work in the field, (2) balancing flexibility and accountability, (3) building the capacity of facilitators, and (4) what and how we measure change. The fourth issue challenges the use of M&E frameworks based on linear and mechanistic paradigms when it comes to assessing the impacts of donors and NGOs on market systems.

The Systemic M&E Initiative

In 2012, a series of online discussions, an e-consultation, and a plenary during that year’s SEEP Conference brought together the voices and experiences of hundreds of SEEP and MaFI members. It created significant momentum around one of the issues proposed by the MaFI-festo: what changes we pay attention to and how we measure them.

These events—which took place thanks to the support of USAID, SEEP, and FHI 360—led to the production of a synthesis paper4 that proposed seven principles to inspire better design and more appropriate management of M&E frameworks that embrace and leverage the systemic and complex nature of markets.

Among the messages that came across clearly from the people who participated in the discussions during the 2012 conference and those who kindly provided comments and criticisms of the paper, two in particular inspired this case study:

• Continue reflecting upon the ideas in the paper to improve them, clarify them, and increase the agreement around them.

• Document examples where the principles and ideas proposed in the paper are making a real difference in the ability of development organizations to monitor and adapt to change and assess the impacts of their interventions.

In addition to being a direct response to the above requests, this paper is also a step toward the long-term vision of promoting appropriate applications of complexity and systems thinking to development practice and policymaking, in particular from the perspective of monitoring and evaluation of inclusive markets development.

3. http://slidesha.re/mafifesto2

4. http://www.seepnetwork.org/monitoring-and-measuring-change-in-market-systems---rethinking-the-current-paradigm-resources-937.php

Background

Theoretical Background

A basic premise of systemic M&E is that markets are complex adaptive systems. In order to properly interpret the principles and analyze them in the context of market systems, a basic knowledge of complex systems theory is helpful.

Complex adaptive systems have a large numbers of components, often called agents or actors, that interact, learn, and adapt to changes around them. In the case of markets, these agents can range from individuals and small, informal groups to highly sophisticated, large public institutions and private corporations; even community-based organizations and grassroots organizations can play important roles in these systems. Other common market actors are buyers, processors, cooperatives, intermediaries, wholesalers, retailers, consumers, technical training and R&D institutions, transporters, agricultural extension agencies, consultants, banks, microfinance organizations, etc.

The actions of agents in complex adaptive systems usually depend on the signals they receive. This means that they are constantly reacting and adapting to their environment and, in turn, sending out signals that prompt other actors to react and adapt. This leads to intricate feedback processes and further changes in the agents and the structures that connect them over time. These changes are usually adaptations that improve the performance of specific agents.5

Complex adaptive systems exhibit a number of important characteristics that are relevant for systemic M&E:6

• The interconnections, interactions, and interdependencies among heterogeneous agents can lead to situations where minor changes produce disproportionately major consequences. Conversely, big interventions might have no significant effect.

• The system has a history, and the past is integrated with the present, essentially because the actors’ strategies have co-evolved with one another and with their environment.

• The interactions between the agents lead to emerging properties that can only be observed on the level of the whole system, not when looking at an individual agent.

• Solutions cannot be imposed; rather, they arise from the circumstances and compromises between agents.

Therefore, in the context of complex systems, we talk about emerging practice, rather than good or best practice.

• Adaptive agents react to their context and to each other. Without a central power, they self-organize into functional units.

• Though a complex system may, in retrospect, appear to be ordered and predictable, hindsight does not lead to foresight because the conditions constantly change.

Overview of the Principles

This section provides a brief overview of the seven systemic M&E principles. For a more elaborate summary, please refer to the systemic M&E synthesis paper.7

Principle 1: Indirectness of Impact. When a systemic approach is used, the endgame is to improve how the system works so that it enhances the livelihoods of marginalized people and increases productivity and efficiency on an ongoing basis. The objective, by definition, cannot be the direct delivery of inputs and services to the poor.

5. John H. Holland (2006). “Studying Complex Adaptive Systems.” Journal of Systems Science & Complexity 19: 1–8.

6. Marcus Jenal and Shawn Cunningham (forthcoming). “Gaining Systemic Insight to Strengthen Economic Development Initiatives--Drawing on Systems Thinking and Complexity Theories to Improve Developmental Impact.” Mesopartner Working Paper.

7. http://www.seepnetwork.org/monitoring-and-measuring-change-in-market-systems---rethinking-the-current-paradigm-resources-937.php

Principle 2: Depth of Impact. Changes in a system can be of different types according to how structural and long-lasting (how “deep”) they are. They can range from the most superficial changes, such as variations in incomes and number of jobs, to the deepest changes, such as those related to self-awareness (i.e., when the system questions its own nature and evolution). All types of change provide useful information, but an exclusive focus on the most superficial ones will have negative consequences on sustainability and scalability. Closer attention must be paid to deeper changes such as the creation of new relationships, increased access to information, and shifts in behavior and power dynamics.

Principle 3: Network-driven Change. A market actor is not a dot on a market map or a monolithic entity that can be managed by the most senior director, owner, or representative. Both public and private market actors are in themselves systems or networks with varying degrees of complexity. Facilitators must work with the most influential members of these networks (the collaborators) before they start driving changes in the system. Members of these networks are first-movers or innovators who try to do things in different ways to improve the system; this is why the networks are called “precursor networks.” Facilitators must work to promote the dissemination, copying, and adaptation of the successes or lessons generated by the precursor networks.

Principle 4: Unpredictability. The behavior and evolution of complex systems resist prediction; this undermines the effectiveness of the mechanistic and rigid tools and approaches that are predominantly used in market development initiatives. Flexibility, rapid learning, and effective collaboration between facilitators, NGOs, and donors are key requisites to deal with and navigate this uncertainty.

Principle 5: Sensitivity to External Signals. The moment an NGO or a donor declares its intention to intervene in a market system, both public and private actors adapt their behavior and strategies, seeking to maximize benefits in many different ways (e.g., for economic gain, reputation, and influence). Facilitators must be very careful about the signals they consciously or unconsciously send when they engage with market actors, design and implement strategies, and monitor and measure change.

Principle 6: Information Deficit. No matter how much we know about a market system, there will always be bits of information that we ignore that can have important consequences on how the system performs and evolves.

This challenges our ideas about how much should we know before we intervene in the system and begs for true participation (where co-creation is promoted), continuous learning, effective communication and collaboration among key stakeholders and partners, opportunism and flexibility throughout the design, implementation, monitoring, and evaluation.

Principle 7: Sustainability as Adaptability. The adoption of a systemic approach requires a shift in our understanding of the idea of “sustainability,” from the permanence of a given set outcomes through time to the capacity of a system to benefit from new opportunities and to minimize negative impacts through time. Therefore, it is necessary to develop M&E frameworks that can detect whether a system is building its capacity to adapt to shocks and new trends such as changes in consumer tastes and technology, the introduction of radical policies, natural disasters, and even socio-political revolutions.

Case Overview: The Kenya Market Assistance Program

The Market Assistance Program (MAP) in Kenya is a multi-donor-funded program that aims to reduce poverty in Kenya through the application of the Making Markets Work for the Poor (M4P) approach. The institutional home for the program is the Kenya Markets Trust (KMT). MAP is implemented by a consortium led by Adam Smith International (ASI), which includes KMT, SNV Kenya, and FIT Resources Ltd. The main donors of MAP are the UK Department for International Development (DFID), the Gatsby Charitable Foundation, and the Royal Dutch Embassy.

The origins of KMT can be traced to the establishment in 1991 of the Kenya Gatsby Trust (KGT), one of four African trusts created and supported by the Gatsby Charitable Foundation in London. After becoming the institutional home for MAP, KGT entered a year of significant organizational restructuring and strengthening of its strategic focus, after which it was rebranded as KMT.

MAP was launched in 2011 and is expected to run until 2018. It focuses its interventions on the economic sectors of cotton, water, input supply, dairy, aquaculture, and media. MAP has a program budget in excess of £23m (USD37m) for the years 2013–2018 and plans to expand to other sectors such as oil and gas, livestock, and seeds.

The program’s main objective is to improve the income growth of poor and disadvantaged people in selected market systems. Its theory of change is that it will reach this goal by implementing catalytic interventions that target the underlying systemic constraints that hinder the participation of poor people in said markets. The interventions are designed to realign the incentives, rules, relationships, and support services that shape the way markets work and to change the way poor people participate in and access markets.

MAP has been showcased on various occasions as a program leading innovation in systemic development and approaches to measure systemic change, which makes it an ideal candidate for this case study. MAP is organized into three teams:

• The Portfolio Team is headed by Mike Field, working in coordination with sector managers responsible for the individual sector teams. This team focuses on improving the functioning of selected market systems through the engagement of strategic market actors; raising awareness about market system approaches among influential stakeholders; and promoting policies that enable private-public partnerships.

• The Operations Team is headed by Richard Waddington and focuses on M&E, capacity building of facilitators, organizational learning and knowledge sharing. The Knowledge and Results Team (KRT) is part of the operations team and is responsible for the design and management of M&E processes in close collaboration with the sector teams.

• The Finance Team is headed by Richard Carter and provides financial guidance and oversight of budgets and expenditure. This team was not interviewed for the case study.

Role of the Knowledge and Results Team

The Knowledge and Results Team (KRT) is responsible for managing how KMT uses knowledge and learning effectively to achieve better results. Its goal is to operate innovative and effective approaches to knowledge management and results measurement for private-sector development programs. By combining both knowledge management and monitoring and results measurement in one dedicated team, KMT champions an innovative approach that it believes can become good practice for private-sector development programs in Kenya and other countries.

In summary, the main roles of KRT are as follows:

1. Design the knowledge and results frameworks and their supporting processes, tools, and products.

2. Act as an advisory unit to members of each portfolio team in applying the knowledge and results frameworks in their work. This recognizes that the portfolio teams are best placed to capture data and information as they are, in effect, “closest to the action.”

3. Ensure the quality of the information collected by the portfolio teams and project partners and the compliance with the quality standards built into the frameworks.

4. Act as a peer reviewer for each sector strategy, promoting reflection and challenging the sector teams about strategic direction and implementation performance.

5. Facilitate the dissemination of knowledge and lessons learned within KMT and among other change agents such as donors, governments, and NGOs (in collaboration with the communications and learning manager).

Objectives The objective of this case study is to explore the concrete applications, implications, and challenges of the seven principles proposed in the systemic M&E synthesis report. The case study describes how the principles manifest in the planning and execution of the monitoring and evaluation processes of the Kenya Market Assistance Program (MAP) and its host organization, the Kenya Markets Trust (KMT). The findings of this case study are relevant for NGOs, donors, and policymakers working in inclusive market development.

Methodology

Process

The case study followed a three-step process:

1. Preparation. Through a desk study of program documents, the case study team familiarized itself with the program and developed the plan and interview guidelines for the field study based on that knowledge.

2. Field study. The team spent five days with the program staff in Kenya, interviewing individual staff members and facilitating group discussions. A list of people interviewed can be found in the acknowledgments section at the end.

3. Report writing. Based on the findings of the desk and the field studies, the case study team prepared this report.

Feedback on draft versions of the report were solicited from MAP, as well as from the donor.

The focus of the three steps was very much concentrated on M&E issues. Specific strategic issues that were related to the principles but not to M&E in particular were not taken into account in order to keep the scope of the case study restricted to systemic M&E.

Structure of the Report

The report is structured around three main sections. Part 1 describes the application of the principles on the ground. It first looks into how the principles are manifested in the program teams’ assumptions and hypotheses of how change will happen through program interventions (manifestation of the principles in the theory of change). Second, it looks at how the principles manifest in how the teams observe and measure change, both in formal and in informal ways (manifestation of the principle in monitoring and results measurement). Third, it describes how the teams learn and adapt based on the logic of the principles (learning and adaptation in the program teams). This section comprises key findings and insights provided by the interviewees and program documents. In some cases, the findings and insights relate to more than one principle or to the principles collectively. For these aspects, a section on overarching aspects has been added at the end of this section.

Part 2 features a discussion of the principles based on the findings of the case study and the discussions with the team. It also includes the impressions of the case study research team on how the principles were understood by the program team members and how they related the principles to their own work.

Part 3, the final section, draws a conclusion by presenting an assessment of the value or usefulness of the principles as an analytical framework for this particular case study. It also includes a number of possible future activities to move the process of developing a systemic M&E framework forward.

Discussion of the Methodology

Due to time constraints, the phase dedicated to interviews and collection of data and other types of information was rather short but nevertheless very intense. The case study team collected large amounts of information and data from individual and group interviews that proved challenging to analyze, filter, and categorize due to the cross-cutting nature of many of the issues discussed.

There is the possibility of a confirmation bias, as two of the authors of this case study were also authors of the reports that proposed the original and revised versions of the principles. We tried to minimize this bias by adding a third member to the case study team, who helped to critically discuss the findings, and by soliciting feedback from the program and the donor on a draft version of the case study report.

Part 1: Application of the Principles on the Ground

Principle 1: Indirectness of Impact

Manifestation of the Principle in the Theory of Change

This principle clearly manifests in the program’s theory of change, as the program is not working with the target populations directly but rather on the parts of the system that can influence them. As Mike Field, MAP’s director of portfolio, put it, the program teams are looking for nodes in the system that will “reverberate change.” In other words, the program targets points that create knock-on or ripple effects beyond the program’s sphere of influence.

According to Field, these nodes are hardly ever the marginalized farmers. Instead, the work that the teams do is changing the environment around the farmers and other beneficiaries, with the assumption that the system will influence them to change their behavior. This assumption is then tested in the monitoring framework.

Good examples of how the program works can be found in the water, media, and cotton sectors. Chiranjibi Tiwari, the water access portfolio manager, said that the team is not bringing water directly to the people, which in practice would mean working directly on the water point. Instead, the team intervenes at a higher level, for example, by changing policies and promoting the participation of the private sector in water access.

Hannington Isiaho, the knowledge and results intern assigned to the media team, said that they see the ultimate beneficiaries as the listeners of the local radio stations. However, the interventions do not target the listeners directly, but instead target the radio stations. The team believes that when they motivate the radio station to engage more with its listeners, the listeners will eventually benefit by receiving the information they need. The team’s assumption is that radio stations are central to promoting changes in behavior, attitudes, and perceptions of the listeners. Their success, however, depends on how much the listeners trust the institution and whether they think it promotes issues that matter to them. So far, the radio stations have been more oriented toward the sponsors and selling them time to advertise their products and services. This was much more a one-way model of content provision without a focus on building the trust of the listener.

Also, Francis Muema of the cotton sector team supports an indirect intervention approach on the basis of logistics, but also because of power relations and distortion of incentives. Regarding logistics, Muema pointed at an obvious but often downplayed truth: If we are interested in impact at scale, development NGOs do not have the resources to replicate every possible successful solution across the market system on their own, let alone sustain the innovation required to keep the system competitive. Regarding power relations, he said that “even when you go directly to the community, you are actually interacting with an elite [group] who knows what NGOs are and want. Many of these people have become very good at ‘managing’ NGOs and have strong incentives to distort their reality or [misrepresent] the impacts of a program’s interventions.” Muema concluded that “the alternative to indirect support is not feasible.”

Manifestation of the Principle in Monitoring and Results Measurement

“How acceptable would the idea of indirectness of impact be to funders?” This was the question that Wanjiku Kimamo, the operations director, asked after she was introduced to this principle. The reason for the question was that she was sure that donors “want an absolute number or something that at the end of the day you can show: our beneficiaries were 200,000 farmers or 50 businesses, profitability of so many businesses and incomes of so many farmers.” In other words, the donors’ focus is still on the ultimate beneficiaries and the numbers created by the program. In contrast, the principle of indirectness of impact indicates that the focus of M&E has to shift from counting direct beneficiaries or assessing direct impacts at the beneficiary level to a broader view of change in the structures and dynamics of the market system.

Richard Waddington, head of the operations team and responsible for M&E, explained that MAP created a “twin-track M&E system” in order to resolve this tension. One track serves the needs of its funders for accountability, impact assessment, and reporting to the taxpayers in the donor countries. Another track serves the needs of the teams and is much more focused on internal learning to enable adaptive management and lay the foundations for achieving sustainability. He is convinced that “metrics around trust and the growth of a business don’t really resonate with taxpayers as much as creating jobs and increasing incomes do.”

MAP had lengthy and interesting discussions with its funders about the appropriateness of the indicators against which the program was going to be held to account, including job creation. “From our perspective,” Waddington said, “while the numbers of new jobs created is an easy-to-understand measure of program impact, this metric is not a reliable indicator of lasting change in the market system and for the program’s target groups in particular.” For Waddington, the real story is not told by these types of indicators; it lies further down the results chains, at the levels of the market system where the teams are trying to make change happen. If the program can achieve change at those deeper levels, the target population will be able to benefit indirectly. “Measuring and understanding what’s going on there is essential from a sustainability perspective,” he concluded.

Learning and Adaptation in the Program Teams

Several challenges are related to the monitoring and evaluation of indirect impacts, but two are particularly relevant at this stage: how to define direct and indirect beneficiaries, and how to attribute indirect change to the program activities.

There are no widely accepted definitions of direct and indirect beneficiaries in the context of systemic interventions.

Hence, there are no simple answers to the first challenge other than promoting ongoing debate, learning, and convergence about what these terms mean for different practitioners and policymakers. However, these definitions influence the ways in which the program goes about overcoming the second challenge. This was illustrated by Chiranjibi Tiwari, who used his experience in the water sector to explain, “If we are facilitating the system, we cannot have direct beneficiaries of our interventions. In that sense, it is really difficult to make the ultimate impact tangible.

But I also feel that it is possible to link those impacts to our work to some extent. For example, if we bring the private sector on board to provide water, contribute to tangible changes to accountability and liability norms, and improve water-quality standards, why can’t we say that the benefits [to the ultimate beneficiaries] really came from our program?”

Tiwari’s comments are very interesting because they confirm that embracing a systemic approach to market development entails the impossibility of claiming direct impacts. But they also show how this impossibility does not mean paralysis in terms of impact assessment or accountability. His experience tells him that it is obvious that certain changes in the system—that can be directly attributed to his team’s work—directly build the system’s ability to deliver good quality, affordable, and reliable water to marginalized populations beyond the sphere of influence of the program.

According to Richard Waddington, “Indirectness of impact is where the real impact of our program lies. Just counting the direct beneficiaries reached through our pilot interventions isn’t going to get us very far. We might get some good numbers to inform our funders and we might have better stories to tell, but the real interest is in how the overall system is changing—for example, how [are] crowding-in and imitation happening?” However, things are never as easy as they seem; Waddington confirmed that the different teams have reflected on the results chains and how crowding-in and copying may happen. But they are not yet at a point where they see sufficient systemic changes to really know how they are going to capture and document these. He thinks that the teams will only get answers once they see that momentum in the market is system building. Of course, this can only happen if the teams have a basic understanding of how complex systems behave and of the appropriate indicators and techniques to detect change when it takes place. According to Waddington, staff members are being trained to constantly think about what is happening in the wider system, beyond their immediate interventions and effects, and how they can get the information they capture back into their strategies and to all sector teams.

This discussion also shows that it is, in fact, very difficult to come up with a detailed set of results chains containing wider systemic change from the very beginning of the program. Mike Field said that this would entail “knowing how you would expect the change to happen.” Instead of investing too much effort in knowing or predicting the processes of change in detail, Field recommended that teams pin down some basic components of the results chains.

This means starting off with a relatively simple M&E framework and adding more layers when they are needed. On the one hand, this allows a co-evolution of the intricacies of the monitoring system with the capabilities and level of understanding of team members. On the other hand, it also helps to put the focus on the most relevant aspects of change at any moment of the program. As this aspect of an evolving monitoring system is relevant to all principles, we are further discussing it in the section of overarching aspects at the end of Part 1.

Principle 2: Depth of Impact

Manifestation of the Principle in the Theory of Change

To achieve systemic change, the program puts special emphasis on the “normalization” of desirable new behaviors of the market actors. Normalization is not just a result of an individual’s acceptance of new ways of thinking or acting and making such behavior routine. It goes much further, to entails a collective acceptance of the new behavior and the buildup of peer pressure to conform to it.

Linking back to the indirectness of impact principle, the program is paying special attention to changes at deeper level. Instead of focusing efforts on the most superficial changes, such as increases in income and number of jobs, it emphasizes the set of behaviors needed to make market systems more inclusive, productive, and efficient in the long run. The program asks why these behaviors do not now exist, why some individuals who try to adopt such behaviors are stigmatized, and what the program can do to improve the situation.

Mike Field offered the program’s goal of increased adoption of growth-oriented strategies in businesses as an example. These strategies are defined by better relationships between businesses and their clients, and the businesses’ awareness that to grow they need more loyal clients and suppliers. According to Field, if there is no business incentive to move past trading relationships into alliance-based relationships, then there is no way to get inclusivity.

Wanjiku Kimamo agreed with the principle and is convinced that the program needs to put its focus beyond income:

“I think that depth of impact is a good way of evaluating our programs because if we stop at income alone, there could be factors beyond our control that drive income either positively or negatively, and thereby we would be presenting superficial results. It’s not just about incomes. There is a reason why we are trying to understand if farmers started procuring inputs; if they are able to make the right decisions about those inputs; if they are adapting to new technologies they are exposed to; if they are learning new things from those technologies. So, it could be buying instead of recycling seeds or pruning or harvesting at a certain time or in a certain way. Those things will gradually lead to increased incomes.”

Manifestation of the Principle in Monitoring and Results Measurement

Information about what goes on at deeper levels of the market system can be used to determine whether the system is changing its trajectory toward a horizon of more inclusion, productivity, and efficiency, and whether it will keep this new course beyond the life of the program. The importance of depth of impact manifests manifold in how MAP is monitoring and measuring results. The teams are looking for a whole range of signals that indicate that change is occurring at deeper, more structural levels of the system. These signals or indicators at different levels of depth are both guiding the staff regarding what to look for when they are in the field and gradually being integrated in the monitoring and results measurement framework. The following are examples of the deeper structural levels that the teams are paying attention to (see Annex 1 for a more detailed description of these fields of observation):

• Behavior change

• Trust

• Loyalty

• Consumer awareness

• Business management patterns

• Participation in policy change and advocacy

• Relationships between the actors

• Perceptions and preconceptions of actors

• Knowledge flows

One example of behavior change in the cotton sector can be seen at the level of the ginners. The program is aiming to change their behavior from simply maximizing the margin in every individual transaction to a bigger focus on value addition and long-term investments. Mike Field explained, “Cotton does not have what I call proper processors; they have traders that process a little. So you can tell quite a lot from the ways in which they use and invest in machinery, the business decisions they make, how they treat their suppliers and buyers. Normally, they invest a lot of their effort towards lowering purchasing price and increasing selling prices and not in value addition.”

Loyalty and the repeated interactions between input retailers and farmers, or between farmers and buyers of their produce, is an important determinant of deeper change for the program. For example, the monitoring system looks at the number of times a farmer comes back to buy inputs from a particular retailer. Richard Waddington pointed out that the number of jobs can be very volatile and went on to say that “if you’ve come much further down in the system and you’ve seen smallholder farmers engaging with an agrovet, going back for repeated sales, and purchasing what they need for their farm, then those are indicators that they’ve got the income to make that investment and that they want to make it.”

For Chiranjibi Tiwari, it is clear that “we must look at the relationships among the actors as an important element of our systemic change, in addition to the income change and ultimate benefits. End results are important, but we need to see if there is a self-sustaining mechanism developed and working towards the goals. That must be monitored if we really want to say that this system is working and we are creating a sustainable system.”

In the media sector, perception and attitudes are also central, as Clement Tulezi, the sector team leader, explained: “A lot of what we try to measure [are] perceptions and change of attitude. Especially when you send a message over the radio that is what you are more interested in in the first place. If the perceptions and attitudes are not right, then the likelihood that something else is going to change is very low. We are interested in more SMS, calls, and attendance to live events, but we are [also] interested in what is beneath all that. That is more important to us than the numbers alone.”

These examples show the awareness that the MAP teams have about the principle of depth of impact and its applications in different parts of the program. It is interesting to see how, despite the differences in the ways each interviewee described the indicators used, there is in all of them a committed drive to have impacts beyond the top-level numbers related to jobs and income.

Learning and Adaptation in the Program Teams

The principle of depth of impact has been very much accepted by the teams as what the program seeks to achieve.

However, there is still pressure from the donor side to report…