Logic Model Using a Logic Model to Build an Evaluation Plan Handout Westat CIPP 2021.pdf

Using a Logic Model to Build a Strong Evaluation Plan

Center to Improve Program & Project Performance (CIPP)

Authors:

Jennifer Schaaf, Ph.D.

Margaret Gillis, Ph.D.

Debra Shaver, Ph.D.

Nancy Hartman, Ph.D.

Project Director:

Jill Lammert, Ph.D.

© 2021

Using a Logic Model to Build a Strong Evaluation Plan

Part 1: Why link the evaluation plan to the logic model?

Training Objectives

Participants will gain increased understanding of

The purpose of a logic model

The purpose of an evaluation plan

Benefits of linking the evaluation plan to the logic model

Purpose of a Logic Model

Purpose of an Evaluation Plan An evaluation plan outlines the data collection and analysis methods that will be used to:

Graphically illustrate project investments, planned activities, and expected outcomes

Illustrate the hypothesized causal flow from inputs and activities to short-, medium- and long-term outcomes

Provide a roadmap for project planning and implementation

Provide a blueprint for a strong evaluation plan

Assess how well project components are being implemented

Understand the extent to which outcomes are being achieved

Provide formative feedback for continuous improvement

Provide summative information for reporting project results to funders and stakeholders

Evaluation Plan Components

Benefits of Aligning the Evaluation Plan to the Logic Model

Provides a clear indication of what to measure

Helps to plan and use evaluation resources efficiently

Assists with establishing the sequence of evaluation events

Helps to target assessment of key interim outcomes to determine if the project is on track to reach longer-term outcomes

Allows the use of evaluation results to examine the hypothesized relationships depicted in the logic model

Summary

A logic model graphically illustrates the project’s inputs, activities, and expected outcomes

A logic model can be the blueprint for a strong evaluation plan

Linking the evaluation plan to the logic model helps to target evaluation resources to the outputs and outcomes the project is aiming to achieve

Part 2: Logic Models 101 What are Logic Models?

Logic models are just a graphical representation of your program, from the resources (inputs) and activities that will take place, to the deliverables (outputs) and goals( outcomes) that the program will produce. The graphic usually takes the form of a table, so the information can be filled-in.

Evaluation questions

Expected outcomes

Performance targets

Methodology

Timelines and responsibilities

When developing a new program or trying to figure out what aspects of an existing program need to be evaluated, a logic model may be helpful in showing the various components, activities, and goals of the program.

There are three main reasons to consider developing a logic model:

1. Program stakeholders are requesting details on your measurable goals and objectives, and you are looking for ways to visually display this information.

2. A funder (e.g. grantor) requires a logic model as part of an evaluation plan in your proposal. So a logic model is always required by a funding agency.

3. You want to see a quick snapshot of how your program operates and what it intends to accomplish.

Logic Model Elements Sample Logic Model Our scenario:

Your district or school has as a strategic goal to offer improved services to students with disabilities and their families, especially deeply involving parents in those opportunities.

Inputs

Just as it says, “Inputs” are what goes into a program. When trying to determine your program inputs, consider the critical resources that are required to make your program successful. For example, partners, project team members, advisory groups, volunteers, facilities or equipment. For our scenario, we definitely want families, we need funding, Parent Training and Information Center staff, technical assistance providers and educational staff.

The next core element in a logic model is “Activities”.

Many programs and projects have a plan/timeline in place that outlines the high-level activity areas and tasks. When thinking about what activities to include in the logic model, reference those high-level activity areas.

For example, if one of the outcomes of your program is to increase the parental understanding and involvement in their child’s special education program, your activity column may include “Provide online and in-person training for parents” and not necessarily something as specific as ‘schedule parent training sessions twice per semester.’

Scheduling meetings is a task that will be reflected in your detailed plan, and the “providing of online and in-person training” is a high level activity area that encompasses that – and is more appropriate for the activity column of a logic model.

TIP Since inputs are directly related to the ‘Activities’ of your program, it is helpful to develop an initial list of inputs, then fill in your activities and then go back to the inputs to make sure that you didn’t miss any resources to achieve those activities.

Logic models are not meant to replace a detailed project plan, they are just a way to link your resources to your activities and then to your goals in a succinct manner.

The third critical element in a logic model are the “Outputs”

After you have consensus on the inputs and activities of a program, think about the direct measurable results of those activities, which will be your program outputs. Outputs can come in a variety of forms. Your program may be producing marketing or awareness materials, which can be simply stated in the output column. In our scenario, the output will simply be the number of online and in-person training sessions” or the number of parents participating in training

TIP When determining whether a result is an output or an outcome, ask yourself “Is this result completely controllable by the program staff? If it is, then it is an output. In the example, the number of online and in-person training sessions is completely controllable by the program staff, so that would be an output. However, whether a parent participates in a training session or increases their knowledge of special education programming depends to a great extent, on the parent, so since it is not completely controllable by the program staff, it is an outcome.

The final core element of a logic model is “Outcomes” but it is important to note that outcomes are typically presented broken into 3 types of outcomes: short, medium and long-term.

As expected, short-, mid-, and long-term outcomes differ by the time span in which we expect to achieve them. Typically, the short-term outcomes are more specific and/or more directly measured, and then mid- and long-term outcomes are broader. For example, when looking at learning-focused outcomes, short to mid-term outcomes can be characterized by an increase in knowledge, skills and abilities when participating in activities (short-term), then a gradual shift in behavior or performance occurs (mid-term), followed by a complete change in behavior/actions (long-term). If we look at our scenario, out short, mid and long-term outcomes would look something like this:

Our short-term outcome might say: Increased parental knowledge of their rights under I-D-E-A, or the Individuals with Disabilities Education Act. Our medium-term outcomes would say:

Increased parental ability to help their child succeed and the long-term outcome would say:

Parents and educators collaborate to provide support services to children with disabilities.

TIP Since outcomes are often learning-focused, they can many times be evaluated using quantifiable data, such as retention rates of special education teachers or graduation and employment rates of students with disabilities, etc.

However, the outcomes might not be able to be quantifiably measured. Even so, there are ways for an evaluation process to capture perceptions, feelings, beliefs and reactions of stakeholders and by gathering these “qualitative” data, evaluators can gain an understanding of whether or not the outcome goals have been met.

When talking about outcomes within a logic model, it is critically important to align your activities and your intended outcomes then during the evaluation process, data collection systems will align to both the activities and outcomes, thereby connecting everything within the logic model.

From: What are logic models, and when should you use them? | NC State Industry Expansion Solutions (ncsu.edu)

Part 3: What Makes a Good Logic Model?

Training Objectives Participants will gain increased understanding of

Uses of a logic model

Characteristics of a strong logic model

How to select meaningful outcomes for a logic model What is the Purpose of a Logic Model?

Graphically illustrates project investments, planned activities, and expected outcomes

Illustrates the hypothesized causal flow from inputs and activities to short-, medium- and long-term outcomes

Provides a roadmap for project planning and implementation

Provides a blueprint for a strong evaluation plan

Describes the project and results as a linear simple process https://www.ies.ncsu.edu/blog/what-are-logic-models-and-when-should-you-use-them/ https://www.ies.ncsu.edu/blog/what-are-logic-models-and-when-should-you-use-them/

What are the Key Components of a Logic Model

Adapted from OSEP IDEAs that Work https://osepideasthatwork.org/sites/default/files/CIPP2_Logic_Model_Outline_03-13-15.pdf

If the project is a journey, the logic model is the map

Example: What if you went on a trip but didn’t use a map?

Nick is planning a nice vacation to Reykjavik. He has a plane ticket, rental car, gas, and GPS. Everything he needs to get where he is going.

He lands and the airport, picks up the rental car, and puts the address for his hotel in the GPS He drives…and drives…and drives

After a 5.5-hour drive, he arrives at a charming blue house in the small fishing village on the north coast of Iceland. The kindly woman who answers the door lets him know that the address on his reservation was spelled wrong and he should have arrived at his hotel after a 45-minute drive.

Her house is on Laugarvegur street, but his hotel back is Reykjavik is on Laugavegur street. The mistake of the extra R has caused him to waste much of his vacation. However, his story was featured on the local news and Icelanders embraced him with drinks, meals, and free tickets to attractions.

Does the logic model include the right “landmarks”?

What outcomes must be accomplished by the project on the way to the ultimate project goal?

– Short-term outcomes

– Medium-term outcomes

– Long-term outcomes

What groups are these outcomes expected for?

What outcomes are meaningful?

In addition to quality, relevance, and usefulness, the logic model should include outcomes that address important expected project results, such as changes in

– Knowledge or skills

– Attitudes or beliefs

– Use of practices or technologies

– Collaboration

– Policies or procedures

Example: Parent Training and Information Center Logic Model

Summary

A logic model graphically displays an entire project as a simple linear process

A strong logic model contains the critical elements of the project: inputs, activities, outputs, and outcomes

A logic model maps the path the project will take to the desired goals

Part 4: How to Link the Evaluation Plan to the Logic Model Training Objectives Participants will gain an increased understanding of

How to select what to evaluate

How to align evaluation questions, data sources, and methods with the logic model

How to ensure language in the evaluation plan and logic model are aligned

Start with the Logic Model

If an element is important enough to be in the logic model as an activity, output, or outcome, consider including it in the evaluation.

It may not be possible to evaluate everything in the logic model

– Consider the scope of the evaluation, the resources, and what is practical.

– Consider the evaluation timeline—long-term outcomes expected to occur beyond the timeline may not be measurable

– Prioritize which logic model elements will be included in the evaluation plan

Focus on Essential Information Focus on identifying the essential information needed to inform continuous improvement and demonstrate project results.

What are the most critical elements of the logic model?

Which outcomes are most important to the project, funders, and other stakeholders?

Which logic model elements are easy to measure?

What existing data sources can help to evaluate logic model elements?

What are the resources and expertise available for the evaluation?

Determine the Methods for Evaluating Each Element

Sample Logic Model

CIPP Logic Model https://osepideasthatwork.org/sites/default/files/CIPP2_Logic_Model_Outline_03-13-15.pdf

Evaluation Questions

Data Sources

Measures

Data Collection Procedures

Analysis Procedures

Aligning Evaluation Questions to the Logic Model Sample Logic Model: Evaluation Question

What is the total staff FTE applied to the project?

What is the total monetary investment in the project?

What successes & challenges are experienced?

What TA activities are implemented, changed, and/or added?

-How many online and in-person training sessions are held?

-How many parents participate in online and in-person training sessions and individual TA?

percentage of participating parents have increased knowledge of -the nature of their children’s disabilities, -their rights under IDEA, and -special education systems?

What percentage of participating parents collaborate with teachers to provide improved services to children with disabilities?

percentage of participating parents are better able to -help their children succeed, -navigate special education systems, and -use effective modes of collaboration with educators?

Aligning Data Sources/Methods to the Evaluation Questions and the Logic Model Sample Logic Model: Methodology (Data Sources & Methods)

-List of online and in-person trainings completed -Training attendance sheets -List of parents receiving individual TA

-List of project activities planned, completed, and changed -Focus groups of staff members and parents

-Budget/ financial reports -Staff FTE reports

-Parent survey on parent-teacher collaboration administered at first training or TA and 9 months later

-Parent survey -Interviews with a subset of participating parents

-Parent survey -Interviews with a subset of participatin g parents

Considerations for Selecting Data Sources and Methods

What data sources already exist?

What new data needs to be collected?

Will the method/data source:

– Be feasible?

– Produce valid and reliable information?

– Provide timely and relevant information for decision-making?

– Can the data be collected and analyzed often enough to inform project progress?

Ensure the language in the evaluation plan is aligned with the language in the logic model Are these Aligned?

Outcome Evaluation Question Performance Indicator

Short-term: Parents will increase their understanding of strategies to support their child’s development.

Do parents increase their awareness of strategies to support their child’s development after engaging with Parent Center resources?

80% of parents who engage with Parent Center resources report increasing their awareness of strategies to support their child’s development.

Medium-term: Parents will use a variety of strategies to support their child’s development.

Do parents have skills to support their child’s development after working with Parent Center staff?

80% of parents report having skills to support their child’s development after working with Parent Center staff.

Better Alignment

Outcome Evaluation Question Performance Indicator

Short-term: Parents will increase their understanding of strategies to support their child’s development.

Do parents increase their understanding of strategies to support their child’s development after engaging with Parent Center resources?

80% of parents who engage with Parent Center resources report increasing their understanding of strategies to support their child’s development.

Medium-term: Parents will use a variety of strategies to support their child’s development.

Do parents use a variety of strategies to support their child’s development after working with Parent Center staff?

80% of parents report using a variety of strategies to support their child’s development after working with Parent Center staff.

Knowledge Check Exercise 1

Logic Model Element Which Data Source Is Aligned?

Output:

Trainings are provided

Ratings of quality of the training

Number of trainings provided

Pre-post assessment of knowledge on training content

Answer: Number of trainings provided

Exercise 2

Logic Model Element Which Data Source Is Aligned?

Short-term Outcome:

Participants in training increase their knowledge of the content

Ratings of quality of the training

Training attendance

Pre-post assessment of knowledge on training content

Answer: Pre-post assessment of knowledge on training content

Exercise 3

Logic Model Element Which Data Source Is Aligned?

Mid-term Outcome:

Parents report feeling better able to navigate the special education system.

Parent focus groups

Website analytics

Observation of parents implementing strategies

Answer: Parent focus groups

Summary

The logic model identifies what to evaluate

Evaluation questions, data sources, and methods should be aligned with the logic model

The language in the evaluation plan should be aligned with the logic model

Thank you for accessing this CIPP training series. For more CIPP information, please go to www.cippsite.org.

© 2021 Center to Improve Program & Project Performance (CIPP)

The Center to Improve Program and Project Performance has been funded with Federal funds from the U.S. Department of Education, Office of Special Education Programs, under contract number ED-ESE15- A-0016/0004. The Project Officer is Dr. Kristen Rhoads. The content of this publication does not necessarily reflect the views or policies of the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government.

http://www.cippsite.org/

Summary