REL_NEI_Logic_Model_Workshop_Toolkit_slide_deck.pdf

Regional Educational Laboratory at EDC relnei.org

Logic Models to Support Program Design, Implementation, and Evaluation

Session I: Learning about Logic Models Hosted by the [insert alliance here]

Moderator:

Presenter:

Date Time relnei.org

Introductions

Workshop facilitator:

Participants:

Name and affiliation relnei.org

Agenda

Introduction and goals

Introducing the cases

What is a logic model?

Elements of a logic model

The logic in a logic model

Next steps relnei.org

Session I – Goals

• Introduce logic models as an effective tool for program and policy design, implementation, and evaluation

• Practice the elements of a logic model

• Provide guidance on the appropriate steps for building a logic model for a program or initiative relnei.org

Case Examples

College-Ready and Blended Learning Programs

Activity I.1: Discussion of Cases

Consider one of the cases:

• What are the goals of the program or policy?

• What might we want to know about it?

relnei.org

What Is a Logic Model?

• Where are you going?

• How will you get there?

• What will tell you that you have arrived?

relnei.org

What Is a Logic Model?

A logic model:

• Provides a simplified picture of the relationships between the program inputs and the desired outcomes of the program

• Is a framework for:

o Planning o Implementation o Monitoring o Evaluation

• Is a graphic and explicit representation of relationships, assumptions, and rationale relnei.org

What Is a Logic Model?

A logic model is not:

• A strategic plan or a fully developed plan for designing or managing a program or policy

• An evaluation design or an evaluation method relnei.org

What Is a Logic Model?

Types of logic models:

• Theory approach model: Conceptual, emphasizes theory of change (program design)

• Activities approach model: Activities and relationships, detailed steps (program management and implementation)

• Outcomes approach model: Connects resources and activities with results and outcomes, may break up outcomes and impacts over time segments (program evaluation) relnei.org

What Is a Logic Model?

The simplest form of a logic model:

INPUTS OUTPUTS OUTCOMES

relnei.org

What Is a Logic Model?

The simplest form of a logic model:

INPUTS OUTPUTS OUTCOMES

Inputs: What is invested in the program (e.g., money, people, time, and space) Outputs: What is done in the program (e.g., program strategies and activities) Outcomes: What results from the program (i.e., short- and long-term outcomes) relnei.org

What Is a Logic Model?

Case: Blended-Learning Program Inputs Outputs Outcomes

Existing technology Infrastructure audit Teachers’ reported use infrastructure completed of diverse instruction

Technology integration Six days of summer strategies increases.

staff person for three teacher professional Student engagement schools development increases.

Teachers’ enthusiasm completed Student achievement in three schools Six blended-learning on districtwide

Technology integration classrooms established assessments improves.

grant relnei.org

What Is a Logic Model?

Activity I.2: Inputs – Outputs – Outcomes Inputs Outputs Outcomes relnei.org

What Is a Logic Model?

Case: College-Ready Program Inputs

Staff

Volunteer mentors

School space and resources

Teacher time

Outputs

Course for parents

Mentoring for students

Guidance meetings

Student meetings

Outcomes

Parent involvement increases.

College applications increase.

College acceptances increase.

College attendance increases.

relnei.org

Elements of a Logic Model

Problem Statement

Resources (inputs)

Strategies and

Activities

Outputs Short-Term Outcomes

Long-Term Outcomes

Impacts

Assumptions relnei.org

Elements of a Logic Model

The elements of a logic model:

• Problem statement

• Short- and long-term outcomes

• Impacts

• Outputs

• Strategies and activities

• Resources (inputs)

• Assumptions relnei.org

Elements of a Logic Model:

Problem Statement Problem statement: The problem or challenge that the program or policy is designed to address

Questions to ask in defining the problem:

• What is the problem or issue?

• Why is this a problem?

• For whom does this problem exist?

• Who has a stake in the problem?

• What is known about the problem (through previous work, research, etc.)?

relnei.org

Elements of a Logic Model:

Problem Statement Problem statement: The problem or challenge that the program or policy is designed to address

Case: Blended-Learning Program

• Students are not actively engaged in their learning.

• Courses are sometimes monotonous.

• Students have limited one-on-one attention from adults.

• Students’ courses are not personalized.

• Students are all expected to work at the same pace.

relnei.org

Elements of a Logic Model:

Problem Statement

Problem statement: The problem or challenge that the program or policy is designed to address

Activity I.3: Problem Statement

• Articulate a targeted and specific problem

• Avoid a problem statement that restates the program as a need relnei.org

Elements of a Logic Model:

Outcomes

Outcomes: What difference does it make?

relnei.org

Elements of a Logic Model:

Outcomes

Outcomes: What difference does it make?

Short-term Long-term Impacts relnei.org

Elements of a Logic Model:

Outcomes

Outcomes: What difference does it make?

Short-term Long-term Impacts

Most immediate and measurable results for participants that can be attributed to strategies and activities relnei.org

Elements of a Logic Model:

Outcomes

Outcomes: What difference does it make?

Short-term Long-term Impacts

Most immediate and measurable results for participants that can be attributed to strategies and activities

More distant, though anticipated, results of participation in strategies and activities relnei.org

Elements of a Logic Model:

Outcomes

Outcomes: What difference does it make?

Short-term Long-term Impacts

Most immediate and measurable results for participants that can be attributed to strategies and activities

More distant, though anticipated, results of participation in strategies and activities

Desired outcomes of long-term implementation of strategies and activities, dependent on conditions beyond the scope of the program relnei.org

Elements of a Logic Model:

Outcomes Outcomes: What difference does it make?

Case: College-Ready Program

Short-term Long-term Impacts

Increased contact with parents or guardians

Improved attendance and academic performance

Increased percentage of students graduating from postsecondary institutions relnei.org

Elements of a Logic Model:

Outcomes

What is the desired change?

(action verb) Increase

Who is the target?

High school seniors in three urban comprehensive high schools

In what?

(results) Applications to postsecondary institutions

By when?

By June 2014

Outcomes: What difference does it make?

Activity I.4: Focus on Outcomes relnei.org

Elements of a Logic Model:

Outcomes

Outcomes Checklist

• Important

• Reasonable

• Realistic

• Unintentional, possibly negative relnei.org

Elements of a Logic Model:

Strategies and Activities

Strategies and activities: What you propose to do to address the problem

Activities, services, events, and products:

• Are designed to address the problem

• Are, together, intended to lead to certain outcomes relnei.org

Elements of a Logic Model:

Strategies and Activities Strategies and activities: What you propose to do to address the problem

Example: Blended-Learning Program

Activities Sequence Strategy Develop teacher training materials 1st Professional training Deliver summer institute 2nd Professional training Conduct technology audit 1st Infrastructure

And so forth… relnei.org

Elements of a Logic Model:

Strategies and Activities

Any questions so far?

relnei.org

Elements of a Logic Model:

Resources

Resources (inputs): The material and intangible contributions that are or could reasonably be expected to be available to address the problem Examples:

• Money, materials, and equipment (material/tangible)

• People, time, and partnerships (intangible) Resources are the inputs that enable the creation of the strategies and activities that are designed to respond to the stated problem.

relnei.org

Elements of a Logic Model:

Resources

Resources (inputs): The material and intangible contributions that are or could reasonably be expected to be available to address the problem

Case: College-Ready Program

• Community mentors

• Local university space for parent meetings

• Volunteer college admissions directors for application workshop

• Student volunteers for childcare at parent meetings relnei.org

Elements of a Logic Model:

Resources

Intangible resources: What intangible resources are at your disposal?

Activity I.5: Intangible Resources

Brainstormat least five nonmonetary resources that are available to you in a program you operate or manage.

relnei.org

Elements of a Logic model:

Assumptions

Assumptions: Beliefs about participants, staff, the program, and how change or improvement may be realized

Make explicit all implicit assumptions:

• Assumptions can be internal and external.

• Ask: What is known, and what is being assumed?

relnei.org

Elements of a Logic model:

Assumptions

Assumptions: Beliefs about participants, staff, the program, and how change or improvement may be realized

Case: Blended-Learning Program

Internal Assumptions External Assumptions

• The participating school leadership will continue to support the program.

• Three staff members will be sufficient to support the program in three schools.

• Access to a range of modalities will increase student engagement.

• Increased student engagement will increase academic achievement.

relnei.org

Elements of a Logic model:

Assumptions

Assumptions: Beliefs about participants, staff, the program, and how change or improvement may be realized

Activity I.6: Uncovering Internal and External Assumptions

What assumptions are you or your program making?

relnei.org

The Logic in a Logic Model

The theory embedded in the model…

A series of if-then statements across the model relnei.org

The Logic in a Logic Model

The theory embedded in the model…

A series of if-then statements across the model Case: Blended-Learning Program

If:

District invests in blended learning in three schools

Then: and If:

Instruction will be personalized and participating students will be more engaged

Then:

Student achievement will increase as measured by standardized assessment relnei.org

The Logic in a Logic Model

Activity I.7: If-Then Statements

Order the if-then statements in the example from the College-Ready Program case relnei.org

Next Steps

What we have accomplished so far?

• Discussed the purpose of a logic model

• Presented the elements of a logic model

• Considered the logic embedded in a logic model relnei.org

Next Steps

Ask yourself the following:

• Do I understand the elements of the logic model and how they differ?

• Who should I consult in developing the model? What colleagues and stakeholders should be participants in developing the logic model?

• Who will shepherd or see through the development of the logic model?

• How do I know we have captured the theory of action?

• How will we use the logic model?

• How will we ensure we make it a living document?

relnei.org

Next Steps

Your next steps…

Activity I.8:

What Are Your Next Steps with Regard to Logic Models?

relnei.org

Final Thoughts on Logic Models

Some final thoughts… •Logic models are tools for program design, implementation, and evaluation.

•The process of developing a logic model is important:

Engage stakeholders in developing a logic model.

•Logic models should be living documents and returned to frequently.

•Logic models are useful for evaluation but best when developed at the program design phase.

relnei.org

Thank You!

For any questions about this workshop, contact:

relnei.org

Logic Models to Support Program Design, Implementation, and Evaluation

Session II: From Logic Models to Program and Policy Evaluation

Hosted by the [insert alliance here] Moderator:

Presenter:

Date Time relnei.org

Introductions

Workshop facilitator:

Participants:

Name and affiliation relnei.org

Agenda

Introduction and goals

Review of logic models

Introducing evaluations

Moving from logic models to evaluation questions

Generating indicators

Building an evaluation design

Putting it all together relnei.org

Session II – Goals

• Reintroduce logic models as an effective tool, specifically for evaluation.

• Practice using logic models to develop evaluation questions and indicators of success.

• Provide guidance on how to determine the appropriate evaluation for a specific program or policy.

relnei.org

Review of Logic Models

A logic model is:

• A graphic representation of theory of change

• A framework for planning, implementation, monitoring, and evaluation

A logic models is not:

• A strategic plan or a fully developed plan for designing or managing a program or policy

• An evaluation design or evaluation method relnei.org

Review of Logic Models

Problem Statement

Resources (inputs)

Strategies and

Activities

Outputs Short-Term Outcomes

Long-Term Outcomes

Impacts

Assumptions relnei.org

College Ready Logic Model Excerpt

Problem Statement: Low-income high school students in selected communities attend college at a lower rate than their middle-class peers, leading to more limited opportunities, higher rates of unemployment, and lower earnings.

Resources

Partnership with 3 public high schools

Strategies and activities

Establish local college mentorship program.

Outputs

Recruited adequate number of mentors for student cohort.

Short-term outcomes

Participating students apply to at least one college on time.

Long-term outcomes

Participating students are accepted to and attend college, remaining enrolled into the third semester.

Impacts

Low-income students in participating communities attend college at the same rate as middle-class peers.

Assumptions: College attendance is desired goal for participating communities; high school leaders will remain consistent and support program; parents will show interest and participate in program.

relnei.org

Review of Logic Models

Questions to ask about your logic model:

•What elements of the logic model were hardest to develop?

•Is the problem statement the right “grain size”?

•Within the strategies and activities, did you identify overarching strategies?

•What assumptions did you uncover?

•What is the time frame for your outcomes?

•What are the impacts?

•What was your process for developing the model?

•What requires further explanation or discussion?

relnei.org

Introducing Evaluation

Evaluation asks the questions:

• Are we successful?

• Have we had an impact?

• What are the most influential aspects of the program?

relnei.org

Introducing Evaluation

Consider:

• Is the program or policy effective?

• Is the program or policy working as intended?

• What aspects of the program are working? What aspects are not working?

Timing:

• Ask these questions at the onset of program design.

• Involve stakeholders in the evaluation design.

• Invest early in designing a good evaluation.

relnei.org

Introducing Evaluation

Consider:

• Is the program or policy effective?

• Is the program or policy working as intended?

• What aspects of the program are working? What aspects are not working?

Activity II.1: How Will You Know?

relnei.org

Introducing Evaluation

Most evaluations are designed to improve or prove:

• Improve: Formative evaluations focus on strategies, activities, and outputs. They are also called process or implementation evaluations.

• Prove: Summative evaluations focus on outcomes.

They are also called results or outcomes evaluations.

relnei.org

Introducing Evaluation

Four types of evaluation:

• Formative o Needs assessment o Process evaluation

• Summative o Outcome evaluation o Impact evaluation relnei.org

Moving from Logic Model to Evaluation Questions

Formative questions:

• Asked while program is operating

• For program improvement or midcourse correction

Summative questions:

• Asked at completion or after the program

• What was the result?

• Was it effective?

relnei.org

Moving from Logic Model to Evaluation Questions

Guidelines for good evaluation questions:

• Can the questions be answered given the program?

• Are the questions high priority?

• Are the questions practical and appropriate to the capacity you have to answer them?

• Are the questions clear and free of jargon?

relnei.org

Moving from Logic Model to Evaluation Questions

Activity II.2: Formative and Summative Evaluation

What is a formative evaluation question you have about a program or policy?

or

What is a summative evaluation question you have about a program or policy?

relnei.org

Moving from Logic Model to Evaluation Questions

When considering an evaluation, keep your audience in mind:

• Audience: Who wants to know?

(participants, funders, staff)

• Questions: What does the audience want to know?

(Is the policy helping? Did we reach the target population? How could the program be improved?)

• Use: How will results be used?

(continued participation, program changes, funding) relnei.org

Moving from Logic Model to Evaluation Questions

Audience Typical Questions Evaluation Use Program staff Are we reaching our target population?

Program operations

Participants Is the program helping people like me?

Participation

Public officials Who does the program serve?

Support, commitment, scale-up, and duplication

Funders Is the program meeting its goals? Is the program worth the cost?

Ongoing funding, accountability

Source: W.K. Kellogg Foundation, 2006 relnei.org

Moving from Logic Model to Evaluation Questions

Audience Typical Questions Evaluation Use

Program staff

Participants

Public officials

Funders

Activity II.3: Generating Questions for Different Audiences relnei.org

Moving from Logic Model to Evaluation Questions

Any questions so far?

relnei.org

Generating Indicators

Activity II.4:

How Do We Know If a Child Has the flu?

relnei.org

Generating Indicators

How will we know the program is successful?

Indicators of success… relnei.org

Generating Indicators

Indicators are:

• Specific, measureable targets

• Seen, heard, read, and felt

• Connected to strategies, activities, outputs, and outcomes

• Evidence representing phenomenon of interest relnei.org

Generating Indicators: Using the Logic Model

From the logic model

Inputs

For example, resources

(tangible and intangible)

Outputs

For example, strategies or activities, participation

Outcomes or Impact

For example, short-term, long-term, impact relnei.org

Generating Indicators: Using the Logic Model

From the logic model

Inputs

For example, resources

(tangible and intangible)

Outputs

For example, strategies or activities, participation

Outcomes or Impact

For example, short-term, long-term, impact

Indicators

Amount of resources used relnei.org

Generating Indicators: Using the Logic Model

From the logic model

Inputs

For example, resources

(tangible and intangible)

Outputs

For example, strategies or activities, participation

Outcomes or Impact

For example, short-term, long-term, impact

Indicators

Amount of resources used

Number of workshops, number of participants relnei.org

Generating Indicators: Using the Logic Model

From the logic model

Inputs

For example, resources

(tangible and intangible)

Outputs

For example, strategies or activities, participation

Outcomes or Impact

For example, short-term, long-term, impact

Indicators

Amount of resources used

Number of workshops, number of participants

Number & percent who learned material, overall improvement relnei.org

Generating Indicators: Using the Logic Model

From the logic model

Activity

Deliver parent education classes

Outcome

Parents understand college application process

Indicators

Number of classes delivered, number of parents attended

Number of parents reporting increased understanding relnei.org

Ask these basic questions:

• What would achieving the goal reflected in the outcome look like?

• How would we know if we achieved it?

• If I were visiting the program, what would I see, hear, or read that would tell me that the program is doing what it intends?

Generating Indicators: Using the Logic Model relnei.org

Generating Indicators: Using the Logic Model

Activity II.5: Process and Outcome Indicators

Look at the logic model and map a path:

Activity Output Outcome Indicator relnei.org

Generating Indicators: Using the Logic Model

Example: College-Ready Program

Activity Deliver a set of parent workshops for college readiness

Output Six workshops developed and delivered, 100 parents recruited to participate

Outcome Parents increase their understanding of college application process

Indicators Process:

Outcome:

relnei.org

Generating Indicators: Using the Logic Model

Example: College-Ready Program Activity Deliver a set of parent workshops for college readiness

Output Six workshops developed and delivered, 100 parents recruited to participate

Outcome Parents increase their understanding of college application process

Indicators Process: 70 percent of parents attend at least five of the six workshops.

Outcome:80 percent of students in program complete at least one college application by deadline.

relnei.org

Generating Indicators: Identifying the Right Indicators

From the logic model

Inputs Outputs Outcomes or Impact

Indicators

Amount of resources used

Number of workshops, number of participants

Number & percent who learned material, overall improvement

Evaluation questions

Were the inputs sufficient?

relnei.org

Generating Indicators: Identifying the Right Indicators

From the logic model

Inputs Outputs Outcomes or Impact

Indicators

Amount of resources used

Number of workshops, number of participants

Number & percent who learned material, overall improvement

Evaluation questions

Were the inputs sufficient?

Were workshops implemented as intended? Well attended?

relnei.org

Generating Indicators: Identifying the Right Indicators

From the logic model

Inputs Outputs Outcomes or Impact

Indicators

Amount of resources used

Number of workshops, number of participants

Number & percent who learned material, overall improvement

Evaluation questions

Were the inputs sufficient?

Were workshops implemented as intended? Well attended?

Did program change participant knowledge? Skill?

relnei.org

Generating Indicators: Identifying the Right Indicators

• One indicator to measure drop-out reduction = relnei.org

Generating Indicators: Identifying the Right Indicators

• One indicator to measure drop-out reduction = o Graduation rate relnei.org

Generating Indicators: Identifying the Right Indicators

• One indicator to measure drop-out reduction = o Graduation rate

• Several indicators to measure parent involvement = relnei.org

Generating Indicators: Identifying the Right Indicators

• One indicator to measure drop-out reduction = o Graduation rate

• Several indicators to measure parent involvement = o Attendance at school meetings o Participation in parent–school organization o Parent calls made to the school o Attendance at school functions relnei.org

Generating Indicators: Quantitative and Qualitative

Quantitative: Outcomes focused and summative

Qualitative: Process focused and formative relnei.org

Generating Indicators: Quantitative and Qualitative

Indicators can be quantitative:

Evaluation Question Indicators

Did the program increase students’ interest in college?

Number of college applications completed relnei.org

Generating Indicators: Quantitative and Qualitative

Indicators can be quantitative or qualitative:

Evaluation Question Indicators

Did the program increase students’ interest in college?

Number of college applications completed (quantitative)

Guidance counselors report increased student interest (qualitative) relnei.org

Final Considerations about Indicators

Remember, indicators may:

• Match the outcomes of interest or questions asked

• Be singular for a given outcome or question

• Be quantitative or qualitative

• Vary based on the audience relnei.org

Building an Evaluation Design

Consider:

• Purpose of evaluation: Formative? Summative? Hybrid?

• Audience: Who is the audience for the evaluation? what do they want to know? How will the information be used?

• Capacity: Who will conduct the evaluation? What resources will be use? What is the time frame?

• Priority: What do you need to know?

relnei.org

Building an Evaluation Design:

Identifying Appropriate Data Sources

Data collection:

• What are pre-existing data sources (e.g., school attendance records, existing survey data)?

• What are existing instruments (e.g., existing surveys measuring same constructs)?

relnei.org

Building an Evaluation Design:

Identifying Appropriate Data Sources

Types of data:

• Administrative data

• Focus groups

• Interviews

• Observations

• Surveys

• Student test scores and grades

• Teacher assessments

• Other data sources relnei.org

Building an Evaluation Design:

Identifying Appropriate Data Sources

Activity II.6: Consider Data Sources

Data Source Brainstorm

Consider your own program:

What relevant data sources do you already collect?

relnei.org

Building an Evaluation Design:

Creating a Data Collection Framework

Strategy or

Activity

Output or

Outcome

Formative Summative Indicator Data Source

Data Collection Instrument

When Collected

By Whom

Data Collection Framework relnei.org

Data Collection Framework Example: College-Ready Program

Building an Evaluation Design:

Creating a Data Collection Framework

Strategy or

Activity

Output or

Outcome

Formative Summative Indicator Data Source

Data Collection instrument

When Collected

By Whom

Parent education strategy

High rate of parent attendance at workshops

✔ 70 percent of parents attend five out of six workshops

Admin-istrative data

Attendance log at workshops

At beginning of each session

Program director

Parent education strategy

Increased parent under-standing of college application process

✔ 85 percent of parents who attend more than four workshops report increased understand-ing

Parent feedback

Survey and interviews

Beginning of program, end of program

Program staff relnei.org

Building an Evaluation Design:

Creating a Data Collection Framework Strategy or

Activity Output or Outcome

Indicator Data Sources

Data Collection Instrument

When Collected

By Whom

Short Term

Parent education strategy

Increased parent understanding of college application process

85 percent of parents who attend more than 4 workshops report increased understanding

Parent feedback

Survey and interviews

Beginning of program, end of program

Program staff

Long Term

Student education strategy

Increased student understanding of the college application process

80 percent of students who attend the workshops apply to college and get accepted to at least one college or university

Student feedback

Survey and interviews

End of program

Program staff relnei.org

Putting It All Together

If you have:

•Developed a logic model in collaboration with stakeholders •Clarified who the audience is for the evaluation and how it will be used •Identified and prioritized evaluation questions based on the logic model •Selected indicators based on the outcomes of interest •Identified data sources and a data collection plan •Considered evaluation design, with awareness of resources, capacity, and timeline

Then… relnei.org

Putting It All Together Create an evaluation prospectus:

• What are you going to evaluate?

• What is the purpose of the evaluation?

• How will the results of the evaluation be used?

• What specific questions will the evaluation answer?

• What data sources will be necessary to answer these questions?

• How will the data be analyzed (evaluation design)?

• What resources are needed to conduct this evaluation?

• What is the timeline for the evaluation?

• How will the results be shared or disseminated?

• Who will manage the evaluation?

relnei.org

Putting It All Together Timeline: Gantt chart

January February March April May June July

Develop survey

Select sample

Administer survey

Analyze survey

Compare data to indicators

Write up findings relnei.org

Review

• Logic models are a useful tool for program design, implementation, and evaluation.

• Planning for evaluation at the onset of program or policy development ensures an evaluation that is relevant and, potentially, more rigorous.

• Engaging stakeholders in the process of developing the logic model and evaluation encourages support and buy-in and increases authenticity.

relnei.org

Next Steps

Your next steps…

What is one thing you’ve learned or will take back with you to your colleagues?

relnei.org

Thank You!

• Contact Information:

For any questions: [Place name title and email here]