Getting Started with Program Evaluation: Chapter 4
Planning on Outcome Evaluation
What can evaluating outcomes tell you about your program?
Outcome evaluation seeks to answer several crucial questions: “Is my program working? Does it have the desired effect on the youth we serve?”
If you have a process evaluation in place, as well as tools that can help you measure relationship quality, you will be better equipped to accurately interpret the outcomes you measure in your program.
Key terms and concepts we’ll explore in this chapter:
- Correlation vs. causation
- Spectrum of outcome evaluation
- Guiding principles of outcome evaluation
-
preparing to evaluate program outcomes 
Outcomes are the changes in individuals that result from participating in a mentoring program. They are most often categorized as shifts in attitudes, behaviors, or knowledge/skills.
Before programs like Super Scholars jump in and start evaluating program outcomes, it is important to acknowledge that many factors can affect the results of outcome evaluation. So far, we’ve considered two important evaluation components that should be in place before youth-mentoring programs evaluate outcomes:
A process evaluation helps identify program implementation issues that can affect youth outcomes. For example, Super Scholars discovered that its mentors were not consistently completing the required one hour of ongoing training. Measuring relationship quality helps identify the factors that contribute to positive relationships between mentors and mentees. For example, Super Scholars identified that 60 percent of its mentors did not feel confident in their ability to provide the academic support their mentees need and that mentor confidence has an impact on the quality of mentoring relationships.
These two components of program evaluation provided Super Scholars with valuable insights into some of the factors that could affect program outcomes. Using these insights they were able to make programmatic improvements which increased their ability to accurately measure their program outcomes. -
spectrum of outcome evaluation When it comes to evaluating outcomes, it’s helpful to picture the different approaches on a spectrum.
On one side, there is outcomes monitoring, in which programs look at pre- and post-program participation data (or sometimes just post) to determine whether youth are making progress toward desired outcomes. Outcomes monitoring could help Super Scholars determine whether mentees’ grades are improving.
On the other side is impact evaluation, which uses different types of comparison group studies to help programs demonstrate that they are the cause of desired outcomes. Impact evaluation could help Super Scholars determine whether mentees’ grades are improving as a result of participating in the program.
-
monitoring outcomes Super Scholars is feeling some pressure to demonstrate results. The new district superintendent has started to question the value of continuing to fund the program, citing uncertainty that it is improving participants’ academic performance. Specifically, the superintendent wants to know:
- Are mentees’ grades improving as a result of program participation?
- Are mentees more likely to think about (and eventually attend) college as a result of participating in the program?
The simplest approach to help address the superintendent’s questions would be to start monitoring outcomes related to academic achievement, high school graduation, and college enrollment. If Super Scholars can start collecting the same data at the beginning and end of program participation, it can monitor mentees’ progress toward these outcomes.
Take a moment to read about the data Super Scholars started collecting to help monitor these important program outcomes: Reading 4: Monitoring Outcomes
-
setting up a comparison group study If Super Scholars truly wants to demonstrate that mentees’ grades are improving as a result of participating in the program, it will need to set up a study with comparison groups. Here are two options Super Scholars could pursue:
Option 1: Quasi-Experimental Design
Compare outcomes data for students in the program with a group of similar students.Option 2: Experimental Design
Use a method such as random assignment, in which two groups of similar students are enrolled in the program, but a randomly selected group of students (the “control group”) would not receive the program’s mentoring services. The outcomes of the control group would be compared with those of students who received the intervention. -
considerations for using comparison groups 
Capacity: Setting up a well-designed study that uses comparison groups requires time, resources, and expertise. Although some programs may have the internal capacity to do this, most will need to work with an experienced external evaluator.
Ethical considerations: There are a lot of ethical considerations when studying human subjects. Using random assignment means that some students would be technically “enrolled” in the program but not receiving the support they thought they would. This could have unforeseen damaging effects on youth. For this reason, some experimental designs only delay the service for a short period of time, such as a school year. But if your program serves only a narrow age range, such as eighth-graders transitioning to high school, it can result in students never getting that needed support. Programs should carefully consider their options and find a design that feels strong but is not beyond their comfort level. -
evaluating impact 
To evaluate the impact of the program on youth outcomes, Super Scholars ultimately settled on what is called a matched comparison design.A principal at a nearby school with no mentoring program agreed to administer the same pre- and post-program survey to similar students in their school. To do this, staff members matched each Super Scholars mentee with a student at the other school who was like that mentee in as many ways as possible (e.g., age, family circumstances, academic ability, relationships). By tracking these two similar groups of youth through the duration of the program, Super Scholars should be able to tell whether it is truly the program that is making a difference or if these groups essentially remain the same over time.
This shift from basic outcome monitoring to a more rigorous evaluation design that can demonstrate causality is precisely the point where a professional evaluator will likely be needed. -
instruments to measure outcomes 
To help find research-based instruments to measure outcomes, programs can look to the National Mentoring Resource Center’s Measurement Guidance Toolkit.Here, you can find recommended instruments for measuring key youth outcomes in mentoring programs:
- Mental and Emotional Health
- Social and Emotional Skills
- Healthy and Prosocial Behavior
- Problem Behavior
- Interpersonal Relationships
- Academics
- Risk and Protective Factors
-
planning for outcome measurement (activity) Review your logic model and identify research-based measures for a few of the outcomes you would want to evaluate.
- Review the outcomes you identified in your logic model.
- Transfer three to five of your outcomes to the “Outcome Evaluation Planning Worksheet.” If you have more than five outcomes in a given category (e.g., short-term), start by listing the five you are most interested in measuring.
- For each outcome, identify a potential measure. For example, if an outcome is academic achievement, possible measures could be mentees’ self-reported grades (there is a good measure of this type in the toolkit) or, if feasible, mentees’ actual report cards.
- For each measure, think about how and when you could administer the evaluation.