About this Evaluation Tool Kit

This tool kit is a compilation of materials developed for The Domestic Violence Prevention Enhancement and Leadership Through Alliances Project (DELTA)--a cooperative agreement made possible by the Centers for Disease Control and Prevention (CDC) with 14 state domestic violence coalitions that receive DELTA Program funds. World Bridge Research works with the North Dakota Council on Abused Women’s Services/Coalition Against Sexual Assault in North Dakota (NDCAWS/CASAND) Empowerment Evaluation Project and the Wisconsin Coalition Against Domestic Violence (WCADV) Empowerment Evaluation Project.

Empowerment Evaluation is about helping programs work better by using evaluation results to guide strategic planning and decisionmaking. Specific tools (e.g., Getting to Outcomes) are used to guide planning, implementation, and evaluation.

Introduction to Evaluation

During the past 30 years program evaluation for social programs has evolved into a field known as “evaluation research.” “Evaluation research is the systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of social intervention programs,” (Rossi & Freeman, 1996).

Let’s pick this apart - To conduct an evaluation we must create a system or logical way of planning, designing and putting into place activities and strategies that help to change people’s lives. Then we must document what we do, what participants experience and the changes that take place in both the program and participants. Evaluation has steps before, during and after program implementation.

So, what are these evaluation procedures that we use before, during and after the program implementation? We call them research or evaluation methods and they fall into two categories: quantitative and qualitative.

Quantitative Methods:

Quantitative methods are a formal, objective, systematic process in which numerical data are utilized to obtain
information about the world. Quantitative methods:

Quantitative methods are usually understood to include:

Qualitative Methods:

Qualitative methods are useful compliments to quantitative and participatory methods in order to:

Increase understanding of WHAT is happening.

Contribute to understanding of WHO is affected in which ways.

Analyze WHY particular impacts are occurring.

Sometimes qualitative methods lead to quantitative approaches to understanding and vice versa.

Example 1: Qualitative lead to Quantitative

There is an early intervention/prevention program that recruits at risk mothers for child abuse to participate. It is a 2 year program that begins when the child is born. The program uses psychoeducational approaches and parenting skills training. There is also a coaching element. The program designers later began a “mentorship” component to assist parents' transition out of the program giving program “graduates” a role in the program.

At first, program designers weren’t exactly sure what kind of results the mentorship component was producing. The social workers had some intuitive ideas, but wanted more definitive information. After a series of indepth interviews (qualitative method), the most striking benefit was found in enhancing the graduate mentors’ leadership skills.

Now when they evaluate their program they have a group of leadership questions that they ask on their pre- and post-test survey(quantitative method) to collect more information they learned about with the qualitative investigation done earlier.

Sometimes qualitative methods fall short and quantitative approaches lead to better understanding.

Example 2: Quantitative was better option

There is an organization that designed re-entry programs for incarcerated men. Before release from prison the organization taught parenting classes to help inmates prepare for their fathering roles and responsibilities. At the end of each class the inmates were asked, “Did you learn anything from taking this class? If so, what did you learn?” 100% of the students said they learned something and the students would write about the topics they learned something about.

When the inmates were asked to take a “test” at the end of the course based on the curriculum content - there often was no change in their knowledge of parenting scores from pre-test to the post-test. In other words, students would answer on average 13 out of 20 parenting knowledge questions correctly on the pre-test and about 13 out of 20 questions were answered correctly at the post-test.

Most Important Message:

When you select a method for any part of your evaluation it should be a reflection of which approach is most suitable enhancing your understanding about what is going on with your program. A mistake that we sometimes make in evaluating our program is picking a method because it is happens to be a method that they have experience with, it is the latest “fad” or it just sounds good to us.

Focus groups are a great example of the latest fad. It seems like everyone wants to do focus groups no matter what they are trying to evaluate. Focus groups are great when we are trying to assess need or when we want to know more about participants’ life experience. Focus groups are not so great if we are trying to assess outcomes. This is because it is difficult to prove that our program is working (or isn’t working) without some quantifiable numbers. Knowing how best to answer these questions and picking the right methods is the first step in doing great evaluation work.

References:
Rossi, P.H., Freeman, H.E. Evaluation: A Systematic Approach. Newbury Park, CA: Sage Publications; 1996.