Evaluation Methods
- Observations
- Environmental Scanning
- Secondary Data Analysis
- Document Review
- Focus Group Interviews
- Community Surveys and Questionnaires
- Interviews
- Public Issues Forums
Observational techniques are methods by which an individual or individuals gather firsthand data on programs, processes, or behaviors being studied. By directly observing operations and activities, the evaluator and the evaluation team can develop a holistic perspective, i.e., an understanding of the context within which the project operates. Observational approaches also allow the evaluator and the team to learn about things the participants or staff/facilitators may be unaware of or that they are unwilling or unable to discuss in an interview or focus group.
Observations can be useful during a process evaluation or needs assessment. For example observations can be useful in determining whether or not the project is being delivered and operated as planned. In the hypothetical project, observations could be used to describe how learning sessions operate, examining the extent to which participants understand the concepts, ask the right questions, and are engaged in appropriate interactions. Such observations could also provide valuable insights into the teaching styles of the presenters and how they are covering the material.
Advantages
- flexibility - can change approach as needed
- it measures behavior directly, not reports of behavior or
intentions
Disadvantages
- limited to behavior - it cannot be used to study perception
One kind of observational approach is environmental scanning. The purpose of this environmental scan is to assess how public images portray attitudes about sexual violence and/or partner relationships in your community. Thus the people who do the scan will look for a) media and other images that portray positive relationships between intimate partners and/or opposition to sexual violence and b) negative images of relationships between partners or sexual violence. Steps in Environmental Scanning:
- The environmental scan team should bring a camera to assist in the documentation of your findings.
- While your team is scanning the environment, be sure to include both positive and negative messages.
- Orient the scan team to any tools you will use to track and document your observations. Determine what issues or factors you will incorporate into your scan, for example sexually explicit images, objectification of women or men, strict gender roles.
- Determine if you will conduct an inside or outside/windshield
survey (or both)
- Inside surveys can be conducted in any pertinent location such as a school, business, workplace, mall, etc.
- Outside or windshield surveys can encompass any geographic area designated by the scan team.
- Walk or drive through the designated area and carefully look around and observe billboards, business signs, business types and names, marquees, advertisements, window displays, wall posters, banners, window or wall displays, etc.
- There are many types of signs and ads for clothing, beauty products, movies, entertainment, magazines, alcoholic beverage products, tobacco products, vehicles, and other items that may not be named here. Look for both professional and home-made advertising, displays, and signs. Don’t forget neon signs, stickers on doors or windows, or mini-billboards in or around any location.
- Record the type of advertisement, sign or display, how they are sexually suggestive or overtly sexual in nature, and/or how they promote or normalize sexual violence, gender violence and/or interpersonal violence. Note the brand names/sponsors and location of the items on your tracking tool. Record your observations of positive messages as well.
- On your tracking tool, also record notes about what specific images are observed. Note that some of the content of displays, signs or advertisements may not have appeared to contain these types of material or messages upon first glance but as your awareness increases, the messages become more obvious.
Secondary data is data that is collected about a particular audience without having direct contact with that audience. These often come from administrative sources like the U.S. Census or from published (or unpublished) studies. Secondary data can often provide insight about emerging trends or issues in a particular community.
Advantages
- data already exists
- fast & easy to access
- data available for many geographic levels
Disadvantages
- may not be accurate
- lots of gaps in data
- requires groups to infer about what the data means
- lack of richness in the data
Another kind of secondary data are program or historical documents. Documents review analysis techniques used for qualitative or word-based data and includes an analysis of material generated by a program such as rules, policies, regulations, mission statements, program reports, applications, meeting minutes and records on services and participants.
These documents are a useful source of information on program activities and processes, and can generate ideas for questions that can be pursued through observations and interviews. In addition, program documents can provide valuable information that may not be accessible by other means.
Documents are a part of a program’s history and can include plans that have not been realized in actual program’s performance.
Focus Groups: Focus groups are a way to gather the opinions/ideas from a small, targeted group of residents. The intention of focus groups is perhaps more to build a synergy of thoughts and ideas than it is to make projections about the community or program.
Advantages
- easy to conduct
- provides detailed info
- allows for issue probing
- stimulates thinking and discussion
Disadvantages
- ideas generated not necessarily shared by rest of the community
- small number of participants
- difficult to analyze
- quiet folks suppressed
- requires participation
Steps in Conducting Focus Groups:
- Work with a committee to develop a guide - questions to be asked
in the focus group
- start with a warm-up, then ask easy questions
- then ask questions that are about the topic
- finally ask closing questions
- Select a trained facilitator
- Set a time and place:
- seating arrangement should encourage dialogue (circles)
- have snacks
- give compensation ($20) to participants
- make sure the location is neutral
- Recruit people to participate using already existing groups (i.e., parents who attend PTA meetings) or get a group together that are not really connected in any way
- 6 - 12 participants per group is a good plan
- Holding 8 - 10 groups around the community helps with hearing from diverse perspectives
- Conduct the groups and take detailed notes or tape the groups
- Write a report
Community Surveys and Questionnaires:
Community surveys help one to gather info about local attitudes regarding precisely defined issues, problems or opportunities. Surveys are usually multiple choice while questionnaires are open-ended. Pre- and Post-tests are a form of survey for the community of participants in your group.
Advantages
- wide distribution
- good for data analysis
- strong sample of the population
Disadvantages
- expensive
- requires skill/time
- suffers low return rate
- not a good qualitative tool
- Define objectives, identify what you want to know
- demographics, socioeconomics and other personal attributes: age, race, education, gender, income, family size/structure, location of residence, years living in the area
- cognitive - what do they know, beliefs (what do they think is true or false)
- affective - attitudes, how do they feel about something, preferences, likes and dislikes, evaluations
- behavioral - what have they done in past, are doing in present, expect to do in future
- Decide how to administer
- mail, interview, during an activity or existing group, etc.
- Decide who takes it
- Develop and refine your questions - pilot test it
- Decide how to get the surveys back and follow-up procedures
- Decide how to compile and analyze the data
- Writing a report
Survey Monkey is an easy-to-use tool for the creation of online surveys. Its primary strength is its intuitive web interface, which makes it easy for even nontechnical folks to create surveys and export collected data.
An interview is a conversation with a purpose, designed to help you gather information about people’s assumptions/perceptions of activities in your community. They are useful when looking for in-depth information on a particular topic.
Advantages
- detailed information
- allows for clarification/ followup on questions
- personal contact builds rapport
Disadvantages
- timeconsuming
- requires skill/time
- information is hard to summarize
- limited quantitative value
Steps in Conducting Interviews
- Determine who is to be interviewed
- Select a skilled interviewer
- Select questions
- Decide on an interview format/delivery method:
- formal or informal structure
- face-to-face
- telephone
- email delivery
- Inform the participants
- Practice
- Steer / guide the interview session (take notes)
- Write a summary report
- Thank the interviewees in writing
Community forums are public meetings that allow residents to express their concerns about community issues, problems, and needs.
Advantages
- diverse members can share ideas
- provides quick look at community issues
- involves local residents
- inexpensive
Disadvantages
- requires skill/time
- opinions obtained only from those who attend and speak
- may generate more questions or conflict
Steps in Hosting a Public Issues Forum
- Work with a committee to identify pressing issue(s) to form the focus of forum
- Select a trained facilitator
- Set a time and place:
- seating arrangement should encourage dialogue (circles)
- have snacks
- make sure the location is neutral
- Publicize the forum in all the major media outlets
- At the forum:
- Facilitator outlines process, what the results will be used for, and their role as facilitator.
- Do an icebreaker to build trust.
- Brainstorm: The purpose of brainstorming is to generate ideas (if issues have already been identified go to next step)
- Prioritize issues (you may want to do this via voting)
- Once issues have been identified, next step is to mobilize action groups (allow time for groups to meet and plan).
- Provide each group with action planner worksheet which gets them to write goal, objectives, planned activities, timeline, and persons responsible for activities. This step is meant to be short-term in action planning.