Choosing the Right Evaluation Instruments
Sarah M. Dunifon
Two students work together on a circuitry project
Welcome back to our Insights! Earlier this year, we surveyed our newsletter subscribers on their level of interest in various evaluation topics. In this second installment of the series, we’ll take a look at the process of choosing the right instruments for your evaluation.
Selecting the right instruments for your evaluation can be tricky, but with a few priorities in mind, you can make better decisions. In previous features, we’ve explored how to determine what to measure in an informal STEM learning evaluation, how to craft surveys and align them with your priorities, and even offered quick tips on choosing evaluation methods. In this feature, we’ll tie some of these insights together to further explore how to choose the right evaluation instruments.
There are several priorities you’ll need to consider before selecting instruments or methods:
Your evaluation questions (e.g., what you hope to learn)
Your audience (e.g., who the evaluation centers)
The context of your program (e.g., where it occurs, when it occurs)
Your intervention or program (e.g., what type is it, how long does it last, what it focuses on)
You’ll want to be very specific about each of these priorities going into the evaluation design process. Consider that without clearly defined evaluation questions, you may not know if you’ll need to collect qualitative data or quantitative data. Or, without specifying the audience you want to hear from, you may miss the fact that the age of your participants (for example) will seriously impact method selection and instrument design. Thinking critically about the location of your program (the program context) may influence your decision to conduct evaluation activities on-site or digitally. And the intervention or program itself will influence which methods or instruments will be most realistic. For example, a 10-minute drop-in interpretive program on your zoo grounds might not necessitate a 60-minute focus group.
To identify your priorities, it may be helpful to start by listing all the information you know about the program you plan to evaluate. We put together the following worksheet to help with this exercise (you can also download it here).
Evaluation questions (what do we hope to learn?):
Question 1
Question 2
Question 3
Audience (who do we want to hear from?):
How many participants?
What is the ideal sample size?
What is/are the age(s) of participants to be included in the study?
Are there language, developmental, or other needs to consider?
What is the cultural context? How can we employ cultural responsiveness?
Program context (when and where does it happen?):
Where does the program take place?
What time of year does the program happen?
Program or intervention (what does it look like?):
How long is the program?
Does the program repeat or happen once? With different participants, or the same?
What type of program is it? How would you describe it?
What does the program focus on?
After reflecting on these priorities, you’re ready to start identifying the right method(s) to employ. You may also wish to do a little research on common methods that align to your program and context (e.g., What evaluation methods are commonly used in museum camp programs?) or with your specific audience (e.g., What evaluation methods are developmentally appropriate for 10 year olds?). You can also start to hone in on what types of data might best answer your evaluation questions. For instance, if you’re hoping to answer the question, “To what extent do youth take action to support their environment?” you might decide a combination of quantitative and qualitative data would be appropriate. Here, you could design an approach that uses a quantitative-heavy survey instrument with a qualitative-focused interview protocol. Or, maybe your question is “How do youth describe their pro-environmental actions?” In this instance, qualitative data will better answer your question, so focus groups, journal entries, interviews, or open-ended survey responses could be a good fit.
It is important to recognize that there is a VAST range of methods and instrument options available. Involving qualified evaluation professionals can help you sort through the options and figure out what best fits your needs. Ultimately, selecting the right instruments or methods for your evaluation comes down to triangulation of the aforementioned priorities, coupled with what is realistic for your staff and programs.
We hope you enjoyed this article! If you’d like to see more content like this as it comes out, subscribe to our newsletter Insights & Opportunities. Subscribers get first access to our blog posts, as well as Improved Insights updates and our 60-Second Suggestions. Join us!