Three Ways to Evaluate STEM Learning Ecosystems

Overhead shot of a river delta with interconnected waterways


In our last feature, we discussed the differences between three commonly used terms in STEM learning: pipelines, pathways, and ecosystems. We also described some basic differences in evaluation approaches to evaluating these models. Today, we’ll explore three ways you might approach evaluating STEM learning ecosystems.

Mapping STEM Learning Experiences

As the interconnection between learning experiences and organizations is of key importance in STEM learning ecosystems, one helpful exercise might be to ask learners to design a map of their own STEM learning experiences. These maps could then be analyzed to see what experiences were most important to them and how one experience led to another. This approach could be used in tandem with interviews to dig even deeper into their perceptions of these experiences and how they have influenced their thinking and choices. 

A second mapping exercise could be aimed at exploring learner awareness of STEM learning opportunities available to them. It is important that learners in an ecosystem are aware of - and feel empowered to pursue - different opportunities. So, a mapping exercise asking them to describe organizations or opportunities in their area (either geographic area or area of interest) would be a great way to get those gears turning. For adults in STEM learning ecosystems (say, informal educators perhaps), you might even take this a step further and ask them to draw connections between organizations that work together, influence each other, or contribute to common goals. 

Views on Science and Identity

Perhaps the most interesting part of the STEM learning ecosystem is the informal STEM learning space (okay, I might be biased). In informal learning spaces there are often less direct connections between STEM learning experiences and what learners perceive to be “learning” or “science.” A visit to the zoo will likely feel less pivotal than a formal learning experience, like a biology class, due to the less structured, more learner-directed nature. Bell et al. (2013) have argued that the images learners hold about science and about themselves will vary based on what they encounter. Therefore, it might be useful to measure what learners count as “science,” their reasoning for including or excluding experiences in that category, and how this reasoning may influence how they see themselves (perhaps, in a simple interpretation, how much they see themselves as “scientists”). 

To measure these constructs, you might employ a method like a card sort activity. Here, each card depicts a science-associated activity and participants are asked to sort these cards into categories. You might choose “science” and “not science” or something more complex. To take this a step further, you could even turn this into a group exercise where learners work together to depict different experiences on the cards and discuss to which category each card belongs. 

In the article described earlier, the authors used a Science Activity Task method where ‘participants rated the frequency of the activities that they did and then reflected on how these activities connected to scientific knowledge, practices, and tools’ (Bell et al., 2013, p.133).

Measuring Collective Impact 

In a STEM learning ecosystem, collaboration is key. Organizations are often working towards big-picture common goals like increasing the representation of underrepresented groups in STEM fields or ensuring youth have access to STEM programs and opportunities. Because of this, ecosystems can be the perfect place to implement evaluation focused on collective impact. This might look like collaborating on research and evaluation efforts, or designing common instruments to be employed at all connected organizations. 

It may be of interest to look into concepts like social network development between STEM learning communities, the opportunity awareness that learners have within an ecosystem, and collective outcomes (e.g., interest, skill development). If collective outcomes are of particular interest, you might reference public, validated tools such as those from the Cornell Lab of Ornithology to see if they align with your needs, or you may work with a program evaluator to design a plan specific to your interests. 

Because of their interconnected and complex natures, there are endless possibilities in evaluating STEM learning ecosystems, and many creative opportunities beyond simple surveys. We hope you found this article inspiring. The most important thing is that you start to engage with data, and that you let your unique needs and interests guide the work. 

References:

Bell, P., Bricker, L., Reeve, S., Zimmerman, H., & Tzou, C. (2013). Discovering and supporting successful learning pathways of youth in and out of school: Accounting for the development of everyday expertise across settings. In Bevan, B., Bell, P., Stevens, R., & Razfar, A. (Eds.), LOST opportunities: Learning in out-of-school time (pp. 119-140). Springer Nature.  


If you enjoyed this post, follow along with Improved Insights by signing up for our monthly newsletter. Subscribers get first access to our blog posts, as well as Improved Insights updates and our 60-Second Suggestions. Join us!

Next
Next

STEM Learning Pathways, Pipelines, and Ecosystems