Should you invest in an RCT or in a teacher evaluation?
Why EdTech evaluations need the whole spectrum of research
Edtech evaluations often fall into two camps: controlled studies (like randomized controlled trials, RCTs) and teacher-driven, contextual evaluations. While many evaluators favour only one or the other approach, the truth is that both approaches are essential. Indeed, leading research frameworks, such as the 5Es Framework used by EduEvidence, integrate efficacy studies (RCTs) and effectiveness studies (real-world evaluations).
There are pros and cons to both evaluation methods. RCTs provide rigorous, controlled data, but they miss the nuances of real classroom dynamics. Teacher-led studies offer rich insights but can lack generalizability. So, relying on only one method creates blind spots. A controlled study might show that a tool improves test scores in a lab setting, but without real-world classroom input, it misses how and why it works or if teachers even use it effectively. Conversely, teacher-driven studies may capture classroom realities but lack objective evidence.
Without both perspectives, edtech providers risk misinterpreting their product’s true value. Furthermore, by following only efficacy or only effectiveness studies, edtech companies risk missing key factors in impact measurements, like the interplay between cost-effectiveness and learning outcomes.
So, it’s not about choosing one approach over the other, but about knowing when and at which stage of a product’s development each approach is most appropriate. In the past, randomized controlled trials were seen as suitable only for large-scale, expensive evaluations, but that view is outdated. Today, mini-RCTs can be conducted in cost-effective, rapid-cycle trials. Similarly, teacher evaluations and feedback on usability and contextual fit are not only valuable in the early stages of product development but should continue throughout the entire implementation cycle.
To build truly effective learning tools, edtech companies need to:
Ensure transparency - Report research that shows what works as well as what doesn’t work and use the insights gained to inform and drive continuous product improvement.
Prioritize Quality Over Method - Both randomized controlled trials and classroom-based studies can yield unreliable results if not implemented with quality and rigor. Select the research approach that best aligns with your product goals, and ensure it is carried out with methodological integrity and high standards of execution.
Accumulate Evidence Over Time - Learning is complex, and no single study can capture the full picture. Aim for multiple studies, replication, and continuous learning to better understand what drives usage and outcomes for your product.
This article is a summary version of an OpEd originally published by The 74, a leading nonpartisan U.S. education news organization. Read the full article here.