What 14 Edtech Research Studies Showed Us About The Need for Formative Research

A case for a more iterative, product-focused research

---

Quick Summary

• Declines in NAEP scores have reignited fears of extended learning loss being a post-pandemic legacy. 

• There is a new movement emerging to demand more evidence from edtech companies, particularly those that could provide a salve for such learning loss:

  • Context: Given strong pre-existing research on the effectiveness of high-dosage tutoring, the rapid growth of online tutoring programs are probably the most prolific example of technology built on a strong evidence base. Still, there have been increased calls for more research to explore the effectiveness of these remote approaches to tutoring. 

• Moreso, there appears to be another barrier to implementing evidence-based education technologies at scale; getting consistent usage. A Brookings report cites a study of a tutoring intervention where “Only 19% of students ever logged on and received help from a tutor […] struggling students were only half as likely to ever use the resource than their higher-achieving peers.” However, an implementation modification was able to increase usage.   

• In an analysis of 14 studies that Leanlab Education performed on varying edtech platforms, we unearthed a trend that anecdotally is being confirmed by even the biggest technology companies; low usage due to implementation flaws or non-user friendly design is potentially the largest threat to edtech living up to its promise.   

• We recommend companies and schools look at product focused studies that focus on implementation practices within the contextual realities of real classrooms, leveraging the ESSA IV tier of evidence, sometimes known as the “innovation sphere.”

---

“There’s potentially a larger threat that looms in the edtech space—lack of user engagement.” 

Declines in NAEP scores and recent reports of low uptake of newly implemented education technology programs doesn’t mean that folks are no longer looking to education technology with hope. While there is no denying that the education sector will continue to be digitized, there is a growing call to action for edtech to live up to the promise of expanding access to educational opportunities that can accelerate learning. 

However, while the call for more tech efficacy research is warranted and important, we found there’s an overlooked and perhaps more urgent need for product research that focuses on streamlining user experiences and enhancing product implementation. After all, there’s potentially a larger threat that looms in the edtech space—lack of user engagement.  

 In fact, a Hechinger Report analysis recounts yet another low usage implementation of a tutoring programs: 

“In one large city, Amplify contracted to give almost 1,200 students tutoring sessions three times a week with a tutor delivering sessions over a video call, similar to Zoom. More than 100 kids never logged in to connect with a tutor online. Only 200 students – fewer than 20 percent – received at least two sessions a week throughout the school term. More than 80 percent received less, often far less.” 

And this is consistent with our own findings. At Leanlab, we work hard to match edtech companies with the right school partners. But, even when priorities are aligned between the researchers, the edtech company, and the school, if an edtech tool is too hard to use or teachers don’t understand how to fit it into their lesson plan, we know we’ll see some level of attrition throughout the study.

For products to be successful in creating impact, they need to be easy to use within the chaotic realities of K12 education in its current moment. If nothing else, as school districts continue to become more savvy consumers, they will begin to jettison edtech products that are under-leveraged, no matter how evidence-based.

---

At Leanlab Education, we contracted EdTech Recharge to analyze 14 studies of seven edtech products we conducted in partnership with American public schools in 2021, as students re-entered traditional school environments after the pandemic. What we found is that while EdTech products do demonstrate potential in aiding student learning, more weight needs to be given to formative education technology research that focuses on improving product features and implementation protocols, which are currently impeding student progress on many education technology solutions. 

We studied products focused on social emotional learning, parent engagement, elementary math, Spanish reading comprehension, and teacher and administrator efficiency and administrative tools. 

The types of studies ranged from usability studies (measuring if the product is easy to use), feasibility studies (measuring if the product is feasible to use as intended within the realities of a school day), implementation studies (measuring under what conditions is the product best used, and at scale), and correlational studies (what relationship might exist between usage and student outcomes).

What We Found

• Gamification

  • Gamification is a growing trend featured in edtech products, but companies may need additional support to implement gamification effectively.

• Training

  • Ineffective training for product use was a concern; most companies overloaded content in their training sessions rather than offering it on a bite-sized/need-to-know basis.

• Usability

  • Poor intuitive user design in some instances led to lower uptake of the product.

• Implementation

  • Logistical difficulties such as students being unable to log in, navigating support in a virtual classroom, student absences, or companies not having a usable implementation guide was a relatively consistent finding across four of seven products investigated limiting both product use and related research. 

• Attrition

  • Correlational studies saw drastic attrition and statistically insignificant results

---

These findings suggest that more companies can benefit from research that focuses on user experience and the contextual realities of school communities that can improve the uptake of products.  

Since the 2015 passing of the Every Student Succeeds Act (ESSA), schools and districts have been equipped with the ESSA Tiers of Evidence framework, that supports schools in evaluating interventions and technologies based on the rigor of independent research interventions they have engaged in to evaluate the performance of the tool. 

The Tiers—ranging from Tier I (strong evidence from an experimental study) to Tier IV (demonstrates a rationale)—help companies boost credibility, however they can, and should, be leveraged as important guide posts to inform product development that best serves learning outcomes for students.

This poses a larger question to the field; how might we reframe the ESSA Tiers of evidence so that they support the realities of iterative product development cycles while also focusing on the realities of classroom implementation?