Does ed tech need its own Consumer Reports?

The recent surge of investment and energy in education technology is bringing a new wave of tools to schools across the country. But determining which options are most appropriate and effective for specific classrooms is becoming an increasing challenge for teachers and administrators.

Education blogs and newsletters, as well as social platforms such as Edmodo and Schoology, are helping educators discover new products and services. But a proposal recently presented to the nonprofit Hamilton Project calls for a Consumer Reports-like independent third party to evaluate and rate new learning technologies.

Submitted by two business school professors, Duke University’s Aaron Chatterji and Northwestern’s Benjamin Jones, the nonprofit program, called EDU STAR, would quickly and rigorously assess new learning tools and then report the results to the public. Their goal is to encourage innovation in ed tech by reducing buyer uncertainty on the part of the schools and lowering the cost of marketing for entrepreneurs.

Not ‘if’ but ‘when’

To start, EDU STAR would work with a group of schools (potentially a school district) and entrepreneurs to test instructional content designed to address the Common Core State Standards. Over time, it would expand to learning platforms, such as Edmodo and Class Dojo, that don’t target specific curriculum goals. The proposal says EDU STAR would seek $5 million in funding from U.S. Department of Education grants and other sources. (The Huffington Post has more details on the program’s nuts and bolts.)

Considering the number of new services coming online and the limited resources schools have for evaluating and purchasing them, a third-party evaluator makes a lot of sense.

“This is not an ‘if’ but a ‘when’,” said David Balter, co-founder of ed tech startup Smarterer and CEO of marketing firm BzzAgent.

As long as it gives teachers guidance around their specific learning environment, he said, a good ratings system could help educators figure out where to spend their money and make them more comfortable spending money on new technology in general. And it could raise the bar for ed tech companies across the board.

Can research keep up with developers?

But although ed tech innovators say an independent evaluator is appealing in theory, some worry that it won’t work in practice.

Jeremy Friedman, co-founder and CEO of Schoology, said he thinks a Consumer Reports-style system would be too hard to implement because evaluators would likely be unable to keep up with constantly iterating developers. Adaptive systems that personalize learning are especially quick to update their processes, he said, and by the time a test group uses a product and researchers collect the data, the product might have already moved on to its next version.

That kind of ratings approach could actually harm the system by discouraging those who innovate, he said.

“Many web companies practice agile development, and some practice continuous deployment, meaning they are often pushing code multiple times throughout a single day,” he said. “Imagine trying to make a valid review for a product like that.”

Teachers need context, not just a rating

Alan Louie, co-founder of ed tech accelerator ImagineK12, said the idea for an independent evaluation system isn’t new, as educators increasingly say they need guidance on navigating the expanding world of ed tech. But he said there are different models trying to serve that need.

EdShelf, an Imaginek12 startup, for example tries to address the discovery and filter problem for teachers looking at ed tech with a more Yelp-like approach. It relies on reviews from teachers who supply grade level and topic information, as well as additional notes on the context in which they used the software.

The online resource EdSurge also offers product reviews and reports that include teacher reviews as well as information across a number of categories.

What’s important, Louie and others say, is that a ratings system provide the granularity teachers need to make decisions – from considering class size and learning level to teachers’ and students’ familiarity with technology.

“The big question for education is under what circumstances does it work?,” said Louie. “Educational tools are subject to specific usage and use cases. There are so many variables.”