A Classroom Software Boom, but Mixed Results Despite the Hype - NYTimes.com

The federal review of Carnegie Learning’s flagship software, Cognitive Tutor, said the program had “no discernible effects” on the standardized test scores of high school students. A separate 2009 federal look at 10 major software products for teaching algebra as well as elementary and middle school math and reading found that nine of them, including Cognitive Tutor, “did not have statistically significant effects on test scores.

So much large-scale education research ends with "no discernable effects." I mean, like almost all of it. Seriously.

The linked article examines some of the research claims made about Cognitive Tutor, made and distributed by Carnegie Learning. I'm familiar with this software, and it is probably the best of what is out there in terms of educational software. The content is solid and the system is adaptive to the students. But the research results are mixed, depending on who you talk to.

Looking at the research, one could come to the conclusion that all educational interventions are crap. After all, if an intervention doesn't stand up to the rigor of scientific investigation, we probably shouldn't use it, right?

Well, another way to look at this is that maybe the "gold standards" of scientific rigor are not appropriate for educational contexts. Large-scale, randomized trials make sense for medical trails, but not for educational interventions. Why? Because educational systems are complex and have multiple inputs. But wait, isn't your body a complex system too.

Dr. David Agus was the guest on The Daily Show last night (2 Feb 2012) and suggested that medical thinking has been stunted ever since the discovery of pathogens. They make us think that all diseases have a cause and a cure. Dr. Agus suggests that diseases like cancer and Alzheimer's are different. They don't have a single cause; the body has to be treated as a whole.

Similarly, if we expect to improve educational outcomes for all students, we aren't going to do it by giving the same treatment to all students. These educational treatments need to be delivered at the right time, to the right student, by a trained professional (or computer that can aid a teacher).

The frustrating thing is, who's going to get on board with a program that doesn't meet rigor, especially when the bar might be unattainable?