American Radio Works has recently released a five-part documentary on “personalized learning in the digital age.” In the second chapter they take on one of the more interesting and controversial threads of scholarship on learning and technology: Bloom’s Two-Sigma Effect and the computer tutor.
Everyone knows that working one on one with a tutor is preferable to learning in a classroom of 25 or 30 people. In a famous and widely cited paper, Benjamin Bloom, the eminent educational psychologist at the University of Chicago and Northwestern and creator of Bloom’s Taxonomy of higher-order cognition, attempted to quantify the advantage of tutoring.
His paper, based on the PHD dissertations of two of his students, compared three groups of math learners. The control class had 30 students and one teacher. The “mastery learning” class also had 30 students and one teacher, but in this class, after the students took a test they went over what they got wrong on each test, corrected their errors, and re-took a new test covering the same material with the goal of mastering each lesson before moving on to the next. In the third group, students never attended classes, just worked with an expert tutor in groups of one, two, or three students.
In Bloom’s study, the average student under tutoring scored about two standard deviations above the average student in the typical class–that is, the average of the tutored group was better than 98% of students in the control class. In addition to better scores on the aptitude tests, the tutored students showed more positive attitudes, more interest, and spent 90+% of their time on task. (The mastery group also beat the control by a whole standard deviation, which is impressive considering they had no more resources). This has been dubbed The Two Sigma Effect, and it’s become a cornerstone of the idea that personalized tutoring, or something very like it, is the gold standard we should be aspiring toward when creating educational technology (under the assumption that we’ll never have the human resources to provide actual human tutoring to every single student).
Bloom’s paper is convincing, but updated research suggests that its findings are exaggerated. An updated review of research published in 2011 by Kurt Van Lehn at Arizona State University found, surprisingly, that human tutors were 0.79 sigma more effective than no tutoring, not the 2.0 sigma found by Bloom. Furthermore, intelligent computer tutoring is already almost as good as human tutoring, with a 0.76 mean effect size across studies.
Intelligent tutoring doesn’t just deal with whether an answer is right or wrong. Computer systems today can see into the steps of the process: choosing a method to solve the equation, entering the lines, and then solving the equation.
VanLehn told American Radioworks that it’s not that surprising that computer tutors function nearly as well as human tutors:
You learn by experience, you learn by doing. And a computer tutor just helps a person do. It helps them move forward. It helps them stop getting frustrated and stuck, gives them hints when they need it, gives them prompts, gives them encouragement. That’s pretty much what a human tutor does, too.
This is a counterintuitive claim. Most people don’t want to believe that computer tutoring is quite so far advanced, for otherwise, why isn’t everyone using it? Rich people pay hundreds of dollars an hour for access to live tutors for their children, while computer-based tutoring programs (aka “adaptive learning”) are just at the beginning of the adoption curve. There seems to be a lingering, almost mystical belief in what has been dubbed “the warm-body effect” –the idea that just being in the same room with a human teacher improves student motivation and behavior.
Of course, the kind of experience you get working one-on-one with a coach or tutor is not comprehensive to all kinds of education. It’s still important to collaborate, debate, work solo on a project, and even run up against limitations and conflict. But paying careful attention to what the research tells us about the impact of computer-based tutoring could have a huge upside.