WHY A CONTROL GROUP IS NEEDED TO DETERMINE IF A K-12 EDUCATION PROGRAM IS EFFECTIVE, in 2 simple charts. The example I'll use is Xtreme Reading - a program for struggling 9th graders that provides year-long reading instruction in a small classroom setting (in lieu of an elective).

​

• Xtreme Reading was evaluated in a large, high-quality study, commissioned by the U.S. Institute of Education Sciences (sample of 2,833 students in 17 schools nationwide).

• As shown below, the study found that reading scores of students in the program rose from the 15th to 25th percentile over the course of 9th grade:

• But the study had a randomly-assigned control group of students who didn't participate in the program, and control-group students saw an almost identical gain:

• So the study found that students participating in Xtreme Reading didn't do discernibly better in reading than an equivalent group of students who didn't participate (and received usual school services). In other words, the program was not effective.

• If the program had been evaluated in the usual non-rigorous way - looking at participants' reading gains over time without reference to a control group of nonparticipants - it would've been (erroneously) deemed highly effective.

• You may wonder why the control group's reading scores - like the treatment group's - increased over the course of 9th grade and (and is that atypical?).

• Two likely (and not atypical) reasons: (1) students naturally make reading gains in 9th grade, based on usual services provided in the 17 study schools; (2) students were selected for the study based on low baseline test scores and "regressed toward the mean" when later re-tested.

• CONCLUSION: Simply looking at before and after improvement in student outcomes, without respect to an equivalent group of nonparticipants (controls), often yields the wrong answer about program effectiveness.