top of page

Annenberg Institute posted a quasi-experimental study of a California program to train teachers in the science of reading. Quick take: Despite study claims of positive impacts (covered by NY Times and others), it found no discernible effect on 3rd grade reading achievement in its primary analysis.

Program:

  • The study evaluated California’s Early Literacy Support Block Grant, providing low-performing elementary schools with teacher training in the science of reading, new funding (~$1000/student), spending flexibility, expert assistance, and other supports to improve reading.

 

Study Design:

 

  • The study's primary, prespecified evaluation method (set out in the study registration) was to compare outcomes over 1 year for (i) schools that were just above a threshold percentage of struggling readers (and thus received the program) to (ii) schools just below that threshold (which were ineligible for the program). This type of study is known as “regression discontinuity.”

 

Findings:

  • Using this method, the study found no statistically significant effect on the percent of 3rd graders scoring near-proficient or higher in reading after 1 school year. In fact, the impact table (shown below) shows a slight decrease of 4-7% points (though not statistically significant).

  • The study abstract unfortunately does not mention the disappointing results of the primary analysis, & instead presents the study's findings as unambiguously positive based on post-hoc (vs prespecified) analyses that are far less credible.  

  • They're less credible because, in post-hoc analyses, researchers have wide discretion over key study parameters - e.g., which schools comprise the comparison group, and which analysis methods to use - and can consciously or unconsciously select parameters that yield a desired result.

Comment:

  • Unfortunately, the study received uncritical coverage in the NY Times, Education Week, and elsewhere - coverage which largely repeats the study abstract's positive claims.

Science of Reding CA.png
bottom of page