Back in 2015, I wrote about a study that had recently been released. The study was a ‘gold standard’ randomised controlled trial that began in 2011 and was funded by the Investing in Innovation (i3) initiative of the U.S. government. The trial found struggling first grade readers who had participated in a programme known as ‘Reading Recovery’ made significantly more progress with their reading than similar students who were not offered the programme.
At the time, I argued that this study did not demonstrate much, largely because Reading Recovery consisted of proprietary Reading Recovery techniques bundled together with one-to-one tuition and additional reading instruction. This was compared to the control group which was a complex mix of things where students were given additional support of varying kinds or none at all. It was therefore fallacious to assume that it was the Reading Recovery techniques that made the difference.
A better approach would have been to compare Reading Recovery will a similar dosage of one-to-one structured literacy tuition, because structured literacy is the main competitor for the hearts-and-minds of teachers and schools.
The idea of varying only one thing at a time is a key scientific principle that is critical to good quality educational psychology research — the kind I have attempted to conduct through my PhD work. However, broader education studies are often messier and people tend to accept this messiness. I found myself arguing about this in the comments*.
My assumption was that one-to-one tuition of any kind would be beneficial and that this was likely to be causing the positive effect — my understanding of the proprietary Reading Recovery techniques is that they do not align with the science of reading. However, I never expected these techniques could have a negative effect. Perhaps I should have done.
Over at APM Reports, Emily Hanford and Christopher Peak are reporting the results of a new study presented at the annual conference of the American Educational Research Association (AERA). First some caveats. The study is currently a working paper and has not been peer reviewed. Moreover, it does not report on a randomised controlled trial but instead a ‘regression discontinuity’ study.
What’s a regression discontinuity study? Well, imagine we used a screening test to assign students to an intervention and we set the cut-score on that test at 20 marks out of 40. A student scoring 19 would be placed in the intervention but a student scoring 20 would not. However, scores of this kind are noisy. The student scoring 20 could, on a different day, have scored 18 and the student scoring 19 could, on a different day, have score 22. So we have a population of students who are very similar and their chance of being placed in the intervention is effectively random. We can then compare the results for students in the intervention against similar students who did not receive it.
Why would we use such a design? It allows us to approximate a randomised controlled trial using data that was not collected as part of a randomised controlled trial. Given the expense and complexity of conducting randomised controlled trials, particularly ones that last for years, this is useful. In some cases, it may be the only feasible way of teasing-out long-term effects. This is pertinent in the case of the i3 randomised controlled trial because all participants eventually received Reading Recovery in the first grade, making it impossible to examine long-term effects.
I am no expert on regression discontinuity studies and so I cannot comment on whether the researchers have done a good job. Instead, I will describe their approach.
First, they tracked a similar body of students to the original i3 randomised controlled trial but applied the regression discontinuity approach. They effectively replicated the results of the original study, obtaining a similar effect.
So far, so good for both Reading Recovery and the researchers. Reading Recovery seems to lead to gains in first grade and the regression discontinuity approach appears to be valid.
However, because of the regression discontinuity design, the researchers were also able to track these students in the second, third and fourth grades. This is when they found something shocking — by the third and fourth grade, students who had received Reading Recovery were doing significantly worse in their reading than similar students who had not.
We cannot rule out some systematic difference between the two groups of students. Lots of students where lost from the study as they moved in and out of schools, and different schools used different cut-scores, meaning there could have been some other factor schools were taking into account.
However, one possible explanation is that even though it gives an initial boost, Reading Recovery sets children up badly for later reading. If Reading Recovery teachers are emphasising the use of context to guess unknown words — rather than using phonics to decode them — this may be a strategy that offers diminishing returns as books become more complex, pictures less frequent, and other contextual hints become less obvious. If students have been habituated into these strategies — strategies of last resort used by poor readers — then that could be hampering their performance.
If nothing else, this research casts doubt on the i3 trial. When we triangulate these results with what we know from elsewhere about effective reading instruction, we may have just lost the last reasonable justification for investing in expensive Reading Recovery interventions.
*Professor Jim Thornton, a medical researcher with expertise in randomised controlled trials, seems to have spotted a subtle but perhaps even more fatal error in the design of the study.
Having watched 11 - 12 year olds read, who have done Reading Recovery in the past, I can see how the effect could be negative. Despite the page having no pictures, their eyes dart around the page, perhaps searching for 'clues' rather than focussing on the print in front of them. The habit is hard to shake at that age. Trying to prove whether an intervention is successful by whether intervention is needed again at a later stage might not be the best design for a study. Often students with LD will need continuous intervention and support - no matter what they had when they were younger, just because of the nature of their difficulties.
I’ve seen negative effect size results in studies for others US “reading“ Ed Tech companies — to them this is no catastrophe, just a new assignment for the marketing dept.