I often say, largest educational study ever and the team that won wasn't supposed to, at least that's what the other 21 models and their supporters/funders had hoped. Hence, the results were an inconvenience to many largely, heavily funded interventions. It was a nuisance and so hiding the entire project under the rug must have felt like the only solution to those folks. Had we listened to the results back in 1960s, what would our world be like, especially for those numerous disadvantaged students who would have otherwise thrived under DI because it offered a way to build knowledge. I often get emotional presenting PFT to our teacher candidates. Doing our part by not hiding it anymore.
Greg, thanks for bringing this topic forward. I had the good fortune to learn from the people who led the DI model (S. Engelmann, W. Becker, D. Carnine, and others) and the bad fortune of seeing the Follow Through demonstration diluted and discounted. As a researcher, I understand that there are legitimate questions one may raise about aspects of the evaluation (e.g., it was an evaluation, not a formal experiment). There are also important strengths in the evaluation that are rarely mentioned (e.g., the outcome data such as achievement test scores, were collected by a third party who had no reason to favor the 1000s of children in one of the models over the 1000s in other models; likewise, Abt Associates, which analyzed the data, was an independent party with no connections to favor one group over another; yet another third party assessed implementation).
These (and other) strengths add support for your summary of the findings. And those outcomes were pretty dang clear. Even though the evaluation was set up so that there were measures that would tap outcomes aligned with subgroups of the models, the kids in the 40 DI model schools in 13 different geographical locations had better scores on the outcomes across the domains. They would be expected to be stronger on the "lower order" aspects of academic learning (decoding & computation), of course, but they also did better on the higher-order areas (comprehension and math problem solving) and social emotional outcomes (self-concept and attributions for success).
Now, I might quibble with one or two characterizations in your analysis, but they're mostly accurate. To be sure, for example, there was variability in the outcomes of all the different models. For example, one of the scores from one of the local education agencies that is counted as a DI site were markedly lower than the others; as it happened, soon after the study began, a new leader for that LEA stopped the implementation of DI in those schools, but the data collectors and data analysts for Follow Through included the data from those schools as if DI had been implemented in them.
And, though your post about Follow Through wouldn't have to include it, I think it is valuable to note that even if some people would dismiss the FT results, negating FT does not remove the scores of other studies examining the effects of DI. As summarized (and meta-analyzed) by Jean Stockard and her colleagues (https://doi.org/10.3102/0034654317751919), those studies provide substantial indications that DI helps students learn rapidly, thoroughly, deeply, and happily.
When I was in college studying to be a teacher, I'll never forget getting into a somewhat intense (though respectful) discussion with a professor (the head of the education department). He was -- along with most ed professors -- a strong supporter of progressivist methods. I brought up Follow Through, and he had never heard of it. He had been an ed professor for something like 40 years; he had published numerous articles and books. Yet, he hadn't heard of what was possibly the largest educational study ever. It blew my mind...
Perhaps the best summary I've read is ‘The Story Behind Project Follow Through’ by Bonnie Grossen in which she explains the background, implementation, results, and the takeaways of this study. Since PFT, there exists a mountain of evidence, research, and data (program-specific and independent) to support the efficacy of Direct Instruction programs. DI has always followed the science of how kids learn even before 'the science of learning' became a thing.
In my opinion, the biggest disservice to the engineered content behind DI methodology is not whether the results of a mega experiment like PFT validated it or not (which it absolutely did) but the idea that using a few elements of DI or a mash up of various approaches (insert choice of alternative methodologies, checklists) works better for 'at risk' students.
Here's a sentiment I share from a fellow advocate, "Picking from a DI program is like picking apart a car, laying it all on the side of the road and thinking "Yeah, that should run."
I don't know. I'm sympathetic to the idea that there's it's a gestalt. But the end goal can't be a take-it-or-leave it commercial product. It needs to ultimately be decomposable into its effective elements and their interactions.
(1) I get the interest is the "decomposable elements." There actually were studies during the early days of DI that examined the entire classroom approach. If one examines the actual early studies by B. Rosenshine, one will learn that some of the classrooms he observed in his process-product research that led to his "direct instruction" were—hold on to those hats!—were using Engelmann's methods.
(2) The DI model was more than the instructional materials, indeed. DISTAR was a big part of it, including other practices and procedures (e.g., teacher-directed classrooms; rewarding of school-appropriate behavior) that many critics find anathema.
I think we're in full agreement. That there absolutely are elements of DI that can be taken out and put back together in various ways - and that many of those have fed, directly and indirectly, into modern Science of Learning, explicit instruction alá Rosenshine, etc
I often say, largest educational study ever and the team that won wasn't supposed to, at least that's what the other 21 models and their supporters/funders had hoped. Hence, the results were an inconvenience to many largely, heavily funded interventions. It was a nuisance and so hiding the entire project under the rug must have felt like the only solution to those folks. Had we listened to the results back in 1960s, what would our world be like, especially for those numerous disadvantaged students who would have otherwise thrived under DI because it offered a way to build knowledge. I often get emotional presenting PFT to our teacher candidates. Doing our part by not hiding it anymore.
Greg, thanks for bringing this topic forward. I had the good fortune to learn from the people who led the DI model (S. Engelmann, W. Becker, D. Carnine, and others) and the bad fortune of seeing the Follow Through demonstration diluted and discounted. As a researcher, I understand that there are legitimate questions one may raise about aspects of the evaluation (e.g., it was an evaluation, not a formal experiment). There are also important strengths in the evaluation that are rarely mentioned (e.g., the outcome data such as achievement test scores, were collected by a third party who had no reason to favor the 1000s of children in one of the models over the 1000s in other models; likewise, Abt Associates, which analyzed the data, was an independent party with no connections to favor one group over another; yet another third party assessed implementation).
These (and other) strengths add support for your summary of the findings. And those outcomes were pretty dang clear. Even though the evaluation was set up so that there were measures that would tap outcomes aligned with subgroups of the models, the kids in the 40 DI model schools in 13 different geographical locations had better scores on the outcomes across the domains. They would be expected to be stronger on the "lower order" aspects of academic learning (decoding & computation), of course, but they also did better on the higher-order areas (comprehension and math problem solving) and social emotional outcomes (self-concept and attributions for success).
Now, I might quibble with one or two characterizations in your analysis, but they're mostly accurate. To be sure, for example, there was variability in the outcomes of all the different models. For example, one of the scores from one of the local education agencies that is counted as a DI site were markedly lower than the others; as it happened, soon after the study began, a new leader for that LEA stopped the implementation of DI in those schools, but the data collectors and data analysts for Follow Through included the data from those schools as if DI had been implemented in them.
And, though your post about Follow Through wouldn't have to include it, I think it is valuable to note that even if some people would dismiss the FT results, negating FT does not remove the scores of other studies examining the effects of DI. As summarized (and meta-analyzed) by Jean Stockard and her colleagues (https://doi.org/10.3102/0034654317751919), those studies provide substantial indications that DI helps students learn rapidly, thoroughly, deeply, and happily.
When I was in college studying to be a teacher, I'll never forget getting into a somewhat intense (though respectful) discussion with a professor (the head of the education department). He was -- along with most ed professors -- a strong supporter of progressivist methods. I brought up Follow Through, and he had never heard of it. He had been an ed professor for something like 40 years; he had published numerous articles and books. Yet, he hadn't heard of what was possibly the largest educational study ever. It blew my mind...
Not surprising at all, still sad though!
Perhaps the best summary I've read is ‘The Story Behind Project Follow Through’ by Bonnie Grossen in which she explains the background, implementation, results, and the takeaways of this study. Since PFT, there exists a mountain of evidence, research, and data (program-specific and independent) to support the efficacy of Direct Instruction programs. DI has always followed the science of how kids learn even before 'the science of learning' became a thing.
In my opinion, the biggest disservice to the engineered content behind DI methodology is not whether the results of a mega experiment like PFT validated it or not (which it absolutely did) but the idea that using a few elements of DI or a mash up of various approaches (insert choice of alternative methodologies, checklists) works better for 'at risk' students.
Here's a sentiment I share from a fellow advocate, "Picking from a DI program is like picking apart a car, laying it all on the side of the road and thinking "Yeah, that should run."
Bonnie's analysis is just flat excellent. Thanks for pointing to it.
I don't know. I'm sympathetic to the idea that there's it's a gestalt. But the end goal can't be a take-it-or-leave it commercial product. It needs to ultimately be decomposable into its effective elements and their interactions.
Please, Alex, consider these points:
(1) I get the interest is the "decomposable elements." There actually were studies during the early days of DI that examined the entire classroom approach. If one examines the actual early studies by B. Rosenshine, one will learn that some of the classrooms he observed in his process-product research that led to his "direct instruction" were—hold on to those hats!—were using Engelmann's methods.
(2) The DI model was more than the instructional materials, indeed. DISTAR was a big part of it, including other practices and procedures (e.g., teacher-directed classrooms; rewarding of school-appropriate behavior) that many critics find anathema.
I think we're in full agreement. That there absolutely are elements of DI that can be taken out and put back together in various ways - and that many of those have fed, directly and indirectly, into modern Science of Learning, explicit instruction alá Rosenshine, etc
This reminds me a lot of the Millenium Challenge 2002 with Lt. Gen. Van Riper. He wins, so they change the goal posts.