The Education Endowment Foundation's Toolkit is a complete mess

It is now time to quietly let it go

I think we were all hopeful when the Education Endowment Foundation (EEF) was founded ten years ago in the UK. Here was a new organisation, generously endowed with public money and with a remit to both rigorously evaluate the available evidence on educational approaches and to conduct its own randomised controlled trials. Unfortunately, the reality has been disappointing.

Through its Toolkit, the EEF has chosen to use the method of meta-meta-analysis to synthesise evidence. This involves grouping interventions into categories and then computing an average effect size for that particular class of interventions, which the EEF then misleadingly interprets as ‘months of additional progress’.

I’ll get to the practice later, but this is deeply flawed in principle. Any student of school statistics knows that simple averages can be misleading and so we often use median values or include measure of spread. However, this is only the start of the problem. Effect sizes from different studies will vary according to the quality of the study, the age of the students, whether the outcome assessment is designed by the experimenters, whether the subjects are a representative sample and a whole range of other factors. A high average effect size is therefore more likely to indicate poor quality underlying studies than it is to indicate the likely effect of adopting this approach in the classroom.

The EEF approach is a strange hybrid that illustrates this problem. Each strand started out by averaging effects from a range of meta-analyses not conducted by the EEF. However, over time, the EEF has gradually added the results of any of its own, more rigorous trials.

Practice has compounded the problem. The categories designed by the EEF are a strange grab-bag of methods covering a wide range of subjects and outcomes. I think we all hoped that the EEF would avoid capture by the progressivist ideology that suffocates most educational organisations, not least because this ideology tends to oppose rigorous, quantitative research due to fact that such research tends to prove its tenets wrong. However, it is in the selection of these categories and grouping of approaches that EEF researchers have made the preconceptions known.

The Toolkit strand I have spent the most time analysing is ‘Metacognition and Self-Regulation’. This strand covers everything from sensible approaches to explicitly teaching writing - tellingly, there is no ‘explicit teaching’ Toolkit strand - to the frankly bonkers idea that primary school ‘philosophy’ lessons will improve reading and maths scores. The time lag between the EEF conducting new studies and factoring them in to their Toolkit strands compounds the problem.

For instance, when I first started examining this strand, the positive result of a study called ‘Thinking Learning Doing Science’ was included but the negative result of a study based on the same principles, ‘Let’s Think Secondary Science’ was not. It took a while for the EEF to catch-up with a new revision of the strand that had a lower effect size.

This is similar to the current situation. The strand still includes the positive result from a 2015 trial of ‘Philosophy for Children’ - the study which implausibly appeared to demonstrate the link between philosophy lessons and maths scores. Many of us criticised this trial, not least for omitting any tests of statistical significance, and we appear to have been vindicated by the follow-up trial that found no such result. However, this new study has not been factored in to the Toolkit strand. Yet.

I think the EEF are aware of the problem they have created, because they seem to have shifted their emphasis away from these Toolkit strands in recent years and towards producing guidance reports. This is helpful, but it is not enough. Again, the metacognition and self-regulation strand illustrates why.

Metacognition has become a safe haven for educational pseuds and chin-strokers. It has inevitably become the latest excuse for discredited progressivist methods. There is definitely something of value there. I can buy in to the idea that students might benefit from learning study techniques such as self-quizzing - although I suspect the effect would be less than for a teacher-designed programme. And the recent Ofsted mathematics research review* made an excellent case for teaching subject-specific deep structure in order for students to develop the capacity of ‘knowing when’ to deploy their declarative and procedural knowledge, a capacity we could accurately describe as ‘metacognitive’. Unfortunately, these more solid and prosaic ideas get crowded out by the silly stuff and the headline effect size.

So the Toolkit has to go. It has had its time on the pedalo and now needs to come back to shore. It is wrong in both principle and practice and is misleading. The EEF already has an alternative - its guidance reports - and so the focus should shift to these.

*There was a strange episode where a group of UK maths researchers criticised this report. I blogged about that here and Ofsted have now released their (immensely patient) response.