This is a very clear critique and I think you could have added even a few more things like temporal effects and the file drawer problem. I was duped into this stuff to some degree a few years ago but luckily for me I focused more on the feedback professional learning and the VL platform was a useful framework for building self evaluation in the school. Effects sizes however seemed a pat way to explain evidence based approaches to almost everything and why you might be clever enough to be pushing a strategy in school. But then I started to read more around effect sizes and understood its pitfalls. Like you Greg, this started me on a road of discovery, albeit not a PhD, in relation to the poor evidence for a lot of what was being pushed by academics and commercial companies supported by large scale glitzy companies - I even spoke at a few. The truth is out there and as a school leader, I am now firmly on the path of targeting work promoting explicit teaching, cognitive load theory ( including talking to staff about primary and secondary biological knowledge and fluid and crystallized intelligence) and an in school project we call the Big 5 which is focused on Dylan Wiliam's five key embedded formative assessment strategies. However, this is frustrating work and there is extreme deafness in the system.
Mr.-Dr. Ashman, Mr. Jay, & Mr. F., please allow me to wade into the coversation here. (Sorry if I used mistaken courtesy nouns of address there!)
Thank you! Thank you all for letting me barge in and for your comments.
This is a potentially fruitful conversation, so I'm glad to be able to participate.
Mr.-Dr. Ashman, I think your post includes important points. Those points include (but are not limited to) these:
(a) The influence of standard deviations on ESs (especially when the SDs for compared measures differ; as you rightly note, under that condition, one should use a pooled SD) must be considered; if one changes the denominator of a fraction (e.g., an ES), that makes a difference.
(b) Selection of samples (randomly chosen from what population?) and assignment to conditions (randomly assigned?) can influence outcomes of an study, whether it is a quasi-experiment or even an "randomized control trial"...just how are students randomized into groups (i.e., conditions)? Sequential coin flips or rolls of the dice? Stratefied on some basis (e.g., pretest)? Etc.
(c) How different types of instructional methods might be tested can make differences. Pre- vs. post-test studies might (probably will!) produce different outcomes than RCTs. Correlational (i.e., descriptive studies) are likely to produce different (and more widely variable) ESs than other types of studies.
Well, here's the thing: These are important concerns. They do not, in my view, however, provide sufficient reason for rejecting analyses of ESs. I see them as arguments for good practice in meta-analysis. A meta-analyst could code for these differences: She could (a) have something simple like a 0-1 code for each study about whether it used control SD or pooled SD (and, obviously, the code could be expanded to cover other variations); (b) include codes for sampling and sample assignment methods employed in each study; and (c) categorize (i.e., code) types of research designs for the studies in the corpus of the review.
The big point here is that scientifically savvy meta-analysts _code_ for these sorts of differences in a corpus of studies and then they _analyze_ the ESs according to those codes. That is, they go beyond simply reporting overall effect sizes; they report moderators—not just age, gender, ethnicity, and such, important as they may be—that affect those main affects, but also methodologic moderators.
So, Jay, I'll agree with Prof. Hattie about the importance to look beyond simple ESs. Code one's studies thoroughly and carefully. Get to know the studies like a primary teacher gets to know her first graders. But, do it systematically, so that one can test questions about their ESs (e.g., do correlational studies yield different ESs than RCTs?).
So, should I accept or dismiss any concerns about Hattie's research? Definately maybe. But I'd prefer to defer my decision until advocates of either position demonstrate the actual evidence that objectively shows why I should go left or right, up or down, forward or backward.
Now, if I may, let me add this tag: I hope we as as educators concerned about evidence and research, press to have our researchers use methods of open science so that we can examine results from meta-analyses (and other research) systematically. Publish not just your report of your meta-analysis, but your actual freaking coding system and your data.
When Ollie Lovell challenged Hattie about effect sizes I think he said something like - you can't just look at effect sizes, instead, you have to look at the story. And it would be a mistake to just look at the effect sizes. This seems to defeat Hattie's point of calculating effect sizes. It's also hard because he probably makes a lot of money out of it (he mentioned how extremely well his books have sold). So the best we're left with it's well meaning pseudo science?
Hi Greg, I don't know if it's a coincidence, but the respected science periodical Nature posted a longish piece just yesterday on the EEF and the toolkit's findings: Might be worth a response if you have time. The article is entitled "COVID derailed learning for 1.6 billion students. Here’s how schools can help them catch up" by Helen Pearson.
This is a very clear critique and I think you could have added even a few more things like temporal effects and the file drawer problem. I was duped into this stuff to some degree a few years ago but luckily for me I focused more on the feedback professional learning and the VL platform was a useful framework for building self evaluation in the school. Effects sizes however seemed a pat way to explain evidence based approaches to almost everything and why you might be clever enough to be pushing a strategy in school. But then I started to read more around effect sizes and understood its pitfalls. Like you Greg, this started me on a road of discovery, albeit not a PhD, in relation to the poor evidence for a lot of what was being pushed by academics and commercial companies supported by large scale glitzy companies - I even spoke at a few. The truth is out there and as a school leader, I am now firmly on the path of targeting work promoting explicit teaching, cognitive load theory ( including talking to staff about primary and secondary biological knowledge and fluid and crystallized intelligence) and an in school project we call the Big 5 which is focused on Dylan Wiliam's five key embedded formative assessment strategies. However, this is frustrating work and there is extreme deafness in the system.
Mr.-Dr. Ashman, Mr. Jay, & Mr. F., please allow me to wade into the coversation here. (Sorry if I used mistaken courtesy nouns of address there!)
Thank you! Thank you all for letting me barge in and for your comments.
This is a potentially fruitful conversation, so I'm glad to be able to participate.
Mr.-Dr. Ashman, I think your post includes important points. Those points include (but are not limited to) these:
(a) The influence of standard deviations on ESs (especially when the SDs for compared measures differ; as you rightly note, under that condition, one should use a pooled SD) must be considered; if one changes the denominator of a fraction (e.g., an ES), that makes a difference.
(b) Selection of samples (randomly chosen from what population?) and assignment to conditions (randomly assigned?) can influence outcomes of an study, whether it is a quasi-experiment or even an "randomized control trial"...just how are students randomized into groups (i.e., conditions)? Sequential coin flips or rolls of the dice? Stratefied on some basis (e.g., pretest)? Etc.
(c) How different types of instructional methods might be tested can make differences. Pre- vs. post-test studies might (probably will!) produce different outcomes than RCTs. Correlational (i.e., descriptive studies) are likely to produce different (and more widely variable) ESs than other types of studies.
Well, here's the thing: These are important concerns. They do not, in my view, however, provide sufficient reason for rejecting analyses of ESs. I see them as arguments for good practice in meta-analysis. A meta-analyst could code for these differences: She could (a) have something simple like a 0-1 code for each study about whether it used control SD or pooled SD (and, obviously, the code could be expanded to cover other variations); (b) include codes for sampling and sample assignment methods employed in each study; and (c) categorize (i.e., code) types of research designs for the studies in the corpus of the review.
The big point here is that scientifically savvy meta-analysts _code_ for these sorts of differences in a corpus of studies and then they _analyze_ the ESs according to those codes. That is, they go beyond simply reporting overall effect sizes; they report moderators—not just age, gender, ethnicity, and such, important as they may be—that affect those main affects, but also methodologic moderators.
So, Jay, I'll agree with Prof. Hattie about the importance to look beyond simple ESs. Code one's studies thoroughly and carefully. Get to know the studies like a primary teacher gets to know her first graders. But, do it systematically, so that one can test questions about their ESs (e.g., do correlational studies yield different ESs than RCTs?).
So, should I accept or dismiss any concerns about Hattie's research? Definately maybe. But I'd prefer to defer my decision until advocates of either position demonstrate the actual evidence that objectively shows why I should go left or right, up or down, forward or backward.
Now, if I may, let me add this tag: I hope we as as educators concerned about evidence and research, press to have our researchers use methods of open science so that we can examine results from meta-analyses (and other research) systematically. Publish not just your report of your meta-analysis, but your actual freaking coding system and your data.
Thanks for letting me interrup!
JohnL
John Wills Lloyd, Ph.D.
Professor Emeritus, UVA School of Ed & HD
Co-editor, Exceptional Children
Editor, https://www.SpecialEducationToday.com/
When Ollie Lovell challenged Hattie about effect sizes I think he said something like - you can't just look at effect sizes, instead, you have to look at the story. And it would be a mistake to just look at the effect sizes. This seems to defeat Hattie's point of calculating effect sizes. It's also hard because he probably makes a lot of money out of it (he mentioned how extremely well his books have sold). So the best we're left with it's well meaning pseudo science?
Hi Greg, I don't know if it's a coincidence, but the respected science periodical Nature posted a longish piece just yesterday on the EEF and the toolkit's findings: Might be worth a response if you have time. The article is entitled "COVID derailed learning for 1.6 billion students. Here’s how schools can help them catch up" by Helen Pearson.
https://doi.org/10.1038/d41586-022-01387-7