- Category: Non-fiction
- Rating: 4 out of 5
- Tags: Decision-making, Evidence, Meta-analysis, Statistics, Systematic reviews
- How I learned about it: Author interview on Nature podcast
I might have to retract my criticisms of the medical industry for its late adoption of evidence-based decision-making, which only started in the 1970’s and didn’t really take off until the 1990’s. It’s a late start for a discipline rooted in science, and what the heck were they using before? It turns out there’s more to it than I thought.
Pearson tells how evidence-based decision-making was adopted in various fields by following the people who helped to lead the change. The story quickly becomes familiar – someone new to the field notices there’s little evidence to justify what’s considered best practice, so it falls to the most senior (or the loudest) person in the room to decide how it’s done. Justifications range from “we’ve always done it that way,” to “the textbook says to do it that way,” or “I read it in last month’s journal.” There was no systematic way of gathering evidence to decide what was best. This was true in medicine, social policy, policing, environmental conservation, business management, and education.
It’s a no-brainer to base decisions on evidence, but finding, assessing and summarizing that evidence is hard. For one thing, randomized controlled trials weren’t widely used in medicine until the 1960’s or 1970’s. The data just wasn’t there. And there are different kinds of evidence, from large, high quality randomized controlled trials to lab-based research results. Trials can have different sizes, methods and ways of measuring results, so are hard to combine. Finding the data requires searching journals and theses, which used to mean paging through paper copies in the library. The relevance and quality of the studies have to be assessed, the data extracted and merged, and conclusions summarized.
Meta-analysis and systematic reviews may sound routine now, but it wasn’t until the mid-1970’s that statisticians figured out a way to combine the results of different studies. It was an exciting development, as researchers realized they could finally “extract meaning from a mountain of similar studies.”
Like any innovation that challenges established practice, there was opposition – experts don’t like being challenged with evidence. Evidence can reveal “uncomfortable truths”, like the fact that most social programs don’t help. Of course, we can only discover this if policies are studied to see if they work (or even cause harm), and most are not evaluated. It’s the same for environmental conservation efforts, partly because field experiments are so difficult. And for business management, which is “a largely evidence-free zone,” there are tons of books and articles touting the flavour of the week, but it’s even harder to experiment on humans than on the environment, so there’s little in the way of follow-up studies to show which practices are the best.
We can be grateful to the statisticians who figured out how to combine disparate data. It’s more exciting, and easier to get funding, to do a new study than to gather existing data from dusty archives, so we should also be grateful to those dedicated to finding, assessing, combining, and summarizing the results of multiple studies. It’s getting easier now with internet search and AI summarizing capabilities, but it’s still a daunting task.
Visit my blog at Books I’ve Read, where you can search and filter by category, rating, tag and date.