The world-renowned Cochrane Collaboration has recently published a systematic review of the evidence about mass programmes to treat children in less developed countries for intestinal worms. It found that “deworming children seems like a good idea, but the evidence for it just doesn’t stack up”. Giving Evidence’s blog and book have vaunted deworming, using it to show how some programmes are much cheaper than equally sensible-sounding alternatives, which have also been tested by rigorous randomised control trials (RCTs) conducted by academics and peer-reviewed by others.

“When the facts change, I change my mind. What do you do sir?” – John Maynard Keynes

Deworming is based on the logic that worms share a child’s food, so reduce the child’s nutrition and hence their physical and cognitive development, and, by sapping their energy, also the child’s ability to attend school. Some RCTs have indeed found this, but by looking at all RCTs of mass deworming programmes, the Cochrane study found:

a)      some studies which found that mass deworming had no effect on a child’s weight, cognitive development or school attendance, and

b)      that some large studies haven’t been published. They include one which looked at a million children in India. Obviously they can’t conclude much from an inaccessible study, but withholding a study is normally a sign that it’s found either no effect or an unwelcome effect. This ‘publication bias’ is precisely why it’s valuable for somebody independent and knowledgeable to review all the data systematically, which is why Cochrane was set up.

Cochrane’s study – which concerns only mass treatments for soil-transmitted helminths (worms) and is its third on that subject – has been greeted with criticism from the authors of some of the studies which found it effective, and Cochrane has defended itself. Here’s the BMJ on it.

The striking feature of this row is not the detail or merits of deworming, but that these rows can happen at all. They are precisely how science is supposed to work. Many charities don’t make their impact data public, or publish only small samples, or don’t use control groups in their studies, or have a million other flaws in their data such that these debates couldn’t even happen.

Systematic reviews save lives: for example, premature babies were sometimes given  steroids but sometimes they weren’t because the evidence of whether they prevented ‘complications’ (which Ben Goldacre says is ‘a medical term meaning death’) seemed ambiguous. A systematic review of the evidence showed that steroids do save lives and hence are now normally given. The information to save lives were all there but only a systematic review showed up the real insight. Cochrane’s systematic reviews only consider randomised control trials – because they are a uniquely robust way to identify the impact of an intervention – and amongst them, only those which have sensibly large sample sizes and which are publicly available. That is, it is only because the development people are so sophisticated in rigorously collecting data and publishing it that this analysis can happen.

The organisations whose data are being contested should be proud that their data are capable of being contested in a way that will ensure that the true story of the data can emerge. With Professor Dean Karlan of Yale University, I wrote an op-ed on this topic.

What else can philanthropy learn from science and medicine? Masses–>