Charities need to understand which parts of their activities are working and which aren’t. But to really understand the charity’s impact, we need to know not only what did happen, but what would have happened without the charity’s work.

Imagine a city with poor air quality. Suppose that a charity works there, trying to persuade inhabitants to turn off their car engines when idling at traffic lights. When the charity started work, the air was clean 10% of the time (it has some sensible definition for air cleanliness: perhaps the air has fewer than 10parts per million of particulates), whereas a end late, it was clean 20% of the time. Result! Really? Perhaps the improvement was due to the charity; but perhaps it would have happened anyway; and perhaps more improvement would have happened without the charity. (It’s not unknown for irritating campaigns to provoke people to ‘rebel’ and do the opposite of what’s being asked.)

These kinds of ‘before/after studies’ indicate precisely nothing about whether the charity is doing a good job. It’s scary because they’re very common amongst charities (and others). If you think about it, the example above is analogous to an educational charity saying: ‘At the beginning of the year, the average height of a child we helped was 1.2 metres whereas afterwards it was 1.3 metres’! Or someone saying ‘we banged some pots together and then it stopped raining’ or even ‘we elected an Afro-Caribbean President and then the economy tanked’. We can’t infer causation from simple before/after statements.

To see why something happened, we need a control, that is, a situation in which everything is the same apart from the charity’s work.  This will show which of the scenarios above applies and therefore whether the programme is actually doing anything useful.

Celebrated development economist Professor Esther Duflo of the Massachusetts Institute of Technology says that without control situations for social or charitable programmes, we’re no better than medieval doctors putting leeches on their patients. If right now somebody put a revolting slimy leech on your arm, you too would probably claim to feel just fine. It is only by comparing ailing people treated with and without leeches that mankind learnt that the slimy bugs weren’t helping and could move to more effective treatments. Charities too need to test whether their programmes are working by using controls.

Sometimes getting a control is easy: run the programme in one class at school and not the other (keeping everything else the same between them) and look at the difference between them at the end. Sometimes it requires a bit of imagination: if you (like me) run national campaigns to influence behaviour, then look not only at changes in behaviour of people who have seen the campaign but also in people very unlikely to have seen it.

But without controls, we might as well be handing out leeches: we’ve essentially no clue what’s causing what, and thus we ourselves have no control.

How a proper control trial showed one programme is 25 times better than another–>

(‘Control’ article also published by TrusteeHome)