This two-part series first appeared in Stanford Social Innovation Review.

So what should happen if no one has properly evaluated an idea yet? If it’s important, an independent and suitably skilled researcher should evaluate it in enough detail and in enough contexts for other charities and donors to rely on the findings. The leading medical journal The Lancet cites a tenet of good clinical research: “Ask an important question, and answer it reliably.”

A countercultural implication follows from this. It’s often said that the evaluation of a grant should be proportionate to the size of the grant. It’s also often said that evaluations should be proportionate to the size of the charities. We can see now that both views are wrong. The aim of an evaluation is to provide a reliable answer to an important question. From there, the amount worth spending on an evaluation is proportionate to the size of the knowledge gap and the scale of the programs that might use the answer.

To illustrate, suppose a small company has developed a new drug for breast cancer. The “first-in-(wo)man studies,” as they’re called, involve only a few people, for obvious safety reasons. Relative to the cost of dispensing the drug to those few women, how much should the company spend on evaluating the effect on them? The answer is “a lot,” because the answer is important for many people. So the cost of the “pilot” is irrelevant. So too is the size of the company running the “pilot.” Often, the cost of robustly evaluating a program will exceed the cost of delivering that program—which is fine, if the results are useful to a wide audience. Continue reading →

This two-part series first appeared in Stanford Social Innovation Review.

Most “evaluations” of charities’ work are done by the charities themselves and are a waste of time. Perhaps this is a surprising view for an advocate who thinks that charitable work should be based on evidence—but it’s true because charitable activity should be based on good quality,robust evidence, which isn’t what many charities can reasonably be expected to produce.

What is evaluation?

Before we get into why this is true, let’s get clear about what evaluation is, and what it isn’t.

The effect of a charity’s work depends on the quality of the idea (intervention) it uses, and how well it implements that idea. Both idea and implementation need to be good for the impact to be high; and if either idea or implementation is low, the impact will be low. Think of it as:

impact = idea x implementation

To illustrate the difference, consider a breakfast club in a school for disadvantaged children. The idea is that a decent breakfast aids learning by avoiding the distractions of hunger. The implementation involves having foods that the children will eat, buying them at a good price, getting children to show up, and so on. Continue reading →

The government’s new What Works Centres should be awesome. And they should be a fantastic and free resource for charities and donors and others, which we can use to dramatically improve effectiveness. 

What are you on about?

Wouldn’t it be a good idea if government pr

ogrammes, rather than being based on hearsay and prejudice, were based on some assessment of whether they’ll actually work, for example by looking at what mankind has already discovered about similar programmes?

Well, yes, obviously. So the surprise isn’t that these What Works Centres are being set up now, but that they’ve not been set up before.

Everybody knows that some interventions work and some don’t. Schools vs. trepanning come to mind, respectively. We also know that some interventions work better than others, and that some only work in particular circumstances. But it’s hard to figure out which are which: the evidence is disbursed, complex and varies in reliability. Hence horribly few decisions about government spending are based on decent evidence, and probably horribly few in charities and foundations too. Continue reading →