So what should happen if no one has properly evaluated an idea yet? If it’s important, an independent and suitably skilled researcher should evaluate it in enough detail and in enough contexts for other charities and donors to rely on the findings. The leading medical journal The Lancet cites a tenet of good clinical research: “Ask an important question, and answer it reliably.”

A countercultural implication follows from this. It’s often said that the evaluation of a grant should be proportionate to the size of the grant. It’s also often said that evaluations should be proportionate to the size of the charities. We can see now that both views are wrong. The aim of an evaluation is to provide a reliable answer to an important question. From there, the amount worth spending on an evaluation is proportionate to the size of the knowledge gap and the scale of the programs that might use the answer.

To illustrate, suppose a small company has developed a new drug for breast cancer. The “first-in-(wo)man studies,” as they’re called, involve only a few people, for obvious safety reasons. Relative to the cost of dispensing the drug to those few women, how much should the company spend on evaluating the effect on them? The answer is “a lot,” because the answer is important for many people. So the cost of the “pilot” is irrelevant. So too is the size of the company running the “pilot.” Often, the cost of robustly evaluating a program will exceed the cost of delivering that program—which is fine, if the results are useful to a wide audience.

Conflicted out

So not only are most charities unskilled at evaluations—and we wouldn’t want them to be—but also we wouldn’t want most charities to evaluate their own work even if they could. Despite their deep understanding of their work, charities are the worst people imaginable to evaluate it because they’re the protagonists. They’re selling. They’re conflicted. Hence, it’s hardly surprising that the Paul Hamlyn Foundation study found “some, though relatively few, instances of outcomes being reported with little or no evidence to back this up.”

I’m not saying that charities are corrupt or evil. It’s just unreasonable—possibly foolish—to expect that people can be impartial about their own work, salaries, or reputations. As a charity CEO, I’ve seen how “impact assessment” and fundraising are co-mingled: Charities are encouraged to parade their self-generated impact data in fundraising applications. No prizes for guessing what happens to self-generated impact data that isn’t flattering.

To my knowledge, nobody’s ever examined the effect of this self-reporting among charities. But they have in medicine, where independent studies produce strikingly different results to those produced by the protagonists. Published studies funded by pharmaceutical companies are four times more likely to give results favorable to the company than are independent studies. It’s thought that around half of all clinical trial results are unpublished, and it doesn’t take a genius to figure out which half that might be.

Who should do evaluations?

Skilled and independent researchers, such as academics, should normally take on evaluation of ideas. They should be funded independently and as a public good, such that charities, donors, and others can access them to decide which ideas to use. It’s no accident that The Wellcome Trust, the UK’s largest charity, requires that all the research it funds is published in open-access journals.

The charity itself can normally take on monitoring of implementation.

Useful resources and ideas for funders

Academics and others have evaluated ideas well and published the results. Smart funders use that material to avoid funding something that is known to be useless or even harmful. To be fair, much available information could be easier to find and understand. But some of it is already easily accessible:

On first world education, The Education Endowment Foundation (a £125 million fund of UK government money to improve education for 5-16 year olds in England) has collated and analysed evidence on interventions in many countries, and created a wonderful toolkit (“menu”) that shows the quality of the evidence and the apparent strength of the intervention. The foundation is rigorously evaluating all the interventions it funds, publishing the results, and adding them to the toolkit.

In international development, several entities publish all their evaluation findings. They include the Abdul Latif Jameel Poverty Action Lab at MIT (J-PAL) and Innovations for Poverty Action, which have collectively run some 400 impact evaluations. Also, the International Initiative for Impact Evaluation (3ie) database includes more than 600 impact evaluations, as well as systematic reviews of all the evidence on particular topics. Recent systematic reviews have looked at small-scale farmers, female genital cutting, and HIV testing.

In health, The Cochrane Collaboration uses a network of more than 28,000 medics in over 100 countries to produce systematic reviews of many types of intervention. Its database already has more than 5,000 such reviews, including some related to health care in disaster and emergency situations.

Various universities have centers that produce and publish such research. One example is Oxford University’s Centre for Evidence-Based Intervention, which looks at social and psychosocial problems.

The UK government’s new What Works Centres will create libraries of evidence about crime and policing, aging, and various other sectors in which UK charities and donors operate.

These are free resources. Smart donors and charities use them. And they publish their own evaluations—with full methodological detail—so that others can learn from them. It’s essential that charities’ work is evaluated properly so that resources can flow to the best. That means appropriately skilled and independent people should evaluate a charity’s work only when necessary.

What does decent evidence look like? Like this–>

Most “evaluations” of charities’ work are done by the charities themselves and are a waste of time. Perhaps this is a surprising view for an advocate who thinks that charitable work should be based on evidence—but it’s true because charitable activity should be based on good quality,robust evidence, which isn’t what many charities can reasonably be expected to produce.

What is evaluation?

Before we get into why this is true, let’s get clear about what evaluation is, and what it isn’t.

The effect of a charity’s work depends on the quality of the idea (intervention) it uses, and how well it implements that idea. Both idea and implementation need to be good for the impact to be high; and if either idea or implementation is low, the impact will be low. Think of it as:

impact = idea x implementation

To illustrate the difference, consider a breakfast club in a school for disadvantaged children. The idea is that a decent breakfast aids learning by avoiding the distractions of hunger. The implementation involves having foods that the children will eat, buying them at a good price, getting children to show up, and so on.

Assessing the quality of implementation is relatively easy. Do children come, what is their feedback on the breakfast club, how much of the food gets wasted, etc.? This is monitoring. It’s vital, and by rapidly providing feedback to staff, it enables the organization to improve its processes and improve them dramatically (as Bill Gates discussed in his annual letter). This monitoring (or “process evaluation”) should happen almost always, and the charity can normally do it themselves.

Notice, however, that monitoring looks only at the performance of that one organization, so it will never (on its own) tell you whether a charity is good relative to other places you might put your money—that is, whether funding that charity is a good idea.

Assessing the quality of the idea is rather harder. That involves investigating whether a decent breakfast actually does aid learning. And that requires isolating the effect of the intervention from other extraneous factors. This is impact evaluation, and “evaluation is distinguished from monitoring by a serious attempt to establish attribution,” says Michael Kell, chief economist at the (UK) National Audit Office.

It’s hard. In our example, we can’t look simply at whether the children who attend breakfast club are now learning better than before: Perhaps the club starting coincided with a new teacher arriving, or better books arriving, or children suddenly watching more Sesame Street. (Organisations could look at these factors in a pre/post study, and many charities do, but they prove little if anything.)

Neither can we look at whether children in breakfast club learn faster than those who don’t, because it’s highly likely that there will be major differences between those who come and those who don’t: Perhaps only the worst learners attend. To get around that, we’d have to do a randomised control trial. That throws up complicated questions such as whether to randomise children, or schools, or towns; and what sample size to have. But even then we’re not out of the woods. We might have to deal with “spill-over effects” (benefits to children who don’t go to club—for example, those who do go may be less disruptive in class); and “cross-over effects” (such as children who attend the club giving food to children who don’t go). These and many others are all normal questions in such research.

Hence, establishing attribution—which is integral to evaluating an idea—is a whole field of social science research.

Monitoring is of the implementation.

Evaluation is of the idea.

Immediately we see that monitoring and evaluating are totally different exercises, despite the fact that they are often used as though they are identical.

Most charities aren’t comprised of social scientists

It’s reasonable to expect charities to monitor their work: How many trains do you run, are they on time, and what do passengers think of them? As with companies reporting the number of units they’ve sold, we might audit those figures, but there’s not normally anything technically difficult in monitoring.

By contrast, evaluation is social science research, and it is hard: What effect do trains have on economic growth? Most charities aren’t able to run evaluations because they don’t have the skills in-house. We can see this in a recent review by the Paul Hamlyn Foundation of reports from grantees (possibly the only analysis of its type): 70 percent of evidence presented by charities ranked below what the foundation called “good.”

The good news is that most charities don’t need these research skills. Once we know whether and when breakfast clubs work—once they’ve been evaluated rigorously—then we know and don’t need to evaluate them again (unless the context is very different). To take a medical analogy, we don’t expect every hospital to be a test-site. The clinical trials are done somewhere—properly, with luck—and then published so that everybody else can use the results.

Often, the ideas used by charities don’t need to be evaluated again, because they’ve been amply evaluated already. Charities—and funders and others—can use those existing evaluations to choose effective interventions. All the charity then needs to do is run the programs well. Charities need to be skilled at implementation—at running breakfast clubs, or community transport, or drug rehab centres. By the long-established law of comparative advantage, we should let them do what they’re best at and not ask them also to get good at the totally unrelated skills of social science research.

Part two: So who should evaluate anything then?

Giving Evidence is launching a new project to improve the effectiveness of education in less developed countries, by seeing what and how it can learn from evidence-based medicine. Performance has improved dramatically in medicine with the rise of evidenced-based practice: for example, deaths from infectious diseases in the US fell by over 95% during the 20^th century. By contrast, educational learning levels are often abysmal: for example, in India, a third of 8-9 year olds can’t recognise simple words.

The move to evidence-based practice in medicine took decades. What can education emulate which will enable similar performance improvements? This project aims to find out. And inviting you to get involved.

Is medicine analogous to education?

Yes, up to a point. For example, in almost every country, education and healthcare are both delivered universally by government (with private alternatives for those who can afford them) and hence are major parts of government policy and spending. They share difficulties with incentives: funded by tax, neither the institutions (hospitals and schools) nor practitioners (doctors, nurses and teachers) are paid directly by the people they serve. Both have information asymmetry between providers and consumers (pupils don’t know what they need to learn, and patients don’t know what they need to take or do). Both have thousands of practitioners who go through a system for training and qualification, normally run or overseen by the state. 

But they’re not identical, and hence lessons from health can’t automatically be transferred. For example, the goal of medicine (healthy people) is easier to define and less contentious than that of education (People who can pass exams? People able to get jobs? People who are fulfilled?) Education is normally a batch-process and proactive, whereas most healthcare is individual and reactive. Teachers don’t self-select for ability in science, like doctors do, and education is more context-dependent than many medical interventions are.

Where the project will focus

Giving Evidence will look at:

1. Current practice in evidence-based medicine. What enables the helpful parts to work, such as: training and skill-levels, institutions (e.g., academic journals, watchdogs, regulators), funding mechanisms, politics, relationships (e.g., between practitioners and academics), tools (e.g., metrics and analytical frameworks), traditions, ‘customer’ expectations and norms. What hinders good practice, which education in less developed countries should not emulate, such as distortion of the evidence-base by private operators.

2. How did medicine get to work this way? How can education in less developed countries transition to evidence-based practice faster than healthcare did? What short-cuts and blind alleys are now visible?

Get involved

If you are involved in education in less developed countries – as a provider, funder, academic, regulator, trainer or commentator – we’d love to talk to you. The project involves many senior medical practitioners and academics.

Giving Evidence is seeking further funding for this project: the research part is funded by an American family foundation, though we know that research reports are insufficient to change practice and hence need to engage actors in education to mobilise them (e.g., through creating toolkits, and running seminars, webinars and workshops). We are seeking funding for this latter part.

Clearly, lessons from evidence-based medicine pertain to many parts of charitable and philanthropic activity, not just education, and we are open to discussions about translating them to these other areas.

What can giving learn from physics and law and history? —>

This is the first analysis which shows (doesn’t just argue) that high-performing charities spend more on administration costs than weaker ones do.

So it’s unarguably wrong-headed of Parliament’s Public Accounts Committee to be considering limiting charities’ admin costs. It’s unarguably wrong of donors such as Gina Miller to suggest that admin costs be capped. The data indicate that such caps would nudge donors towards choosing weaker charities, at untold cost to their beneficiaries.  It’s time for this to change.

The analysis

Judging a charity’s quality  is hard. Some of the most rigorous analysis is by GiveWell, a US non-profit run by former Wall Street analysts, whose analysis is often dozen of pages. GiveWell looks for various sensible indicators of quality, including: a strong documented track record of impact; highly cost-effective activities; and a clear need for more funds.

Charities which GiveWell reviewed in 2011 and recommended, spend 11.5% of their costs on administration, on average. Charities which GiveWell reviewed and didn’t recommend spent less on administration, only 10.8% on average.

This is no freak result. The same pattern was true in 2009:

In 2009, GiveWell had four levels of ranking, and the pattern is even more pronounced if we use those:

The clear conclusion is that low admin costs do not signal that a charity is good. They signal the converse. So it’s wrong of charities to parade their low admin costs, as we’ve argued here before. Rather better is looking at what the charity actually achieves. 

Admin costs are discussed in detail in It Ain’t What You Give, and when we understand what’s included, we can understand the pattern these data show. ‘Admin’ includes systems for capturing learning, for improving, for reducing costs. It’s spending on those things which enables good performance. Scrimping on them is often a false economy.

Assessing a charity by its admin spend is like assessing a teacher on how much chalk they use, or assessing a doctor on how many drugs they prescribe: they’re easy measures but don’t relate to performance. As Einstein said: ‘Not everything that counts can be counted. Not everything that can be counted counts.’

This isn’t to say that there isn’t waste in charities. There is: masses, much of it avoidable, and good charities try to avoid it. But don’t expect to find it clearly labelled in the financial statements.

Why is this the case?

—

The data behind these graphs and analysis are all here. Press release from 2nd May 2013 is here.

—

A ton more insight to inform your giving–>

Want advice on your giving (or your company’s giving)? Here’s how–>

What does good evidence of impact look like? Not like this–>

Social impact bonds (SIBs) are a high-profile innovation in funding public services. The pilot SIB in Peterborough, UK, which aims to reduce recidivism, has been widely watched and—despite not yet producing results—already widely emulated.

Given the international interest in SIBs and similar payment by results (pay-for-success) schemes, it’s important to determine whether the Peterborough SIB works. The Ministry of Justice describes the program’s evaluation method as “the Rolls Royce of evaluation.” However, Professor Sheila Bird of Cambridge University and the UK Medical Research Council says: “[It] might well be a brilliant success; it might achieve little. But we aren’t going to know either way.”

This article examines three aspects of determining whether the SIB works.

The first is straightforward: whether the investors should be repaid. Determining this will be easy, because it depends solely on the re-offending rate and the contractual terms—both of which will be clear.

Second, whether the intervention itself works to reduce re-offending—a central question. Determining this will be more difficult, because this first SIB is using a variety of interventions—only some of which have been evaluated rigorously and the combination has never been evaluated.

The issue is attribution: figuring out whether the re-offending rate amongst the Peterborough prisoners has anything to do with the charities’ work which the bond funds. Both sides agree that the way to see what the charities have achieved is to compare:

1. The one-year re-offending rates of men with whom the charities work.
2. The one-year re-offending rates of a group of similar men with whom the charities haven’t worked. This “control group” screens out effects of, say, changes in society, the law, or sentencing procedures.

It’s essential that the “treatment group” and control group be effectively identical beforehand; if they are, the sole difference between them is the program, which alone must account for differences in re-offending rates between the groups. Bird would have liked the treatment group and control group to have been selected at random to ensure that the groups were effectively identical. But this isn’t what is happening. Social Finance says it was impossible: within the prison, the program is advertised and open to anybody whose sentence is a year or less. Prisoners are used to—and exasperated by—being apparently arbitrarily excluded from things, and neither Social Finance, the nonprofit company that invented social impact bonds and is running the Peterborough pilot, nor the prison governor wanted this program to generate ill-will in that way. Social Finance says that its “investors wouldn’t tolerate excluding some people.” Sheila’s view is that random selection inside prisons (as outside them) is not only possible, but also pretty common.

If randomising prisoners wasn’t possible, the next best option would have been randomising prisons: In other words, several randomly selected prisons would get the program while others wouldn’t, and the re-offending rates of their populations would be compared. Social Finance says that this wasn’t possible either, because the Ministry of Justice would never have allowed a pilot in several prisons at once.

Interestingly, Peterborough prison wasn’t chosen at random, but rather because the prison governor was willing to engage. As Bird remarks, that may indicate an usual trait in the governor, which itself may influence the results. It’s not impossible that a prison governor willing to take on this innovative project is unusually progressive in other respects too: perhaps Peterborough prison offers other unique programs that could skew the results.

To construct a control group, the bond evaluation uses Propensity Score Matching (PSM), a system often used when samples can’t be randomised. With PSM, you start by figuring out what indicators have historically correlated with eligibility for the treatment (propensity to be eligible). In this case, prisoners at institutions other than Peterborough who have the same “propensity scores” as the treatment group serve as a control group. Social Finance is doing an unusually elaborate PSM by having about ten “control” prisoners for each “treatment” prisoner.

Nonetheless, there are major objections to PSM as a way of attributing any effects observed. One is that PSM can only ever look at indicators that are observable, such as age, background, and criminal history. Yet it’s often unobservable factors—such as attitude or resilience—that drive behaviour.

Another problem is that the only data available for the PSM are what’s stored in the Police National Computer, which is surprisingly basic. For instance, it can’t distinguish whether somebody has mental health problems or a history of heroin use, which obviously would influence their behaviour and the care they need.

Astonishingly, even the Ministry of Justice explicitly acknowledges that the control group may be pointless (see page 7 of this Ministry of Justice document about the evaluation).

The third respect is whether the bond structure itself works. Social Finance says that the mere existence of this first bond proves that it is possible. It prove possible to  define performance criteria against which a public body agreed to repay, and to find private donors willing to provide funding based on those criteria.

But when we eventually see the re-offending rates of the treatment and control groups, we won’t know whether to attribute any differences to:

• Social Finance’s particular mix of interventions
• The money. The SIB brings in about £1,667 pounds per prisoner. Bird thinks any prison governor could use that amount to dramatically reduce re-offending. It’s possible that the prison governors could out-perform Social Finance’s program.
• The new financing mechanism itself. We won’t know whether it produces better outcomes than if that money had been put into that intervention through, say, a grant program.

The core problem might be that Social Finance is delivering on a contract: it isn’t doing social science research, to which distinguishing between possible causes is central. So does the difficulty of seeing the effect of the financing mechanism itself matter? Well, not for Social Finance or its donors in this first instance. Their proximate issue is delivering the contractual obligations such that they get paid. But surely it would have been helpful to Social Finance’s future work to see the effect of the SIB mechanism itself.

It certainly matters to the Ministry of Justice, which 1) may end up paying for a service that didn’t achieve anything beyond what that particular prison governor would have achieved without that money, and 2) won’t therefore know what service they should roll out to other prisons if the Peterborough service does apparently succeed.

It matters even more to UK taxpayers who are funding all of this—as well as hoping not to be burgled or mugged. Yet they’re unlikely to object because the intricacies of randomisation and PSM for determining attribution are a shade too complex.

“All these problems could have been averted,” says Bird. She says, for example, that this first SIB could have been tested against a known intervention with a conventional funding mechanism.

And yet, we should not let the best be the enemy of the good. Clearly, we are likely to get better public services when the interests of the provider and purchaser are better aligned, and SIBs are a step in the right direction. Despite the Peterborough SIB’s curious design choices, it has taught us many things—and will teach us many more.

Why asking ‘what does this achieve’ doesn’t usually help much—>

Birthday Price! Just £12.99 (+ P&P. This is for the UK: for RoW, see below.)

“A terrific read” – The Guardian

Ben Goldacre: “You’ve been waiting for this:  Evidence-based charitable giving”

“Indispensable … dispels the fog … no-nonsense …exactly the guide that donors need … refreshingly rigorous … long overdue” – Spears Wealth Management

“Refreshing … relentlessly logical … fresh and forceful … an important contribution” – Center for Effective Philanthropy

“Very helpful” – a letter in The Guardian

“Thank goodness … engaging, informative, irreverent … compelling evidence and logic” – Alliance Magazine / Institute of Philanthropy 

“Strongly recommend … read this book and absorb its lessons” – Stephen Lloyd, UK’s top charity lawyer

“Huge amounts of insights … this is money well spent” – Philanthropy UK

“Cocky and quirky in equal measure …inspiring … engaging … different and refreshing …thoroughly recommend … a must-read” – Whitebarn Consulting

“Wonderfully easy-going and informative” – reader

“Intelligent, methodical and respectful … I love your sense of fun too” – reader

“Very annoying. Made me miss my tube stop  ” – reader

All that is on top of other reader reviews and endorsements on the physical book, incl.:

The Freakonomics of the charity world – with better cartoons’ – Martin Houghton-Brown, CEO, Missing People

‘Great advice: inspiring, entertaining and much-needed’ – James Caan, Dragons’ Den panellist

‘A unique and very clear guide to a very complex topic, with insight for any donor’ - The Body Shop

‘Caroline Fiennes explains how to balance heart and mind for serious philanthropy. She emphasises with clarity the importance of evidence and economics for to maximise good deeds per dollar’– Simon Singh, science writer.
For orders in Europe (excl. UK):

For orders outside Europe:

Here’s how to read a load of it for free–>

The government’s new What Works Centres should be awesome. And they should be a fantastic and free resource for charities and donors and others, which we can use to dramatically improve effectiveness. 

What are you on about?

Wouldn’t it be a good idea if government pr

ogrammes, rather than being based on hearsay and prejudice, were based on some assessment of whether they’ll actually work, for example by looking at what mankind has already discovered about similar programmes?

Well, yes, obviously. So the surprise isn’t that these What Works Centres are being set up now, but that they’ve not been set up before.

Everybody knows that some interventions work and some don’t. Schools vs. trepanning come to mind, respectively. We also know that some interventions work better than others, and that some only work in particular circumstances. But it’s hard to figure out which are which: the evidence is disbursed, complex and varies in reliability. Hence horribly few decisions about government spending are based on decent evidence, and probably horribly few in charities and foundations too.

The What Works Centres are being set up by government to collect and sort out the evidence in five areas of UK domestic policy: Ageing Better, Reducing Crime, Local Economic Growth, Early Intervention, and Education. They’ll be funded by central government and to some extent by the Big Lottery Fund, and run independently of government.

It’s free?

Yes. The Centres will make freely available the data they find and their analysis of it. Charities and donors in those sectors will get access to top-notch analysis of the evidence on which they can base decisions – for free. You could hardly make it up.

The education centre will be run by the Education Endowment Foundation (which has £125m of government money, run outside government, to figure out how to improve education for 5-16 year olds in England). They’ve already started analysing the evidence on loads of education interventions and publishing it for free, and it’s very handy for many charities, donors and schools.

This all sounds a lot like NICE (the National Institute for Health & Clinical Excellence) 

It is. NICE collates, analyses and publishes evidence on health interventions, in order to decide which should get funded by HM Taxpayer, and the idea of the What Works Centres was first dubbed ‘a NICE for social policy’.

NICE will be one of the What Works Centres. There will be four new centres, which with EEF, make six in total.

But it’s not as simple as saying that something ‘works’ or doesn’t

Indeed. So the modelling on NICE – and health care generally – is important and instructive. Often in health, something will work differently on some people than in others: Asian bodies metabolise some drugs differently from European bodies, for example. Hence the What Works Centres will look at what works when, and why, and what needs to be in place to get it to work. This is pretty standard in social science research as well as in health: the systematic reviews by the International Initiative for Impact Evaluation in international development, for example, look at this.

Isn’t this the government telling us what to do?

No. It’s the government making public the evidence for what it’s doing, and if we want to use that evidence to inform our decisions, then we can. If we read their analysis and think it’s rubbish, we can ignore it. And if we’d prefer to make decisions based not on evidence but on hearsay and prejudice, then I guess we can, though it’s hardly likely to optimise performance.

Won’t all this inhibit innovation?

No. It will focus it, improve it and amplify it.

Clearly, the purpose of innovation is to find whether a new idea works better than the best existing idea. So we need rigorous evaluations to see whether any particular innovation is worth having. If it is, we should share the idea; if not, we should kill it. The What Works Centres should be very valuable here, in doing that evaluation and getting out the mega-phone when we find an improvement.

For those of us keen that charities and giving be based on evidence, this is a boon.

Why asking ‘does this work’ or ‘what’s our impact’ is pointless–>

It starts well enough, with a surprising ‘fact’ (is this true?) that the Giving Pledge generated seven percent of all traffic on Twitter in 2010.

But then it becomes clear that the author isn’t talking about why philanthropy matters: he’s talking solely about why American philanthropy matters to America. He appears unaware of – or at least unmoved to comment on – any other country anywhere. It’s fine to write books about America, but don’t pretend that they’re universally applicable.

“To understand philanthropy is to understand the American psyche…” Well not if you’re understanding philanthropy elsewhere.

“When philanthropy is absent, rent-seeking flourishes”. This is a pretty weird claim: philanthropy is higher in countries which have a small role for the state and low taxes – it’s low in France, for example, not because French are evil or selfish but because the French, unlike the Americans, think that all roads should be cleaned, not just those which a donor wants to clean. (Much more on the red herring of bald comparisons of national giving rates here.) Is France – or other big-state places like Norway – full of outrageous rent-seeking? It’s hardly what they’re famous for.

Back on the Giving Pledge, he talks of it having been signed by “some American business leaders”. Like David Sainsbury, a Brit; Richard Branson, another Brit; Chuck Feeney, an Irishman; Mo Ibrahim, who’s Sudanese; the Russian Vladimir Potanin; Chris Hohn or Michael Moritz, more Brits; Hasso Plattner who’s German; or George Lucas one doubts defines himself as ‘a business leader’.

Then it has lots of statements which just seem untrue or garbage:

“Only philanthropy that results in economic and social opportunities…has positive externalities”. So support for churches or architectural preservation or, species conservation, or most arts, or even soup-kitchens don’t have positive externalities (economist-speak for ‘benefit other people’)? Neither does caring for the dying?

“In France, you must leave your money to your family.” Must? You can’t leave it to a charity or the church? Really? Shame there’s no reference for that curious statement.

“Philanthropy has all along been an unstated principle at the heart of American-style capitalism”. Really? Odd that it gets such scant press as such.

Towards the end of the first chapter, he ‘fesses up: “This book is a reflection on contemporary American-style capitalism”. In which case, perhaps the title should have said so.

Philanthropy is fundamentally about noticing other people and responding to what they need. So it’s depressingly ironic when people discuss it without noticing the 99% of humanity outside their own borders.

A book entitled ‘why philanthropy matters’ could have been really interesting: about the important things which need doing in a society which cannot be done by corporate or commercial money and which therefore must be done philanthropically. Basic scientific research, promoting rights for disabled people or abused people, environmental protection, pushing for legislative change: in fact anything where the timescale is too long, or risk too high, or benefit too diffuse, to enable a commercial return, and/or where the work involves influencing government. Philanthropy is vastly important sometimes to prevent something becoming privately owned: galleries are good examples, and the work funded by Wellcome Trust to ensure that the human genome is freely available is perhaps the best example. Current work on defending free speech is literally vital and couldn’t happen other than philanthropically.

A list of inventions and developments funded philanthropically which influence us all would be fascinating, and inspiring, and would amply show why philanthropy matters. It could even feature the surprising result of a scholarship to a Kenyan student: his son, America’s first black US President.

Peter Singer on why people should give & how to get them to give—>

Caroline Fiennes on how to give well—>

“Ask an important question, answer it reliably, and publish the results promptly, irrespective of the findings”: good advice for any experiment, from an article in The Lancet about how results of the biggest clinical trial ever were withheld for five years, basically because they were surprising. It’s good advice too for charities and donors thinking about ‘monitoring and evaluation’, much of which addresses questions which aren’t important (e.g., ‘what was our impact’) and uses research too ropey to provide decent answers.

How to design “unreasonably effective” development programmes: this great paper draws on research into psychology and behaviour to show how we sometimes misdiagnose problems and hence design ‘solutions’ which don’t work. Gives several principles which have helped create many more effective programmes. Paper from the Center for Global Development.

How to raise money: behavioural insights. Two studies (RCTs) show that donors give more if: (a) the mail-out includes a pre-filled bank form, and the donor gets a reminder, and (b) the charity cites a major donor who’s already committed.

Why philanthropy is no substitute for tax: Mega-gifts are much flaunted and vaunted in the US. But “of the 50 largest individual gifts to [American] charities in 2012, not a single one went to a social-service organization or to a charity that principally serves the poor. More went to elite prep schools than to any of [the] largest social-service organizations, including United Way, the Salvation Army, and Feeding America (which got, among them, zero).

Underlying our [America's] charity system—and our tax code—is the premise that individuals will make better decisions regarding social investments than will our representative government. Other developed countries have a very different arrangement…our charity system is fundamentally regressive, and works in favor of the institutions of the elite.” Important article in The Atlantic.

Data quality: The quality of data in charities’ reports to funders is generally pretty poor, found the Paul Hamlyn Foundation. In a study of reports it has received, data in 15% was poor, only 30% was good, and some claims had no evidence at all. 

Counterfactuals: “Perhaps my donations (to a Ugandan water project) are simply relieving the government of its obligation to provide water, and allowing it to buy missiles instead”, says a donor. The donor complains that they’ve asked the charity to investigate… though of course they don’t get a good answer from the charity because the incentives are all wrong: you wouldn’t expect independent system analysis from a protagonist.

What to analyse? Psychologists discover that educated American minds are quite different to everybody else’s. They’re much more prone disaggregate a problem, analysing the bits in isolation from each other. Asians (and others) are more holistic. No wonder (a) they have a small state and hence high giving, (b) comparing US giving to that elsewhere is widely misleading, (c) their style of giving is quite different (more flashy etc.) (d) Western minds aren’t good at analysing the whole situation, which is what the Ugandan water donor above was finding, among other things.

How do get people to act on evidence:  some lessons from J-PAL‘s work with NGOs, governments and donors on precisely this. In short: it’s hard but possible.

Why is encouraging giving like selling chocolate? –>

Evidence comes in many forms, some distinctly better than others. Below is a hierarchy produced by NESTA. Is it any good?

Level 1 is essentially having a plausible theory of change. This is a rather odd inclusion on a hierarchy of evidence because it’s not evidence: it’s a story. Still, a coherent theory of change is (usually) necessary for effectiveness.

Level 2 is some supportive-looking data. Notice that this is as far as most charities & donors ever get.

Level 3 requires ‘demonstrating causality’: this NESTA (rightly) says requires a control group. Notice that very few charities or charitable donors get this far. Notice too that it’s often impossible to have a control group: if you run a national campaign, or work to change the law, then you’re working on a sample of 1 (there’s only one nation, or one set of laws). If you work on a very rare disease, you may have too few people in the sample for the work and any comparison group to be statistically significant. So, interestingly, under NESTA’s view, you can never get very good evidence on those numerous cases. I’d have expected Level 3 to require that the study be robust, e.g., that the comparison group is actually comparable (selection bias having been removed by randomising, and removing other biases e.g., survivor bias, and cross-over effects etc.), and that the treatment and control groups be statistically significant.

Level 4 requires having studied the process in more than one location/times, and having the studies conducted independently. This is useful to ensure that the result is robust and real: repeatability is crucial to any scientific discovery (we can all claim that we got cold fusion to work in our bathroom last Tuesday but mysteriously can’t do it again). Note that the studies don’t have to be in different locations: despite deworming children not increasing school attendance in Scotland, it does help in India, so we need to be a bit careful that the replication  studies are really relevant.

And it’s obviously important that the results be independently verified – though again, the overwhelming majority of ‘evidence’ used by charities and donors isn’t, and hence may well be garbage.

Level 5 is frankly a surprise. Having a manual on how to do something doesn’t, to my mind, constitute evidence: it’s just a manual. That may be useful for implementation, but it doesn’t of itself add to the certainty that the intervention actually works.

What I’d expect to see as the top level is a systematic review / meta-analysis: that analysing all the studies of this intervention as though they were one (amalgamating the samples of all of them), giving more weight to big studies than to small ones, produces a positive result. This is a guard against cherry-picking: often some studies will show an intervention to work, and others show it to not work. That situation is pretty confusing for a practitioner who might conclude that the evidence base is inconclusive. Not so! Often, a meta-analysis will show that the answer is there, just hidden. Medicine is awash with such examples - where interventions looked alright, but careful analysis showed that they were in fact fatal. (The analysis can be shown in a ‘blobogram’ shown here.)  
Hence medicine would put systematic reviews at the top of the hierarchy of evidence: and they are the most frequently cited form of clinicial research. Medicine has a huge global network entirely dedicated to doing them - The Cochrane Collaboration - and they also happen in disaster & emergency relief, and international development via 3ie and others.

Notice that you could get to the top of NESTA’s hierarchy with a manual for doing something for which the evidence looks mixed - even if that mixture actually contained proof that the intervention is harmful,  and even fatal.

Getting charities and donors to the point that their data can be scrutinised by systematic reviews is tough, but we must.  These are the most rigorous form of evidence, and hence I’m delighted to be on a board of The Cochrane Collaboration.

Actually, ‘what is our impact’ is the wrong question. Here’s why–>