Charity begins with admitting we got it wrong

We need to be scientific and fearless about assessing whether proposed solutions work

This article first published in the FT

The Bill & Melinda Gates Foundation seems to have wasted six years and $1bn, having initiated a programme to improve teaching effectiveness in US schools. An evaluation released last month showed that it had a negligible effect on its goals — some of them worsened — which included student achievement, access to effective teaching and dropout rates.

Much the same happened to Ark, the UK-based charity founded by the hedge fund industry. It created and co-funded a programme in 25,000 schools in the Indian state of Madhya Pradesh, supporting the government to improve school performance. It was based on sound research about how to reduce teacher absence, improve teaching and create more accountability through school inspections. Ark has worked on it since 2012.

The results of an evaluation of that programme, presented at a conference in Oxford last month, found it had no effect on learning, as measured by how well children perform in tests after 18 months, nor on teacher absence.

Should we berate these foundations for their profligacy? No. On the contrary, we should applaud their willingness to investigate properly what their programmes actually achieve and, moreover, their willingness to publish those findings, even if it embarrasses them. These two programmes both happen to be in education, but evaluations in plenty of sectors show null results.

This is science. This is how we learn. If a programme isn’t working, it’s surely better to find that out after six years than after 20 — and better after 20 years than to persist obstinately forever. Kudos to these foundations for asking the questions.

Susannah Hares, director of Ark’s international operation, says: “Evaluations aren’t simply pass/fail. Rather, they help us learn what does and doesn’t work. Big, ambitious education reforms like this one need proper evaluations that generate truly objective evidence to guide next steps and to inform accountability initiatives elsewhere.”

The approach by Ark and the Gates Foundation here contrasts with the normal practice in philanthropy of asking the charity (or non-profit) which receives the funding to evaluate itself. In that system, charities themselves produce the material which informs major decisions about their future funding and quite often about their very existence.

The awfulness of this system is so obvious it’s a wonder it survives. When I ran a charity, we had to produce this kind of “impact research”. We published the bits that flattered us and hid the bits that didn’t. I didn’t know then that this behaviour has a name: in science, it’s called “publication bias” and is considered a serious research fraud. I had simply noticed that graphs that go up make for good meetings with funders, whereas graphs that go down make for bad ones.

Beyond the incentive problem, most charities lack the skills or funds to evaluate their work properly. Social science is a specialism which makes no sense for most charities to have.

Moreover, few charities have a “sample size” large enough to distinguish the effect of a programme from that of other factors, including random chance. For instance, when the Institute for Fiscal Studies and the National Children’s Bureau wanted to see whether breakfast clubs in schools improve learning outcomes, their study needed to include 106 schools. That is far more sites than most charities serve.

Consequently, much of the impact research produced by charities is awful and simply doesn’t show impact reliably. Neither donors nor trustees should rely on it. I have seen many such studies by charities where the conclusions simply don’t follow from the premises, and/or which provide zero confidence that any observed change is due to the programme rather than to something else. When the Paul Hamlyn Foundation, a UK grant-maker, assessed the impact research it received from grantees over a five-year period, two-thirds failed its pretty low threshold for “good quality”.

What Ark and the Gates Foundation did was to have their programmes evaluated properly by expert and independent researchers. Though those funders are not alone in doing this, it is still too rare: we need many more such studies.

Many donors intuit the solution to some broader social problem. It seems not to occur to them that they might be wrong, nor that there might be better variations.

We patently do not yet know how to solve many social problems. We need to discover — and acknowledge — our ignorance here, and be scientific and fearless about assessing whether proposed solutions actually work.

It is science that doubled life expectancy in the West within only about a century, and moved us from carrier pigeons to mobile phones. It will be science, and the attendant humility of donors and public policymakers to their own ignorance, that will enable us to solve the longest-standing social problems.

 

 

Advertisements
Posted in Uncategorized | Leave a comment

Charities (gasp!) using and producing sound evidence

This article first published in the Financial Times.

Anne Heller has done something that I had never previously seen in my 18 years in the non-profit sector. She identified a social problem, scoured academic literature to find a solution, and then set up a non-profit to implement it. That approach sounds jaw-droppingly obvious, but it is in fact very rare for a charity to design itself around existing evidence.

Ms Heller had worked for the city of New York when Michael Bloomberg was mayor, running shelters for homeless families. She noticed that about 10 per cent of the families who use the shelters returned repeatedly. In other words, the services which the shelters provided were not solving these people’s problems. Continue reading

Posted in Uncategorized | Leave a comment

Mapping the existing evidence about preventing child abuse in institutions

We are producing a map of the existing evidence about child abuse within organisations

New project! Giving Evidence is working to produce a map of the existing evidence (and gaps in it) about what is effective at reducing child abuse – particularly child sexual abuse – within organisations, such as youth clubs, sports clubs, churches etc. We’ll be working on it with the Centre for Evidence and Implementation, and Professor Aron Shlonsky of the University of Melbourne, who have undertaken considerable research around this topic. Continue reading

Posted in Uncategorized | Leave a comment

The Big Money Questions

Wide-ranging chat about charitable giving, with the Daily Mail.

What to give to, what to avoid, one of Caroline’s favourite charities, how to choose a charity… (~20 minutes) Please click on the picture below to watch the full video.

big money

Posted in Uncategorized | Leave a comment

Charities and donors: What would you like more research about?

Giving  Evidence and think-tank Charity Futures break new ground in researching what matters to charities and donors

Survey here! Pls take ten minutes to tell us what you would like to know!

Charity Futures, the new sector think tank led by Sir Stephen Bubb, is running a major consultation to find out the topics on which donors and charity leaders most want more research to help them in their vital work.  Clearly this is essential for ensuring that charitable activity and giving can be based on sound evidence.

Giving Evidence is running the consultation, which invites input from any charity, foundation, public or private donor in the United Kingdom. Through an open ‘crowd-sourcing’ process, including a series of focus groups across the country (which finished in June 2018) and a couple of rounds of public, open, online survey, the project invites charities, private donors and institutional funders to say where more research would be of most use. The project is supported by a distinguished advisory group of funders, private donors, researchers, charity leaders and umbrella bodies (listed below). Continue reading

Posted in Uncategorized | Leave a comment

Donors left empty-handed on charity data questions

How can you tell if a charity is effective, when there is little independent analysis?

This article first published in the Financial Times

Judging by my inbox, there seems to be huge demand for advice about which charities to support. As donors have followed the turmoil around Oxfam, Save the Children, Kids Company and others before them, they want decent independent analysis along the lines of the data that rating agencies provide for bonds or Which? does for fridges.

There isn’t any.

It doesn’t exist, because of costs, incentives and the genuine challenges involved in nailing it down. Continue reading

Posted in Uncategorized | Leave a comment

How to make an evidence-based intervention

VIDEO: Watch Caroline’s new lesson with the Fitzroy Academy!

https://fitzroyacademy.com/lesson/evidence-based-interventions?playlist=social-impact#-QITpDwHX4A

Full lesson here.

Posted in Uncategorized | Leave a comment

Give one charity donation, get one free

Making a big gift to draw other donors is only useful if the charity is highly effective

This article first published in the Financial Times

The practice of donors offering to “match” gifts made to charities they support has become common in the charity world. In the run-up to Christmas, many have been offering to double donations. In the recent Challenge Campaign, for instance, money given to selected charities was doubled by the Big Give, set up by Sir Alec Reed, founder of Reed, the recruitment firm.

Others schemes propose different deals: one donor was offering to give $2 for every $1 given on Friday to Charity Navigator, a charity rating agency; while PayPal will add 1 per cent to all donations made through its platform until New Year.

The UK’s Department for International Development does it too. Its UK Aid Match offered up to £5m from the UK aid budget to eight charities.

The theory is that the match attracts new donations. If your aim is helping your chosen charity to raise money, is it a good idea to offer to match other people’s contributions? The rigorous evidence suggests that it is.  Continue reading

Posted in Uncategorized | Leave a comment

What the Skoll Global Threats Fund learnt with its $100 million

This article first published in Alliance Magazine.

The first president of eBay, Jeff Skoll, set up his Global Threats Fund in 2010 to ‘make hundredmillioncamper progress against five of the gravest threats to humanity’: climate change, pandemics, water security, nuclear proliferation, and conflict in the Middle East. It closes this month having spent its $100 million, and this week published a report about “what worked, what didn’t work, and what we learned about philanthropy’s role in reducing global threats”. As more foundations ‘spend out’ and publish their learning, we will probably see more such documents. This one is a weird read. Continue reading

Posted in Uncategorized | Leave a comment

Giving suggestions for this Christmas

If you, like millions of people, want to give this Christmas / holiday season, the following are all good bets, in our view:

Medicins Sans Frontieres: They always seem to be to be where the sh*t is worst, e.g., Yemen, Rakhine province in Myanmar (where the Rohinga live/lived). We’ve never formally analysed them but all our contact with them and literature we’ve seen from them implies that they have their act together. Continue reading

Posted in Uncategorized | Leave a comment