We need to be scientific and fearless about assessing whether proposed solutions work

This article first published in the Financial Times in July 2018. 

The Bill & Melinda Gates Foundation seems to have wasted six years and $1bn, having initiated a programme to improve teaching effectiveness in US schools. An evaluation released last month showed that it had a negligible effect on its goals — some of them worsened — which included student achievement, access to effective teaching and dropout rates.

Much the same happened to Ark, the UK-based charity founded by the hedge fund industry. It created and co-funded a programme in 25,000 schools in the Indian state of Madhya Pradesh, supporting the government to improve school performance. It was based on sound research about how to reduce teacher absence, improve teaching and create more accountability through school inspections. Ark has worked on it since 2012.

The results of an evaluation of that programme, presented at a conference in Oxford last month, found it had no effect on learning, as measured by how well children perform in tests after 18 months, nor on teacher absence.

Should we berate these foundations for their profligacy? No. On the contrary, we should applaud their willingness to investigate properly what their programmes actually achieve and, moreover, their willingness to publish those findings, even if it embarrasses them. These two programmes both happen to be in education, but evaluations in plenty of sectors show null results.

This is science. This is how we learn. If a programme isn’t working, it’s surely better to find that out after six years than after 20 — and better after 20 years than to persist obstinately forever. Kudos to these foundations for asking the questions.

Susannah Hares, director of Ark’s international operation, says: “Evaluations aren’t simply pass/fail. Rather, they help us learn what does and doesn’t work. Big, ambitious education reforms like this one need proper evaluations that generate truly objective evidence to guide next steps and to inform accountability initiatives elsewhere.”

The approach by Ark and the Gates Foundation here contrasts with the normal practice in philanthropy of asking the charity (or non-profit) which receives the funding to evaluate itself. In that system, charities themselves produce the material which informs major decisions about their future funding and quite often about their very existence.

The awfulness of this system is so obvious it’s a wonder it survives. When I ran a charity, we had to produce this kind of “impact research”. We published the bits that flattered us and hid the bits that didn’t. I didn’t know then that this behaviour has a name: in science, it’s called “publication bias” and is considered a serious research fraud. I had simply noticed that graphs that go up make for good meetings with funders, whereas graphs that go down make for bad ones.

Beyond the incentive problem, most charities lack the skills or funds to evaluate their work properly. Social science research is a specialism which makes no sense for most charities to have.

Moreover, few charities have a “sample size” large enough to distinguish the effect of a programme from that of other factors, including random chance. For instance, when the Institute for Fiscal Studies and the National Children’s Bureau wanted to see whether breakfast clubs in schools improve learning outcomes, their study needed to include 106 schools. That is far more sites than most charities serve.

Consequently, much of the impact research produced by charities is awful and simply doesn’t show impact reliably. Neither donors nor trustees should rely on it. I have seen many such studies by charities where the conclusions simply don’t follow from the premises, and/or which provide zero confidence that any observed change is due to the programme rather than to something else. When the Paul Hamlyn Foundation, a UK grant-maker, assessed the impact research it received from grantees over a five-year period, two-thirds failed its pretty low threshold for “good quality”.

What Ark and the Gates Foundation did was to have their programmes evaluated properly by expert and independent researchers. Though those funders are not alone in doing this, it is still too rare: we need many more such studies.

Many donors intuit the solution to some broader social problem. It seems not to occur to them that they might be wrong, nor that there might be better variations.

We patently do not yet know how to solve many social problems. We need to discover — and acknowledge — our ignorance here, and be scientific and fearless about assessing whether proposed solutions actually work.

It is science that doubled life expectancy in the West within only about a century, and moved us from carrier pigeons to mobile phones. It will be science, and the attendant humility of donors and public policymakers to their own ignorance, that will enable us to solve the longest-standing social problems.