This was first published by our friends at Evidence Matters.

It’s hard to make evidence-based decisions if much of the evidence is missing, ropey, unclear or you can’t find it. Charities produce masses of evidence about their effectiveness but Giving Evidence suspects that much of it  is wasted because of these four problems.

Research that is poor or hidden damages beneficiaries in two ways. First, donors and other operational charities can’t see what works and therefore what to fund or replicate, so may implement something avoidably suboptimal. And second, the research consumes resources, which could perhaps be better spent on delivering something that does work.

Hence Giving Evidence is this very week starting to study non-publication of research by charities. We aim to understand the extent and causes of non-publication, in order to see what might fix it. Though non-publication of research is a known and major problem in science and in medical research, our project is, to our knowledge, the first ever study of non-publication of charities’ research.

We know that much charity research is unpublished: when I was a charity CEO, we researched our impact, and when the results were good, we published them, and when they weren’t we didn’t. I’d never heard of publication bias (of which this is an egregious example) but I had noticed that bad results make for bad meetings with funders. In our defence, we weren’t being evil: we were just responding rationally to badly-designed incentives.

We suspect four reasons that charities don’t publish their research.

• First, incentives, as outlined. The system is that charities evaluate themselves and use the results in to raise funds. The system is so obviously flawed that when I spell it out at conferences – even of seasoned professionals in this sector – everybody laughs.
• Second, charities may think that nobody’s interested. By analogy, a campaign in the UK to get grant-makers to publish details of all their grants (which few do) found that many foundations were open to doing this but simply hadn’t realised that anybody would want them.
• Third, it’s unclear where to publish even if you want to: there are few repositories or journals, nor standard ways of ‘tagging’ research online to make it findable, so charities may (rightly) think that the traffic to material just on their own websites won’t justify the work in sprucing up the research to publish it.
• Fourth, commercial confidentiality given that many charities compete for government contracts. We suspect that the issue here is a bit different from that in pharma: charities’ interventions are rarely patented, so the confidentiality is around the details of their intervention. That’s the secret sauce on which they compete.

This first study focuses on UK charities supporting people with mental health issues: what research do they do, what do they not publish, and why not. The initial budget isn’t big enough for us to get into publication bias (is the published material different from the non-published material?) nor research quality – though we hope to look at both eventually.

We’ll report back on what we find.

So that’s missing research. We’re also looking at the other three problems – of research being ropey, unclear and unfindable – in other sectors. It’s thought that these four problems result in fully 85% of all medical research being wasted. We need to know if and where charitable resources are being similarly wasted, and make haste to fix it.

Giving Evidence’s work is, we think, making important steps.

This talk describes what we’re doing and why: