What makes a helpful reporting & evaluation system? Learning from an outlier

Funders’ reporting and evaluation systems are rarely loved: they are more often regarded as compliance or ‘policing’. But not so for the Inter-American Foundation apparently: IAFIAF cover received better feedback from its grantees on its reporting and evaluation system than have the ~300 other foundations analysed by the Center for Effective Philanthropy (CEP). Its anonymous survey of grantees includes the question:

“How helpful was participating in the foundation’s reporting/evaluation process in strengthening the organization/program funded by the grant?”

In both 2011 and 2014, IAF got the highest scores CEP has seen for this question. Furthermore, IAF comes top on this metric by some margin. Respondents can answer from 1 (“not at all helpful”) to 7 (“extremely helpful”), and in 2014, IAF scored 6.00; the funders that rated second and third on this question scored 5.80 and 5.72.

Giving Evidence is interested in charitable giving based on sound evidence, so we investigated. Continue reading

Posted in Analysing giving, Effective giving, Impact & evaluation, meta-research | Leave a comment

The Magic Impact Fairy will ensure that your research really changes something

Many charities’ theory of change is: ‘here’s that document you didn’t ask for’

I want to introduce you to someone: the Magic Impact Fairy. Her job is to take all the research that people do and the reports they write, and ensure it all actually changes something – that the conclusions are read and heeded by the people they’re intended to influence. She’s busy.

Many charities, foundations, think tanks, academics and people in government rely on her. Without her, their theory of change is pretty much “here’s that document you didn’t ask for”. Sometimes we don’t even get that far: a friend who worked for a human rights charity recounts how its researchers once created an important report, but never even planned to distribute it; the office was awash with unopened boxes of copies.

Organisations often don’t seem to plan how to get their research used, so it’s lucky that the Magic Impact Fairy can spirit the insights to where they’re needed, and incentivise and empower the key people to act.

Some evil forces are trying to make the Magic Impact Fairy redundant. For example, Innovations for Poverty Action, a US-based NGO which researches fixes to “poverty problems”, has an ABC model. First (A), it finds out what policymakers and practitioners want to know. Then (B) it does research into those questions, and finally (C) it provides support for implementing the answers. It finds that the challenges at stage C are mainly practical: policymakers might like the idea of dishing out anti-malarial bednets, but they then need to know how many nets fit into one box, how many boxes fit onto a truck, and how long the insect-repelling chemical lasts. 

The Harvard surgeon Atul Gawande finds that two types of problems impede good outcomes in many disciplines. First, problems of incompetence, where we don’t know what to do. The research that we all do is meant to solve those. Then come problems of ineptitude, wherein we know what to do but don’t do it. The latter is partly about changing behaviour, which is brutally tough: even getting doctors to wash their hands is famously hard. As the behavioural economist Richard Thaler says: “The daunting realisation is that we don’t know what the hell we’re doing in most fields of life, especially the ones that involve people.” (Maybe fairies are more effective than economists…)

The researchers Abhijit Banerjee and Esther Duflo have gone further and identified three problems in research uptake: ignorance, incentive and inertia. Notice that these problems aren’t solved by the classic solutions of holding events to launch documents, nor policy briefs (glossary summaries). Those things address ignorance. More common is that the research discusses a problem the audience doesn’t think it has, or that it requires action its readers are not empowered to take.

That’s why the Education Endowment Foundation talks not about merely disseminating, but about mobilising. And those problems have spawned a whole discipline called Implementation Science. Southampton University has a centre dedicated to it, and Oxford University runs courses on it.

Now, I’ll tell you a secret about the Magic Impact Fairy. It’s that you have to believe: as with Tinkerbell, only if you really believe in her can the magic really work.

This article first published in Third Sector.

Posted in Effective giving, Impact & evaluation, meta-research, Uncategorized | Leave a comment

2015: Giving Evidence’s Year In Brief

We were busy this year! Pushing forwards substantially on improving the quality and use of evidence in our three areas of interest: assessing interventions, assessing charities, and assessing ways of giving.

Read our 2015 Year In Brief.

Posted in Uncategorized | Leave a comment

NSPCC’s stats on child addiction to porn don’t stand up to scrutiny

Oh dear. The NSPCC seems to have lost the moral high ground to – of all people –Vice magazine, a trendy periodical that is mainly about trainers and unconventional sex.

The children’s charity recently claimed that a tenth of the UK’s 12 to 13-year-olds were addicted to porn. It was an eye-catching story that was picked up by many media channels. The Conservatives immediately pounced on it as an election issue, pledging that they would block internet porn sites that did not have controls to prevent people under 18 from looking at them.

Vice was alone, it seems, in wondering what those numbers actually meant and where they came from. “Such inflammatory findings when published by a respected national charity would usually be accompanied by a full report of the study,” Vice says. “But not in this case. All the NSPCC would offer was an extended press release with some more quotes from concerned parties.” Continue reading

Posted in Fundraising, Great charities | Leave a comment

Talk of charities ‘proving their impact’ is dangerous and misleading

Proof is a big concept, and in social science – which is what impact research is – it almost never happens

Suppose you hear of a new intervention that’s never been tried or tested
voltaire-writer-quote-doubt-is-not-a-pleasant-condition-but-certaintybefore. What are the chances of it producing decent results? Clearly, you’ve no idea: the uncertainty about its results is sky high. Now suppose there’s an intervention with teenage mothers that a well-conducted, rigorous evaluation in the US suggests reduces child abuse and neglect, reduces child injuries by 20-50 per cent and improves children’s educational outcomes. Will it produce those results in, say, Edinburgh? Continue reading

Posted in Uncategorized | Leave a comment

Dear Santa, please bring us some curiosity!

In 1929, Werner Forssmann, a junior doctor in Eberswalde, Germany, found in an obscure 19th century journal a print of a man passing a tube through a horse’s jugular vein into its heart to measure changes in ventricular pressure.

He wondered if it were possible to investigate human hearts in a similar way. His seniors said he was crazy for asking and told him to go back to work. Undeterred, he sneaked into the x-ray room, made a slit in his own arm, threaded a small tube into his own heart and took some x-rays showing the position of the tube.

He was fired. In 1956 he won a Nobel Prize for, in effect, co-founding the field of cardiology. Continue reading

Posted in Uncategorized | Leave a comment

The charity sector should use more systematic reviews to leverage what’s already known

Any single piece of evaluation research, designed to understand the effect of an intervention, has limitations. It will examine the effect of a particular intervention on some particular outcomes in a particular group of people (‘population’), at a particular time. That’s fine, but it inevitably limits the value of the research for organisations using, say, the same intervention on a different population. Studies also vary in their robustness – the chance that their answer is wrong – and even a good study can get a weird result by fluke.

Better, then, to look at multiple impact studies when designing programmes or making funding decisions. This is what the Blagrave Trust, a foundation, recently asked my organisation, Giving Evidence, to do on the subject of outdoor learning, one of its funding areas; the report will be published next month. With University College London, we looked for every relevant study of outdoor learning published over the past 10 years. ‘Systematic reviews’ such as this enable people to stand on the shoulders of myriad giants – and see a long way.  Continue reading

Posted in Effective giving, Impact & evaluation | 1 Comment

Charities should stand on the shoulders of giants

If campaigns to raise awareness of global poverty and progress fixing it are only reaching people already interested, what should we do? The Gates Foundation is one of many bodies concerned about this, and asked various communicators and academics to design some communications and campaigns to reach other audiences. They knew well the literature which shows: how people respond to messages, and that comedy can be a Trojan horse carrying messages beyond their normal reach. It also shows that engagement with social issues is hindered by ‘otherness’, the notion that they only affect people very different and far away; and ‘narrative transportation’ a well-evidence theory created by two psychologists, that when we’re transported deeply into a story, our barriers to persuasion fall.  Continue reading

Posted in Uncategorized | Tagged , | Leave a comment

Effective campaigning, dark matter and Stephen Lloyd

The church in Berkeley in Gloucestershire has a plaque that commemorates “the many virtues and great usefulness of Miss Sarah Merrett Pike of this town”. How delightful: I’m sure we’d all like to end our days reputed for our “great usefulness”. SAM_9918

So it was with our friend Stephen Lloyd, the great charity lawyer who was taken from us in an accident this time last year. It’s taken me a year to be able to write about him. I was fortunate he was a trustee of the charity of which I was chief executive – one of our various shared ventures. Continue reading

Posted in Impact & evaluation | Leave a comment

TV coverage of charity effectiveness and impact evaluations

The closure of Kids Company on August 5th raised the question of charity’s management and effectiveness. Caroline spoke to BBC News about it:

And, once it emerged that there are 60,000 children’s charities in the UK, she spoke about whether there should be consolidation:

Caroline was also on BBC Radio4’s PM programme on the same day, which FeedbackLabs turned into a podcast, here. Continue reading

Posted in Uncategorized | Leave a comment