Dear Santa, please bring us some curiosity!

In 1929, Werner Forssmann, a junior doctor in Eberswalde, Germany, found in an obscure 19th century journal a print of a man passing a tube through a horse’s jugular vein into its heart to measure changes in ventricular pressure.

He wondered if it were possible to investigate human hearts in a similar way. His seniors said he was crazy for asking and told him to go back to work. Undeterred, he sneaked into the x-ray room, made a slit in his own arm, threaded a small tube into his own heart and took some x-rays showing the position of the tube.

He was fired. In 1956 he won a Nobel Prize for, in effect, co-founding the field of cardiology.

Acknowledgement of unknowns and the curiosity to explore them has helped mankind hugely. So I’m surprised that curiosity is scarce among colleagues in charities and philanthropy, given that so much is unknown.

For example, what happens to educational results as we vary class sizes, duration or terms? Or what if, like the Khan Academy, we were to invert the traditional model to give students online lessons at home and spend time with teachers at school doing exercises and problems?

In anti-poverty programmes, does it matter if surveyors who, for example, collect data on people’s income for rigorous evaluations are male vs female, or if they collect that data on paper or on electronic devices? If the measurement tool affects the apparent results, then surely we need to know. What does the literature say about how domestic charities can best collaborate? Are there inexpensive short-term marker outcomes that reliably predict the long-term outcomes we really care about? When can beneficiaries’ perceptions of an organisation be used as a reliable proxy for the effectiveness of that organisation? After an earthquake, is it better to pull people painstakingly from the rubble or to amputate trapped limbs and move on to the next person? And does it matter in terms of outcomes that, as the Stanford Social Innovation Review recently reported, non-profit board places are increasingly being occupied by the rich? These questions might all affect our success, yet all remain unanswered.

So Santa, I’m not sure if you deal in abstract nouns, but what I’d love you to bring this year is more curiosity. Please bring some for donors, foundations and operating organisations, because they all have many practices that probably could be usefully questioned.

On donors, do the expert panels that assess grant applications make better decisions than those made by flipping a coin? Do the type and extent of a funder’s engagement with grantees affect outcomes? When should donors make a few large grants and when should they make loads of small grants? Can donors’ existing data be used to predict whether or not a particular grant or organisation will succeed? So incurious are most donors that Clara Miller, president of the FB Heron Foundation, which helps low-income people and communities in the US, says that “most philanthropy is a culture of bureaucracy; it isn’t a culture of discovery“.

The instinct to see and answer myriad questions like these could help us to understand and achieve much more – but we do need curiosity.

Santa, can you help?

This article first published in Third Sector.

How to save time & money by using research that already exists?–>

Posted in Uncategorized | Leave a comment

The charity sector should use more systematic reviews to leverage what’s already known

Any single piece of evaluation research, designed to understand the effect of an intervention, has limitations. It will examine the effect of a particular intervention on some particular outcomes in a particular group of people (‘population’), at a particular time. That’s fine, but it inevitably limits the value of the research for organisations using, say, the same intervention on a different population. Studies also vary in their robustness – the chance that their answer is wrong – and even a good study can get a weird result by fluke.

Better, then, to look at multiple impact studies when designing programmes or making funding decisions. This is what the Blagrave Trust, a foundation, recently asked my organisation, Giving Evidence, to do on the subject of outdoor learning, one of its funding areas; the report will be published next month. With University College London, we looked for every relevant study of outdoor learning published over the past 10 years. ‘Systematic reviews’ such as this enable people to stand on the shoulders of myriad giants – and see a long way.  Continue reading

Posted in Effective giving, Impact & evaluation | Leave a comment

Charities should stand on the shoulders of giants

If campaigns to raise awareness of global poverty and progress fixing it are only reaching people already interested, what should we do? The Gates Foundation is one of many bodies concerned about this, and asked various communicators and academics to design some communications and campaigns to reach other audiences. They knew well the literature which shows: how people respond to messages, and that comedy can be a Trojan horse carrying messages beyond their normal reach. It also shows that engagement with social issues is hindered by ‘otherness’, the notion that they only affect people very different and far away; and ‘narrative transportation’ a well-evidence theory created by two psychologists, that when we’re transported deeply into a story, our barriers to persuasion fall.  Continue reading

Posted in Uncategorized | Tagged , | Leave a comment

Effective campaigning, dark matter and Stephen Lloyd

The church in Berkeley in Gloucestershire has a plaque that commemorates “the many virtues and great usefulness of Miss Sarah Merrett Pike of this town”. How delightful: I’m sure we’d all like to end our days reputed for our “great usefulness”. SAM_9918

So it was with our friend Stephen Lloyd, the great charity lawyer who was taken from us in an accident this time last year. It’s taken me a year to be able to write about him. I was fortunate he was a trustee of the charity of which I was chief executive – one of our various shared ventures. Continue reading

Posted in Impact & evaluation | Leave a comment

TV coverage of charity effectiveness and impact evaluations

The closure of Kids Company on August 5th raised the question of charity’s management and effectiveness. Caroline spoke to BBC News about it:

And, once it emerged that there are 60,000 children’s charities in the UK, she spoke about whether there should be consolidation:

Caroline was also on BBC Radio4’s PM programme on the same day, which FeedbackLabs turned into a podcast, here.

The problems establishing the effectiveness of KidsCompany illustrate a wide-spread problem in the charity sector, which Giving Evidence and others are actively trying to solve. More here.

References cited in the interviews:

Brief debriefing:

Bangladesh nutrition example:

The micro-credit in NE Thailand example:

Data on relationship between charities’ effectiveness and their admin costs:

KPMG study showing that 83% corporate mergers fail to produce value, here.

10 year study of 3,400 charities (accidentally cited as ‘of 3,400 studies’), by Bridgespan here.

Posted in Uncategorized | Leave a comment

Measuring long-term outcomes: new plan!

Like those of many social programmes, the goals of taking young people on Sail Training voyages are long-term: In this case, to improve life chances, involvement in employment and training, and sound mental health. However, many organisations which provide Sail Training cannot conduct or commission high-quality longitudinal studies that demonstrate an impact on these long-term outcomes, because of complexity and cost. So Giving Evidence is delighted to be working to identify short-term outcomes that, if ‘produced’ by an intervention, have a beneficial effect on key longer-term outcomes. If future research can show a link between the intervention(s) and certain short-term outcomes, and there is a known link between those short-term outcomes and particular longer-term outcomes, then one can make a coherent and evidence-informed claim about the long-term outcomes produced by the intervention.diagramDiagram 2 Continue reading

Posted in Impact & evaluation | Leave a comment

How come this foundation’s grantees love its reporting process so much?

Most charities hate the reporting which funders make them do. Notionally a learning process, it’s often just compliance, box-ticking and a dead-weight cost. But not so apparently for the Inter-American Foundation, an independent US government agency which grant-funds citizen-led community development across Latin America and the Caribbean. IAF seems to be a positive outlier: it has twice undertaken the Grantee Perception Report – an anonymous survey of grantees by the Center for Effective Philanthropy* now done by over 300 funders – and both times got the best rating ever seen for helpfulness of its reporting process. Plus IAF was both times in the top 1% on the all-time list for usefulness of its selection process, and for its transparency. It’s also in the top decile for a whole pile of other indicators.

What on earth is going on? And what (if anything) can other funders learn from this?

Giving Evidence is interested in producing and sharing decent evidence about what makes for effective giving, so we’re delighted to be working with IAF to figure this out. We’ll produce a case study later this year, showing what IAF does and why it seems to be so popular. Continue reading

Posted in Donor behaviour & giving stats, Effective giving, Impact & evaluation, meta-research | Leave a comment

Some grant decisions should be made at random(!)

Don’t laugh – the notion that grants should be given at random rattled around when the National Lottery was set up over 20 years ago. The joke was that since prize-winners are chosen at random, maybe grant-winners should be too.

Perhaps we should resurrect the idea. The medics have studied it. Australian health economists looked at every grant application to the National Health and Medical Research Council of Australia in 2009 – all 2,085 of them – and analysed the scores given by the expert panels that assessed applications.

Now, if there’s one thing we know about experts it is that they’re not very good. For example, the US National Academy for Sciences showed that judges’ decisions about imprisonment varied dramatically and predictably, depending on whether the decision was made before or after lunch. The Nobel laureate Daniel Kahneman reported how, given the same picture on different days, radiologists contradict themselves 20 per cent of the time, as do stock market analysts, pathologists and many others. Extraneous factors often hold sway.

The Australian health economists put a margin of error around the experts’ scores to account for extraneous factors such as these. They found that 80 per cent of proposals were ‘never funded’ – those applications would get binned even with extraneous variations in their favour; only 9 per cent were ‘always funded’ – they scored highly enough for extraneous variations not to sabotage them; and nearly a third (29 per cent) were ‘sometimes funded’ – their fate depended on how supposedly irrelevant factors happened to play out that day.

The health economists note that “strong and weak grant proposals should be identified consistently, but most are likely to occupy a tightly packed middle ground”. This study – the only one of its type I’ve seen, and I’ve looked hard – showed there was still what the authors called “a high degree of randomness”.

So let’s formally introduce randomness – just for the applications that are neither stars nor duds. It might save considerable time and, therefore, money. It is also honest: a 1998 US paper called for some random grants because, it said, “instead of dodging the fact that chance plays a big part in awarding money, the (random allocation) system will sanctify chance as the determining factor”. 

Grant-makers and their trustees and panels might hate this idea because it reduces their decision rights. But it might also reduce “grant rage” from rejected applicants and hence reduce aggravation in their jobs. It might have secondary benefits too, such as discouraging grant-seekers from hanging around at drinks receptions hoping to suck up to grant officers or trustees who might influence their case, because they’d know there was a good chance of a decision being random anyway.

We all know that grant funding is scarce. The experience of running randomised evaluations in less developed countries – in most of which some people get something and others don’t – is that people who are accustomed to scarcity value the transparency of allocation being made at random. It is better than the usual allocation by patronage.

This article first published in Third Sector.

The day after I filed it, somebody at my tennis club happened to recount the uproar when, one year, the club’s batch of Wimbledon tickets were given to the chair-person, the ladies doubles captain, the men’s doubles captain, etc. It only subsided when those tickets were recalled and re-allocated via a ballot in the bar. It seems that we all appreciate the transparency of random allocation when valuable resources are scarce.

Posted in Uncategorized | Leave a comment

Deworming: problems under re-analysis

A flawed study on deworming children—and new studies that expose its errors—reveal why activists and philanthropists alike need safeguards.

The book Zen and the Art of Motorcycle Maintenance, of all things, offers a critically important message for people who work in development and philanthropy. “The real purpose of the scientific method is to make sure nature hasn’t misled you into thinking you know something you actually don’t know.”

Three new papers published today confirm this, by illustrating just how easily we can be misled by what we think we know, and just how much the power of the scientific method can safeguard us from continuing to be misled (and potentially investing significant time and effort on the wrong priorities). That’s because the three papers raise important questions about the practice of treating children for intestinal worms, which, in recent years, has become a darling of international development.

Continue reading

Posted in Effective giving, Great charities, Impact & evaluation, meta-research, transparency | Tagged , , | Leave a comment

What to when when you’re badly treated by a funder?

Jake Hayman was right in his recent blog Not Fit For Purpose: Why I’m Done With the Foundation World – there are major problems with charitable funding.

We can see this just from the fact that charities normally pay between 20p and 40p to raise £1, whereas companies pay between 3p and 5p. We can tell, too, from the remarkable unpopularity of many grant-makers in comparison to most people who hand out money.

But what do you do about it? This isn’t a rhetorical question: I’m asking for actual examples. What have you – you! – done in the past when you’ve felt badly treated by a foundation? Do you write to the chief executive? To the chair? Rant on Twitter? Just bitch about them privately? And what happened as a result? If we collectively had more stories and examples (evidence, of a sort) about what works and what doesn’t in terms of influencing donor behaviour, perhaps we could solve much of this. Continue reading

Posted in Uncategorized | Leave a comment