Charities should stand on the shoulders of giants

If campaigns to raise awareness of global poverty and progress fixing it are only reaching people already interested, what should we do? The Gates Foundation is one of many bodies concerned about this, and asked various communicators and academics to design some communications and campaigns to reach other audiences. They knew well the literature which shows: how people respond to messages, and that comedy can be a Trojan horse carrying messages beyond their normal reach. It also shows that engagement with social issues is hindered by ‘otherness’, the notion that they only affect people very different and far away; and ‘narrative transportation’ a well-evidence theory created by two psychologists, that when we’re transported deeply into a story, our barriers to persuasion fall. 

The result is Stand Up Planet, an hour-long comedic travelogue documentary TV show, “to entertain, to enlighten, and to see people living around the world through a new lens.” Fronted by an Indian born US stand-up comic, it’s hilarious. And the evaluations suggest that it’s effective at influencing attitudes, and the underpinning theories explain why.

Isaac Newton famously proclaimed, in a rare moment of humility, that “if I have seen further than others, it has been by standing on the shoulders of giants.” Being only 5’2”, standing on virtually anyone’s shoulders sounds good to me.

Charities could do more ‘standing on giants’ like this. It would remove two weird and wasteful traps to which we’re prone. The first is thinking that every single charity should produce research into its impact. This dreadful notion has emerged over the last 15 years from the M&E / impact lobby (of which I was part – oops), and is a problem because producing reliable research is hard and a specialism. Few charities have the skills or money for good research – because they’re instead designed to deliver services or run campaigns – and operational charities aren’t incentivised to do research. Better for them to use research which specialist giants have already produced.

The second related issue is thinking that programmes are so innovative that no relevant research exists at all. This is almost never true: there is little so new or off-the-wall under the sun that humankind’s entire experience cannot inform a guess as to whether it’ll work or how to do it well. Take social impact bonds. Though they were unprecedented, they are basically a way to give providers an incentive to achieve particular targets – and a great deal is known about how organisations respond to incentives.

So let’s stop producing so much bad research and instead use existing research produced by relevant giants. Equally, funders need to stop asking every organisation for the impact research it has produced, and instead ask about the research that it uses.

What is the best research to use? It’s a systematic review of all the evidence relevant to a topic. Systematic reviews are increasingly common in social issues. Next month we’ll look at how they work and a systematic review underway commissioned by a foundation to inform their funding. It’s enabling us to see far further than research by any individual organisation ever would.

This article first published in Third Sector.

Individual studies by charities can be pretty ropey—>

Posted in Uncategorized | Tagged , | Leave a comment

Effective campaigning, dark matter and Stephen Lloyd

The church in Berkeley in Gloucestershire has a plaque that commemorates “the many virtues and great usefulness of Miss Sarah Merrett Pike of this town”. How delightful: I’m sure we’d all like to end our days reputed for our “great usefulness”. SAM_9918

So it was with our friend Stephen Lloyd, the great charity lawyer who was taken from us in an accident this time last year. It’s taken me a year to be able to write about him. I was fortunate he was a trustee of the charity of which I was chief executive – one of our various shared ventures.

The scale and range – “usefulness” – of Stephen’s achievements are well known and well documented. Striking in those accounts is just what a nice guy he was: well liked and trusted so widely. Many in our charity dealt with him only slightly and yet, on hearing the news, spoke of how he’d been “very kind and helpful on the few occasions I met him”, or “such a kind and reasonable man and great counsel”, or “kind and generous to even the most junior in the room”.

By the time he met these people, remember, he was kingpin of a City law firm. Not everyone of such seniority is so charming.

And it’s not as though he simply spent his time agreeing with everyone. Stephen was a campaigner – for sustainability to count as a charitable purpose, for legal forms for social enterprises, for disrupting the insurance industry. Most change creates losers as well as winners, so it’s easy to make enemies. Many of us do: another trustee seemed to object to everything all the time, which bothered me until my mentor pointed out that he was a professional campaigner and was angry for a living.

The trap for those of us who campaign is to rely on our arguments, the facts, the logic. These can be valuable for the air war, but the real work of change – changing minds, norms, systems and even laws – is often more like an operation on the ground. It involves engaging with people, and that is much easier if they know you and like you. Two examples illustrate how.

Fritz Zwicky was a Bulgaria-born astrophysicist of Swiss and Czech parentage who worked in the US in the 1920s and 30s. He noticed that galaxies don’t rotate in the way classical mechanics predicts. He postulated that the variance was due to what he called “dark matter”, which might account for a large chunk of the universe. Physics was then fresh from the observational evidence of general relativity and not receptive to huge ideas like this. Zwicky was ignored, not because he was wrong but because he was difficult: he called his colleagues “scatterbrains”, “sycophants” and “plain thieves” who “doctor their observational data”. He did one-arm press-ups in the canteen rather than talk to people. It was another 30 years until his idea came around again.

Contrast him with David Sackett, a pioneer of evidence-based medicine who also died recently. His ideas create plenty of losers – doctors lose their alleged omniscience and must defer to facts – but he succeeded by carting his ideas around. He visited more than 200 district general hospitals in the UK and scores in Europe. For sure, campaigning needs data and logic, and they need to be right. They are necessary but not sufficient for the “great usefulness” we seek.

This article first published in Third Sector.

Posted in Impact & evaluation | Leave a comment

TV coverage of charity effectiveness and impact evaluations

The closure of Kids Company on August 5th raised the question of charity’s management and effectiveness. Caroline spoke to BBC News about it:

And, once it emerged that there are 60,000 children’s charities in the UK, she spoke about whether there should be consolidation:

Caroline was also on BBC Radio4’s PM programme on the same day, which FeedbackLabs turned into a podcast, here.

The problems establishing the effectiveness of KidsCompany illustrate a wide-spread problem in the charity sector, which Giving Evidence and others are actively trying to solve. More here.

References cited in the interviews:

Brief debriefing:

Bangladesh nutrition example:

The micro-credit in NE Thailand example:

Data on relationship between charities’ effectiveness and their admin costs:

KPMG study showing that 83% corporate mergers fail to produce value, here.

10 year study of 3,400 charities (accidentally cited as ‘of 3,400 studies’), by Bridgespan here.

Posted in Uncategorized | Leave a comment

Measuring long-term outcomes: new plan!

Like those of many social programmes, the goals of taking young people on Sail Training voyages are long-term: In this case, to improve life chances, involvement in employment and training, and sound mental health. However, many organisations which provide Sail Training cannot conduct or commission high-quality longitudinal studies that demonstrate an impact on these long-term outcomes, because of complexity and cost. So Giving Evidence is delighted to be working to identify short-term outcomes that, if ‘produced’ by an intervention, have a beneficial effect on key longer-term outcomes. If future research can show a link between the intervention(s) and certain short-term outcomes, and there is a known link between those short-term outcomes and particular longer-term outcomes, then one can make a coherent and evidence-informed claim about the long-term outcomes produced by the intervention.diagramDiagram 2First we will research which short-term outcomes for young people produce which longer-term outcomes and with what probability and in what circumstances. This will be irrespective of how those short-term outcomes are produced, i.e., we will look beyond outcomes produced by (and research about) Sail Training. We will also assess the reliability of the various measures of short-term outcomes. Second will come some empirical, experimental research to explore whether (and when and how) Sail Training produces the short-term outcomes which are good interim indicators.

Sail Training organisations will be able to use this in at least two ways. First, for designing future research: specifically in choosing reliable measures of important short-term outcomes. And second, in providing evidence that their work produces particular long-term outcomes. 

Interestingly, this work is being funded by two umbrella/ memberships bodies in the Sail Training sector, as a ‘communal resource’ for the sector: the Association of Sail Training Organisations which has UK-based members, and Sail Training International, its international counterpart. Since so many charitable sectors have this pattern – of pressure to demonstrate long-term outcomes but inadequate resources to do so experimentally – this kind of communal resource might be a useful tool for other infrastructure bodies to provide.

Giving Evidence is partnering for this work with the EPPI Centre (Evidence for Policy and Practice Information and Co-ordinating Centre) at University College London. Not only is the EPPI Centre expert in this area, but we are already partnering on the systematic review of the effects of outdoor learning more broadly, to which this project is clearly adjacent.

Process and outputs

We have some early decisions about the types of young people on whom to focus etc., and will make them in consultation with an advisory group from the Sail Training sector.

Then we will produce and publish a protocol for our research: this is a pre-specified plan which enables anybody to check our findings or to repeat the study later to see whether the situation has changed.

The secondary research (i.e., the first stage: of exploring the relationship between short- and long-term outcomes) will be a systematic review of the literature. This is a structured investigation to find, critically appraise and synthesise all the relevant primary research. They are less prone to bias, as science writer Dr Ben Goldacre explains:

“Instead of just mooching through the research literature consciously or unconsciously picking out papers that support [our] pre-existing beliefs, [we] take a scientific, systematic approach to the very process of looking for evidence, ensuring that [our] evidence is as complete and representative as possible of all the research that has ever been done.”

The systematic review will be published, probably in early 2016. We will then look to design some primary, experimental research. 

If you are a Sail Training organisation and would like to be involved, or have relevant research, or have relevant experience (e.g. you’ve been a trainee on a Sail Training voyage have taken part in a Sail Training voyage), or have something else useful to contribute to this study, do get in touch:

To ensure that you get the research, sign up here and/or follow on Twitter here.

Posted in Impact & evaluation | Leave a comment

How come this foundation’s grantees love its reporting process so much?

Most charities hate the reporting which funders make them do. Notionally a learning process, it’s often just compliance, box-ticking and a dead-weight cost. But not so apparently for the Inter-American Foundation, an independent US government agency which grant-funds citizen-led community development across Latin America and the Caribbean. IAF seems to be a positive outlier: it has twice undertaken the Grantee Perception Report – an anonymous survey of grantees by the Center for Effective Philanthropy* now done by over 300 funders – and both times got the best rating ever seen for helpfulness of its reporting process. Plus IAF was both times in the top 1% on the all-time list for usefulness of its selection process, and for its transparency. It’s also in the top decile for a whole pile of other indicators.

What on earth is going on? And what (if anything) can other funders learn from this?

Giving Evidence is interested in producing and sharing decent evidence about what makes for effective giving, so we’re delighted to be working with IAF to figure this out. We’ll produce a case study later this year, showing what IAF does and why it seems to be so popular. Continue reading

Posted in Donor behaviour & giving stats, Effective giving, Impact & evaluation, meta-research | Leave a comment

Some grant decisions should be made at random(!)

Don’t laugh – the notion that grants should be given at random rattled around when the National Lottery was set up over 20 years ago. The joke was that since prize-winners are chosen at random, maybe grant-winners should be too.

Perhaps we should resurrect the idea. The medics have studied it. Australian health economists looked at every grant application to the National Health and Medical Research Council of Australia in 2009 – all 2,085 of them – and analysed the scores given by the expert panels that assessed applications.

Now, if there’s one thing we know about experts it is that they’re not very good. For example, the US National Academy for Sciences showed that judges’ decisions about imprisonment varied dramatically and predictably, depending on whether the decision was made before or after lunch. The Nobel laureate Daniel Kahneman reported how, given the same picture on different days, radiologists contradict themselves 20 per cent of the time, as do stock market analysts, pathologists and many others. Extraneous factors often hold sway.

The Australian health economists put a margin of error around the experts’ scores to account for extraneous factors such as these. They found that 80 per cent of proposals were ‘never funded’ – those applications would get binned even with extraneous variations in their favour; only 9 per cent were ‘always funded’ – they scored highly enough for extraneous variations not to sabotage them; and nearly a third (29 per cent) were ‘sometimes funded’ – their fate depended on how supposedly irrelevant factors happened to play out that day.

The health economists note that “strong and weak grant proposals should be identified consistently, but most are likely to occupy a tightly packed middle ground”. This study – the only one of its type I’ve seen, and I’ve looked hard – showed there was still what the authors called “a high degree of randomness”.

So let’s formally introduce randomness – just for the applications that are neither stars nor duds. It might save considerable time and, therefore, money. It is also honest: a 1998 US paper called for some random grants because, it said, “instead of dodging the fact that chance plays a big part in awarding money, the (random allocation) system will sanctify chance as the determining factor”. 

Grant-makers and their trustees and panels might hate this idea because it reduces their decision rights. But it might also reduce “grant rage” from rejected applicants and hence reduce aggravation in their jobs. It might have secondary benefits too, such as discouraging grant-seekers from hanging around at drinks receptions hoping to suck up to grant officers or trustees who might influence their case, because they’d know there was a good chance of a decision being random anyway.

We all know that grant funding is scarce. The experience of running randomised evaluations in less developed countries – in most of which some people get something and others don’t – is that people who are accustomed to scarcity value the transparency of allocation being made at random. It is better than the usual allocation by patronage.

This article first published in Third Sector.

The day after I filed it, somebody at my tennis club happened to recount the uproar when, one year, the club’s batch of Wimbledon tickets were given to the chair-person, the ladies doubles captain, the men’s doubles captain, etc. It only subsided when those tickets were recalled and re-allocated via a ballot in the bar. It seems that we all appreciate the transparency of random allocation when valuable resources are scarce.

Posted in Uncategorized | Leave a comment

Deworming: problems under re-analysis

A flawed study on deworming children—and new studies that expose its errors—reveal why activists and philanthropists alike need safeguards.

The book Zen and the Art of Motorcycle Maintenance, of all things, offers a critically important message for people who work in development and philanthropy. “The real purpose of the scientific method is to make sure nature hasn’t misled you into thinking you know something you actually don’t know.”

Three new papers published today confirm this, by illustrating just how easily we can be misled by what we think we know, and just how much the power of the scientific method can safeguard us from continuing to be misled (and potentially investing significant time and effort on the wrong priorities). That’s because the three papers raise important questions about the practice of treating children for intestinal worms, which, in recent years, has become a darling of international development.

Deworming Programs Have Been “In”

Here’s the back-story. Worms infect people through contact with infected faeces. They live in people’s bodies (they can be a metre long!), eat their food, deprive them of nutrients, and make them lethargic and ill. And in 1999, two US economists, conducting a study in Western Kenya, found that “deworming” a number of school children improved their nutritional intake, reduced their incidence of anemia, and—by making them less ill and lethargic—increased their attendance at school and hence improved their exam results. The economists also claimed that attendance at schools where children did not receive treatment also increased—by 7.5 percent—because those children, living in the same area as the children who were treated, were not infected by worm eggs in feces in soil near their homes. (There are two main types of worms: soil-transmitted worms, and water-transmitted worms known as schistosomiasis or bilharzia. The Kenyan study was mainly of soil-transmitted worms but did pick up some schistosomiasis.)

Consequently, the Copenhagen Consensus made deworming one of its top recommendations. GiveWell named two organizations that focus on deworming in the top four on its list. And development economist Michael Kremer, a co-author of the 1999 Kenyan study, started an initiative called Deworm the World, which has treated 37 million children in four countries to date.

The Scientific Method

Now, the scientific method involves several safeguards against being misled. One is isolating variables to reveal which one(s) matter. Maybe the speed with which a dropped object hits the ground depends on the height from which it’s dropped and gender of the person who drops it. So we experiment by having people of both genders drop identical objects from the same height, thus “isolating” gender as a variable and, when the objects hit the ground at the same time, showing that it doesn’t matter.

Another safeguard addresses bias by replicating an experiment elsewhere, and comparing and combining the answers. If we open our back-to-work program only to motivated people, then we don’t know whether their success getting jobs is due to the program (a “treatment effect”) or the unusual characteristics of the people we chose (a “selection effect”). The latter would create a selection bias. If we interview only the people in the program who stick it out to the end, we don’t hear from the people who quit because it was so arduous, so our user-experience data may suffer from survivor bias. These and other biases mislead us into thinking we know things we actually don’t know. Single studies may also be biased because they may unwittingly involve particularly unusual people or take place under unusual circumstances. They may also simply get freak results by chance.

A third safeguard in the scientific method is repeating the analysis. In other words, checking the maths.

The three papers, now available, used the scientific method to great effect. The Cochrane Collaboration is a global network of medical researchers who do “systematic reviews” and “meta-analyses” (it may well have saved your life at some point). In 2012, the Cochrane Collaboration wrote: “It is probably misleading to justify contemporary deworming programmes based on evidence of consistent benefit on nutrition, haemoglobin, school attendance or school performance.” Recent correspondence with the authors implies that they’ve not changed their minds. And today, the Cochrane Collaboration publishes its fourth systematic review of mass deworming. The group looked at all 45 studies within its scope and concluded that: “There is now substantial evidence that this [mass deworming treatment] does not improve nutritional status, haemoglobin, cognition, or school performance.”

In two additional studies published today, the London School of Hygiene and Tropical Medicine (LSHTM) simply re-analyze the Kenyan data. They found, if you excuse the pun, a can of worms: errors, missing data, misinterpretation of probabilities, and a high risk of various biases. The effects are huge: The claimed effect on school attendance among untreated children seems entirely due to “calculation errors” and effectively disappeared on re-analysis; the claimed effect on anemia statistically did the same.

We shouldn’t be surprised: That people make mistakes is hardly news. What’s impressive is that somebody took this important step of re-analyzing the data, caught the errors, and prevented us being misled by them. As Yale’s Dean Karlan and I noted when the 2012 Cochrane worm study published, this is exactly how science is supposed to work.

The re-analysis papers raise three more subtle issues. First, the choice of analytical method matters (even if the data are complete and accurate). When looking at changes in school attendance, the economists used a method common in economics; the epidemiologists used a different method common in epidemiology and found that “the strength of evidence supporting the improvement was dependent on the analysis approach used”. There can only be one “correct” answer, and it’s not yet clear which method is misleading.

Second is how rare re-analyses are. Open data to enable post-publication review is sexy and funded and increasingly common. But actually doing post-publication review is hard. It’s hard to fund—so hats off to3ie who funded this one; it’s hard to do—the original authors sacrificed masses of time digging up old files for LSHTM to use; and it’s hard to get the results published—pre-publication peer review of LSHTM’s papers took about five months.

Third is just how different this is from most impact research in the social sector. This is often unreported, or reported unclearly or incompletely, and only rarely are the raw data made available to enable inspection. I’ve argued before that most charities shouldn’t do impact evaluations (as has Dean, separately)—eradicating misleading biases is just too hard for non-specialists. But when they do, they should publish the full details and data. The scientific method requires it.

This article first published in Stanford Social Innovation Review.


Posted in Effective giving, Great charities, Impact & evaluation, meta-research, transparency | Tagged , , | Leave a comment

What to when when you’re badly treated by a funder?

Jake Hayman was right in his recent blog Not Fit For Purpose: Why I’m Done With the Foundation World – there are major problems with charitable funding.

We can see this just from the fact that charities normally pay between 20p and 40p to raise £1, whereas companies pay between 3p and 5p. We can tell, too, from the remarkable unpopularity of many grant-makers in comparison to most people who hand out money.

But what do you do about it? This isn’t a rhetorical question: I’m asking for actual examples. What have you – you! – done in the past when you’ve felt badly treated by a foundation? Do you write to the chief executive? To the chair? Rant on Twitter? Just bitch about them privately? And what happened as a result? If we collectively had more stories and examples (evidence, of a sort) about what works and what doesn’t in terms of influencing donor behaviour, perhaps we could solve much of this. Continue reading

Posted in Uncategorized | Leave a comment

A welcome public row about donor effectiveness

Well done Malcolm Gladwell. On Wednesday this week, Harvard announced its biggest gift ever, $400m from the American hedge fund manager John Paulson for its school of engineering and applied sciences. Gladwell ridiculed it: ‘It came down to helping the poor or giving the world’s richest university $400 mil it doesn’t need. Wise choice John!’ Various other financial overlords sprang to Paulson’s defence: ‘My first thought was: ‘Wait a minute, pal, how much have you given?’’ said one; ‘Would they criticize him if he just sat on his wealth and ‘compounded it’ like certain others?’ said another; and a third said ‘Who the f— can criticize a guy who donated $400 million to his alma mater?!”… What’s to criticize? Extremely generous and he is to be applauded.’

Opportunity cost, that’s what to – well not criticize but to question – and effectiveness along with it. Charities vary wildly in how effective they are: with the same amount of resource, some achieve results, some achieve nothing, some achieve masses, some make things worse. The choices which donors make – like the one Gladwell is calling out – are highly consequential. Continue reading

Posted in Uncategorized | Leave a comment

Is grantee / beneficiary feedback a substitute for RCTs?

The short answer is no. At first sight, it seems that randomized controlled trials (RCTs) and Constituent Voice (CV: a good way of gathering feedback from programme beneficiaries or grantees) could substitute for each other because they both seek to ascertain a programme’s effect. In fact they’re not interchangeable at all. An RCT is an experimental design, a way of isolating the variable of interest, whereas CV is a ‘ruler’ – a way of gathering information that might be used in an experiment or in other ways. Continue reading

Posted in Effective giving, Impact & evaluation, meta-research | Leave a comment