Charities should do fewer evaluations; those few can be better

It’s hard to make evidence based decisions if much of the evidence is missing, garbage, unclear, or you can’t find it. Talk given in Barcelona (18 mins)

More examples of important evidence being missing or garbage–>

What Giving Evidence is doing to make charities’ evidence clearer and easier to find--> 

Posted in Uncategorized | Leave a comment

Give Your Best, this Giving Tuesday

Caroline Fiennes explains how to maximize the effect of your donation, even if you have no money at all. [This article was first published by GivingTuesday.]

The basics which you must know about charities before you start

Some charities are miles better than others 

This sounds rather heretical because we often think that all charities are good. But we also think that teaching is good, and so is providing medical care, and yet we know that some teachers are better than others, some doctors, some treatments. It’s the same with charities, so your choice matters.

For example, in Kenya, where diarrhoea from dirty water is a major problem, delivering chlorine to households can prevent diarrhea for a certain cost, but giving people chlorine at the village water source achieves the same result for less than half that cost.

Similarly in North India, free village clinics are pretty good for getting children immunised. But if clinics offer mothers free lentils for every child immunised, immunisation rates increase more than six times.

And in Southern India – where I was once a teacher – children skip school a lot.  Giving the parents cash if their children show up (a respectable and widely used idea, called a ‘conditional cash transfer’) solves some of this, but giving out free school uniforms achieves ten times as much. And that’s peanuts compared to dealing with intestinal worms which many children there have: for the same price, ‘deworming’ can achieve 25 times as much.

Do the maths. Or cheat.  

The catch is that choosing wisely is hard, because charities rarely have these comparative data: a leaflet picturing a school child doesn’t indicate much about an organisation’s performance. So ignore the fundraising literature and look under the bonnet at what the charity actually does and whether it’s good at it. Making a difference depends on having a good ‘idea’ (strategy) and implementing it well. If you’re into formulae, think of it as: impact = idea x implementation.

In fact, since choosing wisely can be fiddly and laborious, find somebody smart and copy their homework! The charity world includes two types of people who’ve already done their homework in detail. First, are independent analysts. GiveWell analyses charities in great detail and only recommends about 1% of the charities it assesses. They’re all in international development, mainly public health. Charity Navigator* is much broader, publishing analysis on several thousand US-registered charities. Its ratings look at the charities’ performance on financial criteria, and transparency and accountability, and it’s adding information about their results. Global Giving is ‘an eBay’ for international development, and lists many grass-roots organisations which it has vetted.

Second, many charitable trusts and foundations employ people to analyse charities to decide which ones the trusts should fund. Some (but not all) of them are robust, and you’ll be pretty safe supporting charities which they back. You only have to find one whose interests match yours. If you’re interested in creating jobs in the US, look at F.B.Heron Foundation; if it’s poverty in New York City, look at the Robin Hood Foundation, for international development, look at Hewlett Foundation. A good signs is when a foundation publishes a sensible-looking strategy and criteria.

Don’t look at administration costs

People often think that low admin costs are a good sign. It turns out that they’re not. The costs which get shown in a charities accounts (and I wrote a whole book once about charities’ accounts, so I know!) include all kinds of useful things like systems to monitor results and evaluate what’s working and to make improvements. It’s more accurate to think of them as management costs: and so it’s rather unsurprising thatanalysis shows that charities with higher ‘admin’ costs tend to perform better.

But I don’t have any money 

Well then, rustle up money for charity from thin air! Try variants of these ideas:

Friends. Starfish, a charity which helps HIV/AIDS orphans in South Africa, is supported by young professionals in the West. They hosted dinner parties in their homes and got guests to donate to Starfish the money which they would have spent if the party had been in a restaurant: money which nobody had earmarked for charity.

Neighbours. Fred Mulder lives in London, UK, and was in a dispute with his neighbors over access to some land that he owns. Rather than all hire expensive lawyers to resolve it, Fred offered to give his neighbours perpetual access if they each (Fred included) donated £25,000 towards an educational charity in Zambia. This generated over £100,000 for charity and improved the neighbours’ relationship, which a legal fight never would have done.

Clients. Fred Mulder is full of these ideas. He’s an art dealer, and sometimes when negotiations with clients get become stuck, he suggests that the difference between his price and the offering price be donated to charity.

Bulk purchasing. A financial services company in a medium-sized British town includes various charities in its IT purchasing processes, so they benefit from the company’s volume discounts.

Hotel toiletries: Some business people who travel a lot give the complimentary toiletries from hotels to a domestic violence refuge. For people on the run from a violent partner, it’s nice if somebody’s provided some decent shampoo.

Things to give which aren’t money

Blood Find your nearest blood donation session at http://www.blood.co.uk/SessionSearcher/search.aspx

Bone marrow Some tissue types are more common in certain ethnic groups of the population, meaning that a patient normally needs a donor from a similar ethnic background to her own. There’s a particular need for stem cell donors from African, African-Caribbean, Asian, Chinese, Jewish, Eastern European and Mediterranean communities. You can register as a bone marrow or stem cell donor when you give blood or at http://www.nhsbt.nhs.uk/bonemarrow/

Business clothes Disadvantaged women trying to get back into work need business clothes – as well as training and confidence – for interviews and when they start work. Dress for Success works in nine countries, and has now helped over 550,000 women. http://www.dressforsuccess.org

Cars Several organisations will collect an unwanted car and turn it into money for charity through http://www.giveacar.co.uk.

Computer equipment Which? has a useful guide to recycling computers http://www.which.co.uk/environment-and-saving-energy/environment-and-greener-living/guides/recycling-computers/pc-recycling-tips/

Coupons and free stuff. You can donate the buy-one-get-one free items you don’t want. I know some business people who are constantly travelling and give the complimentary toiletries from hotels to a domestic violence refuge: if you’re on the run from a violent partner, it’s nice if somebody’s been thoughtful enough to provide some decent shampoo.

Cycles A number of nonprofit organisations refit unwanted bicycles to send to countries such as Haiti and South Africa, in the process training people in the UK to repair bikes. http://www.re-cycle.org and http://www.recyke-y-bike.org

Furniture The Salvation Army will take furniture to sell in its shops or pass on to homeless people settling in a new home. http://www.salvationarmy.org.uk

Gardens Landshare brings together people who want to grow their own food but have no place to do it and those who have land to share but lack time, experience or muscle-power. www.landshare.net

Glasses Visionaid Overseas organises a nationwide recycling scheme for old or unwanted spectacles. http://www.vao.org.uk

Musical instruments can go to school music programmes, senior citizens, talented young students, community groups, and charities can use them at events and as prizes to help raise money.

Paint Community RePaint schemes collect unwanted, surplus paint and re-distribute it to individuals, families and communities in need, improving the wellbeing of people and the appearance of places across the UK. www.communityrepaint.org.uk

You can even give your hair! If you have more than ten inches of hair cut off, take it home and donate it to make wigs for people who’ve lost hair due to medical treatments. www.charityintersection.com/donatehair.html orwww.littleprincesses.org.uk/donate/hair.aspx

The rest?  

  • Charity shops take clothes, books, records, CDs, DVDs and jewellery, and some take furniture and electrical goods. Remember to fill in a Gift Aid form.
  • Primary schools and nurseries can use all sorts of things for craft projects: fabric, knitting wool, rolls of wallpaper, old Christmas cards, jars and bottles. Just ask first.
  • Find a new home for almost anything on Freecycle and save it from landfill.www.freecycle.org
  • Lend it to people in your neighbourhood through http://www.streetbank.com
  • Sell it and donate the proceeds. Through the online marketplace eBay you can donate the proceeds from selling virtually anything to a charity of your choice. Secondhand books can also be sold through Abebooks http://www.abebooks.co.uk and Amazon.

But do check with the charity first. People donate real junk, so much so that aid agencies run an annual competition for Stuff We Don’t Want (#SWEDOW). Past winners have included second-hand knickers(!), and the 2.4 million Pop-Tarts® airdropped onto Afghanistan by the US government in 2002. Far from amusing tales, these items create costs for charities because they need storing and sorting, and simply become a hindrance. It’s not difficult to check that a charity needs an item before sending it.

Posted in Uncategorized | Leave a comment

Lessons during the decade since the Asian tsunami

This article first appeared in Third Sector

It’s 10 years this December since the Indian Ocean Boxing Day tsunami. We salute those who died, those who mourn, those who tended; and we celebrate those who’ve since sought to improve response to disasters and emergencies: they’ve been remarkably effective.

For doctors in unfamiliar situations, the first port of call is The Cochrane Collaboration, a huge set of high-quality reports that collate and synthesise the (reliable) evidence about what to do. Its Cochrane Reviews are produced by more than 34,000 researchers in 120 countries, most of whom do them voluntarily, coordinated by a small band of experts from a tiny office in a residential street in north Oxford.

The day after the tsunami, the former co-chair of the collaboration, Mike Clarke, realised that The Cochrane Library, where the reviews are published, was pretty unhelpful for disaster situations. Reports on fractures might assume you’re in a first-world hospital with several hours to spare per patient. You’re not, and you don’t: you’re in a makeshift field hospital with patients queueing up. Worse, relevant Cochrane Reviews are scattered, filed under umpteen categories, and you’ve got no time to search and a dodgy internet connection. And some reviews are paywalled.

Evidence Aid was born that day. Now based in Belfast, it compiles relevant reports from Cochrane and elsewhere so they’re easy to find. It creates new reviews if NGOs and medics feel existing literature is inadequate. Volunteers are creating 100-word summaries, adapting guidance to the situation: in surgery, for example, the evidence shows you’re as safe washing a wound with tap water as using expensive sterile saline; but if clean water has become precious after a disaster, or is dirty, use saline for washing. It’s obvious, really, but chaos allows no time for thinking.

It seems to be working: Evidence Aid’s advice has prevented the use of various treatments that sound plausible but are shown by the evidence to actually make things worse; NGOs contribute their ideas and requests; and advice went to the World Health Organisation within 24 hours of the Haitian earthquake in 2010.

A few months after the tsunami, a girl turned up in a clinic in Indonesia, apparently with measles – a surprise, because many agencies had worked to prevent measles after the tsunami. It transpired that she’d been vaccinated three times by several different organisations.

At the time, there was no common standard through which charities and government agencies could report publicly about their activities, so it was all but impossible to get data on what other agencies were doing. The International Aid Transparency Initiative was set up to make information easier to find and more useful, and thereby avoid these situations. The Department for International Development was the first entity to publish in IATI’s format (in 2011), since when 280 others have done so.

Much remains to be done. The response in Haiti is infamous for poor coordination and providing inappropriate aid: the International Olympic Committee funded a new stadium – at $18m! When Typhoon Haiyan hit the Philippines last year, many (myself and others) asked us not to help like we helped in Haiti.

DOI: I’m an unpaid advisor to Evidence Aid. Here’s why –>

Posted in Impact & evaluation | Tagged , , , , , , , | Leave a comment

What constitutes good evidence?

Lovely interview, about what constitutes good evidence, which donors is this relevant to, doesn’t requiring evidence impede innovation or encourage donors to focus on short-term outcomes, etc. (Gets into English after about 2 minutes.)

This is the Forbes article which I reference.

Posted in Uncategorized | Leave a comment

Getting people to give better

New initiative aims to get donors to give better

Many people look at getting people to give more. Giving Evidence and the Social Enterprise Initiative at the University of Chicago Booth School of Business are starting work looking at getting people to give better. First, we’re developing a ‘white paper’, to be published early in 2015, to collate what is known about effective giving, what isn’t yet known, and what would be useful for researchers to find out. [The University of Chicago Booth School of Business was recently ranked by The Economist as the best business school in the world.]

The way donors give is important, so perhaps persuading them to give better would have the same social effect as getting them to give more. For instance: the cost of raising capital for charities is about 20-40 per cent, against only about 3-5 per cent for companies, and charities turn away some donors who are fiddly to deal with.  Plus money doesn’t always go where it’s most needed: for example, about 90 per cent of global health spending goes on 10 per cent of the disease burden. And making many small gifts is demonstrably more wasteful than making a few large ones.

Perhaps it’s easier to get somebody to give better than to get them to give more.

We aim to identify questions which non-profits, funders and other practitioners want answered about making giving better, and to encourage researchers to address them. Those questions include the following:

  • How do various donors (including foundations, corporates, individuals) define a ‘successful gift’?
  • Is success affected by (eg):
    • being hands-on?
    • donors working together (eg in giving circles)?
    • gift size?
    • how and whether the grant is tracked?
    • whether the donor gives, lends or invests?
  • What does it cost to raise and manage grants of different sizes
  • How and when can one influence the cause that a person supports?
  • How do donors choose causes, charities or grantees, and how influence-able is that?
  • How do donors choose processes (eg for sourcing grantees, selecting which to support)?

However one defines success for a grant, it would be useful to know (wouldn’t it?) whether and when and how the chance of success is affected by how the donor gives.

Our purpose is to identify questions which non-profits, funders and other practitioners would like to have answered, which would help make giving better, and to encourage researchers to address them.

In terms of scope, we’re looking at all giving: ‘retail’ individuals, endowed foundations, fund-raising foundations, private family foundations, companies – the lot.

Do get involved!

Please send relevant material to jo [dot] beaver [at] giving-evidence [dot] com

Feedback from readers suggests that an example might. We’re interested in what makes for successful giving. So if a particular donor or funder has data on the success rate of its grants (ie., the no. which ‘succeed’, on whatever measure of success that donor uses) and how that success rate varies with (things like) grant size, grantee size, how the donor came across that organisation (e.g., open application process, in the pub, network), how hands-on the grant was, duration, whether it was co-funded… we’re VERY interested in that.

We’re not at this stage looking to do primary research (eg., working with funders through their historical grants, assigning ‘success scores’ to them & cross-tabbing that with things like size) though we may get to that in future.

To be clear, this project isn’t (just) about getting donors to choose high-impact charities. Not because that isn’t important, but because many others are looking at that. It’s about all the other choices which major donors /foundations make which can have just as much impact, and indeed can anhiliate the impact of their grant. A simple example of the effect of how one gives (as opposed to where one gives) is that funders sometimes create so much work for grantees that the grantee would be better off without that grant/relationship at all.  In that case the choice of charity doesn’t matter much!

So questions we’re looking at include: should donors give individually or in groups? should they proactively search out grantees or let grantees find them? how engaged should they be? how many focus areas should they have? That is, which giving behaviours (of those types) drive the success of grants -in whatever way the donor defines success.
There’s no shortage of opinions on these topics, but we’re looking for data.
The Shell Foundation published data on the %of its grants which succeeded when it was, various, spray-and-pray, somewhat focused, and latterly very focused. That’s what we’re after: some empirical basis for ascertaining what makes for effective philanthropy. Obviously the ‘right’ answer may vary between circumstances, just as the ‘right medical treatment’ depends on the patient’s condition, and those variations are interesting too.
Posted in Uncategorized | Leave a comment

Caroline Fiennes: best philanthropy advisor

Newsflash! Giving Evidence’s Caroline Fiennes has been nominated a ‘best philanthropy advisor’ by Spears Wealth Management magazine, here.

The profile of Caroline (here) says: CF Barcelona

“Caroline Fiennes’ work in philanthropy focuses on making giving as effective as possible by basing it on sound evidence. A physicist in a previous career, Fiennes became interested in the fact that some charities are better than others and wanted to figure out which ones are most effective in order to guide donors to them. This is also true of ways of giving.

The founder of Giving Evidence feels there is ‘often a big mismatch between where the money goes and where it’s needed’, and advises clients on using the available evidence to choose issues and organisations to focus on and support in the most effective ways.

Caroline works a lot on the quality of research available to donors, because charities produce a lot of information, but much of it is of too low a quality to be reliable. This has led to some of her clients giving funds to help produce better evidence.

Caroline and Giving Evidence are working on creating a mechanism for anybody to rate a charity with which they’ve had contact, a little like TripAdivsor or Toptable. This ‘opening up of reputations’ would greatly help donors to make much more informed decisions.”

Stay in touch to hear more as this project progresses!

Why is rating charities important?—>

Posted in Uncategorized | Leave a comment

Listening to those we seek to help

This article first appeared in Third Sector.

Unlike in business – where companies must heed their customers because they’re the sole

From a school wall in Zimbabwe, tweeted by Melinda Gates

source of funds – charities don’t normally get funds from beneficiaries and hence feel no financial pressure to listen to them. A recent report by Médecins Sans Frontières shows the result, recounting the apparent abandonment of war-torn areas and emergency situations by most aid agencies, which seem instead to follow funders’ wishes to operate in safer countries.

It’s tough for beneficiaries to tell an NGO, government or funder directly what they want or what they think of what they’re getting. It’s harder still for would-be beneficiaries. There are few feedback loops in our sector, though there is an outbreak of intriguing work to create more.

GlobalGiving UK is “an eBay for development”, directing donors to grass-roots organisations. To enable the organisations to hear and heed the constituencies they’re ostensibly serving, it developed a simple tool that requires a charity to recruit about 20 volunteer “scribes” who go out and interview local people. The charity chooses the questions, which GlobalGiving recommends be very open. Typically, the scribes ask for a story about a need in the community and one about an organisation being helpful (or not). They do not ask “what is your opinion of organisation x?”

There are two clever bits: first, the interviewer is not part of the charity, which probably makes the stories more honest; second, GlobalGiving – using a system designed by a man with a neuroscience PhD – analyses the stories for patterns. For instance, the contexts in which the charity is mentioned and the sentiments expressed about it. Maybe the charity doesn’t get mentioned much, implying that it’s not achieving much. The frequency with which various problems are mentioned can show the charity where it might target its work.

Stories collected this way over the summer by going house-to-house in Lambeth, south London, featured cancer, the passport fiasco and alleged corruption in local government. Charities were often cited unprompted: one advertising executive was so struck by Greenpeace protesters risking jail for their altruism that he’d started volunteering locally and taken his children to protest at Shell’s sponsorship of Lego.

The Department for International Development is piloting ways of getting feedback from beneficiaries into its programmes. The Association of Charitable Foundations recently provided training to foundations on listening to beneficiaries, and the White House hosted a summit last year about improving feedback to US government programmes. And a group of foundations, led by the Hewlett Foundation, has announced a fund to improve philanthropy by “listening to, learning from and acting on what we hear from the people we seek to help”.

We should all get good at doing that.

Posted in Uncategorized | 1 Comment

Publishing the whole truth

This article first appeared in Third Sector.

The C&A Foundation – linked to the department store that closed in the UK but is flourishing elsewhere – joins a small clutch of non-profits this month that publicise the lows as well as the highs in their work.

It ran a programme in 18 garment factories in Asia designed to improve both working conditions and productivity. Some aspects worked in some factories, some aspects worked in others. Rather than taking the conventional option of reporting only the glories, in a kind of self-promoting publication bias, the C&A Foundation is publishing it all: data from each factory, correlation coefficients, statistical significance tests and operational challenges all feature in the report, which is called Frankly Speaking. (Disclosure: I am a co-author.)

Likewise, Engineers Without Borders, a Canadian NGO, has been publishing an annual failure report since 2008. In each one, volunteer engineers recount tales of an error they made in the field. Yet, despite the praise for EWB and the obvious value of hearing the whole truth, EWB remains an anomaly. To my knowledge, it’s the only operating charity that publishes so candidly. When I asked its chief executive why it discloses when others don’t, he said: “Well, if your bridge falls down, it’s pretty obvious.” Indeed. By contrast, plenty of social programmes appear to work but are shown by sophisticated analysis not to work. The crime-prevention programme Scared Straight and some micro-finance programmes are examples of this.

The C&A Foundation encountered something similar – but the opposite way round. When Giving Evidence got involved, it looked as though working conditions in the factories hadn’t improved much, but the inclusion of later data in the analysis showed that they had.

Medical doctor John Ioannidis, now of Stanford University in California, uses complex statistical tests to unearth often shocking stories within clinical trial data and says his work echoes a theme from the ancient literature of his native Greece that “you need to pursue the truth, no matter what the truth might be”.

Dogwood Initiative is an environmental campaign on Canada’s west coast. It was inspired by EWB to publish its own failure report and found an important issue in its internal culture. “Dogwood Initiative could change our relationship with failure,” says the report. “It involves piloting an idea, measuring results, figuring out what works and what failed, adapting and rebooting.” Giving colleagues that right to admit failure can take time. Dogwood Initiative’s first annual failure report took so long to agree internally that it ended up covering the next year too.

The World Bank also holds internal “failure fairs” and finds it needs rules to ensure the discussions are safe – each speaker makes a case but can’t blame anyone except themselves and can’t name anyone else.

The funding and competition in the charity sector undoubtedly discourage confessions of weakness, but if we don’t do it, we won’t learn from ourselves and we’ll bias the literature from which our peers – and funders – can also learn. EWB’s Admitting Failure website lets anybody upload stories.

Go on, I dare you.

Posted in Impact & evaluation, transparency, Uncategorized | Leave a comment

Go see your MP

This article first published in Third Sector.

Why is so little policy based on sound evidence? Many voluntary organisations, academics and others spend time producing research in order to influence the government. There are some successes – but much policy appears to disregard the evidence.

Mark Henderson is head of communications at the UK’s largest charitable funder, the Wellcome Trust, and author of The Geek Manifesto, which calls for a more scientific approach to policy and politics. He says there’s little political price to be paid when MPs ignore the evidence. He also says that, in their constituencies, most MPs know the business people – who, after all, will ensure that people of influence have the benefit of their views – but rarely know the scientists. They probably don’t know the charity sector people, either.

I am an advocate of evidence and I’m often in meetings about the importance of getting evidence into policy with organisations such as the Alliance for Useful Evidence, the Institute for Government, the Hewlett Foundation or the National Institute for Health and Care Excellence. If Henderson is right, we’re all stuck in an echo chamber and missing a trick.

So I went to see my MP. And with haste: she is Justine Greening, a Cabinet minister – for international development, as it happens – and the recent reshuffle was looming.

“Hello, I’d like to talk about how keen I am that government policy be based on robust evidence,” I said – an unusual opening in an MP’s surgery, to say the least; but it led to a spirited conversation. Before my visit, I had, by way of a focus group, asked on Facebook what I should raise with a secretary of state. The doctors all rattled on about distinguishing between good and bad evidence, and everybody cited weariness of politicians cherry-picking data that suited them. Having removed the names, I printed out the responses and presented them.

Most revealing were two interconnected things. First, when I told Greening that the Department for International Development was generally very sophisticated in its use of evidence, she seemed amazed. “Most of my constituents think foreign aid is a waste of money,” she said. Actually, plenty of her constituents don’t think that – I socialise with them and they say so – but, clearly, those views had never reached her.

If we want MPs to act on evidence, we should go and tell them that that’s what we want

Second, not a single person I know had ever been to see their MP. Many of us battle attitudes voiced in what we might call “the uncharitable press” and bemoan MPs who pander to it. They hear and heed calls to continue the Work Programme, to cut the Third Sector Research Centre and so on. If we’ve never told them our contrasting views, we’ve nobody to blame but ourselves.

So go and see your MP – it’s your democratic privilege and weirdly empowering.

Does it make any difference? I don’t know. Perhaps we should gather evidence on this. Ben Goldacre, the broadcaster and science campaigner, and the innovation charity Nesta created an online tool, RandomiseMe, which enables anybody to run a randomised, controlled trial. We could all participate in such a trial: half of us go to see our MPs and half don’t, and we watch for subsequent differences in their voting behaviour. If we want MPs to act on evidence, we should go and tell them that that’s what we want.

What IS good evidence? Not this–>

Posted in Uncategorized | 1 Comment

Does the charity sector have a publication bias problem?

This article first published in Third Sector.

It’s hard to make evidence-based decisions if much of the evidence is missing or ropey. So it’s disastrous that the results of many medical clinical trials are missing, preventing doctors from using them.

It’s thought that fully half of all clinical trials remain unpublished. It’s not difficult to guess which half. Apparently, trials published by pharmaceutical companies are four times more likely to show that the drugs have a positive effect than identical trials conducted independently. So why is that?

Well, trials themselves don’t lie. Magically, however, the negative ones don’t get published. This publication-bias costs lives, yet is perfectly legal. Dr Ben Goldacre, author, broadcaster, campaigner and debunker of bad science, says that the near-fatal effects of the drug trial in Northwick Park hospital a few years ago – when all the men in the trial ended up in A&E with multiple organ failure – could have been predicted from results that were known but not published.

So we should all applaud the AllTrials campaign, initiated by Goldacre and and seed-funded by Simon Singh, to ensure that the results of all trials are published. Goldacre and Singh take scientific integrity seriously: both were accused of libel for highlighting bogus claims, refused to recant, endured horrible, long legal trials – and won.

Does the charity sector have the AllTrials problem? Do we withhold some of our monitoring and evaluation research and publish only unrepresentatively positive and misleading material? I suspect so. I did it myself when I was a charity chief executive: graphs that go up made for good meetings, so we published them; graphs that go down made for terrible meetings, so we didn’t. I don’t believe we were alone.

Monitoring and evaluation is research. It’s not always framed as that – it’s often seen as compliance with the requirements of funders; but it’s there to investigate possible causal links. Whether and when does intervention X lead to outcome Y? Do breakfast clubs improve children’s learning? Does teaching prisoners to read help them get jobs when they are released? These are research questions.

Ideally, that research would not be private but would be published to constitute evidence-based practice, just as clinical trials guide doctors. Any other charity could use it to decide whether the intervention might work in its context and whether it should replicate it.

But we can’t make evidence-based decisions if the literature is incomplete or biased; and, as ever, it’s our beneficiaries who miss out.

Research-withholding and publication-bias are commonly studied in medicine to establish whether and where there’s a problem and work can be targeted to fix it. But to my knowledge, neither has ever been studied in our sector. Not once. One study (itself unpublished) in Canada that was looking at something else found – shockingly – that the proportion of research carried out by charities that was published was 2 per cent.

Investigating the withholding of research and publication-bias is neither difficult nor expensive. It’s time we knew whether we too need a fix.

Charity data is missing and ropey a lot, such as here –>

Posted in Effective giving, Impact & evaluation | Leave a comment