The Bill & Melinda Gates Foundation seems to have wasted six years and $1bn, having initiated a programme to improve teaching effectiveness in US schools. An evaluation released last month showed that it had a negligible effect on its goals — some of them worsened — which included student achievement, access to effective teaching and dropout rates.
Much the same happened to Ark, the UK-based charity founded by the hedge fund industry. It created and co-funded a programme in 25,000 schools in the Indian state of Madhya Pradesh, supporting the government to improve school performance. It was based on sound research about how to reduce teacher absence, improve teaching and create more accountability through school inspections. Ark has worked on it since 2012. Continue reading →
Anne Heller has done something that I had never previously seen in my 18 years in the non-profit sector. She identified a social problem, scoured academic literature to find a solution, and then set up a non-profit to implement it. That approach sounds jaw-droppingly obvious, but it is in fact very rare for a charity to design itself around existing evidence.
Ms Heller had worked for the city of New York when Michael Bloomberg was mayor, running shelters for homeless families. She noticed that about 10 per cent of the families who use the shelters returned repeatedly. In other words, the services which the shelters provided were not solving these people’s problems. Continue reading →
Below: A map of the existing rigorous ‘what works’ evidence about institutional responses to child abuse, and summary of what that evidence says
Giving Evidence and partners have produced a map of the existing evidence (and gaps in it) about the effectiveness of institutional responses to child maltreatment, and a ‘Guidebook‘ which summarises what that evidence says. This document summarises them both. An article in Alliance Magazine written with the funder (Porticus) describes both products, the rationale for them, and also Porticus’ learning from its own daily work. These products were first produced in 2020, and updated versions produced in 2023 – with many more studies! (see below)
What’s included? We looked for effectiveness studies. We included: any primary study with a decent counterfactual (so RCTs and some quasi-experimental designs), and systematic reviews, in settings such as organisations such as schools, youth clubs, sports clubs, churches, and residential care. We included studies from any country and published at any point in time; both academic and non-academic studies; and studies about prevention, encouraging disclosure, response (e.g., legal sanction), and treatment of survivors.
The evidence and gap map. Below is a picture of our map. We ran a systematic search for the kinds of studies that we were including, screened them all for relevance, and then categorised them. We put them on a grid, in which the rows are interventions (from the top: prevention, disclosure, response, treatment), and the columns are the outcomes. Each study that we include is placed in the cell(s) to which it relates (e.g., if the study looked at the effect of Intervention 2 on Outcomes 3 and 4). This is the standard structure of an evidence gap map. As you can see, much of the map is blank, but there are a few marked concentrations. A detailed report about the map is here.
We also made an interactive version of the map. It includes summaries of the evidence: for cells with one or two completed primary studies, there are summaries of the studies; and for cells with 3+ studies, there are syntheses of the studies.
Recent years have seen many more rigorous studies published in this area: (NB, interpreting this needs a bit of caution. Sometimes studies take ages to be published. One published in 2021 gathered its data in 2013! – perhaps researchers used Covid lockdowns to get around to writing up and publishing studies, rather than these all being actual recent studies.)
Giving Evidence and Porticus did a webinar with the Campbell Collaboration in which we talked through this whole project: the logic of why the funder started this, how we made the EGM, what it found, how we made the Guidebook, what that found, the interactive version of both, and what we are doing next. Plus questions from audience members.
What this evidence and gap map shows:
108 completed primary studies, and 20 systematic reviews. There are also eight protocols. Of the 108 completed primary studies, 79 are RCTs.
The studies don’t at all match where the world’s population is: none from India, only three from China, and only 19 from Africa.
The evidence is concentrated -though much less so than in our first iteration, three years ago: mainly in education-based prevention programmes, in early education and school settings. Fully 60 of the 108 completed primary studies look at that.
Most of the studies are about sexual abuse. Sexual abuse was considered by 93 of the studies.
38 studies reported on physical abuse, 19 on neglect, and 27 on emotional abuse. .
Most of the studies are about prevention. Prevention was examined in 120 papers (some studies generate more than one paper).
Treatment was studied in only a handful of primary studies. On disclosure, we found three primary studies of interventions aiming to facilitate disclosure: last time, we found none.
No completed study has assessed interventions with adults to stop them offending within organisations
Only few studies focus on children particularly at-risk
No causal studies conducted in religious organisations
Almost all the studies have appreciable risk of bias i.e., the ‘answers’ that they report may be wrong. The colours on the map above indicate risk of bias: red is high, and green is low.
Only one study had educational attainment as an outcome
What the evidence says. We have produced a ‘Guidebook‘ which summaries the studies, and provides guidance for using the evidence.
Full report about the map. The full report is here which discusses the findings in detail.
How does the evidence align to activity within child protection? We also explored finding data about activity in child protection (e.g, what schools are doing, governments, youth clubs etc.) and mapping it to the EGM frame – to identify where there is lots of activity but no evidence (=priority for new evidence) and areas of little activity but where the evidence is clear that something works (=areas to encourage more activity). Basically, that didn’t work because the evidence about child protection activity is so scant (=important research gap!). Our report into this exploration is here.
Wide-ranging chat about charitable giving, with the Daily Mail.
What to give to, what to avoid, one of Caroline’s favourite charities, how to choose a charity… (~20 minutes) Please click on the picture below to watch the full video.
Giving Evidence has been consulting with UK charities and donors to identify the topics in which they would like further research (/ evidence / data) into charities and philanthropy. This was a large-scale and ground-breaking exercise, using a (variant of a) consultation method developed in medical / health research for eliciting from patients and their carers where they would like researchers to focus. We are today (2nd July 2019) launching the findings, which you can download here.
Judging by my inbox, there seems to be huge demand for advice about which charities to support. As donors have followed the turmoil around Oxfam, Save the Children, Kids Company and others before them, they want decent independent analysis along the lines of the data that rating agencies provide for bonds or Which? does for fridges.
There isn’t any.
It doesn’t exist, because of costs, incentives and the genuine challenges involved in nailing it down. Continue reading →
The practice of donors offering to “match” gifts made to charities they support has become common in the charity world. In the run-up to Christmas, many have been offering to double donations. In the recent Challenge Campaign, for instance, money given to selected charities was doubled by the Big Give, set up by Sir Alec Reed, founder of Reed, the recruitment firm.
Others schemes propose different deals: one donor was offering to give $2 for every $1 given on Friday to Charity Navigator, a charity rating agency; while PayPal will add 1 per cent to all donations made through its platform until New Year.
The UK’s Department for International Development does it too. Its UK Aid Match offered up to £5m from the UK aid budget to eight charities.
The theory is that the match attracts new donations. If your aim is helping your chosen charity to raise money, is it a good idea to offer to match other people’s contributions? The rigorous evidence suggests that it is. Continue reading →
The first president of eBay, Jeff Skoll, set up his Global Threats Fund in 2010 to ‘make progress against five of the gravest threats to humanity’: climate change, pandemics, water security, nuclear proliferation, and conflict in the Middle East. It closes this month having spent its $100 million, and this week published a report about “what worked, what didn’t work, and what we learned about philanthropy’s role in reducing global threats”. As more foundations ‘spend out’ and publish their learning, we will probably see more such documents. This one is a weird read. Continue reading →
If you, like millions of people, want to give this Christmas / holiday season, the following are all good bets, in our view:
Medicins Sans Frontieres: They always seem to be to be where the sh*t is worst, e.g., Yemen, Rakhine province in Myanmar (where the Rohinga live/lived). We’ve never formally analysed them but all our contact with them and literature we’ve seen from them implies that they have their act together. Continue reading →
progress against five of the gravest threats to humanity’: climate change, pandemics, water security, nuclear proliferation, and conflict in the Middle East. It closes this month having spent its $100 million, and this week published 
Yemen, Rakhine province in Myanmar (where the Rohinga live/lived). We’ve never formally analysed them but all our contact with them and literature we’ve seen from them implies that they have their act together. 
giving-evidence.com 