Understanding impact. What would have happened anyway?

Article first published by the Society of Impact Assessment Analysts

In understanding a charity’s impact, we seek to identify the difference which the charity has made in the world. That is, what has happened which would not otherwise have happened. Though this may sound obvious, impact data rarely actually show it.

For example, imagine a city with poor air quality. A charity works there, trying to persuade drivers to turn off their engines when they’re idling at traffic lights. The charity reports that at the beginning of the year, the air was clean 10% of the time, whereas by the end of the year, it was clean 20% of the time.

Great!

Actually this indicates precisely nothing about whether the charity is doing a good job. Perhaps the improvement was due to the charity; but perhaps it would have happened anyway. Maybe engine technology is improving, or drivers are trying to save fuel because petrol prices are rising. Perhaps more improvement would have happened without the charity: annoying campaigns occasionally provoke people into doing precisely what the campaign is trying to curb.

At this stage, all we have from the air quality charity is ‘before & after data’. So we have an attribution problem. We know what happened but we have no idea why, and therefore we have no clue about the charity’s impact.

To determine the charity’s impact we need to ascertain three things:

1. What happened? In the air quality case, the change from 10% to 20% was pretty clear, but often identifying everything ‘what happened’ is pretty complicated.

2. How is that different from what would have happened anyway?

Since we’re normally allocating scarce resources, analysts usually also need to know:

3. How good are those results relative to other charities’ results?

To answer the second question, we need to understand the ‘counterfactual’: what would have happened anyway. That requires having a ‘control’ – that is, a situation in which everything is the same except the charity’s work. Setting up a control is sometimes easy, often tricky and occasionally impossible. How to do it is a big topic for another day.

Notice that ‘before & after data’ get nowhere near the second question. Yet charities often present results which are in fact just ‘before & after data’. For example, we hear statements such as: ‘awareness of HIV transmission is much higher than when we started’, or ‘following our campaign, the law was changed’. This is no better than saying that ‘before our work, the average height of a child was 1.2 metres whereas afterwards it was 1.3 metres’!

We need to watch out because ‘before & after’ data can be impressively complex or detailed: ‘HIV transmission rates are now 14% in the villages in which we work, whereas they were 20% a year ago’ or even ‘every time we go into a village, the transmission rate drops, and every time we leave, it rises again’, or ‘We use the “Complicated tool” to measure a randomly-chosen sample of 10% of our patients, and we find that in 95% of cases we get a drop in xyz behaviour, with a 15% margin of error and 23% standard deviation’. But complexity and detail are no proof of rigour.

Before & after data, on their own, are not useful. Rather, we need to ensure that data about ostensible results show both what happened and how that differs from what would have happened anyway – because only then can we see whether anything is actually being achieved.

How do you find a great charity?–>

Advertisements
This entry was posted in Impact & evaluation and tagged , , , , , , , , , , , , , , , , . Bookmark the permalink.

9 Responses to Understanding impact. What would have happened anyway?

  1. Pingback: The government’s raid on giving makes no sense | Giving Evidence

  2. Pingback: The government’s raid on giving makes no sense | Caroline Fiennes @carolinefiennes

  3. Pingback: Why I don’t back the Give It Back George campaign | Giving Evidence

  4. Pingback: Why I don’t back the Give It Back George campaign | Caroline Fiennes @carolinefiennes

  5. Pingback: Charity Tuesday’s Child Is Full Of Grace « captain [...] elaboration

  6. Caroline,

    The part of me that was formally trained as a researcher and evaluator, and that has done this work for 15 years agrees wholeheartedly with this post. But the part of me that worked as an internal evaluator for a nonprofit wanted to point out a few things……

    First, the critical component of your example, and your whole post, is the attribution / explanatory problem. In other words, what REALLY caused the change. But while RCTs can certainly help to explain the change, they are not the only way to get at a counter factual. In fact, a poorly designed and implemented RCT may only provide the illusion of explaining impact. While a well designed and implemented quasi-experimental design (or other design) could certainly provide some evidence of impact.

    Second, in reality, even a well designed and implemented RCT is not providing ‘true’ evidence or explanation of impact. It is simply describing a situation where most (but never all) alternative hypothesis are controlled for. A single RCT would also need to be replicated several times to attempt to eliminate as many alternative hypothesis as possible.

    Third, there will always be things that RCTs can never control, and your air quality example, if described further, will hint at this. Even if in this situation the nonprofit pays for and completes an RCT to measure air quality, there are countless factors that could never be controlled for but that can influence the results. For example, the city government could offer tax credits to a manufacturing business to move in, and they build a factory that pollutes the air. Or, the opposite could happen, tax credits could expire, and a polluting business could leave town. Or another nonprofit could be inspired by the presence of this program (or by the absence of the program in the control city) and start a similar project.

    I think all of us (funders, governments, nonprofits, and evaluators) need to understand that there is a delicate dance that we must all engage in that maximizes the rigor of evaluation (to explain impact), minimizes the costs for the evaluation, and that can be implemented within the context of the real world. In some cases, RCTs are indeed the best option. In other cases, RCTs may be impossible, or simply much too expensive, and other evaluation designs should be explored.

    Ultimately, the question we should all be trying to answer is how much ‘evidence’ is ‘enough’ given the costs and reality. While I agree that the nonprofit in your example certainly can’t ‘claim’ they are sole reason for improvements in air quality, I don’t think that this is a justification for the nonprofit to pursue an RCT. Rather, I think it is necessary for the nonprofit to carefully couch their findings with limitations – “air quality improved in the areas where we implemented the program, but there could be several other explanations” – while at the same articulating their theory as to why their program may have had an effect. At the same time, funders should learn to ask critical questions about findings and force the nonprofits to explain their reasoning.

    The key here is to do what you can given your limitations, rather than do nothing because there is the thought that your will never be rigorous enough.

    Isaac
    Senior Research Scientist
    Child Trends

  7. Great response, thanks.
    Yes, I know! But please understand that this article was written for another blog who had a 600 word limit, into which the subtleties you rightly raise (and others) couldn’t fit.
    However, they are ALL discussed in the rather longer section about impact & results in my book (from which this article is a rather bastardised excerpt) It Ain’t What You Give, It’s The Way That You Give It: http://www.giving-evidence.com/book

  8. In fact, my original tweet wasn’t quite correct (140 characters is really not a lot!) I said that RCTs are the SOLE way to deal with the counterfactual. In fact, you can sometimes get to it in other ways, e.g., natural experiments (such as several of the examples in Freakonomics) or controlled cohort studies (where you kind of do an RCT only backwards).

  9. Pingback: How philanthropic money makes major change: Moving the tanker | Giving Evidence

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s