This article first published in Alliance Magazine in June 2024

Is your foundation any good? I don’t mean: is it big, or does it give away a lot, or does it run an efficient process, or do the grantees achieve much. Rather, I mean: are you any good at being a foundation? Are you effective? Are you good at finding work that will succeed? Do your grantees think that you help them? Is progress towards your goals faster with your foundation’s involvement than it would be without you?

Few foundations seem to know. We suspect that few foundations analyze to answer these vital questions. Certainly, few publish any. Yet it is perfectly possible to find out, and to identify where and how to improve. Giving Evidence knows this because we have looked at what foundations publish about their own effectiveness, and we have done these kinds of analyses for foundations.

Our research about what foundations publish about their effectiveness

Each year, the Foundation Practice Rating assesses 100 UK-based charitable grantmaking foundations on their practices on diversity, accountability, and transparency. The research is done by Giving Evidence. For our ‘accountability’ criteria, we look at whether each foundation publishes any analysis of its effectiveness, what types of analysis it publishes, and whether it says what it will change as a result.

In this most recent year, only 16 of the 100 foundations were scored as publishing any analysis of their own effectiveness. This is remarkable given foundations’ enthusiasm for assessing other organisations, i.e. ones which they support or might support.

For the Foundation Practice Rating, we count as an analysis of a foundation’s effectiveness analysis such as:

• views of grantees and/or applicants, collected systematically. (We did not count ad hoc quotes or case studies published without a statement that all grantees/applicants were surveyed, because there is no way of knowing whether the foundation has cherry-picked only the most flattering examples);
• analysis of the proportion of grants which at some level succeeded vs. those which did not; or
• analysis of the costs created by the foundation’s funding processes and borne by grantees/ applicants. Ideally this would be expressed as a proportion of the amount given, i.e. the net grant. This matters because clearly if a foundation is a net drain on the sector it seeks to support, then it is not helping.

None of these is a perfect measure of a foundation’s effectiveness, but each gives a line of sight. By analogy, there is no single measure of the health of a nation’s economy, which is why economists use a whole raft of measures. If a grantmaker has none of the three types of analysis listed, we would argue that it cannot know whether it is doing well.

What if the foundation does this analysis but doesn’t publish it?

That’s clearly better than nothing, though there are two weaknesses. First, the foundation is not making itself accountable about its effectiveness. And second, nobody else can learn from it.

Indeed, because only 16 of 100 foundations publish anything on this, foundations can learn only very little from each other’s public material about how to be effective and how and where to improve.

If foundations assess their effectiveness, does that shift even more power to funders and away from operating organisations?

No. Firstly, as mentioned, we count systematic surveys of grantees. That doesn’t imply anything about who is in charge, and indeed, a grantee survey would reveal whether grantees feel that power is stacked against them and show any consequent harms created by the funder. And second, we also count if a foundation reports the proportion of its grants that achieve whatever goals they are intended for. Those goals can be entirely set by the grantee.

The following are not analysis

• A breakdown of where their grants went, for example, by geography or sector is not analysis of effectiveness. Rather, that simply catalogues activities or inputs.
• Nor is citing activities or outputs such as “76 volunteers have received training to help them provide support within their organisations”.
• Reviews of individual programmes. People can learn from those, but they do not assess the foundation as a whole.
• Case studies or anecdotes of feedback from grantees are not analysis because, again, there is no way of knowing whether they are representative.
• Nor do we give credit for reports recounting grantees’ achievements – because those achievements might be despite their funders! This isn’t a rhetorical joke: there are real examples of funders whose laborious processes cost grantees more than the grant is worth. In those cases, the funder is detracting from grantees’ work.

What foundations do publish on this:

Systematic surveys of grantees are the most common analysis for which FPR gave credit. Around 12 foundations publish those. Examples include that by The AB Charitable Trust.

Some surveys were run by external agencies; others appear to have been run by the foundation itself. The analysis published by the Lloyds Bank Foundation for England and Wales is structured around eight lessons that it has learned through its work, including from systematically hearing from grantees, and how it is integrating those lessons.

None of the 100 foundations assessed published analysis of the proportion of grants which meet their goals.

Near-misses and honourable mentions

Two foundations published numbers for their ‘social return on investment’ but without any detail of the calculations, input data used, or what the numbers refer to. Disclosing those details might be helpful and insightful.

Catternach cites a review of its “approach to evaluation and learning”, done by an external entity. We could not find that review published. However, the foundation does cite in its Annual Report some changes made as a result, for example adopting a lighter-touch approach to evaluation, focusing on ‘an evaluation approach that supports organisations in their own improvement process’ and generally using ‘evaluations as a tool to support grantee learning’.

Co-Op Foundation says that ‘95 percent of young people took an action after seeing our Lonely Not Alone campaign’. It cites neither any reference nor what set of young people it refers to, nor whether this number relates to the funding provided: without that, we do not know whether that number is unchanged from before the campaign – or even whether it has fallen!

Other examples come with no analysis of why the foundations have been able to achieve the claimed results, what the necessary conditions or resources are, nor the cost. So the scope for other foundations to learn from them is pretty limited.

One of the strangest statements of ‘impact’ that we’ve seen for a while is this, from the Hunter Foundation: ‘It is almost satirical that despite years of [our] investment in enterprise education and entrepreneurship, Scotland, thanks to our own Centre’s research, has not moved one jot in becoming more entrepreneurial in the past decade. We only console ourselves that the millions of pounds we’ve invested in enterprise education alongside Government and others maybe stemmed a possible reduction in entrepreneurial activity rather than just helping to maintain the status quo.’

How foundations can assess their effectiveness

The three methods listed above are all perfectly possible. In order, those are:

1. Systematic collection of feedback from grantees, covering all grantees. Above are examples of that, and clearly it is what the Grantee Perception Reports have gathered for years.
2. Analysis of the proportion of grants which (at some level) succeeded vs. those which did not. The Shell Foundation did this in a report to mark its first 10 years. Giving Evidence has done this analysis for the ADM Capital Foundation, based in Hong Kong. Another UK-based foundation did this analysis – and compared the eventual success of work that it funded to that of work that it rejected… and actually found no difference at all (!)
3. Analysis of the costs created by the foundation’s funding processes and borne by grantees / applicants. This is the simplest of the three to do. Giving Evidence knows of a UK-based foundation which has done it but never published it.

“Learn as if you were to live forever” – Gandhi

All foundations could usefully do these analyses, in order to learn where and how to improve. We are happy to discuss with any foundation how to do this.

            A new project will collate the measures used in studies of institutional responses to child abuse and show how both researchers and practitioners can use these measures, and the measures’ respective strengths and limitations. Article by Gabrielle Hunt.

A newly updated Evidence and Gap Map (EGM) collated rigorous ‘what works’ studies about institutional responses to child maltreatment: incl. preventing it and responding to it. Although the body of evidence is growing, there are still significant gaps.

The EGM includes randomised controlled trials, quasi-experimental design studies, and systematic reviews. Most studies on it examined interventions that aimed to raise children’s knowledge (e.g., about how to avoid being abused or how to report abuse) or include measures to assess improvements in their well-being. Though this is important for a well-rounded approach to prevention, it places much of the responsibility for safety on children rather than on the adults who care for them. Few studies directly measure the incidence of maltreatment, and only a few studies measured the attitudes or skills of adults working with children or the culture in institutions.

It is essential to determine whether the measures used in the studies actually measure children’s safety or incidence of abuse, and which measures are most useful for research and practice.

Professor Daryl Higgins and Gabrielle Hunt from the Institute of Child Protection Studies (ICPS) at Australian Catholic University will work on a 3-month project to collate details about all the outcome measures used in the studies on this EGM. ICPS is a nationally recognised centre for excellence in child, youth, and family welfare and is committed to collaborative approaches to translating knowledge into policy and practice. Professor Daryl Higgins has researched child abuse impacts and prevention, family violence, and family functioning for nearly 30 years. He has focused on public health approaches to protecting children and child-safe organisational strategies. Gabrielle’s PhD studies, which aim to understand the prevalence and prevention of child sexual abuse in faith-based settings, as well as understanding harmful sexual behaviour and peer sexual harassment across the population, are being supervised by Daryl.  

This project aims to understand how to apply these measures effectively and identify their strengths and limitations (psychometric properties). Our initial review has revealed inconsistencies in how surveys, questionnaires, or tools to measure ‘safety’ are applied in organisations. Many studies focus on ‘proxy’ measures, such as children’s knowledge, but these measures may not accurately predict better safety. We also plan to explore other tools that have yet to be used in causal or experimental research that may be useful to youth-serving organisations.

We hope that by exploring the studies on the EGM, we can provide new insights into the usefulness of the measures used. We would be delighted to hear from other researchers who have done work to mine data in an EGM or other similar reviews to share their insights and experience. We plan to publish a resource outlining our findings and identify measures that leaders, practitioners, researchers, and funders can use in their work.

If you have conducted a similar study, or would like to hear more, please get in touch.

The Foundation Practice Rating rates 100 UK charitable grant-making foundations each year on their practices on diversity, accountability and transparency. The set of foundations which we research and rate changes from each year. A couple of people have asked recently why we do that and whether that compromises the FPR’s rigour. This article is to explain.

Our sample

To be clear, the set of 100 ‘included foundations’, as we call them, each year is as follows[1]:  

1.       The five largest charitable grant-making foundations by giving budget. 
2.       All the foundations which fund the Foundation Practice Rating. (There are currently 13 of them. One is not a charity: the Joseph Rowntree Reform Trust.)
3.       A random sample of: community foundations across the UK (as listed by UK Community Foundations, the membership body of community foundations), and the ~300 largest foundations in the UK (as listed in the ACF’s annual Foundations Giving Trends report).

The sample is organised so that it is stratified, i.e., a fifth is from the top quintile in terms of giving budget, a fifth from the second quintile etc. So, for example, if no foundation funding the FPR is in the 2nd quintile, then all 20 included foundations in that quintile would be chosen randomly; whereas if three foundations funding the FPR are in the 2nd quintile, then 17 foundations in that quintile are chosen randomly. Obviously, at least five ‘slots’ in the top quintile are filled non-randomly (by the five largest foundations), and some other ‘slots’ are filled by foundations funding the FPR, so in the top quintile, not all the ‘slots’ are filled randomly. The foundations funding the FPR vary considerably in size: they are not all at the top.

We re-make the sample each year. The FPR is not a panel study: we do not track the same participants over time. This is intentional.

Notice that our sample is 100 out of about 340 foundations*. Thus we include ~29% of the total set. (*Those are: the ~300 on the list in the ACF report, + about 35 community foundations, + a couple of foundations which fund FPR which are in neither of those.) 

Why do we change the sample each year?

Well, on the first part of our sample, the five largest foundations change around: in the three years that we have been doing this, eight foundations have appeared in the largest five at some point. Looking at the chart below, it would seem rather bizarre to continue to rate, say, BBC Children in Need – now the 11th largest foundation – just because it was in the largest five when the FPR happened to start. We always include the (then) five largest foundations because their practices dominate grant-seekers’ experiences, so it is important to reflect which foundations those large ones are at the time.

On the second part of our sample, the set of foundations funding FPR changes: in the first year, there were only 10 and now there are 13.

On the third part of our sample, the rationale is this. First, we are trying to get a representative picture of progress across the whole foundation sector. And second, part of the ‘intervention’ of FPR is foundations knowing that they might be included at any time. If some foundations knew that they would definitely be included, they would have an incentive to improve their practices in order to improve their grades, but other foundations would not feel that incentive so might not improve, or at least, not make so much effort to improve. Thus the random selection enables FPR to have more influence than if it were a panel study: and our primary goal is to influence practice.

These two reasons interact. If FPR were a panel study, quite probably the foundations included would improve more than those who are not, and we would gain zero information about the set which are not included. They might well diverge over time. We therefore would not get a sense of the sector as a whole.

Given that the sample changes, how can FPR make year-on-year comparisons?

The technique of studying a randomly-selected subset of relevant entities is used in many surveys of public opinion, including consumer confidence and voting intention. Typically, those survey 1000 randomly-chosen adults from across the country. The sample may be adjusted to make it  representative e.g., in terms of age, gender, the four nations of the UK. That is like FPR ensuring that our sample is representative in terms of foundations’ size. So, when you see news stories that voting intention has changed, those are almost certainly based on sequential studies of a small set of people, and that set is freshly-drawn each time.

Professor Stephen Fisher of Oxford University studies public opinion and was on the British Polling Council panel that investigated the 2015 UK General Election polls. He says:

“The methods that FPR uses are very sensible. Dividing foundations into five groups according to how large those foundations are, and then randomly selecting foundations within each group should ensure a broad and representative sample overall. Opinion polls aren’t perfect, but they typically get the share of the vote within a margin of error of +/- 4 percentage points. They come from sampling around 1 in every 30,000 voters. FPR is sampling 1 about in every 3.5 foundations: a much larger proportion of the total, and with much more coverage of the bigger foundations. On that basis, fluctuations in the FPR due to random differences in sampling should be very small indeed.”

Making year-on-year comparisons 

On the basis described above, it is rigorous to compare the full set of 100 foundations year-on-year. We made that comparison in the Year Two report – i.e., the first year when we had a previous year. In that report, we also included comparison of:

• The set of foundations which were included in both years 
• The set of foundations which were randomly included in Year One with the set of foundations which were randomly included in Year Two.

In each case, we assessed the changes in overall numerical scores and numerical scores on each of the three domains (diversity, accountability and transparency), and we looked at whether those changes were statistically significant. 

We will repeat and extend those analyses in subsequent years.  

The FPR Year One (2021/22) and Year Two (2022/23) reports are here.

[1] Foundations can opt-in to the FPR: they can pay to be assessed. They are treated as follows. If a foundation wants to opt-in and happens to be selected randomly for inclusion, then it is treated as a normal randomly-included foundation: it does not pay and its results are included in the analysis of the main 100. By contrast, if a foundation wants to opt-in and is not selected randomly for inclusion, then it pays and is not included in the analysis of the main 100. This is to avoid selection bias in the sample.

Very few UK charitable foundations disclose the diversity of either their staff or their boards, according to research by Giving Evidence for the Foundation Practice Rating (FPR). Of the 100 foundations included in the FPR in 2022-23, only six disclose the diversity of their staff, and only six disclose the diversity of their trustees. In total, nine of the 100 foundations disclose data on either staff or trustees.

Why does Foundation Practice Rating research this?

The FPR researches 100 foundations each year. The process for selecting those 100 foundations is discussed here. The research covers three pillars: diversity, accountability and transparency. In diversity, FPR (currently) looks at whether foundations disclose the diversity of their staff and, separately, their trustees, on various dimensions. FPR does not currently look at what that diversity is.

Nonetheless, during the FPR research, the FPR team notes which of those 100 foundations make those disclosures. It is only a small step for us to then log and collate their actual data on what the diversity is, and so – despite this being outside FPR’s scope – we have done this.

To our knowledge, these are the first data about diversity of UK foundations to be published, so we share them here in the hope that it is a useful contribution to the discussion. We intend to track these data over time, in order to show whether and how the diversity of UK foundations’ teams is changing.

The data about diversity of UK foundations’ personnel were gathered in autumn 2022 using the most recent materials published by the foundations at that time.

The foundations which disclose diversity of their personnel

The foundations that disclose these diversity data are:

*Members of the funders group

We noted where a foundation disclosed diversity of its staff and/or trustees on the following four dimensions. Note that: (a) we used the foundation’s own wording and categorisations; and (b) to make the graphs legible, the graphs show only one data-point on each category:

• Ethnicity (the graph shows people whom the foundation reported identify as something ‘other than white’)
• Gender (‘other than male/man’)
• Sexual orientation (‘other than straight sexual orientation’)
• Disability (self-declared as having a disability)

Not all foundations reported their data in all four of these categories.

We excluded (from the numerator of the figures in the graphs) responses such as “prefer not to say” and show only data provided by individuals who disclosed their characteristics.

We are aware of the limitations of this approach – simply collating foundations’ disclosures in the way that they are reported – but hope that these data are useful nonetheless.

Some Key Findings

Race: Racial diversity exhibited significant variation among the foundations that shared data in this category. The percentage of individuals identifying as a race other than white varied greatly: from zero to above 60%. For instance, on trustees, Blagrave Trust reported that over 60% of its trustees identify as non-white, whereas the Joseph Rowntree Reform Trust reported 0% in this category. On staff, there was also variation, but a noticeable concentration around the range of 30% to 40%.

Gender Diversity: The information on gender diversity highlights the proportion of individuals who do not identify as male within the foundations that provided data in this regard. This category includes individuals who identify as women, non-binary, or another gender. Among the foundations that reported this data, there was a significant representation in the “other than male” category.

Sexual Orientation: Sexual orientation was the characteristic on which fewest foundations reported: only three foundations disclose information about this for trustees, and only one foundation disclosed this data for staff. It is worth noting that there is an ongoing debate within the sector regarding whether or not sexual orientation should be disclosed. This debate primarily concerns protecting individuals from potential harm that may arise from such disclosure versus liberation and challenging heteronormative practices.

Disability: Disability disclosure was also quite surprisingly low, considering 17% of the UK population is disabled (Census, 2023).

Staff diversity: one foundation, Esmee Fairbairn Foundation, splits its staff diversity data by type of staff. This allows for a more granular understanding of diversity within the foundations. The graphs show the split reported by Esmee Fairbairn Foundation.

Conclusion: The data presented provides a snapshot of:

• The amount of diversity reporting by UK grant-making foundations, which is disappointingly little, and
• The reported diversity of staff and trustees in those foundations who do disclose it.

We hope that these findings contribute to the ongoing conversation around diversity and inclusion within the grant-making sector, and help to promote more disclosure and encouraging further exploration of diversity-related initiatives.

More about the Foundation Practice Rating is here.

How much churn is there amongst the largest UK grant-making foundations (by giving budget)? One might expect basically none, because huge foundations don’t get created very often, and foundations don’t compete for resources. Giving Evidence looks at these data each year for our work on the Foundation Practice Rating, and we find that there is a surprisingly high amount of churn. These are the data for the last few years.

Why is that churn there? We haven’t investigated so can’t say. Maybe it’s related to investment income – because success there might enable larger giving budgets. Maybe it’s related to other income, e.g., BBC Children in Need and Comic Relief, which were both in the top 5 in 2019, raise their grant budgets from the public, which one might expect to rise and fall.