Proof is a big concept, and in social science – which is what impact research is – it almost never happens

Suppose you hear of a new intervention that’s never been tried or tested
before. What are the chances of it producing decent results? Clearly, you’ve no idea: the uncertainty about its results is sky high. Now suppose there’s an intervention with teenage mothers that a well-conducted, rigorous evaluation in the US suggests reduces child abuse and neglect, reduces child injuries by 20-50 per cent and improves children’s educational outcomes. Will it produce those results in, say, Edinburgh? Continue reading →

In 1929, Werner Forssmann, a junior doctor in Eberswalde, Germany, found in an obscure 19th century journal a diagram of a man passing a tube through a horse’s jugular vein into its heart to measure changes in ventricular pressure. He wondered if it were possible to investigate human hearts in a similar way. His seniors said he was crazy for asking and told him to go back to work. Undeterred, he sneaked into the x-ray room, made a slit in his own arm, threaded a small tube into his own heart and took some x-rays showing the position of the tube.

He was fired.

Then, in 1956, he won a Nobel Prize for, in effect, co-founding the field of cardiology. Continue reading →

The closure of Kids Company on August 5th raised the question of charity’s management and effectiveness. Caroline spoke to BBC News about it:

And, once it emerged that there are 60,000 children’s charities in the UK, she spoke about whether there should be consolidation:

Caroline was also on BBC Radio4’s PM programme on the same day, which FeedbackLabs turned into a podcast, here. Continue reading →

Don’t laugh. The notion that grants should be given at random rattled around when the National Lottery was set up over 20 years ago: the joke was that since prize-winners are chosen at random, maybe grant-winners should be too.

Perhaps we should resurrect the idea. The medics have studied it: Australian health economists looked at every grant application to the National Health and Medical Research Council of Australia in 2009 – all 2,085 of them – and analysed the scores given by the expert panels that assessed applications.

Now, if there’s one thing we know about experts it is that they’re not very good. For example, the US National Academy for Sciences showed that judges’ decisions about imprisonment varied dramatically and predictably, depending on whether the decision was made before or after lunch. The Nobel laureate Daniel Kahneman reported how, given the same picture on different days, radiologists contradict themselves 20 per cent of the time, as do stock market analysts, pathologists and many others. Extraneous factors often hold sway. Continue reading →