The initiative had been running for years, with a gradually strengthening evidence base, and the CEO felt now was the right time to conduct a randomised control trial (RCT) to capture really robust evidence on exactly what impact the initiative was having.

Thinking that the charity’s most loyal funder was a good starting point, the CEO was surprised and disappointed when they declined to contribute any money. The funder passed it up not because they thought it wouldn’t produce good evidence, but because they thought it would – and that this evidence might in fact show that the initiative wasn’t having an impact, meaning money down the drain for the last decade.

Better that they continue to waste their money indefinitely and unknowingly, than face the more public and unequivocal shame of curtailing ten years of proven waste.

The Centre for Social Action Innovation Fund

Thankfully my experience at Nesta has been very different. I am working with the Centre for Social Action Innovation Fund (CSAIF), providing evidence and evaluation support as part of a partnership between Nesta and The Social Innovation Partnership (TSIP). And whilst funding initiatives that aren’t having an impact is an inevitable risk for any funder, this hasn’t stopped the Fund from spending over £1m on evaluation across its portfolio of 39 grantees.

Appropriate evaluation

The Fund hasn’t just spent its evaluation money on 39 RCTs; each grantee’s evaluation has been co-designed with them, ensuring that it is tailored to their stage of development. On occasion this has meant an RCT or similar, but often it has meant more exploratory evaluations to help the grantees improve the design or delivery of their initiatives, and/or putting systems and processes in place that will allow the grantees to monitor their impact over time.

These evaluations won’t and shouldn’t tell us ‘what works’, but rather ‘this initiative, in the way that it is currently designed and delivered, works approximately this much, and can work better in the future if these things are done’. Less sexy as a phrase, but much more accurate and useful as a statement.

Purpose

But perhaps the word that best captures the Fund’s approach to evaluation, as well as that of other programmes that some of my Nesta and TSIP colleagues are working on together (such as Cities of Service and Helping in Hospitals), is ‘purpose’. The truth is that most funders don’t go as far as refusing to pay for evaluations due to a fear of what they might reveal. But it’s also true that when funders do pay for evaluations, a huge proportion of them end up being seen as tick-box exercises – sometimes by all parties involved!

This often kick-starts a vicious circle by which the evaluations in question are less useful, less enjoyable and ultimately of a lower quality.

Thankfully Nesta has avoided this to date by giving evaluation at the CSAIF a clearly-articulated purpose from the outset – to help grantees get the evidence they need to scale sustainably. This message has been consistently reinforced both internally and externally, and has been vital in creating a positive and engaging environment where TSIP, Nesta and the Fund’s grantees have been able to work together - not just to end up with a series of robust evaluation reports, but to move towards genuine system change in the way that evidence is generated and used in the sector.

In the words of Code Club, one of the Fund’s grantees:

"I think it's fair to say that at the beginning, the Code Club team and I were evaluation novices - at first it felt a bit like learning a new language! The excellent support we have received from the CSAIF has taken us from zero to creating a Theory of Change and establishing a Level 3 impact evaluation in less than a year. We are looking forward to gaining clear insights into the effectiveness of our programme so that we can make it as impactful as possible for the children taking part. We could not have achieved this without funding and mentorship from the CSAIF."

Funders have such a crucial role to play in this movement, and TSIP’s experience has shown that Nesta are one of those leading the way.

Photo Credit: Lisby (lisby1) via Flickr