Standards of Evidence
www.nesta.org.uk/feature/innovation-methods/standards-evidence/
www.nesta.org.uk/feature/innovation-methods/standards-evidence/
The Standards of Evidence model helps to answer an important innovation question: "Is an innovation doing any good? Is it even doing harm? Is the status quo just as good as the innovation?" It provides a robust framework for choosing the right approach to understand whether an innovation is working.
Standards provide a structure – often some sort of scale – for thinking about whether you are making a positive difference. Typically, the lower levels show that there is only limited evidence; higher levels show that more evidence is available. They can, however, vary widely around this model.
The standards were first taken up in health. In medicine, we saw centuries of innovations that turned out to do harm, such as leeches, bloodletting, or smoking cigarettes to cure asthma. The rise of medical research – and the use of hierarchies of evidence – helped to bust these myths. Standards of Evidence take a similar evidence-based approach to social innovation. The standards have spread from medicine to other areas including education, crime, and housing.
The standards take you on an evidence journey. The first step of that journey asks you to describe what you plan to do, in a way that is coherent, clear and convincing. This is often referred to as a ‘Theory of Change’ and is useful for early-stage innovations. A Theory of Change helps you to be explicit about your goals – and how you’ll achieve those goals.
Another benefit of doing a Theory of Change is that it’s a first step in designing an effective evaluation: it asks you to accurately identify all of your desired outcomes, such as reductions in loneliness or increases in health and wellbeing – before going on to try and measure them.
The next stages of that evidence journey ask that you try and capture some data to see if you are making a significant difference. This may involve using control groups, comparing the people benefitting from your innovations with groups that do not, such as through randomised controlled trials.
The more advanced levels of the journey involve repeating the success of your innovation. If your smart idea worked well in Copenhagen or Cardiff, will it work in Paris or Prague? In the jargon, this is called replication, and is a vital part of your growth: showing that this is not a flash in the pan, and that your promising innovation can be repeated elsewhere.
In 2012, we developed our Standards of Evidence to evaluate the evidence created by our investments. Nesta was the first innovation body in the world to use a robust impact evidence framework to assist in funding decisions.
It lays out five levels of evidence, ranging from a Theory of Change (Level 1) to larger bodies of evidence that have ‘ manuals, systems and procedures’ to ensure replication (Level 5). We have helped organisations to move up these levels of evidence over time. It helps to increase our confidence that what we are doing is having a credible and measurable impact.
The Standards help us to seek evidence that is proportionate to the right stage of innovation. So, for instance, early-stage innovation needs Levels 1 and 2. More advanced innovations, requiring more Nesta funding, should seek Levels 3, 4 and 5.
Reaching towards higher levels of evidence, should only be used at the appropriate stage in the development of an innovation. It should only be used when a practice or programme has started to be well-researched and thought through, is being delivered consistently and effectively, and is backed up by early indications of positive impact.
The Standards were first used on our impact investment funds. Since then, the Standards have been used in other parts of the organisation, such as the Centre for Social Action Innovation Fund, a partnership between Nesta and the Cabinet Office. All grantees receive support to create evaluation plans and find good evaluators, as well as finances to improve their evidence too.
We have helped organisations like Code Club undertake randomised control trials, Action Tutoring and Access Project have compared their results against a matched control group, and the British Lung Foundation are comparing against a comparison group from other regions.
If the data is credible, then each of these organisations will be validated at Level 3 on our Standards of Evidence. Standards of Evidence also continue to be championed by Nesta’s Alliance for Useful Evidence. Founded in 2011, this network champions the smarter use of evidence in social policy and practice.
In 2016, we developed the Using Research Evidence practice guide and the Evidence Masterclass programme to help people learn about the Standards, and about evidence-informed decision-making more widely. We have also been an advocate and host for some of the UK What Works Centres, many of which have their own evidence standards.
The Access Project matches business volunteers with motivated students from disadvantaged backgrounds, helping them progress to top universities.
The project was part of the Nesta and Cabinet Office Centre for Social Action. The students involved in the programme were compared to other similar students who didn’t receive the tutoring. They met Level 3 of the Standards of Evidence by using this type of evaluation, called ‘propensity score matching’.
The results were highly positive: six out of the nine evaluated schools showed statistically significant positive raised GCSE grades for the tutored pupils. Tutored pupils were also shown to be more aspirational in their applications to universities, such as Russell Group universities like Oxford, Cambridge and Imperial College.
Data collected before and after the students joined the Access Project found a rise in those attending top universities (Level 2 of the Standards of Evidence).
There has been a proliferation of evidence standards and frameworks in the UK and overseas.
A recent Nesta report found 18 alone in the UK. Some organisations have adapted the Nesta version to their own needs. The housing innovation charity HACT, for instance, created their own version in 2016, funded by Public Health England and a group of leading housing associations.
The Swedish innovation body Vinnova has translated the Standards for what they call their ‘impact logic method’. They have also had value outside the not-forprofit sector; the global education business Pearson worked with Nesta to create a Standards of Evidence framework.
