In March 2013 Nesta hosted the launch of the What Works Network. Led by the ESRC, Cabinet Office and the Big Lottery Fund, these new centres have the ambitious task of improving the links between the supply, the demand, and the use of evidence. These will complement existing bodies - NICE in health, and the Education Endowment Foundation in schools, orchestrating evidence across the areas of active and independent ageing, early intervention, crime reduction and local economic growth.
This short paper recommends five ways that the What Works centres can ensure that they maximise their impact, and avoid the mistakes of past evidence initiatives, which were often overly focused on supply rather than demand and use, and on academic research to the exclusion of other types of evidence.
In summary, the What Works centres should:
- Orchestrate all relevant kinds of evidence - from data to formal experiments, qualitative research to quantitative, practitioner insight to experience: the key is to focus on what's most likely to improve the effectiveness of the users of evidence. That will include communicating what doesn't work as well as what does - and not shying away from being blunt.
- Involve the likely users of evidence in the shaping of work programmes, prioritisation and governance - the more that they share ownership of these organisations, the more likely it is that they will value them.
- Mobilise evidence for multiple types of use - from policymaking at national or local level to management and front-line activity.
- Try to influence the creation of new evidence - to ensure it addresses live compelling problems, and where possible to bring down the costs of evidence creation. Evidence also needs to support innovation and experiment - not crush it.
- Be ready to adapt - the What Works Network should reflect on its own impact, learn from the evidence about evidence, and respond to what is and isn't effective.
1 CONSIDER ALL TYPES OF RELEVANT EVIDENCE
The What Works centres should orchestrate all kinds of evidence - from data to formal experiments, qualitative research to quantitative, practitioner insight to experience. That will include communicating what doesn't work as well as what does - and not shying away from being blunt.
The What Works Network centres cover a range of policy areas, with great variance in the depth of knowledge and evidence available in the field. Each What Works centre will have to deal with interventions and programmes at various stages of development, and all the centres will need evidence for different purposes and audiences. This means the What Works Network will need to consider a range of different evaluation and research methods in order to understand the effectiveness of interventions.
At Nesta we have adapted and developed Standards of Evidence to provide a shared language for understanding the evidence underpinning particular policies or practices. At the higher levels the systematic methods of randomised trials and controls can bring rigour to showing what works. But in many fields there is very little evidence of this kind.
Figure 1: Nesta Standards of Evidence (Source: Puttick and Ludlow 2012)
- Level 1
You can describe what you do and why it matters, logically, coherently and convincingly.
- Level 2
You capture data that shows positive change, but you cannot confirm you caused this.
- Level 3
You can demonstrate causality using a control or comparison group.
- Level 4
You have one + independent replication evaluations that confirms these conclusions.
- Level 5
You have manuals, systems and procedures to ensure consistent replication and positive impact.
As a result, there is no choice but to draw on a wide range of sources. Academic sources can ensure certain quality criteria have been met. But publication lags can delay new approaches being identified, and those approaches which aren't on the radar of academia can be missed out entirely.
It follows that the What Works Network therefore need to consider a broad range of sources, such as from independent research organisations, think tank literature, government reviews, open data, administrative data, and evaluations commissioned from provider organisations, as well as feedback and insights from service users themselves.
These can be mapped against the standards of evidence - but experience suggests that in all fields the strongest insights come from the use of multiple types of evidence. Indeed, a single RCT may be more misleading than multiple findings from other, less rigorous sources that all point in the same direction. The key point is to prioritise usefulness, and to orchestrate access to the most useful evidence which won't always be the evidence with the strongest formal credentials.
The new centres should also be frank and honest about what is not working, just as much as what is. This involves the What Works centres influencing decisions to decommission and stop funding less effective interventions, just as much as they encourage the adoption of what is working. Yet there also needs to be recognition that there are differing degrees of 'failure'. The What Works centres will need to make clear when improvements to an intervention are needed and are sufficient, or when the issues are too great and the intervention should be terminated altogether.
This frankness requires the What Works network to retain independence from government. Only then will they be seen to offer credible advice, and be enabled to constructively criticise decision making when politicians, officials, and others outside of government, go against what is known without clear rationale or justification.
2 INVOLVE THE USERS OF EVIDENCE
Involve the likely users of evidence in the shaping of work programmes, prioritisation and governance: the more that they share ownership of these organisations, the more likely it is that they will value them.
The What Works centres should be demand-led, ensuring evidence is useful and useable. To achieve this there will need to be engagement with a range of different users, from policymakers, commissioners, practitioners, academics, service users, media, and a range of others alongside.
Where do they need evidence? What are their priorities? Where is the biggest gap between what they know and what they need to know? How can evidence be more useful and relevant?
This engagement shouldn't be seen as a one off exercise, instead there should be systematic feedback at all stages a point we will return to.
Encouraging study circles and their equivalents is one way of embedding a culture that takes evidence seriously. Study circles are regular meetings of practitioners that reflect on new evidence and frontline experience. They exist on a modest scale in some schools and hospitals - but could become much more widespread in these fields, and amongst charities or policymakers. They can then potentially provide feedback to researchers on what's most useful.
In addition, the What Works Network should build upon and complement the work already underway. The UK Government is opening up much of its data to be interrogated and used. We have a world-class academic research base to utilise, as well as excellent independent research institutions and foundations, such as the Wellcome Trust and Dartington Social Research Unit.
Alliance for Useful Evidence
The Alliance for Useful Evidence is a network of over 1,000 members, encompassing users and producers of evidence from across academia, third sector, companies, providers, governments, and others, from the UK and around the world. A component of the Alliance for Useful Evidence is to provide a forum for discussion, debate and knowledge sharing, for those working in the same fields, and to enable ideas to be shared across different sectors. There are many events held across the UK, and a number of discussion papers regularly published on a range of issues, from scientific advice in government to the use of evidence by charities, and numerous other hot topics. For further details and to join the Alliance for Useful Evidence see:
www.alliance4usefulevidence.org/
Knowledge sharing and debate is therefore crucial. The Alliance for Useful Evidence, led by Nesta in partnership with the Big Lottery Fund and ESRC, is a network with over 1,000 members which could help facilitate these conversations.
3 MOBILISE EVIDENCE FOR MULTIPLE USES – FROM POLICY TO FRONT-LINE ACTIVITIES
Mobilise evidence for multiple types of use from policymaking at national or local level to management and front-line activity.
The What Works centres will be looking for what is working in policy, programmes and practice, or a mix of these. Although the distinctions are rarely mutually exclusive, the table below begins to outline some of the differences between these, and the audiences involved.
|
What do we mean by this |
Audiences to involve and influence |
| Policy |
Testing, experimenting and learning from what is effective policy, in terms of guidelines, legislation, principles that are implemented to impact and change conditions conducive to human welfare. Social policy has been defined as "public policy and practice in the areas of healthcare, human services, criminal justice, inequality, education and labour.” |
Policymakers at a central and local level. |
| Programmes |
The approaches and models being developed to address social challenges, either within the public sector or outside by providers, for instance, Family Nurse Partnerships. Tasks could involve developing, testing and evaluating different programme models. |
Commissioners and other funders, such as philanthropic foundations. |
|
|
Providers across third sector, private sector and public sector. |
|
|
Service users |
|
|
Front-line practitioners |
| Practice |
Best practice skills and culture. Tasks could involve training or creating communities of practice. |
Front-line practitioners |
|
|
Providers |
|
|
Service users |
| Interventions |
Types of products, such as technologies, developed to address specific challenges or to enhance ways of working. |
Front-line practitioners |
|
|
Commissioners |
|
|
Service users |
Identifying effective interventions is not the end result. The recent Nesta study on adoption of NICE guidelines in primary care points to a need for much greater attention to adoption and speed of adoption across public services. This will help both to encourage faster take-up of proven models, as well as better understanding of why and how effective models spread.
The What Works Network also needs to ensure that the identified interventions are implemented correctly to help maintain impact when used elsewhere. This requires clarity and advice on how much the intervention can be changed and adapted to suit the new context, and what elements are core, to help increase the chances of success when rolled out.
To support implementation and adoption, the engagement and involvement with commissioners, practitioners, and others, is again absolutely vital. This active communication could take the form of events, training sessions, newsletters, and other forums that enable ideas and issues to be discussed and shared. Nesta is developing tools through its Open Workshop which will help practitioners and policymakers to interpret, use and make the most of evidence.
4 INFLUENCE EVIDENCE CREATION TO HELP ADDRESS LIVE PROBLEMS
The What Works centres should try to influence the creation of new evidence - to ensure it addresses live compelling problems, and where possible to bring down the costs of evidence creation. Evidence also needs to support innovation and experiment - not crush it.
If the What Works Network is going to find the best interventions then it is vitally important that research production is not ignored. With the opening up of public services there is a plethora of new approaches and interventions that may be more efficient, making experimentation and testing ever more important. Yet many providers - particularly smaller, charitable organisations - need help and support to evaluate their work. This should be addressed. We do not want to be in a position where those provider organisations who can afford to pay for an evaluation make it onto the list of What Works at the expense of potentially better alternatives.
This means the What Works centres should be leading the way to make it easier to evaluate. Our understanding of what are effective programmes and practices has evolved greatly over the past decades, yet the research methods used to evaluate them has remained fairly static. There is therefore a great need to innovate and advance new methodologies.
It also means that the What Works centres support the funding of primary research, or work with others, such as research councils, to signal and fill the gaps in evidence when they are identified.
In response to the need for innovation in research methods, Nesta is supporting the development of Randomise Me, an innovative, free to use online trials generator that will strip the complexities out of randomised controlled trials (RCTs), enabling anyone to set up and run their own trial (see the text box). The What Works centres should build upon this to explore and develop other methods that enable providers to evaluate their work in flexible but rigorous ways.
Randomise Me
Nesta is working with Ben Goldacre to create Randomise Me, a free to use online platform that will enable any individual or organisation to set up and run a trial to test interventions and answer questions relevant to them. The aim is for Randomise Me to help strip out the complexities commonly associated with the method and enable RCTs to be more commonly used, helping to robustly test different issues and interventions in an agile and low-cost way. The Randomise Me platform will launch at Nesta in summer 2013.
As well as innovating with how evidence is generated, there is also a need to ensure that it is accessible, useable and easy to interpret. This will require the What Works centres advancing evidence visualisation, communication and dissemination.
A related point is that evidence needs to support, not work against, systems of innovation. All parts of public policy and practice need experimentation and creativity to discover better alternatives. By its nature, innovation has to go beyond what's already known - and experimentation needs space to adapt and sometimes to fail. Applying evidence criteria too soon can stifle innovation.
But all experiments should involve measurement and clarity about what counts as success. In short we need both more open and creative experimentation to generate new ideas - and more rigorous evidence to determine what deserves to be scaled-up.
5 BE READY TO ADAPT
The What Works Network should reflect on its own impact, learn from the evidence about evidence, and respond to what is and isn't effective.
The What Works network should learn from similar evidence institutions around the world that also broker links between research and practice. At Nesta we sought to find evaluations of these 'arbiters of evidence', hoping to find the institutions and mechanisms effective at improving the use of research and evidence in decision making. Yet although these institutions may appear to be effective at identifying what is working in different fields, we only found a couple of evaluations that show how effective and impactful the institutions are at actually ensuring this information changed decision making.
The What Works Centres should therefore build in evaluation of their own work from the outset. They should explicitly test different mechanisms for generating and disseminating evidence, and monitor the impacts this is having, with the knowledge then shared across the What Works Network.
Evaluating Project Oracle: a London-wide evidence generation campaign
To increase our understanding of how evidence can be generated at low cost, and then effectively incorporated into decision making, Nesta is evaluating Project Oracle. Project Oracle is the only city-wide evidence generation campaign of its kind in the world. We are interested in the impact that Project Oracle is having through testing the different mechanisms that are being trialled. Details of the work streams are available on the Project Oracle website. Nesta's evaluation of Project Oracle will be published in autumn 2013.
For the same reasons the network will need to continually test, evaluate and experiment with different approaches, as well as refreshing the lists of 'what works'. Finding 'what works' implies that once identified the work is done. It isn't. Our problems change, we change, and prevailing wisdom will be overturned. What works in one period may cease working in another, equally what works in one location might not work when tried elsewhere. Instead we should be looking at 'what is working now,' recognising that the collation of convincing evidence is just the beginning.
Nesta has been involved in the What Works Network since its inception and we are delighted that they are now becoming a reality. We will be watching their progress closely and hope that they have the impacts anticipated.
If you have any thoughts or comments please do let us know.
Contact [email protected]
Nesta
1 Plough Place
London EC4A 1DE
[email protected]
www.twitter.com/nesta_uk
www.facebook.com/nesta.uk
www.nesta.org.uk
Nesta. May 2013
Nesta Operating Company is a registered charity in England and Wales with company number 7706036 and charity number 1144091. Registered as a charity in Scotland number SCO42833. Registered office: 1 Plough Place, London, EC4A 1DE.
