ABOUT THE AUTHORS

Francis Ruiz is Senior Adviser at NICE International. Previously he worked in the Guidelines team at NICE as a technical adviser since, and prior to that within the Institute's Technology Appraisal programme as an analyst. He has a specialist interest in health economics and has worked on a number of technology appraisals and guidelines in a variety of therapeutic areas. Prior to joining NICE, Francis had over six years experience in clinical data management and health economics within the pharmaceutical sector.

Jonathan Breckon is Head of the Alliance for Useful Evidence. The Alliance for Useful Evidence champions the use of evidence in social policy and practice. It is an open-access network of more than 2,000 individuals from across government, universities, charities, business and local authorities. He joined Nesta from the Arts and Humanities Research Council where he was Director of Policy and Public Affairs. He has also worked in policy and research at the Royal Geographical Society, the British Academy, and Universities UK. He is a member of The Society of Evidence Based Policing executive committee and Board member of Campaign for Social Science.

THE NICE WAY: LESSONS FOR SOCIAL POLICY AND PRACTICE FROM THE NATIONAL INSTITUTE FOR HEALTH AND CARE EXCELLENCE

In his first media interview as Cabinet Secretary, Jeremy Heywood mooted the idea of setting up a 'NICE-type' body in social policy.¹ Although details were lacking, the proposal received widespread interest. Nine months later, Coalition Government ministers Oliver Letwin and Danny Alexander launched four new What Works centres in social (and local economic) policy at Nesta.² NICE has been an inspiration for these centres and is formally part of the What Works network. New institutions are planned in areas such as wellbeing policy. Yet, arguably there is insufficient understanding surrounding the workings of the 'NICE model'³ and its potential application in domains beyond health and social care. Even the champions of NICE recognise in the model, features that might limit its applicability in social policy fields.⁴

What can be learned from NICE that might work for the What Works centres or other institutions? There is a lot of misunderstanding about the NICE approach outside of health expert circles. This paper aims to bust some of those myths. We hope that clearly articulating the NICE model will inspire others to follow the approach.

Our key recommendation is for an approach that engages with wider social values and interests - getting service users, providers and others involved is vital to success. Any new NICE-type institution aiming to be an evidence intermediary must avoid only working in a "navel-gazing” technocratic, academic research-focused silo. There is a need to engage with wider audiences, and the difficult and messy politics that goes with making tough decisions relating to crime, education and other areas of social policy.

A SHORT HISTORY OF NICE

NICE was founded in 1999, issuing its first guidance in 2000. When it was set up, one of the biggest clinical healthcare issues was the so-called 'postcode lottery', i.e. different availabilities of treatments in different areas, and widespread variations in the quality of care. There was public and government concern as well as media criticism about this situation, and this coincided with government seeking to invest more money into the healthcare system as a whole. NICE was created and designed to tackle those issues.

The initial principles were that it should be national, it should produce guidance both on effectiveness and cost-effectiveness. It should be very inclusive, particularly with patient and public involvement. And critically, it should be independent.⁵

(Professor David Haslam, Chairman, NICE)

In 2005, NICE took on a major public health role, including issues such as obesity, smoking and exercise. From April 2013 NICE also took on the responsibility for quality standards in social care. To reflect its broader remit, NICE now stands for National Institute for Health and Care Excellence. However, whilst the title has shifted, the key methodological principles have stayed broadly similar. Overall, NICE objectives are to reduce inappropriate geographical variation, set quality standards for the practice of medicine, encourage innovation in health technologies, and promote efficiency in resource allocation. Using evidence was core to this variety of work.

NICE now has activities from the molecular through to the multinational, from kindness in care homes to the molecules in cancer drugs. It's an extraordinarily broad organisation."

(Professor David Haslam, Chairman, NICE)

RATIONAL, NOT RATIONING - WHY NICE IS RELEVANT OUTSIDE HEALTH

Critical to some of NICE's success has been its focus on facilitating prioritisation, making choices based on the benefits and opportunity costs. This has at times been very different from the public perception of NICE.

Folk who only know (NICE) from the news stories think it's just the organisation who says no to the drugs you would like your family and friends to have. As an aside, [NICE] actually says yes to around 80 per cent of the drugs looked at but you'd never guess that because yes-es never make news. People often think of the job as one of rationing and I don't see it as rationing at all. It's trying to be rational, which is quite different, and the key to that is prioritisation."

(Professor David Haslam, Chairman, NICE)

Resources are not limitless, decisions need to be made about where resources are to be allocated. The pressures concern limited resources coupled with powerful drivers for increasing spending, such as new drugs and other innovations, and our increasingly ageing population. NICE offers a more rational and transparent way to make those choices - rather than on the whim of politicians, or what is pushed by big pharma companies or other interested groups, or on the basis of where you live.

In fact NICE operates according to a set of self-declared 'core principles' (see below). Some of these will be explored in more detail later in this document.

• Independence: while there is a formal referral of topics from government, the Institute operates at 'arm's length'. A declaration of interests policy applies to staff and to contractors.
• Methodological robustness: a commitment to using and developing methods that are internationally recognised and also pragmatic.
• Using the best available evidence on clinical and cost-effectiveness: it's not just about randomised controlled trials.
• Using expert opinion: recommendations are made by independent committees with a wide range of expertise.
• Public involvement: all decision-making committees include lay members (members from the general public).
• Transparency, stakeholder consultation and contestability mechanisms: recommendations are always consulted on to allow a wide range of stakeholders (such as industry, patient organisations and academia) to comment. Stakeholders can also help prioritise topic areas for a NICE evaluation. A formal appeal process against final recommendations also exists.
• Review: guidance is regularly reviewed at appropriate intervals.
• Application of social values and equity considerations: recommendations aim to take into account social values, informed by NICE's Citizens Council.

Such principles (among others), and their application, have been described as necessary for the successful operation of similar institutions in health more generally,⁶ and arguably are relevant in non-health contexts too.

That is not to say that NICE has had a smooth ride since its creation 15 years ago, and the application of "rationality" and transparency in decision making is not without its problems. There is a difficult balance to strike between the need to deliver outputs (guidance) quickly, but in a methodologically robust and open way. Indeed it has been argued that explicit decision making can serve only to 'increase tensions, conflict and instability'.⁷ However, despite the controversies, which have sometimes required the Institute to make changes in its methods and processes, NICE retains broad political support. Its approach to multi-stakeholder engagement and open, evidence-based priority setting is often regarded internationally as the gold standard to be emulated where possible as an alternative to implicit and often 'conflict increasing' priority setting approaches.

Certainly NICE can be seen as providing 'body armour' for politicians, who when challenged about, for example, the availability of a particular drug or treatment, can refer to NICE as having made the decision based on all the available evidence. However, politicians can, at times, step into more contentious areas, and while this can prove difficult for NICE, arguably they have every right to do that.

An example of this at the moment is the Cancer Drugs Fund... from a political perspective cancer is very noisy and, therefore, Government has felt that it's important to put money into cancer drugs and that is absolutely their prerogative."

(Professor David Haslam, Chairman, NICE)

AVOIDING 'SLAVISH ADHERENCE' TO HIERARCHIES OF RESEARCH

NICE uses a variety of evidence sources to make decisions. There is no one single piece of research that will sway decisions. All sources of research - and the views of stakeholders - are put into the mix.⁸ While experimental research methods are vitally important, particularly when looking at the effectiveness of a new drug, it is a myth to think that they are the only approach that informs NICE.

NICE has developed a more sceptical view of the untargeted use of formal hierarchies of evidence, which usually places experimental studies such as randomised control trials, and systematic reviews of such studies at the top.⁹ The appropriateness of the evidence to the question is more important, not its place in any such hierarchy, according to former NICE Chairman Michael Rawlins:¹⁰

The fundamental flaw with the development and use of hierarchies of evidence is that they fail to recognise that it is not the method that matters, but whether the particular method is appropriate to answer the particular question. In many instances questions about the effectiveness of a pharmaceutical product can only be answered by randomised controlled trials. But this is not invariably the case; and practitioners of HTA harm themselves, their discipline not to mention the patients they seek to serve by slavish adherence to hierarchies of evidence."

(Michael Rawlins, 2014)

The available evidence base is often both incomplete and complex, and in order to make best use of it, NICE and its committees place considerable emphasis on understanding and defining the problem to be addressed. Considerations of cost-effectiveness, and the use of the QALY, or quality-adjusted life-year as measure of health gain from an intervention, have a major role to play in priority setting. They allow an explicit examination of the consequences of adopting one intervention, for one disease or condition, compared with another intervention for a completely different disease or condition.

NICE is essentially a decision-making body. We bring people together in a multi-stakeholder way. We develop processes where we start with the decision or problem that we have to address, and then evidence is identified to help address the problem. So we'll look at randomised control trials, we'll look at observational data, we'll look at the whole spectrum of available evidence and that includes expert opinion, of varying quality... All have their failings, all have their problems, but all are thrown into the mix when it comes to decision making because there is no single source of evidence, there is no single randomised control trial that will give you the answer. So all these data is thrown into a deliberative process involving people like us round the table looking at the evidence and forming a judgement, which is essentially a political process. But thankfully, I would argue, an evidence-informed political process."

(Francis Ruiz, Senior Adviser, NICE International)

NICE-type decision-making processes therefore question how much better new drugs or other interventions are compared to those already on the market, and what such improvement is worth. This can ultimately help in getting a better deal from industry. The United States does not have a body like NICE; the FDA (Food and Drug Administration) legal standard for approval is merely to determine if a drug, device or test is "safe and effective." Without these, in United States, the pricing of drugs and devices has become "unmoored, and some experts argue, free of health logic altogether”.¹¹

SCIENCE ALONE IS NOT ENOUGH - REFLECTING SOCIAL VALUES

In addition to its technical processes, NICE also has a policy on what might be termed the application of social value judgements,¹² covering ethical issues relating to efficiency, equity and autonomy. This is crucial, as while evidence supporting the use of interventions can be from robust scientific research, these alone are not sufficient in decision making and prioritisation.¹³ For example, those interpreting a body of scientific evidence with a view to addressing a 'live' policy or practitioner question, must apply both technical and value judgements, and the latter should broadly reflect the views of society.

Scientific judgements relate to the interpretation of the science. Social value judgements are concerned with the ethical principles, preferences, culture, and aspirations of society, and these must ultimately be informed by the general public."

(Michael Rawlins, 2014)

This interplay between data and research, and social values is one of the core aspects of NICE. The balance between optimising the benefit to the individual and the benefit to society is one of NICE's ongoing challenges, and has been a subject of debate by its Citizens Council.

NICE is about supporting better resource allocation when it comes to NHS spend. But to do that effectively with evidence you need to involve the end user, you need to have independent committees... (NICE does) that through having doctors, nurses, patients actually on committees. They're the ones who ultimately make the decisions and we tailor all our outputs to different audiences as well to support user uptake."

(Francis Ruiz, Senior Adviser, NICE International)

(NICE is) also very keen on ensuring that or trying to facilitate service users, patients, being able to uphold their right to accessing what we've said is good for them. So it is an interplay... There was an initial government top-down policy or imperative but the way that NICE approached it is that it's got to be an inclusive, interactive, multi-level engagement. And I suspect that's also going to apply to What Works centres."

(Francis Ruiz, Senior Adviser, NICE International)

THE IMPLEMENTATION GAP: THE CHALLENGE OF GETTING PRACTITIONERS TO USE EVIDENCE-BASED GUIDANCE

Initially, NICE concentrated on providing good quality evidence and good quality recommendations, rather than on implementation. It was assumed that the evidence and recommendations would be accepted and taken up, but this was not always the case.¹⁴ A big challenge for any evidence institution synthesizing research evidence for practitioners is getting listened to. And getting their advice used in a consulting room, surgery, crime scene, classroom or other real-world scenario. Forcing practitioners to use evidence-based recommendations is likely to be resisted.¹⁵ So how best to get evidence-informed approaches into practice?

Informal networks and personal contacts continue to hold immense sway. Doctors rely on one another to share knowledge and advice so that high standards are maintained across the profession. A recent survey of the 174 GPs found the majority indicated that they rely on other doctors as a source of information and method of promoting innovation adoption.¹⁶

Getting evidence into practice is now a major part of NICE's work, but much of this is done in partnership with other bodies, such as Health Education England which is responsible for the education of health professionals, or in encouraging the Royal Colleges to implement NICE findings and guidelines. There is also the NICE Implementation Collaborative,¹⁷ which brings together academics, and the life science and pharmaceutical industries to explore why certain pieces of advice have not been taken up, and how this might be rectified in future.

Relatively recently we did guidance around a type of anticoagulant for people who'd had strokes or were at risk of stroke, when they don't have to have blood tests every few weeks. It's of great benefit for the individual, but uptake is incredibly low compared to what we expected, so the NICE implementation collaborative has been trying to explore why is that, what's the block, what's going on? I'm afraid that for all the wonders of the health service it takes... around about 16 years for a really good idea to move from being proven to being taken up by everybody."

(Professor David Haslam, Chairman, NICE)

Involving stakeholders helps with implementation

Having stakeholders setting some of the NICE agenda and being involved in the guidance development process throughout, has been important in getting its guidance implemented, even in situations when not all stakeholders agree with the findings. Likewise, it has been very important to get patients and the public involved in driving the NICE agenda.

You're far more likely to get it [implemented] if you solve a problem for someone. So identifying what their problem is and then finding a way towards solving it. So doctors struggle with, what on earth do we do now with high cholesterol or high blood pressure. So to have someone who will do all that work for them, who assesses the current state... solves the problem for them... Whereas if we come up with a solution for something you didn't realise was a problem, who's going to be interested in that?”

(Professor David Haslam, Chairman, NICE)

At the roundtable on the topic in May 2014, there was some discussion about the extent to which stakeholders are consulted or whether they actually drive the NICE agenda and process. One attendee noted that stakeholders had certainly driven the systematic review that they had been involved in, and believed that this was one of the things that had aided NICE's success.

It was really important because they were saying that the driving force, the customer, were these stakeholders, and they framed the question and what was done... The evidence is that push on its own rarely works, you do need this pull. So for me the really wonderful aspect of NICE is this key ingredient, that it really is much more of a pull process, less of a push process."

(David Gough, Professor of Evidence Informed Policy and Practice, Institute of Education)

Encouraging demand: the UK What Works centres

As with NICE, the What Works centres are also working out ways to be closer to frontline practice. Producing guidance and information is not enough; practitioners need tools and programmes that enable them to engage with and use the information effectively.

It needs to be translated for practice before it can be used in schools or on the beat. And different What Works organisations do that to varying extents, but it's as much about changing practice, changing the culture as it is about actually having the evidence on what works. The big caveat for me in establishing the new set of What Works centres has always been the lack of focus on the demand side. These are primarily supply side organisations and I think even with NICE, whilst you can say what should be available on the NHS it's still up to doctors to prescribe, and that evidence-based culture is still not embedded throughout every part of the NHS."

(Carole Willis, Chief Executive, National Foundation for Educational Research and former Chief Scientific Adviser)

There are different ways to tackle this issue, and approaches might include practitioners themselves identifying the questions to be addressed by research organisations/research commissioners, which would provide a ready and receptive audience for the answers. There are potential roles for regulatory and inspection bodies in monitoring how different organisations - hospitals, schools, local authorities, local police forces - are or are not engaging with the evidence, while avoiding a tick-box approach. There is an overlap between the role of organisations like NICE and the What Works centres, and the role of professional judgement, and so characterising the system in terms of demand or supply may in fact be too rigid.

If we have more practitioners engaged in different stages, either doing or contributing towards different research activities, then we might have people more willing and more informed and able to make judgements and actually use the outputs from those pieces of research."

(Carole Willis, Chief Executive, National Foundation for Educational Research)

USING THE LAW TO ENCOURAGE TAKE UP OF EVIDENCE

So if you can't get commissioners and clinicians to follow your evidence-based guidelines, what about forcing them to do so through legislation and national policy? Policy levers were used in 2002 when it was brought to the government's attention that commissioners were tardy in taking up positive NICE guidance that supported new treatments. The reaction from the Department of Health was to issue a direction to the NHS that (unless directed not to in special circumstances) commissioners had to make the funding required to implement positive NICE appraisal decisions within three months of publication.

This dramatically changed the nature of NICE appraisal guidance, for while it remained advisory to clinicians, it became national policy.¹⁸ While patients, doctors and the industry welcomed this development, local health commissioners were anxious that local health priorities would have to be forgone to fund new (mainly cancer) drugs of marginal cost effectiveness.

Ministers themselves are also obliged by law to give due 'regard' for NICE. The quality improvement role of NICE is now part of primary legislation, as set out in the Health and Social Care Bill 2012, enacted April 2013:

The Secretary of State must...secure continuous improvement in the quality of services provided to individuals...In discharging the duty...the Secretary of State must have regard to the quality standards prepared by NICE under section 234 of the Health and Social Care Act 2012."

But many NICE guidelines and recommendations have no specific legal backing, and the NHS has, to some extent, only a moral obligation to follow them. However, the expectation of NICE is that their recommendations will generally be followed unless there is a good reason for them not to be, an example of which might be patient autonomy. In fact, this expectation has been tested in the courts: recently a Clinical Commissioning Group was described as acting ‘unlawfully' in denying NICE-recommended fertility treatments. In this instance the court ruled that the CCG was under obligation in public law to 'have regard for NICE guidance' and it must 'provide clear reasons for any general policy that does not follow NICE guidance'.¹⁹

One attendee pointed out that with the 'moral authority' that NICE has developed, through its transparency, openness, clarity of mission and how it operates overall, legal measures are perhaps not necessary for guidance to be widely adopted, and may in fact cause a negative reaction. This could be a lesson for other policy areas. It was noted that credibility is a key driver for uptake and momentum.

I do think in the new joined-up world of social media we're going to get a change in the power balance between recipients and givers of service and much greater sharing of what good looks like... which will be tense and difficult for all of us, I suspect, but it's a sign of a maturing society.”

(Professor David Haslam, Chairman, NICE)

Nonetheless, in the UK, it is unlikely that NICE would have survived beyond its first few years without government backing, given initial concerns and occasionally outright hostility by some key stakeholders, including the pharmaceutical industry. Similar problems are emerging in other parts of the world that are trying to set up evidence-based processes and organisations with a NICE-like remit (see the NICE International sections below).

We conclude that NICE does a vital job in difficult circumstances...Healthcare budgets in England, as in other countries, are limited. Patients cannot expect to receive every possible treatment. Demand outstrips resources and priorities have to be determined... NICE requires the backing of the Government. NICE must not be left to fight a lone battle to support cost- and clinical effectiveness in the NHS."

(House of Commons: London Jan 2008)

But it is not clear if forcing compliance on clinicians through law, policy and legislation works. A fear in social policy is that formal pushes to follow guidance and rules, such as through an Ofsted-type body, could create a backlash.²⁰ A resistance to what might be perceived as top down rule-following is also happening in the medical field: a new campaign has been launched in the British Medical Journal to resist what GPs see as the overly bureaucratic approach to evidence-based medicine. They want to see a return to 'expert judgement rather than mechanical rule following' and stronger focus on what they see as the 'the human aspects of care'.²¹

RELEVANCE OF NICE OVERSEAS

NICE International was established in 2008 by the NICE board, to be a professional arm to handle overseas interest in the NICE model. It is a non-profit programme housed within NICE which, based on NICE's experience in the UK, offers foreign governments and other organisations including donors, a range of support:

• Advice on using evidence to inform policy.
• Technical support on assessment, guideline development, public health guidance and implementation support.
• Process advice to increase transparency, enhance public and stakeholder involvement and improve consultation.

What NICE has done in England and what it aims to do internationally is to make decision making more rational when dealing with complex sets of problems. However, countries have different healthcare systems and some are very fragmented. NICE International works to support countries to do things better within their own healthcare context, and to reflect on how they interact with a broad set of stakeholders and manage conflicts of interest.²² For example, in some countries in Latin America, choices of what the health system should publicly subsidise has often been fought in the courts, with judges making decisions about the availability of drugs and other interventions. In this instance, and in contrast to the UK, there is no process or mechanism for the various professional organisations to work with stakeholders, i.e. end users, industry, and potentially conflicted healthcare professionals. NICE is trying to work with these countries to support them in developing processes where varied interested constituencies can engage in decision making - have their say, but not necessarily their way.

NICE International's work to date has helped support the creation of NICE-like institutions and processes in other countries, most notably the Institute for Health Technology Evaluation in Colombia. This is a new institution and its future is far from certain, but the fact that it has been established represents a positive step forward.

While organisations similar to NICE already exist in some other countries, including Germany and France, none of these as yet have an international arm, giving NICE a uniquely influential role overseas.

The vice minister of health in China, for instance, is absolutely clear that NICE is world leader in this stuff and I've been to half a dozen countries in the last six months, all of them do not see there's anybody else in the world who's doing things the way NICE is.”

(Professor David Haslam, Chairman, NICE)

WHAT CAN BE LEARNED FROM THE NICE INTERNATIONAL EXPERIENCE?

NICE is not an easily replicable model that can be exported to other countries and contexts, however:

(Many) elements of its methodology, process, evidence base and products... can be adapted to other countries' local realities to improve health care organisation, public health and clinical practices, and develop new financial mechanisms that link budget allocation with improvements in service delivery and better health outcomes."

(Marquez et al., 2010)²³

Hence NICE International's engagement is concerned with sharing learning and providing technical expertise to other countries interested in setting up similar organisations, rather than in exporting and franchising the NICE model elsewhere. A key driver for engagement with NICE is the growing ambition of many countries to achieve affordable, universal healthcare coverage (UHC) for its citizens, which suggests the need to make some tough prioritisation decisions.

Countries were interested because they were facing very similar prioritisation problems and we've seen this most notably in some of the emerging economies that are having to deal with choices over drugs that (NICE) would have difficulty in saying yes to. So they were very interested in how NICE goes about doing things and how we are apparently a success, so they were keen to find out more."

(Francis Ruiz, Senior Adviser, NICE International)

The issues that other countries are interested in when they start to engage with NICE International include technical questions such as:

• How do you go about looking at the evidence?
• How do you go about reviewing the evidence, which trials do you include?
• Do you include observational data?
• How do you interpret the data; how do you critically appraise it?
• How do you go about doing economic modeling?
• Do you (i.e. NICE) routinely undertake modeling?
• Where does the data come from?
• How do we judge value for money?
• What is our willingness to pay threshold?

While these questions are important, and there are undoubtedly data gaps to address, there are also important process and institution issues that many countries have not considered, and a large part of NICE International's engagement is concerned with highlighting this. Hence while the robust technical approach of NICE is important, before it can be effectively implemented there must be some sort of legislative framework and institutional processes to house it.

If you're going to start to think about getting evidence into practice in any serious or real way you need to think about setting up some sort of institution or some sort of government-backed process to make it happen. Without that it's not going to go very far, you're not going to get change.”

(Francis Ruiz, Senior Adviser, NICE International)

This suggests that there is a lesson not only for health, but other sectors as well, including social care, criminal justice, and education. There must be institutional processes, political backing, and perhaps also a legislative framework of some kind in order for NICE-like institutions to survive and to enable evidence-informed practice.

BROADER USE OF THE NICE MODEL IN PRIORITISATION AND DECISION MAKING, E.G. THE BILL & MELINDA GATES FOUNDATION

Much of NICE International's work is funded through donors, such as the Rockefeller Foundation, which seeks to support countries develop effective evidence-based priority setting mechanisms on the journey to UHC. In addition, donors themselves, including the Bill & Melinda Gates Foundation, are becoming increasingly interested in ensuring value for money when making funding decisions. NICE International is now drawing on lessons from how NICE makes prioritisation decisions in England to support its donors to prioritise and make more cost-effective investment decisions.

The way that this is talked about is, what's a life year worth? They call it a disability-adjusted life year (DALY). When you're running a poor country healthcare system, you can't treat a year of life as being worth more than, say $200, $300 or else you'll bankrupt your healthcare system immediately... Someone in the society has to deal with the reality that there are finite resources and we're making trade-offs, and be explicit about that.”

(Bill Gates)²⁴

The Gates funded Methods of Economic Evaluation Project (MEEP) was a six-month project to improve the quality and transparency of economic evaluation and to guide researchers in undertaking and reporting well-conducted and robust analyses. It was a collaboration led by NICE International and included partners from institutions around the world including the Health Intervention and Technology Appraisal Program (Thailand), the University of York, the London School of Hygiene and Tropical Medicine, and the University of Glasgow. A key output of MEEP was the Gates' Reference Case, a set of principles, methodological specifications and reporting standards designed to support health economic evaluations funded by the Bill & Melinda Gates Foundation, but of general applicability since it covers fundamental concepts relevant to all healthcare policymakers tasked with resource allocation decisions.²⁵

A more localised overseas example is the Relentless Monetization programme run by the Robin Hood Foundation, a New York City philanthropic organisation concerned with tackling poverty. In 2012 it awarded $132 million using a cost-benefit analysis system adapted from the NICE prioritisation and decision-making process.²⁶

IS THE EVIDENCE BASE OUTSIDE HEALTH TOO SMALL?

There is a fear that the depth and breadth of evidence is far too underdeveloped in non-health areas for a NICE-type body to be feasible. Research in areas such as policing or education is limited in scale and relevance for NICE-type organisations outside health.

But it is worth remembering that health didn't always have such a strong evidence base. NICE has helped to catalyse more relevant health research. Although NICE does not have a research commissioning function, it does have an important role in setting out the key evidence gaps and research questions, and encouraging other organisations to invest in growing that evidence base

Also, we should not think that NICE only survives on evidence judged of the highest quality. In fact, based on internal NICE research at the time when individual recommendations were still graded according to a hierarchy of evidence (an approach abandoned in 2006) it was found that only about 20 per cent of recommendations in NICE clinical guidelines came from 'high quality' studies, and around a quarter relied on consensus amongst experts.

Education

Other What Works centres in social policy - and their sponsoring bodies - are addressing the gaps in the evidence. The Department for Education has published a series of evidence gaps papers, setting out key evidence gaps and key research questions for each of the 14 education policy areas, such as academies, assessment or adoption.²⁷ The Department for Education has provided £125 million to the Education Endowment Foundation (EEF), a What Works Centre, to build the evidence base and to trial different approaches to stimulate demand in the classroom. The funds awarded to EEF are small compared to those available in health, but it is a good start. In the past the EEF has both championed and funded the use of experimental research, which has helped to raise the level of debate about the type of evidence and the amount of evidence required in order to support classroom practice.

One area less advanced in social policy than health, is the evidence on cost and value for money. Health economics is now a strong subdiscipline. But we have less analysis of costs and benefits, and comparable approaches to QALYs in social policy.

...one of the big gaps I think is the evidence on cost. It's not a trivial issue to try and collect data on costs. It's neglected... (exploring the relative cost effectiveness of different approaches) is a controversial area and it's an area where both methodologically I think we need to develop in terms of the research but also practitioners and the public – it's back to that word trust – need to trust the organisations who are not only setting out what the evidence says but also they're setting out whether it's value for money.”

(Carole Willis, Chief Executive, National Foundation for Educational Research)

Policing

In policing, as in education, the evidence base is small compared to the health sector, and it contains more on policing strategy, less on specific tactics. To build the evidence base to any significant degree, the sector recognises that those who are going to use it must be involved in building it, actively co-producing the evidence. This also has potential benefits for implementation and informed practice.

Our theory is that if practitioners can be involved in building their own evidence base and owning it, they are more likely to use it and less likely to rely solely on practitioner experience and thinking that experience has all the answers."

(Julia Morris, Research Programme Manager, College of Policing)

The What Works centres are active on the 'supply side', of producing and synthesising evidence, but are also giving careful consideration to building the 'demand side', and how practitioners can be encouraged to be research practitioners. The College of Policing is in some ways an unusual What Works centre as it has the training curriculum for policing, and holds the guidance standards for policing. As such, it has a number of levers that can be used to embed evidence. It also houses police exams, selection and assessment process, which provides an additional lever - to insist that a requirement of promotion is an understanding of the evidence base and also to provide some evidence that professionals have helped to build it. Focusing on better understanding of these 'pull' factors, and ensuring that professionals are evidence producers is important, as the university sector does not have the capacity to build up the evidence base alone.

RECOMMENDATIONS

What has allowed NICE to become the go-to organisation and not just nationally but internationally as well? I think it's fair to say that [of] organisations which try to provide evidence in the social policy context, there aren't any that have got the wide presence and the trust of the sector that NICE has managed to develop."

(Danni Mason, Head of What Works team, UK Cabinet Office)

What do we recommend others learn from the NICE model? Rather than picking and mixing certain elements of NICE, it would be better to learn from the overarching approach, which has combined a strong focus on scientific rigour (based on a close working relationship with the National Institute for Health Research, the Medical Research Council and the university sector) with careful attention to the process of decision making, which includes openness, transparency and inclusivity within a defined social and ethical framework. Other recommendations are listed below.

NICE-type bodies should listen to users and understand social values

NICE has placed great emphasis on engaging public services and customers or service users. This experience is valuable when considering how other kinds of centres might work in practice. In education, for example, stakeholder engagement would involve developing information or guidance with the help of teachers, parents, governors and children. There are a number of potential benefits that could result from engaging parents, in particular:

• Parents could be more actively engaged with their children's education.
• It could raise parents' expectations of schools and create more grassroots support for some of the approaches that schools might want to adopt.
• If parents knew more about Inset days and the importance of continuing professional development, schools would have a greater mandate to engage with evidence and research and strengthen practice.
• There may also be less resistance from parents and the public when evidence-based approaches which go against the grain are adopted, for example, increasing class sizes.

There are considerable challenges in getting a profession to adopt evidence and to secure buy-in from service users. NICE has gone to considerable lengths to involve the public in the process of developing evidence. Not all What Works centres have the resources to emulate this, but it is something to be considered.

Greater understanding of demand and diffusion

It is very important to get better at understanding the demand side, i.e. what influences professionals to take up evidence, and the best routes to that audience, especially because it will be different by sector. We may be able to learn from the literature on this diffusion in innovation; for example, Nesta has recently done some work looking at which doctors take up new ideas, and the factors that influence them to do this, as well as the barriers that exist.

NICE may be able to provide lessons in how to manage the role of probability in evidence, that is, how to communicate messages clearly to the public, in the absence of any evidence which can conclude that an intervention is always going to work and in all contexts. There may also be lessons to be learned from NICE in how best to combine clarity of guidance with a handling of risk and probability.

A project to look at how to increase schools' use of research has recently been funded by the Education Endowment Foundation. It is a multi-arm trial looking at different ways of communicating research and engaging with schools. It will include some passive approaches, some more active approaches, linkage approaches, and it will also explore the role of brokers in linking research. The 'pull' aspect will also be considered, looking at building capacity in schools, and at giving time and space and training for schools to be able to find and use evidence.

Take local implementation and context into account

A major difference between the What Works centres and NICE is that, notwithstanding the challenges of working with multiple stakeholders including patients, the NHS and the pharmaceutical industry, many other social policy fields are more disparate and dispersed. For example, there are over 160,000 organisations in the charity sector, and this presents considerable challenges in identifying interventions and assessing evidence.

It's great to hear that the Education Endowment Foundation has looked at 33 (interventions) so far, but there are thousands of organisations delivering education interventions. How do you ensure you've found the best interventions? And then, how do you assess their cost-effectiveness? One organisation may be delivering an intervention fantastically well and cost-effectively and another doing it quite differently. It's harder to assess this, given they're not within the same system in the way that a doctor in one part of the NHS and a doctor in another part of the NHS are."

(Charlotte Ravenscroft, Head of Policy & Research, National Council for Voluntary Organisations)

Just as doctors listen to other doctors, implementing good evidence-based practice relies on close engagement with the practitioner market. But in many areas of social policy, it is not just practitioners but also public service commissioners and procurement staff who need to understand the evidence, and these are an enormous cadre of people spread out around the country with very different provider markets at a local level."

(Charlotte Ravenscroft, Head of Policy & Research, National Council for Voluntary Organisations)

Hence local implementation also needs to be considered, as does procurement and commissioning processes within local authority and public bodies, as this is a key decision point for many charities.

Typically, you can't get head teachers to take anything up in schools unless another head teacher recommends it to them... Head teachers listen to head teachers. In terms of implementing good evidence-based practice, in an awful lot of areas of social policy, it relies on first engaging with the practitioner market but also public service commissioners and procurement staff who are an enormous cadre of people spread out around the country with very, very different local service provision and provider markets at a local level."

(Charlotte Ravenscroft, Head of Policy & Research, National Council for Voluntary Organisations)

In conclusion, discussions at the roundtable in 2014 revealed that while important differences certainly exist between NICE and the What Work centres, and between policy areas (different contexts, variations between geographical areas, different policy regimes) these differences may not be as large as might be imagined, and there is much room to learn from NICE's experiences.

There is a great deal of variation in how the What Works centres themselves are developing. Some are very well funded but some have small budgets. Some have been given to academic centres, while others will be stand-alone organisations. This in itself may provide insight into what works for What Works centres, but we must avoid diluting focus and unnecessary duplication.

It may be a good thing because we don't know what works and we'll understand a bit more. But... in a way, I'd rather have had two or three and really try hard to make a difference in areas that mattered."

(Dan Corry, Chief Executive, New Philanthropy Capital)

We need to continue to explore the success factors for NICE and the extent to which these might be successfully transferred into other policy areas.

ACKNOWLEDGEMENTS

We would like to thank the following for their comments, quotes and help with the preparation for this report: Hasan Bakhshi; Sir Iain Chalmers; Dan Corry; Carole Willis; Gill Edelman; Phil Sooben; Professor David Gough; Professor David Haslam; Ryan Li; Danielle Mason; Julia Morris; Charlotte Ravenscroft; Emma Stone. The views and any errors in this report are, however, the authors own. We would also like to show our appreciation for Jane Aston who drafted the first version of this report.

NICE International's contribution to this work received funding support from the Rockefeller Foundation.

ENDNOTES

Alliance for Useful Evidence 1 Plough Place London EC4A 1DE

www.alliance4usefulevidence.org

October 2014

(Creative Commons Attribution-NonCommercial-ShareAlike license)