About Nesta

Nesta is an innovation foundation. For us, innovation means turning bold ideas into reality and changing lives for the better. We use our expertise, skills and funding in areas where there are big challenges facing society. Nesta is based in the UK and supported by a financial endowment. We work with partners around the globe to bring bold ideas to life to change the world for good.

nesta.org.uk

About Centre for Public Impact UK

Centre for Public Impact UK (CPI UK) is a not-for-profit, founded by Boston Consulting Group, that helps government and public sector organisations to prepare for the complex challenges they face. We help their systems and cultures adapt so that human relationships matter, so that power is shared and decisions are made closer to those impacted, with those impacted.

www.centreforpublicimpact.org

Published September 2020 nesta.org.uk/new-operating-models-handbook

Design: Joana Pereira Editor: Emily Reynolds

Nesta is a registered charity in England and Wales with company number 7706036 and charity number 1144091. Registered as a charity in Scotland number SCO42833. Registered office: 58 Victoria Embankment, London, EC4Y 0DS.

Authors

Jenni Lloyd (Nesta) on behalf of the Meaningful Measurement workgroup.

Workgroup facilitator & lead contributor

John Burgoyne (Centre for Public Impact).

Contributors

Adrian Brown (Centre for Public Impact), Anna Randle (Collaborate), Carrie Deacon (Nesta) and Nadine Smith (Centre for Public Impact).

Thanks to Vicki Sellick and Sarah Mcloughlin for their help reviewing and editing this paper, and to Katherine Zscharnagk for all her support.

What are new operating models?

New operating models describe a new way of working for local authorities that acknowledges the complexity and interconnectedness of social issues and the people and organisations that aim to tackle them. The work of the local authorities participating in the Upstream Collaborative, and the experiences of the communities they serve, has informed the development of a framework that characterises what new operating models look like in practice. The framework incorporates the often 'hidden' qualities which underpin this work, such as mindset, values and behaviours, alongside new practical capabilities and enabling infrastructure.

Find out more in 'Introducing New Operating Models for Local Government'¹, part one of this Handbook.

If you'd like this publication in an alternative format such as Braille or large print, please contact us at: [email protected]

About the New Operating Models Handbook

'Meaningful Measurement' is part of the New Operating Models Handbook, a set of learning products which explore the new operating models emerging in local government – how they work, what they look like and the key features needed to promote success elsewhere. It draws on the experience of the twenty pioneering Local Authorities participating in the Upstream Collaborative, which was led by Nesta in partnership with Collaborate from 2019 to 2020. The Handbook is made up of six parts:

Introducing New Operating Models for Local Government How to create the conditions for new operating models to thrive

From the Margins to the Mainstream: How to create the conditions for new operating models to thrive

Reframing Risk: How to adopt new mindsets around risk that enable innovation

Asset-Based Community Development for Local Authorities: How to rebuild relationships with communities through asset-based approaches

Meaningful Measurement: How a new mindset around measurement can support a culture of continual learning – notes from the field

A Catalyst for Change: What COVID-19 has taught us about the future of local government

"We could end up with a far, far better system if this ever gets to be normal."

Mark Smith, Gateshead Council

00. Introduction

Over the last few years, local authorities across the country have been pioneering new ways of addressing the increasingly complex challenges communities face. They've developed ambitious, high potential programmes of work to try to tackle the underlying causes of social problems, heading 'upstream' to create the conditions for citizens and communities to truly flourish.

Their initiatives are highly varied, but share common features – partnership working, connection to a wider transformation programme, high degrees of localisation, a focus on more personalised outcomes for individuals, and smart use of data.

To deliver these initiatives, councils have had to adopt new ways of working, as detailed in 'Introducing New Operating Models for Local Government'¹. A key part of the transition to new operating models is realigning measurement and evaluation practices to better fit the complexity and interconnectedness of social challenges and enable learning.

This paper, based on the insight and practical experiences of members of Nesta's Upstream Collaborative, outlines how local authorities are experimenting with measurement practices that help them learn about complex problems, the people experiencing them, and how their work can make a difference. These practices help them adapt and improve, co-creating the conditions from which better outcomes are more likely to emerge.

The Upstream Collaborative supported local government innovators to share, assess and accelerate new operating models. During the programme participants formed workgroups, collaborating around a challenge or opportunity relevant to their work and sharing ideas and stories to amplify and improve practice. The Centre for Public Impact facilitated the Meaningful Measurement Workgroup, who co-produced this paper. The workgroup included eleven local authorities and their partners:

Derbyshire City Council, Gateshead Metropolitan Borough Council, Greater Manchester Combined Authority, Huntingdonshire District Council, Kirklees Council, Leeds City Council, London Borough of Barking & Dagenham, Middlesbrough Council, Oxford City Council, Oxford Hub, Redcar & Cleveland Borough Council, City of York Council.

What we mean by meaningful measurement

During our discussions with the workgroup, the terms 'measurement', 'evaluation', 'evidence' and 'performance management' were used extensively and often interchangeably. Within this report we have taken a deliberately inclusive view of evidence, from research and evaluation studies to expert knowledge and stakeholder consultations. For us, good – and useful – evidence is robust and appropriate for the issue at hand.

Importantly, evidence is more than just data. Local authorities often collect a great deal of monitoring data – the number of people who use a service, where they live, what they thought of the service they received, and so on. But data like these give no information about what actually works, what has an impact, or insights into cost effectiveness. The key is turning such monitoring data into evidence.

Quantitative data is something that has been counted or measured – it expresses quantity, amount or range. Qualitative data is gathered from opinions, views, notes or observations. Often it's words, expressions, drawings or symbols. Neither form of data is better than the other – they have different uses – and in their raw form, neither is particularly useful.

For example, say a local authority is collecting data on how many people use their services, how often, who they are, and what they think of it. Data alone won't tell us much. The starting point for interpreting data is to ask questions of it, and to code it and analyse it accordingly. This produces information, and once we have information we have something useful; information can tell us something or shed light on a question. It's at this point, when information is giving insight and answering a question, that we have evidence.

Good evidence can provide insight into what Nesta has described as the 'six Ws': what works, for whom, when, where, and with whom². This paper will explore the move to more meaningful measurement being made by some local authorities as part of their adoption of new operating models, and how this can provide the evidence that helps them make progress as they head upstream.

"The focus of performance and impact measurement is too often heavily weighted towards traditional, often regulatory-led and service-specific frameworks and measures. We are interested in an alternative approach that can be focused around what is important to a person and supports us to experiment and learn continuously."

Rhodri Rowlands, London Borough of Barking and Dagenham

01. Working upstream calls for a new approach to measurement

Measurement can be used to fulfil many different purposes, depending on the context in which it's being used. Measurement can help organisations judge the relative success or failure of their activities; provide accountability to a range of different stakeholders; increase transparency around performance, clarifying and signalling which outcomes are important; and help them learn how to adapt, improve, and understand their progress.³

The Treasury's Magenta Book⁴, the handbook containing central government guidance on evaluation, states: "There are two main purposes for carrying out an evaluation: learning and accountability.” Accountability is a key instrument of democracy, ensuring that public money is well spent, that public services are effective and that any regulation strikes the right balance between burden and protection.

Unfortunately, in the experience of the members of the Meaningful Measurement workgroup, there is an over-emphasis on accountability data collection, in particular the focus on ensuring actions are generating best 'value for money'. This frequently overshadows how measurement and evaluation can be useful and support learning.

Measures are used mostly to keep score. They are generally for the benefit of others (inspectors, commissioners, distant leaders) and allow a justification of current working and/or the attribution of any obseruable problems to other causes (put another way, 'we did our bit, the problem is that other service/agency etc').

Mark Smith, Gateshead Council

1.1 Complicated vs complex

In his work on Developmental Evaluation⁵, Dr Michael Quinn Patton categorises different types of evaluation approaches as either ‘traditional' or 'developmental'.

For a complicated problem, where interventions are expected to have a predictable, linear, cause-effect relationship, 'traditional' approaches, which rely on high degrees of central control, are appropriate. For complex problems, where it is difficult to know in advance what impact an intervention may have, a 'developmental' approach based on learning and adaptation is more appropriate. Printing passports, for example, where you can set targets to evaluate success, is complicated⁶; the education system, trying to understand how an individual child learns and develops, is complex⁷.

While both evaluation approaches can be useful depending on the context, the experiences of the workgroup suggest that traditional approaches are far more common and have become deeply embedded within the values, mindset, processes and practices of the public sector.

1.2 Using traditional approaches in complex domains is problematic

In a range of complicated scenarios, where causal events can be predicted, the traditional approach to measurement can and has helped to drive public sector increases in performance and reductions in cost.

Chasing performance targets prevents people from doing the right thing. When they're driven by performance targets people can lose sight of better understanding those they are working with and what they need.

Becky Lomas, Derbyshire County Council

However, a lack of understanding about how to measure impact in complex systems has led to an overreliance on traditional measurement approaches in local government. Particularly where targets are used and incentivised, this can result in situations that, often unintentionally, prevent local authorities and their partners from being able to learn, adapt and innovate in response to the challenges and opportunities their communities face. Additionally, the members of the Upstream Collaborative who formed this workgroup highlighted that a target-driven culture can skew evaluation approaches to prioritise the measurement of things that are simpler to measure, rather than what matters. The desire to reach targets can also lead to programmes that avoid engaging with groups of citizens deemed 'harder to reach', distorting who benefits from services.

1.3 Mindset shift is at the core of new operating models

This presents a problem for those practitioners trying to address complex challenges by heading upstream and understanding the root causes of demand. Complex challenges, by their very nature, require experimentation and iteration to enable actors across a system to act responsively. In these circumstances the approach to evidence needs to be rebalanced in ways that allow for, and encourage, a continuous process of innovation and learning.

Innovative local authorities in the Upstream Collaborative are adopting new ways of working that move attention and resources upstream of service delivery to focus on creating the economic, social and community conditions that enable citizens to thrive. To do so, they are adopting new operating models – central to which is a mindset that fundamentally changes the way these councils put their infrastructure and capabilities to use.

This mindset is based on core beliefs about the way the world works – a worldview that then influences how the work of local government is done and how progress is measured.

Figure 1: Exploring core beliefs, Centre for Public Impact

1.4 Supporting a measurement culture that prioritises learning and improvement

To embed a measurement for learning approach requires local authorities to consciously explore a cultural shift based on a collaborative, systemic, long-term mindset that will enable a more fundamental change than simply introducing new tools alone.

The workgroup paid particular attention to the underlying beliefs, values, and principles⁸ of their measurement practice, using them to develop a set of shared values and principles¹⁴ which, from their experience, enable measurement for learning.

Measurement for learning relies on creating the conditions under which people can report results honestly, sharing what hasn't worked and needs to be improved in addition to what is working well. Trust, authenticity, and curiosity can create a new virtuous cycle of learning.

Curiosity means seeking out the views of those who are different than you

We need a level of open-mindedness to be wrong about our own assumptions. Too often people struggle to let go of assumptions.

Sara Fernandez, Oxford Hub

A key outcome of this culture is that members of staff feel comfortable speaking truth to power, sharing their authentic experiences, and iteratively adapting and improving based on what is learned. When staff feel that what they share will be listened to and acted on, they are more likely to be open and constructive about their experiences, viewing failure as an opportunity to learn and improve⁹.

Figure 2: Values and principles of measurement for learning, Centre for Public Impact

"It is important to articulate how you are a learning organisation and what this looks like at the frontline and strategic leadership level."

Hannah Elliott, Kirklees Council

02. Measurement for learning: putting it into practice

Changing organisational culture to value measurement for learning can feel like a huge task. However, taking a developmental approach based on innovation practice can be a good first step.

Trying out new tools and building new practices into the work routine of a single team or department first can create a climate of reflective practice and open up new possibilities.

New measurement practices can start with questions that sense check the activity being undertaken, and whether the approach is merely habitual or has really been chosen to maximise the learning opportunity.

This requires a reconsideration – not just of why we are measuring the activity, but who determines what we measure, how we measure, and who the measurement is truly for. To fully enable the approach it's important to take all of these things into account. The framework below sets out the key considerations for doing this.

Figure 3: A framework for measurement for learning, Centre for Public Impact

This chapter sets out how local authorities can experiment in measurement for learning through a series of case studies.

2.1. Case studies: experiments in measurement for learning

The following examples of measurement for learning from the workgroup members represent just a small sample of the ways this approach is taking hold across the UK. The workgroup's 'Measurement for Learning' blog series¹⁰, published throughout the process of developing this paper, describes additional approaches - Community Builders¹¹ in Leeds or storytelling as an evaluation methodology¹² in Oxford. Other local authorities outside the workgroup, members of central government, and civil society organisations are also using approaches for measurement for learning in their own contexts.

By experimenting with and adapting these tools and others shared by expert evaluators, practitioners can develop new capabilities and start to build a measurement for learning approach that works in their context.

Below are five examples of approaches that members of the workgroup have used to practically develop the capabilities and infrastructure that can unlock measurement for learning.

2.1.1 Learning Pods

Internal reflection tool that uses a set of open-ended questions for staff to share learning in a dynamic, adaptive way.

Learning pods are one of the internal measurement methods that have proven effective at building a learning-oriented culture. The method, inspired by Chris Bolton's viral blog post¹³ on deploying learning and innovation teams in response to COVID-19, pairs staff who interact directly with citizens but don't normally work together to reflect on what they have experienced and learned over the past week.

Oxford Together, a collaboration between Oxford City Council and Oxford Hub, which was focused on helping those in need during COVID-19, incorporated learning pods as a regular reflection mechanism into their work. Led by Becky Willis, a project manager for the City Council, Sara Fernandez, who leads Oxford Hub, and the team at Centre for Public Impact, a group of around thirty staff identified important challenges and opportunities they could act upon.

Learning pods use a set of open-ended questions so staff participating can share what has emerged in a dynamic, adaptive way.

Questions include:

• What have you done differently this week?
• What did you learn?
• What enabled that learning?
• What has gone wrong?

Knowing they are free to candidly share their feelings, staff feel safe opening up about what they think could have gone better, insights that are then used to inform decision-making about how to adapt and improve moving forward. After reflecting on the questions with a learning partner, staff come together for a group discussion to understand perspectives across different pods. In addition to enabling learning, the exercise builds empathy as staff are exposed to a wide range of perspectives.

Some examples of ideas generated in a learning pod session were:

• The need to create spaces for relationships and trust to be cultivated across teams and organisations
• To better enable information flow
• To make the council more accessible, especially for minority groups
• To increase flexibility and resilience all emerged as key priorities from the first sessions.

Participants described learning pods as an opportunity to understand and learn from different perspectives, to reflect on their experiences, and to be open and honest. As one participant described: “Being in a non hierarchical space without our 'usual' teams or line managers is helpful. It feels like you can be completely open and honest. You don't feel like you need to censor your answers. It is so helpful to hear from others."

2.1.2 Values and Principles Check-In

Regular meetings where staff can reflect on the values and principles underpinning their approach to measurement.

Another way to enable learning and improvement within your team is to explicitly check in on the values and principles underpinning your approach to measurement. The workgroup has recommended a set of values¹⁴ - trust, authenticity, and curiosity – and principles that flow out of these, but you can develop your own to embody and hold yourself accountable for. The method draws inspiration from Confirmation Practices¹⁵, simple routines for systematic reflective practice developed by Andy Brogan at Easier Inc¹⁶.

Our adapted version¹⁷ simply involves bringing team members together on a monthly basis to reflect on the following questions for each principle that has been defined:

• How well are we adhering to this principle from 1 (not very well) to 4 (very well)? Don't overthink your response, just go with your gut reaction.
• Why do we deserve that rating?
• What action can we take (if any)?

Like the learning pods, the purpose is to genuinely understand how everyone feels the team is doing to enable collective adaptation and improvement. This approach enables you to reflect on how well you are living out your values as a learning organisation.

For a downloadable version of the values and principles check-in, see here¹⁸.

2.1.3 Life journey mapping

Life journey mapping: Proactively reaching out to citizens for meaningful conversations that identify opportunities for early intervention.

In Huntingdonshire City Council, Oliver Morley, Corporate Director (People), has been working with Claudia Deeth, Community Protection and Enforcement Officer, and a range of partners from across the system to create life journey maps.

By combining quantitative data points with qualitative insights they are designing a powerful way for local authorities to understand when proactive action should be taken (i.e. high risk score) and how to give individuals the agency to achieve their goals (i.e., build on an individual's unique strengths). Life journey mapping is an experimental tool designed to enable:

Preventative intervention: by meaningfully engaging with citizens and understanding stressful experiences that may trigger crises, tailored and early support can be provided to prevent future harm. As Oliver Morley describes: "Bad things do happen, but most of the time the signs are there. Quantifying, preventing, and addressing risks reduces poor outcomes."

Learning from failure: the life journey map strengthens understanding of the complex series of life events that contributed to a crisis in someone's life. By examining how these interrelated actions compounded, we can see where we could have intervened earlier and adapt how we handle future situations based on what we learn.

Systems change: a life journey map illuminates different life events that siloed actors may not have seen otherwise. For example, if an individual applied for social care, but did not meet the criteria to receive support, the map enables consideration of other forms of support that individuals may need. Instead of treating it as a transaction where the person is turned away, a more holistic view of a person's needs allows us to treat it as a relationship and refer them to other services within the system, besides formal care, that may be of use.

The council and partners start by providing staff with “meaningful conversations" training, based on the Making Every Contact Count training¹⁹ commonly used in health, to help them have conversations with citizens that move beyond presenting issues and get to the root cause. Trained staff then proactively reach out to citizens (e.g. those who have missed a council tax payment) to have a meaningful conversation and identify key moments in citizens' lives, their perception of these events, and any impact it had on them.

From these conversations, staff create a visual life journey map that plots out life events and quantifies the level of risk associated with each event using tools such as the Holmes-Rahe Stress score²⁰, which weights the stress level of a life event. A total "risk score" of an individual is then calculated by adding together the risk levels of each event. The more traumatic events an individual has experienced, the more likely they are to be susceptible to a crisis (e.g. homelessness, mental illness or arrest)²¹.

Each event is also associated with the cost to various systems within local government (e.g. police, social care, education), using the cost benefit analysis tool²² developed by fellow workgroup member Dave Kelly and colleagues at Greater Manchester Combined Authority.

The risk score and costs can be combined with qualitative data on strengths each person has (e.g., gifts, passions, skills), drawing inspiration from Asset Based Community Development (ABCD)²³.

See a detailed version of the map here²⁴.

2.1.4 From Checklists to Conversations

From checklists to conversations: moving social care assessments away from tickboxes and towards meaningful dialogue.

Mike Richardson, Transformation Programme Manager at York, led an adult social care transformation programme that took a strengths-based²⁵ approach to the social care needs assessment process, designed to influence the level of demand for the service and the length of time the assessment process takes. Central to the approach was enabling staff to have open-ended conversations with citizens rather than completing transactional checklists that have set thresholds for standardised service provision.

As Mike describes: "professionals are free to have meaningful, collaborative conversations promoting the use of strengths, resources, and potential to achieve desired outcomes and solutions to maximise a person's quality of life.”

While this programme embodies all of the values and principles of measurement for learning, the emphasis on empathetic conversations enables social workers to put authenticity at the heart – capturing the authentic voices of citizens and understanding what matters to them.

Figure 4: From checklists to conversations at City of York Council, Centre for Public Impact

2.1.5 Deep Listening

Deep listening: an approach to better understand the real needs, experiences and perspectives of local communities.

Mark Adams, Joint Director of Public Health for Middlesbrough Council, has been working with Catherine Parker, a consultant on public health for Public Health South Tees, to develop an approach to deeply listen to citizens in Middlesbrough. The programme is aimed at better understanding the perspectives of those disproportionately affected by COVID-19, such as people experiencing homelessness, domestic abuse, or mental health issues.

Middlesbrough Council has also partnered with Teesside University and a set of voluntary and community sector (VCS) organisations to put these citizens at the heart of their approach to understanding what and how they should measure.

The approach draws inspiration from the Mayday Trust's Wisdom From the Street²⁶, in which a qualitative review of over 100 people delivered powerful insights into what people thought of the services designed to support their move out of homelessness and toward independence. The results were so powerful they prompted changes not only in how support was delivered but in how the entire organisation thought, acted and responded.

The work will involve:

1. Trained VCS representatives having a series of listening conversations with citizens to better understand the impact COVID-19 has had on their lives, what matters to them, and what recovery would mean to them and their community.
2. A thematic analysis to understand what is most important to individuals, how the partnership can create a baseline from which to measure progress towards identified outcomes, and what changes they need to make based on what they learn.
3. Ongoing conversations with citizens to understand what progress is being made and what must adapt and change over time.

Catherine expands on a critical component of the programme – the nature of the conversations. “The conversations are intended to be open and driven by the citizens - not transactional, where we simply extract information. We aim to arm people who have positive relationships with groups we want to hear from with the tools and training to have a conversation that will provide us rich feedback and learning.”

Mark reflects on the potential of this programme to offer citizens an authentic chance to determine what the council should be accountable for: "Our aim is to have open-ended conversations that allow citizens to tell us what matters to them. It is a way for people to tell us what they care about and what the council should be accountable for delivering."

They are still in the early planning stages of this programme, but Mark and Catherine are optimistic that it will surface powerful insights, strengthen relationships with citizens, and, importantly, support cultural change within the council. Moving away from a service-led culture with predefined processes, this programme requires a more open learning culture that is able to use authentic listening to plan activity, check progress and to adapt based on what is learned.

03. Conclusion: learning how to learn

A developmental approach to measurement and evaluation based on experimentation, learning, and adaptation is a key enabler of new operating models in local government.

While this has always been a part of good evaluation practice, in the experience of the workgroup measurement based on traditional frameworks and metrics is deeply embedded in the culture and practice of local government. They are working to develop alternative approaches through a culture shift that better equips them to understand their progress on complex challenges, centre what is important to the individuals and communities they're working with, and support teams to experiment and learn continuously.

Despite the promise these innovative practices show, they are not yet systematically applied and embedded in business as usual. For them to be sustained and grow requires local authorities to invest in developing both a learning culture and a set of knowledge and skills, including listening deeply, building empathy, cultivating authentic relationships, and designing rigorous experimentation methodologies. These are capabilities that are necessary to create truly open learning environments within and across local authorities, partners and communities.

These skills must not live only in siloed performance and improvement departments, but rather embedded across the entire local authority so that those doing the work are able to assess their progress towards desired outcomes – listening, learning, and adapting to the complex challenges they face.

04. Acknowledgements

With great thanks to the members of the Upstream Collaborative who took part in the Meaningful Measurement Workgroup, without whose experience, insight and openness this paper would not exist:

Becky Lomas Becky Willis Dave Kelly Hannah Elliott Lisa Keenan Mark Adams Mark Smith Mike Richardson Oliver Morley Rhodri Rowlands Sara Fernandez

Additional thanks to the thought leaders whose work has inspired the participating councils and influenced the practice referenced within this guide:

Dr. Ailsa Cook Alexis Pala Danny Buerkli Dr. Toby Lowe

05. Endnotes

58 Victoria Embankment London EC4Y 0DS

+44 (0)20 7438 2500

[email protected] facebook.com/nesta.uk nesta.org.uk @nesta_uk