Innovation teams and labs around the world are increasingly being tasked with building capacity and contributing to cultural change in government. There’s also an increasing recognition that we need to go beyond projects or single structures and make innovation become a part of the way governments operate more broadly.

However, there is a significant gap in our understanding of what “cultural change” or better “capacity” actually means.

At the same time, most innovation labs and teams are still being held to account in ways that don’t productively support this work. There is a lack of useful ways to measure outcomes, as opposed to outputs (for example, being asked to account for the number of workshops, rather than the increased capacity or impact that these workshops led to).

Consequently, we need a more developed awareness and understanding of what the signs of success look like, and what the intermediary outcomes (and measures) are in order to create a shift in accountability and better support ongoing capacity building.

One of the goals of States of Change, the collective we initiated last year to build this capability and culture, is to proactively address the common challenges that innovation practitioners face again and again. The field of public innovation is still emerging and evolving, and so our aim is to inspire action through practice-oriented, collaborative R&D activities and to develop the field based on practice rather than theory.

With this as our core premise, over the past 18 months we’ve been collaborating with innovation practitioners from across the world to develop and test an impact framework for cultural change in government that tackles some of the issues above. Multiple leading innovation organisations and practitioners have been involved in this process – including Laboratorio de Gobierno, MaRS Solutions Lab, Lab para la Ciudad, La 27e Region, MOM Design Lab (Singapore), Snowcone & Haystack, FutureGov, Public Policy Lab, UK Policy Lab, InWithForward, the MBRCGI (UAE government), UsCreates, Pides, OPM Lab, Y-Lab and the Colombian Department of National Planning’s innovation team EiP.

In the rest of this blog, we share the draft framework we’ve created, the process for how we did so, and our thoughts on continuing challenges and next steps.

The development process began at a practitioner gathering in London in September 2016. This was the first of multiple off- and online sessions that have focused on exploring and testing new ways of enabling strategic impact planning and assessment for cultural change in government.

At this first stage, we wanted to focus on 1) framing and developing the right focus, 2) exploring the potential use-value and 3) and testing possible approaches and tools relevant to this context.

Consequently, we asked the questions:

• What are we trying to change and what kinds of value do we provide? (What should we measure?)
• What are our theories for impact and how can we create the mandate for what we want to do? (How do we justify our priorities?)
• How do we collect data to track this change? (How do we measure?)
• How do we share our results? (External reporting formats and processes?)
• What kind of approaches and accountabilities should we help each other build and legimitise?

We focused on prototyping various versions of what an impact framework might look like and explored its use-value. A selected prototype (see below here) was then chosen as the practical starting point for the learning collective.

This led to a next phase of testing the framework in different innovation teams and labs around the world to qualify the content, format and design principles. Following this, a new version was developed and used to analyse, develop and communicate impact stories of selected projects and their overall theories of change connected to building the capacity of government organisations.

These impact stories were presented at a gathering in New York City in the Summer of 2017 to enable peer-learning about how the framework was useful to understand and support impact assessment in very different contexts.

The insights gathered through the impact stories are now informing a series of parallel experiments around the world to test the framework’s value-contribution to creating theories of change, accountability management and programme assessment. Beyond this, the Nesta Skills team is using the framework to assess and measure the development of organisational readiness and capacity development in the States of Change programme.

The main output produced so far is the impact framework illustrated below. It consists of a range of levels in which to observe cultural change and capacity developments (individual, team, organisation and ecosystem), and combines these with overall elements that should be considered when building (and assessing) innovation capacity (attitudes, abilities, behaviour, discourse, roles, relationships, environment, outputs and ripple effects).

Download the framework as a PDF.

Importantly, the framework focuses on spotting, mapping and assessing behaviour change. It is not about filling every box; rather, the framework enables a constructive dialogue about the sorts of areas we should prioritise and develop over time, and focuses on tracking intermediary outcomes. In this way, it is meant to support the ongoing customisation and design of your own contextual approach to impact assessment, and can develop over time.

At this early stage in the development process, the framework has mainly been applied as:

• An assessment of organisational readiness and capacity development tool. To inform and develop assessment criteria for capacity-building (States of Change learning programme in Victorian States government, Australia and Canadian federal government).

• A strategic dialogue tool. Used by host-organisations or beneficiaries for continuous planning and development of accountabilities to support and assess the impact of innovation teams and labs (multiple labs and teams).

• Input for theory of change. Enabling structured focus on what elements should be prioritised in capacity-building efforts (multiple labs and teams, including in Argentina, Canada, Colombia and the UK).

• Model for inquiry. Informing the focus and model of inquiry for mapping and collecting data on innovation capacity-building and cultural change (Laboratorio de Gobierno in Chile, EiP team in Colombia).

• Case production and knowledge sharing. A new way of opening up case studies for knowledge sharing and mutual learning across borders and sectors (multiple teams and labs).

While the overall methodology for the impact assessment approach is still being developed, we are learning about what combination of methodologies will be part of it. For now, it is safe to say that we will build on developmental evaluation approaches.

In this light, it is worth stressing (at least) five overall principles that underpin the work of assessing the impact of innovation teams and labs:

1. Working with an intentionally ambiguous theory of change that use theories as assumptions to be tested and adapted rather than using prescriptive evaluation criteria.
   
   
2. Resetting expectations from measurement to learning about, documenting and assessing which focus areas should be prioritised and which approach will be most effective in and sensitive to the context and purpose of the work.
   
   
3. Going from reporting on projected outputs to learning about possible areas of impact by involving practitioners and stakeholders in the development of assessment criteria and adapting questions, priorities and criteria as we learn .
   
   
4. Experimenting with ways of creating continuous organisational self-reflection on how different cultural change indicators in combination and/or sequence support the journey towards cultural change.
   
   
5. Using peer-to-peer reflection and review to enable continuous fine-tuning of strategy and quality assurance, seeking a diversity of professional perspectives on your practice to expand perceptions and possibilities.

Since innovation labs and teams are by default working to challenge existing ways of working in government, there are significant challenges and limitations to how much existing impact schemes can be influenced at this stage.

Here are three selected key attention points of focus going forward:

1) Creating clarity on assessment purposes is key.

We see a significant lack of clarity about the purpose of impact assessment approaches. It is crucial to be clearer on whether it is about overall accountability (theory of change), documentation (tracking attributable impact and generating intelligence), justification (of resources spent and strategic directions chosen) or learning (for improving practices, strategies, processes, etc.).

2) How to assess and measure the process of “unlearning”?

A lot of capacity-building and cultural change in government implies unlearning existing ways of thinking and working, so how do we best deal with this premise? Focusing on behaviour change - the transition from traditional habits, mindsets and procedures to embracing and embedding new ones - is not a smooth process of abandoning one set and adopting an alternative set, making it a challenge to understand and assess.

3) Dealing with the trade-offs of impact assessment

Attempting to embed innovation processes and structures involves a large range of trade-offs that will need to be considered when shaping your approach to impact assessment for cultural change. Selected trade-offs include:

• Stable criteria vs. continuous development: how to find and leverage a healthy balance between stable measurement criteria and more fluid and continuous growth in value?
• Non-linear value-creation: how and when to attempt to create causality in a context where there is a continuous lack of linear causality between activities and impact?
• Taking credit: what part of the value-creation chain can/should you take credit for?
• Time horizon: what can you assess and when can you assess what in a context of lack of indicators for evaluation?
• Not one kind of change: how to deal with multiple datasets and measurable elements on different levels?
• Big picture vs. smaller scale outputs: how to define, illustrate and measure (or communicate) the big picture while creating smaller assessable outputs?

Our immediate next steps will be to, as part of the States of Change initiative, continue to develop a dedicated suite of practical tools based on the framework to deal with impact assessment and evaluation for cultural change.

Alongside this, we will continue to facilitate experimental user-testing with innovation teams and labs focused on redesigning their approaches to assessment and evaluation (among others Y-Lab, Health Lab and the EiP team in Colombia). We will also communicate some of the impact stories developed using the framework to enable a wider discussion about the future of impact assessment for innovation in government.

We are very interested to engage with interested fellow practitioners facing similar challenges so please get in contact to hear more and explore potential synergies ([email protected]). For more information, please also see states-of-change.org.