Could we soon see a radically more ambitious way of organising innovation? One that can ride and drive the great waves of change of the next few decades in transport, energy and health?

In this post I'll show why I think this could be the case, and how the answers could address some of the dilemmas of innovation policy: how to strike the balance between strategic top-down direction and bottom up entrepreneurial creativity; how to turn the UK's highly centralised governance structures into an advantage rather than a weakness; and how to mix consensual, corporate institutions and ones able to take decisive risks.

Innovation debates around the world continue to polarise in the same ways they have for the best part of a century. On one side there is the camp which believes that all new ideas come from entrepreneurs, and that these are bound to be misunderstood by bureaucrats and bankers. The role of policy is to make things as easy as possible for them by removing barriers and to make government porous to outsiders' ideas.

On the other side there is the view that technologies depend on direction - whether from governments or big firms, or from partnership bodies. The more these can orchestrate the heavy early investment in ideas long before they'll be attractive to venture capital the better. The more these can then mobilise and scale, the better the chances of any nation or region reaping the rewards.

Both sides draw as much on myth as evidence (the first group mythologise entrepreneurialism to such an extent that they ignore the fact that the US has had the most activist industrial policy of any government anywhere in history, the second so mythologise strategic wisdom that they ignore the vast number of blunders and wrong turns that bureaucracies have followed). 

But both are also right in important ways, and both accounts can be seen in every stage of technological history of the last two centuries.

But the key point I want to emphasise is that how right the standard positions are - laissez faire v strategic to caricature a century of argument - may depend on historical circumstances. During some periods, and for some industries, the entrepreneurial, bottom up approach is much more effective; in other periods big choices have to be made and it's not feasible to let a thousand flowers bloom.

There are growing signs that in at least some key industries we may need to make some big bets, and that how we do so may have profound implications for the organisation of technology policy - some of which may play to the UK's advantage.

Systemic innovation

The thesis is that the next few decades will bring a series of radical changes to whole systems, from energy and health to transport and food (for more on this see my piece for Global Brief).

Many of our current systems look increasingly anachronistic - but past sunk investments make them hard to change. Indeed in many cases we are living with the fruits of past successes - systems that worked extraordinarily well for many decades, so well in fact that they became hard to challenge. We still drive internal combustion engines on roads not much different from a century ago. Our energy systems are centralised national electricity grids, again the vision of a century ago. Our healthcare is organised around big hospitals - the assumption of the early 20th century. We even still have copper cables providing access to telephony.

Yet in every case radically different alternatives are now becoming visible, often enabled by the next generation of information technologies. The problem is that to realise these alternatives many things need to change in tandem: not just bringing new technologies to market but also changing the rules of the game, the laws, regulations and complementary technologies that have all been designed to make the old system work.

Health is an obvious example, soaking up ever more resources in systems built around hospitals that no longer fit either patterns of disease (which is primarily long-term) or patterns of treatment (which depend far more on self-management). Our centralised energy systems look ever less suited to a world in which far more complex networked management is possible. Our transport systems have yet to make the move to self-drive cars or comprehensive road changing - and still default to plans for building new roads, or new rail tracks, as the top priority for investment.

If true systemic change is on the way great advantage will accrue to those places which can align the elements of the new system. Some of these will be technologies and business processes, pricing models - but just as important will be public policies, safety rules, environmental regulations and public attitudes and behaviours. The crucial point though is that the systemic change will bring with it a swarm of innovation, as happened during previous periods - indeed it's only through creative experiment around the system that the world will discover just how to make new systems work.

None of this can be done through blueprints but it can be done through conscious iterative enabling of the new system.

If this argument is right, then what has traditionally been a UK weakness could become a strength. That weakness is the constitutional structure of the UK state, which is built around powerful silo-ed departments. These have made it impossible to ever pursue a coherent overall technology and industrial policy. The bulk of resources have been controlled within these departments - notably defence and health - which have never ceded power to attempts to coordinate them, whether by a ministry of technology in the 1960s or BIS today.  

But if systems change really is the priority, then these may actually be easier thanks to strong departments - most of the big systems align fairly well with inherited departmental structures.

The problem is that we lack institutions designed to make the most of this. There are strong capacities in defence and health - though not traditionally linked up to broader economic objectives, and rarely galvanised by a vision of the future. In other key sectors the institutions are very weak. The good news is that we do now have some of the building blocks that would be needed - in the form of Catapult Centres (only 30 years after a major party first put them into its manifesto!). These are well suited to linking academic research and industry, and forging consensus within industries.

But they may be less suited to big bets and risks. This is where the interest in DARPA and ARPA comes in. They have long attracted admiration for their scale, their authority and their appetite for backing fundamental technological breakthroughs. There is much misunderstanding about how DARPA actually works. The vast majority of US defence R&D funding comes from other sources, which allows DARPA to do much riskier things - of which probably 19/20 projects fail - and to empower individuals to follow their hunches. But it does have important strengths which may be particularly relevant to fundamental systems change - the freedom to pursue visions, and to ignore the consensus of the existing incumbents.

Link these arguments together and it becomes possible to map out a new generation of institutions that might complement the emerging Catapult network (and other existing capacities such as Foresight). 

These would have the overt goal of presiding over systemic change in key fields. They would be deliberately linked into departmental power and money. They would develop pictures of where the world is heading; commission important elements of technology; influence the policy environment; run exemplars of the new systems in cities and towns, sharing both the risks and the rewards with them; mobilise SBRI and procurement to maximise the contribution of entrepreneurship in and around the emerging system; and above all act as champions for the future. 

Models of this kind are most likely to be relevant in fields where the alignment of systems is most crucial: like driverless cars and road charging; social networks and data supported self-managed healthcare; smart grids and decentralized energy.

Future Agencies

Since these would have the primary goal of bringing the future into the present they might be named Future Agencies, or perhaps more prosaically Advanced Systems Agencies - their budgets would need to be in the hundreds rather than tens of millions or billions. They would require a degree of confidence on the part of government that may be missing at the moment; they would require careful institutional design, and even more important, selection of the right people and teams. But their case would be that if there are billions to be spent on things like HS2, a smarter investment, and a far greater pay-off will come from leaping to the next generation systems rather than rolling out existing ones.

[We have published extensively this year on systemic innovation, including lessons from history and policy tools (ranging from Whole Systems Demonstrators in the NHS to eco-cities).]