Social innovation is a lot more visible than a decade ago. There are funds, university research centres, offices in government, events of all kinds. The idea that societies need innovation as much as fields like medicine or science is becoming more mainstream. But it’s still very far from being embedded. In some parts of the world, the climate has become markedly hostile – the last thing authoritarian governments like Russia, Turkey or Brazil want is to fuel social creativity. But even the more benign liberal societies are much more comfortable supporting innovations in hardware than in society.

The UK election is an obvious example. The big parties are competing with big promises to invest in physical stuff - roads, rail, bridges. But they have no policies or ideas on social innovation and few ideas on how to solve acute social problems other than by throwing money at them.

The contrast with science is very striking. In the late 19th and 20th centuries science went through a remarkable transition. Before it had mainly been the province of enthusiastic amateurs – like Charles Darwin or Michael Faraday and long-forgotten inventors dreaming up new ideas in their sheds.

But then it became much more of a system. Governments invested large sums in research laboratories and universities. Big firms like Du Pont and General Electric set up R&D labs. The shares of GDP devoted to R&D crept up, to around two per cent in countries like the UK and closer to four per cent in others like Korea and Finland. The result was the tide of new technologies that has changed every aspect of our lives.

But rapid innovation in hardware was never matched by equally restless innovation in society, and the imbalance has continued. Large sums of money go to advancing aerospace or pharmaceuticals but much less to innovation in homelessness or isolation.

During the 20th century society was increasingly left behind by technology and one important reason was that it had no comparable system for innovation, with significant financial and human resources devoted to incubating ideas, improving them and then taking them to scale (despite some pioneering efforts to promote social R&D by big foundations in the 1950s and 1960s).

In the UK, for example, the big funding agency – UKRI – has no dedicated resources for social innovation or social R&D. The same is true of the US. France and Canada are beginning to shift – and Trudeau’s reelected government recently announced $1bn for social innovation. But they remain the exceptions.

The internet is a good example of what goes wrong as a result of this imbalance. It is, of course, an extraordinary technology, the product of public R&D (at DARPA and then CERN) followed up by furious commercial innovation. But the years it became central to daily life were also years when the proportion of the US population that could count on anyone to support them in a crisis sharply fell (from 92 per cent to 75 per cent) and when a host of new pathologies arose, from the accelerated spread of fake news to compulsive behaviour. Very little money was spent on innovations directed to solving these pathologies, mainly because it was no-one’s job to do so.

Too many of the pathologies of our times – from political alienation to inequality – are the result of this profound imbalance between those fields that invest heavily in innovation and those that don’t. The former include the military, surveillance, and much of business, including the click through advertising that dominates the platform economy. The latter includes almost everything to do with society.

The World Happiness report earlier this year showed why this matters. It concluded that relationships explained more of the differences in well-being between nations than anything else. People’s answers to the question: ‘If you were in trouble, do you have relatives or friends you can count on to help you whenever you need them, or not?’ – explained 34 per cent of the well-being score, more than income (26 per cent), or healthy life expectancy (21 per cent). But the systems governing R&D have barely started to reflect these patterns.

For me, social innovation is the lens through which we can think about how to put this imbalance right. The problem is not a lack of good ideas, or creative people, or inspiring projects. The problem – as with science in the late 19th century – is the lack of a system that can help promising ideas evolve, develop and then spread.

In my book I set out some of what such a system would and could look like, showing: how it taps into the energies of social movements and social entrepreneurs; how it experiments and incubates; how it generates capital to develop and scale ideas; how it connects to the power of government; and the role it can play in big challenges like the transition to a zero carbon economy.

Today’s context makes it all the more urgent that these ideas become common sense. A fourth industrial revolution is driving change at breakneck speed – threatening millions of jobs; harvesting data without consent; and leaving people feeling like observers with little power.

There is a clear parallel with the history of the first industrial revolution. In the first half of the 19th century industrial technologies and early capitalism developed at breakneck speed, but with often disastrous social and environmental consequences. Millions moved into cities that were wracked with poverty, ill-health, crime and pollution. In cities like Manchester, for example, death rates from diseases like smallpox and measles were four times higher than in the countryside.

Then in the second half of the century social innovation slowly caught up. Public sanitation and water; universal education; a wider suffrage; trade unions; microcredit; adult education and many others reduced the harms and spread the benefits. Now, too, social innovation can help us shape another industrial revolution in ways that give us more of the benefits and fewer of the harms.

Geoff Mulgan is CEO of Nesta and author of ‘Social Innovation: how societies find the power to change’ published this month by Policy Press. He will be speaking about his book at Nesta on Tuesday 26th November at 6pm. Register for the event.

Use code is POGMJ19 to get a 30% discount on the book from Bristol University Press valid until 31/1/20.