As the UK limbers up for a general election - and a potentially turbulent few years - I share some observations on what might happen to both old and new parties over the next few years.

In the early 2010s we at Nesta started to take an interest in the parties emerging in the wake of the financial crisis (an offshoot of work we were doing on innovations in democracy). New parties tended to be much more open to innovations than incumbents, perhaps an unsurprising pattern since the main beneficiaries of systems are easily convinced of their virtues.

Many of the new parties we found ourselves talking to looked very different to the incumbent ones, and were trying out novel ways of selecting candidates, deciding policies and campaigning.

So we gathered them for events and introduced leading figures from the old parties to see what they could learn. The parties we were interested in included M5S (Five Star), Podemos, Pirate Party, more obscure but imaginative ones like DemocracyOS in Argentina, the movements around President Tsai Ing-Wen in Taiwan, and outsider winners (like Won Soon Park in Seoul who initially stood as an independent). More recent interesting examples included Emmanuel Macron’s En Marche and Volodymyr Zelensky’s ‘Servant of the People’ party (and of course there are many others, from AfD to Vox and the True Finns, to the AAP in India).

Each had strengths and weaknesses. Some made a fetish of completely open Internet-based policymaking which we were sceptical of. Others appeared very democratic but were actually tightly controlled, or even privately owned. An interesting theme was the role of comedians - from Jon Gnarr in Iceland to Beppe Grillo in Italy and Zelensky – and as Gnarr showed as Mayor of Reykjavik, comedians are not necessarily ill-suited to government.

All were at least attempting to adapt democracy to the 21st century, and in ways that ran opposite to the other current trend of authoritarian corporatism – tight links between the state, party, business and media.

The best examples were refreshing to anyone concerned about British politics. The UK remains dominated by a Conservative Party whose members are mainly over 60, and a Labour Party whose processes (composites, conferences, motions etc) come from a century ago, and much of whose current leadership were schooled in the Leninist principles of tight central control. Both have been happy to use new methods for campaigning, but have resisted any serious innovation in their core processes. This became very clear earlier this decade when Nesta hosted events with all the main parties – including, in the House of Commons – but found very little appetite for reform (with a handful of exceptions such as Caroline Lucas and Liam Byrne).

Many of the new parties quickly found themselves holding serious power. M5S now runs several big cities in Italy and is part of the national government; the various parts of Podemos won control of Barcelona and Madrid; the Pirate Party at one point reached over 40 per cent in the polls in Iceland; En Marche swept the board in France, as did Volodymyr Zelensky’s new party in the Ukraine this July.

Leadership/Attention: First, to succeed they needed to grab attention. That usually meant having a charismatic leader without political baggage – like Iglesias or Grillo – who were already well known, could gain the attention of national media and project freshness and novelty. A grassroots movement on its own was not enough.

Internet/Organisation: Second, they needed strong internet-based organisation and technical data skills, to recruit and then organise a large membership or supporters group from scratch.

Narrative/Sticky messages: Third, they needed a simple diagnosis of what was wrong, and a simple prescription as to what needed to be done.

These were the essentials. There were plenty of other important elements – including money, and an irreverent, ‘up yours’ ethos – as well as clever tactics. But these were the key ones, and they could make up for the inevitable raggedness, unprofessionalism and contradictory messages that are endemic to insurgent parties of this kind.

Emmanuel Macron was unusual in achieving some of this while coming from the heart of the system (helped by several huge doses of luck with the implosion of the centre right front-runner, a disorganised left and a far-right adversary still seen as beyond the pale).

Nigel Farage has been the closest equivalent in the UK (twice, first for UKIP and then for the Brexit Party), and of course he studied M5S closely. He fits all three conditions. He also had the backing of the most powerful media barons. The new centrist party Change UK, by contrast, met none of the criteria. It had no charismatic leader; very weak Internet capacity; and only the vaguest prescription – which made its failure all too predictable.

Wikipedia has a useful, and I think comprehensive, list of all the political parties in the UK – from the Women's Equality Party to English Democrats, Christian Democratic Party to the Libertarian Party. Although quite a few of them had a strong diagnosis and prescription none of them has met the first two criteria suggested here.

It always used to be argued in the UK that the electoral system made it impossible for new parties to thrive. But Farage’s successes with UKIP and then the Brexit Party shows just how much influence can be achieved by insurgents, even if it remains harder for them to achieve power.

British politics looks set for a series of combustions and implosions, which could further corrode the conventional wisdoms. The main parties may go through further occasional surges (as both Tories and Labour did in 2017) and crashes, but the space for new parties looks set to grow, not shrink as electoral battlegrounds become more competitive. And we could see some new options in both the centre and the left that better pass the three tests.

Then the big question will be whether they can consolidate. The internet tools used by the insurgent parties are fantastic for campaigning, but much less well-suited to government (as M5S has discovered).

So the future probably lies both with old parties that can reinvent themselves, adopting the best tools from the new ones; and new parties that are smart enough to adopt some of the vital capabilities of the old, including sustaining a local base of councillors and activists, resolving policy dilemmas and handling the inevitable compromises of government.