The world abounds with both opportunities and crises. We live in a time of unparalleled progress – scientific breakthroughs offer to achieve everything from a cure to cancer to self-repairing glass. We are more connected than ever before. We have the opportunity to replace many low quality jobs through automation. And we have more information about the world than in any previous era. But these opportunities are clouded by the rise of political chauvinism and threats ranging from climate change to antimicrobial resistance.

Citizens are more educated than ever before. But we have legacy systems of decision-making. We still vote once every 4 or 5 years and, while governments often consult on what they do, this involves relatively small numbers of people in a meaningful way. For all its achievements, consultation is failing to match the expectations of a demos which expects high levels of inclusion in decision making through the market and in the workplace. The popularity of participatory budgeting schemes and other manifestations of participatory politics – such as the Brexit Citizens’ Assembly – hints at the potential for wider deployment of participatory methods. These methods have been taken up by the House of Commons Housing, Communities and Local Government Committee to investigate the future of adult social care.

While there are many causes for optimism, we know that, left to trawl the internet for information, people may be taken in by ‘fake news’ and conspiracies. But we also know that when people are deliberatively engaged in the process of decision making – through the sorts of participatory techniques that underpin Participatory Budgeting and Citizens’ Assemblies – that those people can properly assess the opportunities and threats, and have access to rational consensus.

When the political theorist Edmund Burke wrote that an MP “owes you, not his industry only, but his judgement; and he betrays, instead of serving you, if he sacrifices it to your opinion” he did so in a context where most were unable to assess and judge the great issues of state.

This is especially so in a society where basic education is universally available and citizens are continually engaged in judgement and decision-making through much more autonomy in the workplace and regular consumer decision-making. Bringing these skills into public decision-making is not just the right thing to do, it is now a necessity for the survival of a democratic society.

There are a number of areas where this is particularly important. When it comes to harnessing the opportunities of ubiquitous and pervasive data, we urgently need a participatory process that allows a realistic assessment of the risks of government use of data. The Nuffield Council on Bioethics has pioneered some of the techniques required in this task (and is considering expanding the discussion to data policy), but this needs to be applied much more widely, and with levels of participation that reflect how widely the effects of any decision will be felt.

We can understand the underlying issue as a failure in the public sphere - where the intersection of public institutions, media and citizens is incapable of synthesising the available information into appropriate courses of action. We have both the tools and the opportunity to move beyond the public sphere and towards a deliberative society. Now is the time to seize this opportunity.

Recent technological developments make this opportunity even more important. The last decade has been marked by a transformative increase in the availability of data. This new data comes in many forms: it is easier to track steps using a phone or personal fitness device than it is to count them yourself. It is easier to count mobile phones passing a turnstyle than employing someone with a clipboard to do the same. It is easier to assess what economic activity there is in an area by web-scraping job adverts than it is to undertake a detailed study. This new data should turn our understanding of the world on its head. Where previously when making a decision we needed to go into the world to actively pursue information, now much of that information is close at hand. While we still need to access it, that is much easier than it once was.

Collective intelligence offers us the opportunity to make human interventions of more value. Instead of expending time and effort measuring what’s happening, devices are doing this for us. We can bring these data sets together to make sense of the world. This can greatly improve decision-making.

And improved decision-making is becoming ever more important. We all know that there are a variety of serious challenges facing the world today. These stretch from antimicrobial resistance to climate change, and on to the ageing population. Our cities are struggling to deal with air pollution and some are even at risk of running out of water.

A different way of solving problems comes when we set them out as challenges. A challenge approach works by setting out a problem, such as regulation of urban drone use or managing variable supply in a renewables-based energy network, and inviting researchers, developers and citizens to pitch ideas on how to solve these problems. It can break down disciplinary boundaries and administrative silos, create understanding, investment and, ultimately, solutions that we might not otherwise have found.

These challenges could be of direct political contestation - as with abortion rights in Ireland - or they could be more local - on approaches to air pollution in a particular municipality. Or they could be longer-term issues, such as the regulation of artificial intelligence. There will be particularly significant opportunities for bringing arts-led approaches to help understand, represent and interpret the evidence and arguments that are required to involve participants in the citizens' assemblies and to more broadly communicate the work of the Institute. The opportunities to bring music and art into debates about the future will increase the impact of this approach and allow alignment with creative imaginings of the future.

In the Republic of Ireland Citizens’ Assemblies have been used to deal with a variety of issues, ranging from equal marriage to abortion. These issues have proved intractable through traditional political structures, which are marked by self-interested approaches. By bringing together demographically balanced groups that reflect the major views, a resolution emerges.

At the same time, we have more methods for bringing people into decision-making processes. Both online and in-real-life we can build more effective ways of including people. Where social media has connected people much more effectively than ever before, so participatory approaches can make use of digital tools to make better decisions. By better identifying the challenges that people see, and matching this with the many exciting possibilities, we can rebuild our public sphere.

These approaches are particularly relevant to institutions of learning. With universities now measured on their impact (and with this measurement becoming more significant), and seeking new ways to make sure their research is more relevant to challenges, a new opportunity arises. A process of identifying great global challenges, assessing different approaches to addressing these challenges through participatory methods and using this to inform research guarantees impact. It is a win-win, allowing academics to validate their research and citizens to be involved in the process of commissioning and design of research.

How could this work in practice?

The opportunity to deliver on a programme of this sort is enhanced by the substantial investment in estate and curriculum that has been delivered through the City Deals for both Edinburgh and Glasgow. In Edinburgh, the rhetoric of ‘challenge’ has already been taken up in the design of the Edinburgh Futures Institute. This adds physical manifestation to an already existing public sphere and creates the conditions to catalyse the move towards a deliberative society and participatory democracy. The University of Edinburgh already has substantial expertise in this area through the work of Oliver Escobar, Co-Director of What Works Scotland and the Smart Urban Intermediaries programme.

The dominant political approach in any era is manifested in its architecture. The German theorist Jürgen Habermas identified the Palace of Versailles as a bricks-and-mortar example of absolute monarchy. So physically vast and so overpowering as to leave the subjects of the French King in no doubt of who wielded power on behalf of god. We have the opportunity to create a concrete manifestation of the participatory society.

By bringing together a challenges approach with a citizens assembly methodology, we can identify, test and surface the areas where change is needed. When we bring researchers and practitioners together we can begin to address these areas where change is needed. By including citizens at every point, we demonstrate that research aligns with popular concerns. The process has impact woven through it and will begin the process of prioritising problems, accounting for social, environmental and economic change, and rebuilding trust in society.

This is not an attempt to replace curiosity-led research, but rather to allow researchers and practitioners to better understand which questions should be addressed and to - if they choose - direct their research to answering these questions. The approach itself will be open to experimentation, iteration and development. By comparing the effectiveness of different ways of identifying challenges, building participatory techniques and measuring impact we can create learning that can be widely shared.

If the Palace of Versaille was a manifestation of the feudal political order intended to awe subjects into submission through the sheer scale and majesty of the buildings, so the aim of Edinburgh Futures Institute should be to create a deliberative space where citizens can realise a participatory democracy. It can be a built manifestation of the participatory society. And through harnessing the information now available to us, and the insights of citizens, we can create a public sphere worthy of awe.