But our systems for achieving the SDGs face three interconnected challenges:

First, there is a fundamental challenge of learning that development shares with many other fields. Although Google searches make it appear that you can easily find the best and most proven ideas, this actually turns out to be hard in practice and our systems are still remarkably poor at learning from successes and failures;

Second, there is a challenge of getting the right kind of innovation. Most funding for innovation focuses on the military and business rather than social needs. Meanwhile, the small sums invested in social innovation are often ill-suited to the complexities of extreme poverty (like excessive confidence in the power of the right app, or an idea that has been validated by a single RCT).

Third there is a challenge of technology. As the Fourth Industrial Revolution sweeps across the world, much of the development field, and the wider public sector and civil society, are bystanders, unable to use new technologies like AI and drones, let alone influence their development.

These are soluble challenges. But they need new institutions, methods and mindsets. Here, I focus on some new tools that are available to address them, most of which were not an option even five years ago, let alone in the era of the Millennium Development Goals (MDGs). All can be loosely described as tools for collective intelligence – ways of harnessing more data and more brain power of all kinds and at every level and learning faster. They form part of a broader programme to accelerate innovation for development – innovation that is disciplined, focused and creative and involves everyday iterative improvement and use of evidence; harvesting ideas from any source, including the grassroots; and faster ways to make the most of the potential of emerging technologies.

I argue that these point to options that would build on the best of the United Nations Development Programme (UNDP) history, but also mark a significant evolution; to becoming much more like a platform. This word is often used vaguely. I try to show here how the UNDP could become a platform in quite precise senses – all of which would make it easier for local communities, agencies and governments to act effectively in pursuit of the SDGs, improvising, innovating and adapting to fit goals to local circumstances.

Development used to be thought of primarily in terms of flows of money. But increasingly we recognise that it depends on new knowledge and new capabilities, and that, without these, money is likely to be wasted. This why the UNDP and other agencies have shifted towards more direct roles in implementation and knowledge brokering.

This shift focuses attention on the question of how a nation, or region, can mobilise all available resources of data, knowledge and intelligence to achieve the SDGs, make fewer mistakes and direct energies where they’re likely to be most effective.

The answers are of course complex. But they can be broken down into three clusters which, added together, can make strategies to achieve the SDGs more collectively intelligent (read an overview, developed with UN colleagues).

1. Seeing: A far wider pool of data and inputs

The first priority is to improve observation of the key facts, whether about water or infant mortality or any other SDG, ideally in real time. There are now many more potential sources of data that are relevant to decisions and actions: from sensors and satellites; commercial data like mobile phone records which track travel patterns or economic activity; and citizen generated data on everything from floods to corruption.

Often the less obvious data are more valuable than the official data. For example, in many countries, mobile phone data is a better indicator of economic activity and its shifting location than anything else. Recent initiatives like Global Pulse have shown how valuable these can be. Often, too, very local, tacit data is more useful than formal data. Other ways of thinking about data can also be useful: asset mapping (which looks at the resources, skills and capabilities within a community); mapping positive deviance (i.e. seeing which groups are succeeding against the odds in improving education or incomes) and so on. Simple ideas – like the Mayor in Indonesia who requires public officials to share a daily picture of their work to counter corruption and non-existent roles – can be remarkably effective.

What’s often missing at a local level is the institution to harvest and analyse this data to help in decision-making, recognising the value of different sources. National statistical offices were set up in a different era and with a different culture. They play a vital role in establishing sound facts. But few have embraced the new roles they could be playing.

Yet in every country or region, and for many of the SDGs, this is a soluble problem: to ensure that there is a curator or organiser of a transparent and open body of data, some local, some regional and some national, available for everyone to use, and pragmatic about combining a very wide range of sources.

2. Shaping: Design and creation

The second priority is to harness a much wider pool of people and capacities to make sense of the information and contribute to solutions. In the past, and often in the present, strategy and policy formulation has involved very small numbers of people, often quite removed from direct engagement with the issues, based in offices in the capital cities.

Circumstances vary greatly as to the role of governments, national, regional and local; the strength of civil society; the engagement of business in relation to SDGs. Generalisations are unwise. But in widely varying circumstances, processes of decision-making can be opened up using at least three complementary routes:

• First, harvesting ideas from a far wider range of sources. This may mean identifying ‘positive deviants’ achieving success against the odds, or grassroots innovations. It can mean using open innovation platforms like challenges.org which is already used for issues like farming, diseases or antibiotic resistance, specifying problems to be solved and then encouraging, and incentivising others to help solve them. The SDG promise to ‘leave no one behind’ brings with it an imperative to involve marginalised, illiterate individuals and communities in collective intelligence initiatives, which involves more work but often a harvest of rich insights (as well as playing its own role in growing capacity and power). These harvesting exercises work best when i) they are carefully curated and managed, since solutions rarely appear ready formed, and ii) when global problem-solvers are closely linked to people on the ground.

For almost any SDG anywhere, some version of these processes can be useful, including mobilising very local communities to contribute to very granular parts of problems.

• Second, using consultation and decision-making platforms like DCENT, Consul and the many in use globally, to mobilise a community to devise, interrogate and improve responses, whether at the level of policy or implementation. Again these have wide application from water conservation to primary healthcare, literacy to human rights to corruption. They tend to work best when there is a combination of online and face to face engagement (and of course online is impossible in many places). They usually require a willingness to engage on the part of politicians.
• Third, through platforms to support experiment. When it’s not clear what will work well the best strategy is to experiment. Experiments can take many forms: they can be quick and dirty, or formal randomised control trials; they can be deliberately iterative, adapting fast in the light of experience, or sticking closely to a particular design. They can involve users and consumers or be led primarily by professionals or providers. They can focus on policy or implementation.

There is no single best approach, and it is as wrong to fetishise overly formal knowledge from RCTs as it is to fetishise user or citizen insights (at Nesta we try to use both). The key is to embrace rapid and open experiment as an ethos, and to make this the norm whether for policy design or implementation. The basic insight of collective intelligence is that there will always be value in mobilising latent sources of intelligence outside the system, rather than relying only on what’s inside, or on the contribution of consultants.

3. Learning: Monitor, adapt and improve

The third requirement is an architecture for monitoring and learning, tracking what is being done, what is succeeding and what is failing and why. Here there are many new (and old) tools to build on. For example, what works centres try to synthesise knowledge and experience about what’s effective in primary education or forestry management. At best, these are embedded in communities of practice – like teachers or public health workers – so that their demand for evidence is paramount. They are then used in regular rhythms of peer learning so that, for example, each month the people working on a particular SDG in a region take stock of what has been learned, what’s surprising, or what new knowledge from elsewhere may be useful. The least successful models simply accumulate evidence hoping that it will be used and lack any processes for systematic learning.

This latter point is key. For progress to be sustained there needs to be systematic processes for taking stock of what’s happening and why, and therefore what should be done differently. Here the many methods of strategy formulation, implementation and iterative improvement are relevant, including recent formulations like PDIA (problem-driven iterative adaptation). None of these are complex but they tend to be ignored in bureaucracies unless they are made part of daily practice.

For each of these levels the best approaches build on what already exists and combine the globally curated insights and knowledge of the system with local knowledge that’s generated by communities themselves. Again, a balance has to be struck. Higher tier bodies – including national governments – can set goals while allowing local agencies discretion as to how to meet them, helped by common information pools and peer to peer learning. The UNDP already works extensively with local governments to help them achieve SDGs. This iterative co-evolution has always been how development happens in practice, as new capabilities are grown through action (the key lesson of Yuen Yuen Ang’s work, and the dynamic interaction of environments, capabilities and business models).

None of this is rocket science. But it is challenging. Some of the challenges are eternal ones – politics, vested interests, corruption and disinformation, laziness and complacency.

Others are less obvious. A crucial one is that intelligence works more effectively when its joined up. As with the individual brain, the value comes from linking things up. There needs to be an integrator or curator, or ‘network facilitator’ - a role that can be played by partnerships linking national government, UN agencies and others.

I describe the best of these as ‘intelligence assemblies’ that link together data, analysis, prediction, memory, creativity and judgement into systems for learning. There are promising models emerging in many fields, including contagious diseases and the natural environment. The key insight is that the benefits are much greater if the elements of intelligence are linked up. A flaw of some recent initiatives has been to focus on just one element – like data or evidence – rather than making them part of a system.

Everything described so far already exists in some form, but is also an innovation in that none exist in a fully developed form, and all make use of technologies that did not exist 10-20 years ago.

In public organisations, innovation often risks being a fad, an aerosol spray to be applied lightly, but not taken seriously, or an indulgence. Yet for these new methods to be adopted and made a success requires energetic innovation and experiment, ideally with clear goals (specific parts of SDGs), milestones and a hunger to use ideas from other places, other sectors and organisations.

There are now plenty of good examples of bold, ambitious and successful innovation in large organisations. Most combine a clear remit from high level leaders; a distinct capacity at one remove from day to day management and delivery, to allow for more creativity; and then well-shaped routes for integrating the successful innovations back into everyday activity. Most also combine the sense of mission of a lead group, able to move fast and fearlessly, with sufficient attention to bringing along the rest of the organisation so that they don’t become barriers.

To support and make the most of new networks of innovators, the UNDP now has a unique opportunity to inject new energy into the SDGs by growing a more systematic infrastructure to support local initiatives that mobilise collective intelligence in these ways.

At a local level it can mean creating and growing a series of problem-solving teams or labs that act to push back the frontiers of innovation, showing by example.

At a global level it could mean the UNDP evolving more into a platform, or series of platforms, that support national and regional strategies around SDGs. This would offer:

• Curated knowledge: here the UNDP can build on decades of work on how to make evidence useful and used, and moving beyond the creation of repositories of evidence to focus on the everyday conditions of use by policymakers and front-line practitioners. Related to that is managing memory to organise knowledge across complex systems, both through data tools and through simple measures to make the people with direct relevant experience findable. A good example is finance – ensuring easy access to experience in using innovative types of finance. The simple goal here is that someone facing a new task in one place should be able to easily access both formal records, but also an individual who worked on something similar.
• Digital tools: providing modular digital elements on an open source basis for governments and others to use to create tools, from identity management to healthcare. The best way to do this is as a curator – seeking out the best available tools and then making them available as a default, either through bulk contracts or links to Github.
• Skills: promoting the skills necessary for officials, governments and NGOs to adapt, contextualise ideas and generate innovations. All experience shows that mindsets and cultures are as important as formal skills. This, the ‘softer’ side of being a platform, is as important as the others (and is the reason for creating States of Change as a practice led network to boost the skills of officials worldwide). But there is also a job to be done in growing more specialised skills, particularly in fields like data harvesting, analysis and AI.
• Experiment: providing platforms to make it easier to design and run experiments and then ensure results are useful (e.g. the tools provided by the Innovation Growth Lab and the formal software of projects like Predictiv), with the results (both positive and negative) shared across the system.
• Assemblies around SDGs: the medium term goal should be that for individual SDGs there are curated bodies of evidence, cases, and communities of practice where the key metric is use and perceived usefulness for practitioners on the ground, and where data is gathered to benchmark speed of achievement of SDGs, breadth of participation and relevant measures of impact. The UN agencies have no monopoly on this – and many independent initiatives are underway to do some of this. In some fields, the key will be to amplify or link these up rather than competing with them. But the key must be to see SDGs from a user perspective – how to provide them with information, money and networks they need.

The organisational challenge will be to balance strategy, innovation and local freedom. Some parts of the system need to be given the freedom to experiment. But this is a very different culture and skill set from coordination, knowledge management or being a central nervous system. Both are also in tension with local agency and accountability.

So the management task is to keep this triangle in balance rather than pretending that all the tasks are the same in nature. Some roles need to be very much at the corners, and others closer to the middle, and with clear recognition that the cultures will both be in tension with each, and complement each other.

This is a radical programme that would make the UNDP and its partners much more explicit mobilisers of data and knowledge in service to communities. In a sense, global agencies would be turned inside out – judged by usefulness to people working directly on SDGs on the ground rather than the other way around, and opening up resources to be used, and combined, by people on the frontline.

This could be a very energising direction of travel – and one that will also make it more likely that the SDGs are achieved.

But it would certainly challenge many vested interests and requires thinking in ways very different to those of traditional bureaucracies.