Computers, sensors and networks are everywhere. As they get smaller, cheaper and more powerful, they are being installed in places and for tasks we could not have imagined a few years ago.

This explosive growth in devices has led to an explosive growth in the amount of data gathered. This data is a huge, growing and untapped resource with the potential for transformative effects far beyond what it is currently used for.

Large computerised datasets already do incredible things. They have improved existing services by adding extra certainty and extra data points. Weather forecasts are far better than they used to be, thanks to larger and more accurate datasets and the improved computer models they enable.

Transport planners have been able to shift from passenger surveys and educated guesswork, to tracking individual passengers’ journeys, revolutionising our knowledge of how public transport networks function.

Big data has also allowed for completely new ideas. Researchers have looked into monitoring the spread of flu outbreaks through the language people use on social networks, for instance (though their attempts have not been entirely successful).

There is huge potential for data that is deliberately gathered for a specific purpose - for instance, air quality sensors feeding data on harmful nitrogen oxides into air pollution models. But in addition to this, datasets that were created for one purpose are now often put to work for another. Google Translate, for instance, was initially trained on the multilingual documents of the United Nations and European Union, rather than on a dictionary.

At the Challenge Prize Centre, we have a longstanding interest in the power of data to create socially useful innovations

We ran a series of Open Data Challenges, to leverage public-sector data for social good. One outcome of this was the FoodTrade Menu, a tool to help food professionals flag potential allergens to their customers.

We recently launched the multi-million pound Open Up Challenge, which uses financial transaction and product data to create tools that give customers greater choice in business banking.

In our current and forthcoming projects, we’re particularly interested in the potential of data-driven solutions to transform lives in the developing world.

Many governments still don’t know much about their people. Statistics and datasets are unreliable, incomplete or poorly understood - and this hampers their ability to efficiently address critical needs. We believe that innovators can help fill this gap with a smart use of data.

We’re not alone in this. Initiatives in this field just now include Ushahidi, a crowdsourced crisis mapping tool in Kenya; mobile phone data in the form of call records have been used to track mobility and social interactions, tackling issues like malaria in Kenya and emergency migration in Haiti; Kenyan farmers now have drought insurance paid automatically based on satellite data. Use of mobile phone airtime vouchers has even been used to estimate household income in developing countries such as Cote d'Ivoire.

In this context we’re currently involved in two challenge prize programmes which we hope will bring valuable data-driven insights to people working in agriculture in the developing world.

We’re working with USAID on the Data Driven Farming Prize in Nepal. This prize, which recently closed to applicants, is seeking tools and approaches that source, analyse and translate data into actionable, timely and context-specific information for smallholder farmers to improve value from agricultural productivity in Nepal.

We think that the solutions that are created as part of this will improve on the ground decision making of farmers. It should increase production from rice, cereal and vegetable farming, as well as other crops, while supporting environmentally sustainable growth in the commercial agriculture sector.

Further south, in India and Bangladesh, we’re leading a consortium designing a series of challenge prizes for the fish farming sector. Here, too, data is likely to play a role. This could be in providing smart recommendations to fish farmers on what varieties to grow, what feed to buy or what techniques to use, much like we’re hoping to achieve in Nepal.

We’re also exploring how open data, combined with new, cheap and easy accurate diagnostic tests, could monitor and control the spread of fish diseases. As fish farming becomes progressively more intensive, disease is spreading more easily, causing economic losses to farmers and harming productivity.

Data in and of itself does nothing: it’s what you can do with it that counts

It can empower you to make better decisions if you are equipped to use it: good data is essential, for instance, for policymakers and large businesses that employ economists and statisticians who understand it.

Good data can allow monitoring of results from initiatives and programmes: it tells us what works and what doesn’t, helping governments and businesses target their money where it will have a long lasting positive impact.

But in the field, data is only helpful if it is married with useful products and services that make sense of the data and translate it into recommendations that you can act on - that’s why these two programmes are focused as much on creating new tools that use data, as on creating new datasets.

There is clearly a pressing need for tools that can use and learn from data in smart, innovative and useful ways.

Top photo: large-scale fish farming in Andhra Pradesh, India - a sector in which Nesta is exploring the possible role of data and sensors. Credit: Olivier Usher

Bottom photo: the Data Driven Farming Prize team meeting farmers in Nepal. Credit: Eleonora Corsini