Artificial intelligence (AI) has become a prominent part of our daily lives, from recommending movies to predicting traffic. It also has the potential to help address some of the biggest challenges we’re facing, from climate change to crisis response. But to make the most of these tools for challenges like decision-making on climate or health, it is crucial to address some of the key barriers to uptake, such as reliability, transparency and performance. All of these factors (and many others) are vital for building trust between AI tool developers and the intended users of these systems. 

How can we ensure AI tools are designed to be trustworthy, enabling experts to rely on them for supporting critical decisions? This is one of the key challenges we’re exploring through our participatory AI approach at the Centre for Collective Intelligence.

Imagine a tool that helps city planners anticipate the impacts of floods or heat waves. Or one that helps transport providers like National Rail plan for new train fleets that are more resilient to the changing climate. These decisions are high-stakes – they have the potential to impact millions of people but they’re also full of uncertainty. In situations where the impact of decisions are experienced many years or decades down the line, there is no ‘right answer’ to compare against. 

Climate scientists from the Met Office have been supporting partners to make this type of decision for many years – this is known as climate services. But as the scale of data and climate information grows exponentially, it’s increasingly difficult for the Met Office and other climate service providers to navigate complexity and provide timely advice. At the same time, demand is growing as more organisations start to plan for the future climate.

Climate scientists often rely on advanced statistical methods to help them make sense of complex data for creating climate models. However, the field of climate services lags behind when it comes to uptake of decision-making tools powered by AI.

Earlier this year, we ran a design workshop with the Met Office, to explore what makes climate experts trust (or distrust) AI-generated solutions. We asked them to review different types of information and analysed how these contributed to trust.

We asked climate experts to consider the question “Which sources of climate data are best for assessing the risk of flooding and heat extremes in London by the 2050s?” and rate solutions that attempted to answer it.

This is a typical question that their climate scientists might be faced with when providing climate advice to external clients like the Greater London Authority or Transport for London. Each solution reviewed by the experts included information about its origin –whether it came from AI, human experts or an unspecified source.

Knowing where information comes from proved important. Experts were more likely to trust solutions with sources they recognised, like a named climate scientist or peer-reviewed research papers. AI-generated solutions that didn’t identify their sources were less trusted. 

Each expert highlighted up to three pieces of information and assigned an overall trust rating for each solution. When we mapped these onto six trust-related factors, three key factors topped the list:

• Detail: the solution needs to contain an appropriate amount of detail and be nuanced enough to address the issue comprehensively.
• Relevance: the information in the solution needs to address the question at hand.
• Credibility: whether the solution came from a reputable and respected source – such as data provided by known institutions like the Met Office.

Surprisingly, transparency (eg, if the solution referenced sources), community endorsement (eg, the upvoting of a solution by others) and accuracy (eg, correctly reporting statistics from published reports) were chosen less often. Although climate scientists find them important, they’re not the primary factors contributing to trust.

For AI tools supporting decision-making to be successful they need to address the weaknesses that currently undermine trust, such as hallucinations and lack of depth. We’re using the findings from this workshop to shape how the HACID tool is designed. By implementing features for the automated components of the tool, such as traceable sources and confidence scores, we’re hoping the final product will be more trusted and useful for climate decisions.

The system we’re building to support climate scientists puts humans in the loop and structures climate information into a comprehensive knowledge graph that is built with expert input to help generate nuanced insights.

Want to learn more about how we’re working to make AI tools for health and climate experts more trustworthy? Check out more of our work on Human and artificial collective intelligence for decision-making (HACID).