Patients may have multiple health conditions, medications and/or a long medical history.

A primary care clinician (first point of contact for patients seeking medical care) told us "There are so many dimensions to health - you give medication for depression, but you don’t understand how that’s impacting something else (eg, their stomach)".

These patients are challenging because their symptoms may be associated with an existing health condition or a side effect of their current medication. Additionally, they may have multiple health providers preventing carers from having a holistic view of the patients' health.

AI tools are well-positioned to ingest large amounts of information (eg, medications, symptoms, medical history) and make recommendations or flag risks that can aid clinical decision-making

Some patients require specialised expertise or clinicians from several different specialities to work together to decide on the most appropriate care. 

This challenge was particularly common among primary care clinicians. They reported very limited access to expert advice and felt under pressure to make the right diagnosis and refer patients to the right specialist. Similarly, secondary care clinicians go through a slow process to seek advice from other specialists (eg, ‘up to three months for dermatology’), which can delay care for the patient. 

AI tools can improve access to rare expert knowledge, through smarter knowledge management and recommendation systems. 

Some biases or errors might be due to clinicians’ personal experiences while others might be the result of systemic failures, eg, biases in the training healthcare professionals receive.

We have all heard about the risks of biased AI systems, but humans also make assumptions or biased judgements based on their own experiences. A clinician explains that “Doctors may also have a bias when they are doing clinical judgements - someone walks in and I can see that they are homeless and make a judgement that a rash is a product of their lifestyle. This could mean ignoring an important symptom.”

Clinicians are under time pressure and need to make decisions quickly which can cause errors. They may also unconsciously introduce bias through judgments based on visual cues. This bias can lead to overlooking crucial symptoms, affecting the accuracy of diagnoses and potentially compromising patient care. 

AI tools could help counteract human bias to ensure fair and comprehensive assessments, fostering more equitable and effective healthcare outcomes.

Start by speaking to patients and healthcare professionals to identify the specific problems that they face. Discuss with users what challenges may be addressed with AI tools and the potential limitations. This can improve uptake of the tool in the long run.

Work out what users want to get out of the AI system early on. Aim to elicit user values during the early stages of the project to understand what ‘good’ looks like for them. For example, we learned that healthcare practitioners valued “understanding the confidence of a recommendation” the most.

Healthcare is a high-stakes domain, therefore it is crucial for clinicians to calibrate their trust in the system and account for errors. This insight could lead to changes in the AI model or the user interface.

Consider how the system will fit into or replace clinicians’ existing strategies and decision-making aids. Exploring how the tool could enhance the patient-clinician relationship or collaboration between healthcare teams means that the value and utility of the tool is multiplied. 

All technology development needs to go through multiple iterations before it can be deployed. Involving different people in testing and evaluating the tool could help to mitigate against the worst risks but remember that errors will still occur. Therefore,  design the system to provide for ongoing accountability mechanisms.