Health care is one of the key areas machine learning is being used to support decisions. While algorithms can help make better clinical decisions, they are not completely objective in the way they make decisions but informed by both data and preferences. Geraint Lewis argues this has important implications for how we use and work with machine learning algorithms.

As a medical student, I was taught the “dark art” of prescribing a medicine called warfarin. Warfarin is used to prevent strokes and other complications but, rather frustratingly, its correct dose varies widely between different patients. Indeed, the right dose of warfarin often changes over time, even for the same patient. For that reason, a bit of sorcery is required to establish the correct dose: clinicians use a combination of a special blood test (called the “INR”) plus a combination of guesswork and intuition to adjust a patient’s warfarin dose up or down as needed.

It won’t be a surprise to hear that there is growing evidence that algorithms do a far better job at gauging the right dose than this finger-in-the-air approach. So, the NHS is increasingly using algorithms such as Heartbeat for warfarin prescribing. These algorithms make their recommendations based on factors such as the patient’s previous doses of warfarin and their impact on the INR blood test, the patient’s age, their physical build, other medications, alcohol intake, and so on.

Advances in machine learning mean that the accuracy of these tools is likely to improve considerably in the coming years, leading to improvements in the safety and effectiveness of care. However, there is an issue lurking inside these tools that we all need to be aware of.

On the surface, it would appear that decision algorithms and machine-learning tools are purely information-driven: they take historic information, merge it with current information (the patient’s latest test results) and process it to make a recommendation such as which treatment to choose or what dose to take). The reality, though, is that all decision tools actually make their recommendations based on a combination of information and preferences.

For patients taking warfarin there are hazards associated with taking too low a dose (e.g. danger of stroke and other forms of thrombosis) or with taking too high a dose (e.g. danger of haemorrhage). The question is whether the algorithm err on the side of giving too high a dose or should it err on the side of giving too low a dose? This preference between the risk of one side-effect over another is personal, and so the preferences must be made by the patient and their clinician, not pre-determined by the manufacturer of a machine.

Even more troubling is the potential for gaming within these tools. Warfarin itself is actually a very cheap drug, but there are stories of unscrupulous companies building loaded tools that have an in-built preference for prescribing more expensive treatments.

So my recommendation is that whenever you come across a new decision technology, you should ask yourself three questions: first, are the preferences the tool uses to make its recommendations clear (remember that all decisions are made on a combination of both information and preferences)? Second, are the weightings for these preferences explicit – can you see them? Finally, can the patient and clinician adjust these preferences to suit their personal values?

Unless the answer to all these questions is ‘yes’, the manufacturer’s preferences will be swaying the decision – and potentially swaying it in ways that suit the manufacturer’s interest rather than the patient’s.

Disclaimer: the views expressed in this blog are those of the author alone and do not necessarily represent those of NHS England.

This blog is part of the Shadow of the Smart Machine series, looking at issues in the ethics and regulation of the growing use of machine learning technologies, particularly in government.