A core selling point of machine learning algorithms is that they are adaptable: they improve in light of new data. But when used in real-world scenarios, this feature can have complex effects: when their predictions are used to make decisions, they aren’t just predicting the future— they’re also changing it, and then learning from those changes. Michael Veale argues that we need better tools to measure and understand the real-world impact of machine learning if we are to see the full benefit of their use and, to do this, computer science will need the help of the social sciences.

Most machine leaning algorithms understand the world through example or training data. The algorithm ‘learns’ to predict the correct outputs by modelling the relationship between the inputs (e.g. personal, financial characteristics) and outputs (e.g. defaulted on a loan or not) in the training data. We then test its predictive performance with data it hasn’t seen before—how much does it get right? This gives a good snapshot of performance.

But in practice, algorithms don’t exist in snapshots: their environments are constantly changing. When algorithms make decisions, they affect these environments, which in turn affect new inputs—a feedback loop. Decisions over important areas such as insurance, employment, health, and our perceptions of the algorithm, will influence the way we act, spend or write. Guides to improving your credit score attest to this.

Snapshots of performance ignore these broader systems. Hyped algorithms such as Google Flu Trends, an algorithm which guesses at illness hotspots by watching patterns in searches for symptoms, have been foiled by ‘unexpected’ changes in our searching habits and styles.

Consider predictive policing, currently being piloted by forces in London and around the world. These algorithms take a punt at tomorrow’s urban crime hotspots, hoping to enable more effective deployment of officers. This might seem like an easy win, but in connected systems such as crime, things are never that easy.

Letting algorithms govern policing could have a wide impact on: people’s day-to-day expectations of security, the behaviour of patrolling policemen and their managers, or the strategies of criminal organisations that might want to ‘game’ the system. Recent research already shows how we might exploit the ways machine learning algorithms ‘think’.

Using algorithms may shake-up whole systems of policing. What happens to the knowledge tied up in the current systems used to set deployment patterns? Is the system reliable in the face of new data, new areas, or following a serious shake-up or crisis? There are reports of officers in Chicago intimidating people who have done nothing wrong, simply because they have come up on an algorithmically-created ‘heat list’ designed to identify those most likely to commit a crime in the future. Society sees crime reduction as a much more nuanced goal than an algorithm’s optimisation-mindset does.

This doesn’t mean we should give up on using algorithms to achieve societal aims. But it compels us to invent better ways of measuring and understanding the algorithms’ impact over time, and their interaction with the social, technical and environmental systems we connect them to. Currently, we really lack useable tools to do this. Algorithmic analysis in the computer sciences is narrowly focused on computational performance- on speed and narrowly defined accuracy, without proper regard to whether they have the intended affect on society. Meanwhile algorithmic analysis in the social sciences generally considers algorithms as foreign, distant ‘black boxes’.

Machine learning is tougher to understand than older decision-making systems, but it’s not impossible. Better performance often comes at the expense of interpretability — our own human brains are an extreme example. But still, the real unknowable ‘black box’ is often smaller and less scary than we imagine. Connecting technical expertise to social expertise can help whittle it down to a more manageable size, and place it within our understanding of social systems to think of it in a more lucid way. The social and the technical have to work together if we want to really understand algorithmic impact.

Making holistic, usable tools for algorithmic impact assessment is my current research focus at UCL STEaPP. Don’t expect panaceas or one-size-fits-all tools—there is no magic button that makes socio-technical algorithmic systems easy to manage. But I am confident we can make better, integrated methods to assess the effects of algorithms in society. But that leaves another question—who will we put them in the hands of?

This blog is part of the Shadow of the Smart Machine series, looking at issues in the ethics and regulation of machine learning technologies, particularly in government.