As the boom of text-to-image generative artificial intelligence (AI) continues, artists are threatened not only by cheaper, AI-fuelled competition but by the risk that their work is scraped from the web and used to train AI models, without consent or compensation. Recently, some artists have started to fight fire with fire, trialling ‘data poisoning’ as a technical solution: deliberately inserting corrupted or malicious data into the training material for AI models to prevent its proper function. Could this presage the bust to follow AI’s boom?

In October 2023, University of Chicago academics revealed a new tool for artists to imperceptibly ‘poison’ AI models which seek to exploit art published online. Nightshade makes very subtle changes to the pixels and attached data (metadata) of an image. These changes affect the parts that only machines can ‘see’, distorting the associations between images and their descriptions. This pollutes AI models' ability to generate accurate new material in response to a prompt.

Nightshade is a more targeted version of Glaze, the team’s original, award-winning data-poisoning tool, which was released in February 2023 following an approach from artists, including campaigner Karla Ortiz. Glaze works defensively by applying a ‘style cloak’ to prevent AI models ‘reading’ artists’ personal styles. For example, a charcoal drawing that humans see in a realist style might be interpreted by a machine as a modern abstract piece, meaning that when the machine is prompted to reproduce something in the artist’s style, the machine reproduces the wrong style. As of late August, Glaze had clocked over one million downloads. The emergence and popularity of these tools show that artistic communities faced with the risk of replacement by powerful AI image generation, including ‘style mimicry’, are serious about fighting back.

Data poisoning isn’t a new concept. It’s long been the fear of developers whose data-hungry models rely on masses of open-source material.

Computer scientists have been proving the technical feasibility of data poisoning for several years. In 2019, a study by Chinese tech firm Tencent showed how data poisoning could influence the behaviour of Tesla autonomous vehicles, for example, turning windscreen wipers on without rain, or driving outside lane markings. Earlier this year, some of the leading minds in AI demonstrated that large-scale poisoning of some large-language models can be done relatively cheaply by anyone with access to the internet.

Before that, the corruption of Microsoft’s Twitter chatbot Tay in 2016 was a different, more rudimentary form of data poisoning. Tay was designed to learn through interactions with other Twitter users; when trolls coordinated efforts to inundate it with explicit, racially abusive and misogynistic language, this ‘poisoned’ its training data, resulting in Tay adopting similar language.

Glaze and Nightshade are a different kind of beast. Their emergence suggests that as generative AI programs continue to spread, we could start to see more examples of technically complex, synchronised instances of data poisoning. These could be much slicker than a loosely coordinated group of trolls, and more destructive than controlled research experiments.

People might sympathise with artists using data poisoning techniques to protect their livelihoods, at least until there is more consensus and regulation of the ethical use of images and other data taken from the internet.

However, the weaponisation of AI training data in other arenas could have significant negative impacts on people’s lives. The prospect of widespread AI failure is alarming as these technologies are becoming increasingly integrated into the systems and services on which we rely. This is particularly the case where the integrity of training data has implications for health and safety for example, in the NHS, policing or even the military.

If data poisoning is indeed the ‘next big threat’ in cybersecurity, we can expect industry to devote a lot of energy towards mitigations in 2024. These could range from counter-attacks and poisoning detection tools (already being termed ‘antidotes’), to the use of closed models that don’t rely on open-source data that could be manipulated. Instead, tightly controlled ‘synthetic data’ might be used to feed models with safe, curated training material.

This brings its own risks, not least the possibility that over time AI might just poison itself. ‘Model collapse’ is when an AI model gets stuck in a loop of imitating its own increasingly narrow, circular outputs. This is a particular risk with over-reliance on synthetic data, which could result in closed systems that eventually distort and become too removed from reality. It could also become a problem for open-source models in a world where so much AI-generated content has saturated the internet that the bulk of training data for new models is AI-generated.

Poisoning tools like Nightshade are in their very early stages. We might yet see cat-and-mouse development of more potent poisoning tactics and defences, or a change in tactics. Artists don’t necessarily need to bring down AI, if they can worsen the quality of AI’s output enough to keep human artistry valuable. Even temporary deterioration buys time to pursue greater legal protections for artists, campaign for regulation and ethical use of AI, and develop new business models for artists to earn a living.

There are also plenty of artists who see opportunity in AI, with calls for playfulness over fear, eminent professors calling on students to embrace the tech in creative practice, and conversations about how the technology could improve inclusivity and access. Perhaps artists can find reassurance from historical fears that photography, film and then digitisation, would each be the death of art and downfall of artists. So far, none of them have been. Expect developments in the relationship between AI and artists in 2024 - the question is whether this will entail collaboration or conflict.