Anthropic's Self-Replicating AI Claim Overstated, Researchers Say
Anthropic researchers' experiment with AI models on Android TV boxes has been widely reported as showing self-replicating AI is months away, but a closer look at the results suggests the claim is exaggerated.
Widely reported claims that self-replicating artificial intelligence is only months away have been substantially overstated, according to a detailed examination of the experiment that sparked them. The research, conducted by two Anthropic employees, involved installing a small AI model across a network of low-powered Android TV boxes, but the results fall far short of the scenario that has captured headlines.
William Zhou and Daniel Freeman, both researchers at the AI company Anthropic, ran an experiment in which they installed a compact model called Gemma4-26B-A4B across 15 Android TV boxes — the kind of inexpensive device used to stream apps on a television. These gadgets are computationally weak, with small CPUs and limited memory, which is why 15 of them were needed to store a single model. Even then, the model produced output at just 0.39 tokens per second, roughly equivalent to 18 words per minute. That is about ten times slower than normal human speech.
The researchers argue that a model distributed across televisions, smart ovens and other internet-connected appliances could endlessly copy itself, creating a form of «digital slime mould» that clogs up a large share of such devices worldwide. However, when they attempted to make a model copy itself in this way, it failed. Three models could not set themselves up at all. A fourth, GLM-5.2, succeeded in four out of five tests, but only by setting up another model, Llama3.1-8B, which is about 100 times smaller. The pair contend this is a fair test because it shows a future smaller model with the same capabilities could set itself up, but this remains far from genuine self-replication. The claim that such a feat might be «months away» comes from extrapolating lines on a graph.
Beyond the technical objections, there is a more fundamental question: why would anyone want to create self-replicating software? The answer already exists, and it does not involve AI. Computer code that copies itself from one device to another is called a worm, and worms have existed for more than 50 years. In the 1970s, Bob Thomas and Ray Tomlinson at the US firm BBN — the latter also invented email — created a program called Creeper that could copy itself. It was not particularly complicated and merely output the message «I'M THE CREEPER: CATCH ME IF YOU CAN». Tomlinson paired it with Reaper, a piece of code designed to find and delete Creeper, making it the first antivirus software. It was not until about a decade later that such self-copying software was called a worm, after a term coined in the science-fiction novel The Shockwave Rider by John Brunner.
John Shoch and Jon Hupp at Xerox Palo Alto Research Center, arguably the Anthropic of its time, used the term in a 1982 paper to describe how an experimental worm «seemed to grow out of control», leaving «dozens of machines dead». The first truly malicious worm was deployed on to the internet in 1988 by Robert Morris, a student at Cornell University in New York state. Thousands of computers were infected and crashed, a significant chunk of the online population at the time, and Morris became the first person charged under the US Computer Fraud and Abuse Act.
Today, worms are deployed by cybercriminals to hijack computing hardware and make money. In 2017, the WannaCry ransomware attack was the result of a worm copying itself across computers and encrypting them, demanding payment in bitcoin to unlock. It infected more than 200,000 computers in a single day, including many in the UK's National Health Service, and caused an estimated $4 billion in economic losses, although only around $100,000 of that was people paying the ransom.
If worms can be that bad, surely an AI-powered worm would be even worse? According to Zhou and Freeman's experiment, an AI worm would find it much harder to replicate. It would take multiple days for their AI worm to copy itself just once. That does not mean such a worm could not scale to large numbers, but it would be slower, taking more than two weeks to achieve the 200,000 devices WannaCry managed in a single day. This is because even small models like Gemma4-26B-A4B are large files — around 14 GB for the version in the experiment, about 4,000 times larger than WannaCry — and these take time to transfer. That would make an AI worm much more noticeable, as network managers would be likely to investigate the source of such large data transfers and shut them down, something Zhou and Freeman admit.
What does an AI worm gain from all that extra data? Not a lot. Zhou and Freeman say that, assuming it could take over more than 2 million Android TV boxes, the computing power available would still be modest. The gap between the narrative and the reality, as ever, is considerable.
4
