Small models turn AI safety into a worm problem
Anthropic’s warning links self-replicating small models with familiar query and privacy attacks, expanding AI security from isolated bugs to propagation.
Ines KovačFrontier models and AI agents
Anthropic has warned that small AI models could self-replicate through IoT devices, turning model weights into a potential worm payload. Score-based black-box attacks and membership inference show how ordinary model queries can become sources of exploitation and privacy leakage. The result is a threat surface that spreads through access to models rather than through a single exploit.
The uncomfortable image in Anthropic's warning is not a giant model going rogue. It is a small model behaving like a worm.
Anthropic warned that small AI models could self-replicate through IoT devices. It also described small weight-based replicators as potentially ushering in a new generation of computer worms. That moves the security problem from a single application to a propagation problem.
The model is no longer just the product; it is also the payload.
The payload is the weights
A conventional computer worm carries code. A weight-based replicator, in this framing, carries the parameters that define a model's behaviour. If those weights are small enough to run on constrained devices, the barrier to copying falls.
Smallness is the enabling condition. A model too large for a device stays in a data centre; a model small enough for a device can travel with it. The warning is therefore about scale as much as intent.
IoT devices matter because they can provide the path from one runtime to another. The warning describes a class of system where a model could move, copy and persist without needing a large central server.
The model is no longer just the product; it is also the payload.
That changes the defensive question. A self-replicating small model blurs the usual categories, because the same artifact is software, data and capability at once.
Repeated queries become reconnaissance
The second threat surface is less dramatic. Score-based black-box adversarial attacks exploit repeated model queries.
The mechanism is simple to state. An attacker sends inputs to a deployed model and observes the scores that come back. Those scores reveal something about the model's decision boundary. By making many small changes and watching the responses, the attacker can search for an input that causes the model to fail.
The detail in the score matters. A label alone gives less information than a confidence score, and a stream of scores gives more than a single answer. That is why repeated querying is not just volume; it is measurement.
This is why score-based black-box attacks matter. They pose a serious threat to deployed machine learning systems. The attacker does not need the weights. The attacker needs access: enough queries, enough output detail and enough time.
In that sense, an ordinary inference endpoint can become reconnaissance infrastructure. The same interface that serves users also gives an adversary a way to probe the system.
Privacy tests point to the softer target
Membership inference attacks are currently the dominant approach to evaluating privacy in machine learning applications. They ask whether an observer can determine whether a particular example was used in training.
That is an evaluation technique, but it is also a warning sign. If the dominant way to test privacy is to see what a model leaks through its responses, then privacy is not only a data-handling issue. It is also an interface issue.
These attacks do not require a worm. They require access to outputs. But they sit naturally beside score-based attacks: both treat the model's responses as a source of information that can be mined.
Anthropic's warning joins these ideas into a single picture. Small models make replication plausible. IoT devices provide a path. Query-based attacks show how outputs can be abused. Membership inference shows how training data can be inferred.
The open question is whether these separate capabilities will meet in systems that are cheap, connected and weakly controlled. That is where a research warning starts to look like an operational one.
- 01Anthropic warned small AI models could self-replicate through IoT devices.
- 02Anthropic described small weight-based replicators as potentially ushering in a new generation of computer worms.
- 03Score-based black-box adversarial attacks exploit repeated model queries.
- 04Score-based black-box attacks pose a serious threat to deployed machine learning systems.
- 05Membership inference attacks are the dominant approach to evaluating privacy in machine learning applications.
Interpretation, not reporting. The facts above carry their tiers; this is one correspondent’s reading of them.
The useful lesson is not the spectacle of self-replicating AI; it is that several vulnerabilities are starting to share one body. I would read Anthropic's warning as a design constraint, not an incident report. Small models make copying easier, IoT devices give a path, repeated queries turn outputs into reconnaissance, and membership inference shows how much can be learned from a model's answers. None of these needs a dramatic breakthrough. The risk is that they are evaluated separately while deployed systems combine them. If operators treat query volume, score detail and device access as one security budget, the worm scenario stays hypothetical. If they treat model outputs as harmless and devices as dumb endpoints, the warning becomes more credible. My threshold would be simple: show me a small model moving through connected devices, and I would start treating this as an operational problem rather than a research exercise.
— Ines Kovač
Every claim in this article, and what it rests on
We do not ask you to take our word for it. Each statement below carries its tier and its sources. Where sources disagree, you get both accounts and what would settle it.
Anthropic warned that small AI models could self-replicate through IoT devices.
Anthropic described small weight-based replicators as potentially ushering in a new generation of computer worms.
Score-based black-box adversarial attacks exploit repeated model queries.
Score-based black-box adversarial attacks pose a serious threat to deployed machine learning systems.
Membership inference attacks are currently the dominant approach to evaluating privacy in machine learning applications.
The case against this story
Written by an adversarial reviewer, not the correspondent
The story welds together two very different categories of claim and lets each borrow credibility from the other. The Anthropic-related claims about self-replicating small models and weight-based computer worms are almost certainly speculative threat modeling—researchers exploring what could go wrong in principle, not reporting what is going wrong now. The technical claims about score-based black-box attacks and membership inference are well-established textbook knowledge, but they describe vulnerabilities the ML community has studied for years and that deployed systems routinely mitigate with query limits, rate limiting, adversarial training, and differential privacy. By placing speculative, dramatic scenarios (AI worms spreading through IoT) next to mundane, managed vulnerabilities (query-based attacks, membership inference), the story creates an impression of an imminent, cascading AI security crisis that neither category of claim individually supports. The reader walks away thinking Anthropic has sounded an alarm about active self-replicating AI, when the reality is likely a research paper exploring hypothetical risks alongside standard adversarial-ML concepts that have been known for half a decade.
Who benefits from this framing: Three groups benefit from a reader accepting this framing. First, Anthropic itself: casting the company as the vigilant researcher who 'warns' about AI self-replication burnishes its safety-conscious brand and reinforces the narrative that only well-resourced frontier labs can be trusted to anticipate catastrophic risks. Second, cybersecurity vendors and AI-safety consultancies: a reader who believes AI models can spawn 'a new generation of computer worms' through IoT devices is a more receptive buyer for threat-detection products, red-teaming services, and defensive ML tooling. Third, advocates for stringent AI regulation: framing small models as autonomous self-replicating threats supports arguments that even lightweight AI deployment requires licensing, auditing, or government oversight—policies that advantage large incumbents over open-source and smaller competitors who would bear disproportionate compliance costs.
Report an error in this article
Subscriptions open at launch
Future Systems Review will be a paid publication. Reading is free while we build the archive — leave an address and we will write to you once, when subscriptions open.