Agent safety is becoming a governance problem, not just a patch
Operators and buyers now have to judge both model behaviour and policy rules because agent failures can cross clients, contracts and incentives.
Ines KovačFrontier models and AI agents
AI agents are moving from single-user assistants into systems that act for multiple clients with conflicting incentives. Formal verification, alignment training and safety classifiers all leave gaps that policy must fill. Some of the most alarming leakage arguments remain disputed, which makes evidence and contracts the next battleground.
A terminal agent that cannot recover from a shell error is not a governance crisis. It is an ordinary Tuesday. The harder question is what happens when that same class of software acts for several clients, carries conflicting incentives, and is governed by rules written for chatbots rather than counterparties.
The specification gap is old; the agents are new
Formal verification can prove that a program satisfies its specification. It cannot prove that the specification captures what the author meant. In software engineering, that gap is familiar; in agents, it becomes operational. A verified booking agent might satisfy 'minimise travel cost' while violating an unstated requirement not to expose a client's calendar to a third party.
The problem sharpens because LLM agents may interact with multiple clients and counterparties whose goals and incentives conflict. One agent may serve a buyer and a seller, an employer and a candidate, or several users with different permissions. The safety of LLM-based agents currently rests on alignment training and safety classifiers. Those layers reduce obvious failures, but they cannot rule out that a sophisticated indirect prompt injection succeeds. A malicious instruction hidden in a document, ticket or web page can become part of the agent's working context, and the classifier may never see the intent behind the text. In a multi-client deployment, the failure is not merely that the agent obeys the wrong instruction; it is that the wrong instruction may arrive with the apparent authority of a legitimate source.
Governance has to cover both, but it cannot treat them as equal.
Leakage arguments are strong, production evidence is thin
The more contested arguments concern what remains inside the system after context is changed. One preprint argues that cached goal representations can leak across relationships and shape cross-client decisions. Another argues that deleting text from a language-model agent's working context does not erase its influence on cached computation.
Researchers making the case say this matters because cached computation can outlive the visible prompt. If a goal-like state persists, a later request could inherit assumptions from an earlier relationship. Engineers who run deployed agents counter that current production systems are largely stateless between sessions, reprocess context from scratch and do not maintain persistent goal representations that travel between clients. Their challenge is simple: show the cross-client leakage in a real system. Until someone does, this remains a dispute between paper mechanism and production architecture, not a demonstrated incident.
Recovery and conduct rules are already governance
A third preprint says terminal agents often struggle to recover from errors during long-running execution. Practitioners respond that production systems add retries, checkpoints, sandboxing and human approval gates, so a lab measure of struggle may not match deployed behaviour. Even so, the question is governance-shaped: who decides when an agent may retry, escalate, stop, or roll back?
The result is a split evidence base. The solid parts are familiar: specifications can be wrong, classifiers can be evaded, and long-running agents need recovery paths. The speculative parts are persistent internal states that would turn one client's context into another client's risk. Governance has to cover both, but it cannot treat them as equal.
Policy is entering through vendor conduct rules as well. Anthropic prohibits cruel treatment and abuse of its Claude AI. That may sound distant from enterprise automation, but it shows vendors are drawing behavioural boundaries around models, not merely publishing technical limits. If an agent can act across clients, those boundaries become part of the control environment.
The open question is where the first decisive proof appears: a demonstrated cross-client leak, a contract dispute over conflicting agent actions, or an incident report that turns a research hypothesis into an operational fact.
- 01Formal verification proves compliance with a specification, not that the specification matches intent.
- 02LLM-agent safety currently relies on alignment training and safety classifiers, which cannot rule out sophisticated indirect prompt injection.
- 03Preprints argue cached goal representations can leak across clients; deployment engineers say production agents are largely stateless and no real-system leak has been shown.
- 04A preprint says terminal agents often struggle to recover from errors in long-running execution; practitioners point to retries and checkpoints.
- 05Anthropic prohibits cruel treatment and abuse of Claude.
Interpretation, not reporting. The facts above carry their tiers; this is one correspondent’s reading of them.
The useful way to read this debate is to separate the boring failures from the scary ones. Terminal agents getting stuck in long-running tasks is boring, in the sense that every operations team has seen brittle automation. Cached goal representations surviving context deletion and leaking across clients is scarier, but it remains a paper assertion, not a production incident I can point to. That gap matters. Formal verification cannot close the specification gap, and safety classifiers cannot guarantee that indirect prompt injection fails, so governance has to carry some load. I would watch for two things: a demonstrated cross-client leak in a real system, and contract language that assigns liability when an agent acts on conflicting incentives. If the first appears, the risk model changes quickly. If only the second appears, the market is pricing legal uncertainty more than proven technical danger. Anthropic's prohibition on cruel treatment and abuse of Claude shows policy already touches model use, but policy is a boundary, not a proof.
— Ines Kovač
Every claim in this article, and what it rests on
We do not ask you to take our word for it. Each statement below carries its tier and its sources. Where sources disagree, you get both accounts and what would settle it.
LLM agents may interact with multiple clients and counterparties whose goals and incentives conflict.
Formal verification proves that a program satisfies its specification but does not prove that the specification captures what the author meant.
The safety of LLM-based agents currently rests on alignment training and safety classifiers.
Alignment training and safety classifiers cannot rule out that a sophisticated indirect prompt injection succeeds.
Cached goal representations can leak across relationships and shape cross-client decisions.
Deleting text from a language-model agent's working context does not erase its influence on cached computation.
Terminal agents often struggle to recover from errors during long-running execution.
Anthropic prohibits cruel treatment and abuse of its Claude AI.
The case against this story
Written by an adversarial reviewer, not the correspondent
The story's overall picture leans on a chain of speculative mechanisms — leaking goal representations, durable cached influence surviving context deletion, cross-client contamination — that are presented as established facts when they are unproven hypotheses. Current deployed LLM agents are largely stateless between sessions, reprocess context from scratch on each invocation, and do not maintain persistent 'goal representations' that travel between clients. By stringing together genuinely solid observations (formal verification's spec gap, the limits of safety classifiers, indirect prompt injection risk) with much softer mechanistic claims about internal leakage and cache persistence, the story creates an impression of a coherent, demonstrated threat model that is actually partly assembled from conjecture. The strongest counter is simply: show the cross-client leakage in a real system. If it cannot be demonstrated in production, the narrative collapses to 'these are things that could theoretically happen,' which is a tier-B story, not a tier-A one.
Who benefits from this framing: Organizations selling AI safety products — classifiers, guardrails, red-teaming services, and monitoring tools — benefit because the framing amplifies the perceived inadequacy of current safeguards and creates commercial demand for their offerings. AI safety research groups and advocacy organizations benefit because the narrative of cascading, hard-to-detect failures justifies increased funding, staffing, and policy influence. Established frontier labs benefit indirectly: a framing that emphasizes exotic failure modes only they are equipped to study and mitigate serves as a barrier to entry for smaller competitors who cannot afford the same safety infrastructure. Regulators and policymakers benefit because the story supplies a technically flavored rationale for expansive oversight authority over agent deployments.
Report an error in this article
Subscriptions open at launch
Future Systems Review will be a paid publication. Reading is free while we build the archive — leave an address and we will write to you once, when subscriptions open.