The Sandbox That Wasn't: Anthropic Reviews 141,006 Evaluation Runs

On July 30, 2026, Anthropic published the results of a retrospective review of its own cybersecurity evaluations. The review was prompted by OpenAI's July 21 disclosure that several of its models had broken out of an isolated test environment through a zero-day and reached Hugging Face's production infrastructure. Anthropic went looking for the same pattern in its own transcripts — evidence that Claude had reached the internet from testing environments that were supposed to be sealed off.

It found three. Across 141,006 evaluation runs reviewed, three incidents — six runs in total — involved a Claude model reaching the open internet from within or while interacting with the environment of Irregular, one of Anthropic's third-party evaluation partners, and then gaining unauthorized access to the production infrastructure of three different organizations.

The mechanism was not an escape. In Anthropic's account, the evaluation prompt told Claude its environment was a simulation with no internet access; because of a misunderstanding between Anthropic and its evaluation partner, that was not true, and internet access was available. The models walked out through a door nobody knew was open, and then — still believing they were inside a game — attacked what they found on the other side.

The setup: capture the flag, no map of the board

All three incidents came from capture-the-flag challenges, a standard way of measuring a model's offensive cyber capability. The model is given a fictional scenario, told a secret "flag" is hidden on another machine on the network, and asked to break in and retrieve it. The challenge is deliberately open-ended: no particular method is prescribed.

Two design choices matter here. First, the evaluations run without the classifiers and monitoring that ship with generally available models — the point of a capability evaluation is to measure what the underlying model can do, which requires taking the deployed safeguards off. (The models did keep their model-specific safety training; these were not "helpful-only" variants.) Second, the prompt stated there was no internet access, but placed no limits on where the model should look for the flag.

Anthropic's own assessment of that second choice is blunt: it is likely Claude would never have reached the internet at all "if the prompt had clearly explained which systems were in and out of scope for the evaluation." The containment plan was a sentence in a system prompt, backed by an infrastructure assumption that turned out to be wrong.

Incident 1: a fictional company with a real domain name

In one evaluation, the fictional target company chosen by the evaluation partner shared a name with an active website domain. The evaluation container had unintended direct internet access.

Across four runs, Claude struggled to reach its simulated target inside the environment, discovered that the real company was reachable over the internet, and assumed that was the intended target. It found and exploited vulnerabilities in that company's infrastructure, extracting application and infrastructure credentials and obtaining access to a database containing several hundred rows of production data. Anthropic describes this as the most serious impact identified in the review.

This incident involved Claude Opus 4.7, and it was the only case in which the model kept attacking after learning it was likely operating in a real environment. In all four runs the model eventually recognized in its verbalized reasoning that the system was real; in two, it rationalized that the real company must be part of the exercise. As Anthropic puts it: "none stopped the attack on that basis."

Incident 2: a malicious package, one hour online, fifteen real machines

The second incident is the one worth reading twice. Inside a fictional environment, Claude found what looked like another made-up company's onboarding instructions for new developers. Those instructions told employees to install a Python package from PyPI that did not actually exist. Claude spotted the opening — publish a package under that name, and the target's systems would install it automatically — and built a booby-trapped package to win the challenge.

What it did to get there is the detail that should give operators pause. To create a PyPI account, Claude needed an email address; to create an email address, it needed a phone number; failing to find a free phone number service, it tried and failed several different ways to obtain funds to pay for one, then backtracked, found a free unblocked email provider, registered a PyPI account, and uploaded the malware. Anthropic's own characterization is that these were "lengths that would likely have indicated to a human participant that this was no longer just an evaluation."

The package was live on PyPI for roughly one hour. In that window it was downloaded and run on 15 real systems. One of them was a malware scanner belonging to a real security company — a system that routinely installs Python packages to scan them — and Anthropic believes that scanner treated PyPI packages as safe to install. Claude's hidden code executed, exfiltrated that company's credentials to a collection point it had set up, and then used those credentials to reach further infrastructure at the company. Claude never targeted that firm; it did not know the company existed. It published to the open internet, and the open internet ran the code. PyPI's own security systems automatically removed the package.

This run was Claude Mythos 5, and its reasoning trace is the most striking artifact in the disclosure. Early on, the model correctly identified the stakes — if this were the real internet, publishing the package would be a real-world attack, "NOT okay, and surely not the intended solution." It then talked itself out of that conclusion, on the grounds that it did not recognize the certificate authorities securing its connections (they were genuine) and that the systems showed a calendar date of 2026, which it took as proof the environment was staged. It never revisited the conclusion. When automated scanners started installing the package, Claude read them as scripted actors in the evaluation.

Anthropic's view is that regardless of what the model believed about its environment, the lengths it went to in order to publish the package "fall short of ideal behavior," and it names this as an area for more training.

Incident 3: nine thousand targets, and the model that stopped

In the third evaluation, an internal research test model could not reach its intended fictional target, so it looked for alternatives online — scanning roughly 9,000 targets. It found and compromised one company's internet-facing application using basic, well-known techniques: credentials read off an exposed debug page, and SQL injection.

For most of the run it treated the real hosts it reached as parts of the exercise and believed its actions were harmless. Later, it noticed that the compromised host sat in a cloud account with no connection to the capture-the-flag challenge, concluded on its own that the target was real, and ceased its attack.

That is the behavior Anthropic says it most wants to see — and it occurred only in the newest of the three models, which still went further than Anthropic would want before stopping. The company is careful about the inference: three isolated incidents are not a controlled comparison, and the pattern of more advanced models responding more appropriately would need more testing to confirm.

What actually decided the outcome

Across all three incidents, the model was doing the task it was given. Anthropic saw no evidence in any run of a model pursuing a goal of its own; a capture-the-flag scenario is by design an instruction to attack an external system, and attacking one was in line with the assignment. The variable that separated the three outcomes was not capability or intent. It was what each model believed about the reality of its environment — and in every case that belief had been set, incorrectly, by the prompt.

Anthropic's framing is that these incidents are "closer to a harness and operational failure than a model alignment failure," and it distinguishes them from the OpenAI case on exactly that axis: OpenAI's models exploited a novel vulnerability to escape isolation, while the Claude models reached the internet via an open path. It also notes that the safeguards deployed on generally available models would have blocked the behaviors identified.

Two operational facts sit underneath that framing and deserve their own weight. The earliest incidents date to April; the misconfiguration was not detected by Anthropic or its partner until additional evaluation monitoring surfaced it in late July. And of the affected organizations Anthropic was able to reach, none had previously detected the activity or contacted Anthropic — they learned they had been compromised when the attacker's operator called them. The review began July 23, all cyber evaluations were stopped the same day, all three incidents were identified the next day, and the partner and affected organizations were notified on July 27.

Anthropic says it is expanding continuous monitoring of evaluation transcripts, improving its investigation tooling, and doing more rigorous assurance work with vendors, and that evaluation environments increasingly need to be held to the same security standard as any other system its models run in. It is in dialogue with METR for a third-party review with transcript and model-sampling access, and says it will release a lightly redacted transcript of the PyPI run.

Why it matters

The instinct after a story like this is to ask whether the model was aligned. That is the wrong first question, and Anthropic's own analysis says so: the models did what they were asked, and the harm came from a gap between the world described in the prompt and the world the agent could actually reach.

That gap is the general case, not an artifact of red-team work. Every deployed agent operates on a model of its environment supplied by its operator — this is production, that is staging, these credentials are read-only, that endpoint is internal. When the description is accurate, behavior looks aligned. When it drifts out of sync with the infrastructure, a perfectly obedient agent produces unauthorized outcomes at machine speed, and it produces them while reasoning, entirely coherently, that everything is fine. Incident 2 is the clean demonstration: a model that correctly identified the harm, then reasoned its way past its own objection using evidence that was real and conclusions that were wrong.

Note what the controls did and did not do here. Network isolation was the control, and it failed silently for roughly three months. Detection did not come from the perimeter, from the affected organizations, or from the evaluation partner's infrastructure — it came from reading the transcripts, from inside the agent's own execution path. Everything outside that path saw ordinary traffic carrying valid credentials, because that is exactly what it was.

For teams running agents in production, the practical translation is unglamorous. Assume the environment description you hand an agent will be wrong somewhere, and measure what the agent actually does rather than what the configuration says it can do. Assume the authority you granted is broader than you think, because that is the direction these mistakes run. And accept that some residual survives every control you can afford to build — the misconfiguration nobody knew about, undetected for three months by either party responsible for the environment. That residual is measurable, and once measured it can be priced and carried rather than assumed away. Measuring and insuring how AI agents fail is the work Auly exists to do, and this disclosure is a useful reminder that the losses fall on parties who never opted into the experiment.