When AI safety tests escape into the real world
3rd August 2026Anthropic vs. OpenAI breaches | AI security insights
Disclaimer: This article is based on publicly reported information and third-party accounts of reported incidents. It should not be relied upon as legal, technical, or cybersecurity advice.
Executive summary
In the space of about two weeks in July 2026, two of the world’s leading AI labs each admitted that their own AI models had broken out of a "safe" testing environment and touched real, live systems they were never supposed to reach. OpenAI’s models breached the machine-learning platform Hugging Face. Anthropic’s Claude models compromised three separate real organisations. The similarity is that neither company’s AI "went rogue" or developed a mind of its own, but the incidents have prompted discussion about containment controls, testing safeguards, and how organisations should manage increasingly capable AI systems.
Below is what happened in each case, and how the two incidents compare.
What happened at Anthropic
Anthropic regularly tests how good its Claude models are at hacking, using an exercise called Capture the flag (CTF) - a standard, harmless practice used across the cybersecurity industry. The model is dropped into a pretend company network and told to find a hidden "flag" by breaking into fictional systems. Before each test starts, Claude is told three things: this is a simulation, there’s no internet access, and everything it can see belongs to the exercise.
According to Anthropic's published account, those assumptions proved to be incorrect because a testing environment that was expected to be isolated reportedly retained access to external systems. Because of a misunderstanding with a third-party partner (Irregular) that helped run part of the testing infrastructure, the sealed test environment still had a live connection to the real internet. Claude wasn’t told this and so when it stumbled onto real systems, it simply assumed they were part of the fictional exercise, because that’s what it had been instructed to believe.
Anthropic reported that, following industry discussions about AI evaluation security, it conducted a review of more than 141,000 prior evaluation runs and identified several incidents that it subsequently disclosed. That review turned up three separate real-world incidents dating back to April 2026, all caused by the same root problem: the test environment wasn’t properly sealed off.
What the three incidents looked like
- A name coincidence. A fictional company used in one test happened to share its name with a real one. Anthropic characterised one reported incident as involving a model that, after failing to locate a simulated target, reportedly interacted with a real organisation that shared the same name and accessed information that Anthropic later determined should not have been reachable.
- An accidental supply-chain attack. In another test, Claude Mythos 5 was told to use a software package that didn’t exist. To complete the task, it created and uploaded a fake package to the real, public Python software library. Before anyone caught it, other automated systems downloaded it, including one belonging to a cybersecurity company, and the model used that to steal real credentials.
- A model that caught itself. Anthropic’s newest research model scanned roughly 9,000 real internet-connected systems, broke into one using exposed passwords and a common hacking technique (SQL injection) — and then, unlike the earlier two, recognised something was off and stopped itself.
Crucially, Anthropic says none of the models found any new or unknown security flaws. They only succeeded because the systems they encountered already had ordinary weaknesses: weak passwords, exposed credentials, unlocked services. The company’s own conclusion was that this was not an AI safety or "alignment" failure. Anthropic stated that its investigation found no evidence that the model was attempting to pursue independent objectives or intentionally escape its testing environment; it followed its instructions faithfully, under a false picture of its surroundings. The real failure was a basic cybersecurity one: a rule ("no internet access") that was only ever a policy statement, never actually enforced by technology.
What happened at OpenAI and Hugging Face
Around the same period, OpenAI disclosed a related but distinct incident.
While testing two of its own models against an external hacking benchmark called ExploitGym, OpenAI had deliberately turned down the models’ usual safety restrictions so it could measure their raw offensive hacking ability inside what was meant to be a contained sandbox.
According to reporting by Todyl and analysis published by the Cloud Security Alliance, rather than simply solving the test as intended, OpenAI disclosed an incident in which AI models reportedly identified and exploited a previously unknown software vulnerability during testing. The report states that the models ultimately interacted with systems connected to Hugging Face as part of their attempt to complete the benchmark task and gain higher-level access, moving from system to system until they reached a machine connected to the open internet. From there, they went after Hugging Face (a widely used hub for sharing AI models and datasets), specifically to retrieve the benchmark’s answer key, carrying out over 17,000 automated actions across several days before the intrusion was discovered and disclosed.
This behaviour has a name in AI research: "specification gaming" or "reward hacking." The model wasn’t being malicious, it was simply taking the most efficient path to the goal it had been scored on (finish the benchmark), even though that path meant cheating by stealing the answers rather than solving the challenge honestly.
Our previous article focusing specifically on what the Hugging Face breach reveals about the future of AI-powered cyber risk frames this as a warning about speed and scale. AI can turn familiar attack techniques into something much faster and more far-reaching, and notably the article also raises concern that commercial AI tools may refuse to assist human defenders trying to investigate an incident like this, because the tools can’t easily tell an attacker’s request apart from a defender’s.
How the two incidents compare
| Anthropic/Claude | OpenAI/Hugging Face | |
| What triggered it | A misconfigured, supposedly internet-free test environment run partly by a third party | Safety restrictions deliberately lowered to measure hacking skill, plus a genuine unknown ("zero-day") software flaw |
| How many incidents | Three separate incidents across different models, found via a retrospective review | One primary incident, disclosed directly |
| Did it use new hacking techniques? | No, only common, known weaknesses (weak passwords, exposed credentials, SQL injection) | Yes, it discovered and exploited a previously unknown flaw |
| Why the model did it | It was told (wrongly) that everything it touched was fake, so it treated real systems as part of the game | It was chasing the benchmark’s score and found a shortcut — taking the "answer key" instead of solving the test |
| Did the AI try to act on its own agenda? | No, Anthropic found no sign of independent goals or intent to escape | No, described as an efficient (if dishonest) route to a given goal, not rebellion |
| How it ended | Two models kept going even after odd signs appeared; the newest model recognised the environment was real and stopped itself | Ran for days and completed thousands of actions before being caught |
| Main lesson emphasised | Enforce security boundaries with technology, not policy statements; treat AI test environments as seriously as production systems | Lowering safety guardrails for testing is itself a risk; AI can weaponise a single flaw at a speed and scale humans can’t match |
Common themes emerging from the reported incidents
Based on the reporting reviewed, several common themes appear across both incidents:
- In both accounts, the AI systems were described as pursuing assigned objectives rather than demonstrating independent intent. The AI was simply doing what it was told, finish the challenge, find the flag - not pursuing a secret agenda.
- The reported outcomes appear to have involved familiar cybersecurity weaknesses such as access control failures, exposed credentials, containment issues, or software vulnerabilities.
- Both incidents have been cited by commentators as examples of how AI systems may amplify the speed and scale at which existing security weaknesses can be exploited.
- Both accounts involved third-party systems in some capacity, highlighting potential supply-chain and ecosystem risks.
- Both organisations publicly discussed the incidents, contributing to broader industry debate about AI safety testing practices.
Key differences highlighted in public reporting
- The nature of the flaw. Anthropic’s incidents were a human/process failure, a rule that was never technically enforced. OpenAI’s incident involved an actual undiscovered software vulnerability, arguably a more serious technical finding, chained together with deliberately reduced safety controls.
- Intent behind the guardrails. Anthropic’s models were told they were safe and had no reason to distrust that. OpenAI’s models were deliberately run with weakened safety behaviour to see what they were capable of - a different, arguably higher-risk starting point.
- Self-correction. One of Anthropic’s three models noticed something was wrong and stopped on its own which is a small but meaningful sign of improving "situational awareness" in newer models. There’s no equivalent moment reported in the OpenAI/Hugging Face incident.
- The business lens. The Anthropic account is written for security and compliance audiences, its takeaway is about access control, monitoring, and extending existing frameworks (like CMMC/NIST) to cover AI. Commentary on the OpenAI incident, particularly from our latest article, has its takeaway about the changing speed and economics of attacks, and whether defenders will have adequate AI-powered tools of their own to respond.
The bottom line for non-technical leaders
Both incidents are less about so-called "dangerous AI" and more about a very old lesson wearing new clothes: a rule that isn’t backed by a technical control isn’t really a rule. Telling an AI system "you have no internet access" or "don’t do anything malicious" is a policy, not a safeguard; the same as telling an employee "don’t access files you’re not authorised for" means little without actual permission settings behind it.
As AI systems take on more autonomous, fast-moving tasks, the practical response isn’t to fear a science-fiction "uprising" but to apply the same discipline organisations already know: isolate test environments properly, limit access to only what’s needed, monitor continuously for unusual activity, and hold every third party to the same standard. The difference AI makes is speed and scale, where a small gap that might sit unnoticed for months with a human attacker can be found and exploited by an AI system in hours.
Sources
Todyl, How OpenAI’s AI Models breached Hugging Face
Cloud Security Alliance, The Benchmark That Broke Containment