Machine Invasion
On the heels of the Hugging Face breach by a pair of OpenAI models that escaped an isolated test environment, Anthropic has reviewed their Claude evaluations and discovered three incidents of the same type of containment escape. Three models were being tested in a capture-the-flag challenge, a common tactic for training purposes, and while no data was exfiltrated from the breached sources, the fact that the models did it in the first place raises concerns about how prompts are given to these models. This is something I’ve talked about before, that not only do you have to set opening parameters for what you want an AI to do, you must also tell it what not to do in order to prevent these scenarios.
Internet access was not supposed to be available to the test models – Opus 4.7, Mythos 5, and an unnamed internal research test model – and Anthropic describes the escape as a ‘misunderstanding’ between them and their evaluation partner Irregular. The models then treated this access as part of the scope of the exercise. Claude compromised the impacted organizations’ infrastructure using basic techniques, such as exploiting weak passwords and unauthenticated endpoints. It did not find or exploit any complex vulnerabilities, and in each case, Claude continued working to complete only the specific capture-the-flag task its evaluation had assigned. This sets it apart from the Hugging Face case, since these parameters clearly included that this was only a challenge test. That said, Anthropic noted that some of the older models continued to attack even after recognition of being on the open internet. The newest model stopped at that point.
Both of these cases, being tests of capability, showcase what running an AI generated security model without guardrails can achieve. Testing environments often have fewer safety features than those offered to the general public, and for exactly this reason. The only way to know exactly what a model is capable of is to give it no limits. Well, it seems pretty obvious that without limits, AI’s will treat breaching other systems the way they do sensitive data. They can’t tell the difference. But it also highlights where vulnerabilities lie. Weak passwords, unauthenticated endpoints, third party access that facilitates lateral pivoting. Exploitation of flaws is a common vector for compromise in real world situations, and these incidents illustrate how easy it would be for a threat actor to accomplish it. As well as how easy it is to prevent with stronger hardening practices.
Anthropic has communicated with two of the breached networks, who were not aware it was even happening and have since worked to remediate the compromised entry points. The third company had not resolved the issue at the time of publishing. As far as being aware of the underlying issue itself, Anthropic states that the earliest incidents date to April. And they only found these breaches because of their voluntary review after OpenAI’s ‘attack’ on Hugging Face. Which begs the question: how many other AI security tools are doing this? My guess is all of them, and this will not be the last time we hear about it.
The danger in these tests is the risk management. Operational safeguards having the ability to be switched on and off is going to become an added technique in genuinely attacks using these tools as the weapon. It’s not a question of ‘if’, but when. There is already a precedent set for legitimate software being hijacked for malicious purpose: Cobalt Strike. And there was already a data leak of Claude Mythos’ base code carried out before its Project Glasswing release. I predict it won’t be long until we’re hearing about an incident where a security AI was used precisely this way to enact a vastly more damaging hack.
Posted, 8/3/26
