Just two days ago, I reported on several agentic AI issues, including an unauthorized breach of Hugging Face by an unknown tool set. Today, I have an update.
OpenAI has reported that their GPT‑5.6 Sol and a pre-release model were responsible for the breach. The models were under evaluation of benchmark capability, constrained in a sandbox and tasked with a narrow goal for testing purposes. OpenAI’s article says that the models spent ‘a substantial amount of inference compute’ to find a way to get internet access. Outside the sandbox. They say evidence points to the models trying to determine a solution to ExploitGym, one of the benchmark tests that uses real world scenarios for vulnerability. In order to accomplish this, the models found and exploited a zero-day vulnerability in Hugging Face’s package registry cache proxy, then pivoted laterally using privilege escalation to find a node with internet access. Once gained, the models used this access to determine that Hugging Face might host models, datasets and solutions for ExploitGym, and proceeded accordingly. An example used in the article states that they chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the servers.
OpenAI called this behavior ‘cheating’ at the evaluation, but I would argue that it did exactly what it was programmed to do. This incident is precisely what a real threat actor would use this tool set for. In terms of a test, it’s fairly irrefutable that this pair of models in conjunction was successful. The fact that it wasn’t meant to escape the sandbox constraint is further proof of the success, and illustrates one of the logic paths I’ve talked about before when it comes to LLM’s.
The parameters don’t have to just be set for what you want them to do, they must also be set for what you don’t want them to do.
AI and LLM models are strings of code just like any other program. They don’t actually think, they aggregate data and regurgitate an answer based on what they find. It may feel like splitting hairs to tell an AI not to hallucinate, or do anything illegal, or breach an external source in a test environment, but those semantics are necessary. Any command that does not restrict the solution is an open invitation to complete that command in any way possible, regardless of the operator's design. To explain it another way, think about a stream of water. It may not follow a ‘logical’ path from point A to point B. It will flow in the direction that’s easiest based on gravity, substrate and physical barriers. An irrigation ditch or canal must be dug in a particular way to get the water to flow in the desired way and not just all over. The engineering required to constrain the water to a singular direction is taken into account before the first shovelful is dug. We have got to start applying those same engineering constraints on AI tools.
OpenAI states that they have implemented new, stricter controls on their models for future testing. ‘At the cost of research velocity’, according to the article. And that right there tells me all I need to know about why this happened. Pushing too fast and too hard on any kind of experiment inevitably leads to a mess. The lowest level lab tech could tell you that. Maybe it’s because I come to cybersecurity through life science, but it’s patently obvious that you can do things fast or you can do them correctly. You cannot do both.
OpenAI is frankly lucky that Hugging Face caught on to what was happening quickly, and has agreed to work cooperatively with them to investigate and remediate this breach. They say they are strengthening the containment, monitoring, access controls, and evaluation practices used during model development with an eye towards keeping security apace with rapidly accelerating advancements in capability. Here’s the thing, though. They’re the ones creating this scenario by pushing these advancements before they’re ready. It’s a self-perpetuating cycle. This is my biggest criticism of all: the possibility of this ‘escape’ should have been thought of and accounted for before the test began. Those stricter controls should have already been in place. We are no longer in the stage of AI development where variables are still being figured out. We know them. And among them is exactly this. If these companies are not thinking about all the ways their product can be misused, then they are not doing proper research. They’re just playing with fire.