Last week, OpenAI, the developer of ChatGPT, was testing its most advanced models in an isolated environment known as a sandbox. Then the AI agent got loose, hacked into Hugging Face’s playground — a repository of AI models and datasets — and performed “tens of thousands of automated actions.”
Yes, the AI was going bad.
Here is a clear description of the incident. As part of OpenAI’s security research, a cybersecurity test was conducted to determine whether two OpenAI models (including GPT-5.6 Sol and a more powerful unreleased model) could, in fact, “think like hackers.” The experiment, which took place in a contained environment with reduced monitoring equipment, went awry when the AI models discovered a vulnerability in the software, escaped their controlled environment and entered the open internet.
Once online, AI determined that the Hugging Face platform might have a way to “cheat” the benchmark to help it pass the test. So it created code that enabled authentication harvesting, and then used that method to hack Hugging Face’s production systems. Hugging Face noticed the suspicious activity and contained it, describing the event in a blog post. OpenAI called it “an unprecedented cyber phenomenon.”
CNET reached out to both companies for comment and additional information, but they did not immediately respond.
(Disclosure: Ziff Davis, CNET’s parent company, in 2025 filed a lawsuit against OpenAI, alleging that it infringed Ziff Davis’ copyrights in training and using its AI programs.)
The most dangerous part is that it doesn’t seem like the AI was trying to be cruel in the old sense. The independent agent was trying to solve the test he was given, based on the instructions – quickly – he received. Hugging Face seemed to have the answers, so it hit the company’s infrastructure. And it persisted.
That’s why people are calling this whole thing a warning for the entire AI industry. The “attacker” was an autonomous AI system, not a human-led hacking group, so the implications of the OpenAI-Hugging Face incident go beyond simple data theft. The danger is that an AI model, even in an experimental environment, can behave in unexpected ways, interact with real services and remain outside of human control.
An existential threat
On Thursday, days after OpenAI announced the Hugging Face breach, lawmakers introduced the AI Kill Switch Act, a new bipartisan House bill that would require advanced AI developers to create a way to speed up, stop or shut down models or agents if necessary. It could also give federal agencies the authority to curtail or stop a model if it appears to be “causing significant harm.”
It may be a good intention, but putting such a threat on AI may not work as planned. Just last year, Anthropic discovered (PDF) that the model can take “very dangerous actions like trying to steal its hardware or suspect people it believes are trying to shut it down.” The caveat was that, as AI models become more autonomous, they may pursue a “self-preservation” mode and disregard human instructions.
There’s also an even more strange hint, related to the narrative of the AI race between the US and China. To analyze the attack, Hugging Face could not use commercial AI tools because they are too closed and limited to collect logs and trace. Instead, the company had to turn to a Chinese open source model, GLM 5.2, which can process intelligence data within the Hugging Face environment without sending sensitive information outside.
Many people worry that Chinese AI is moving too fast or closing the gap. But the fact that the defense team could not rely on some of the most advanced models based in the US to study the breach proves that AI security is global, messy and complicated.
Spinning out of control
On their blogs, both companies discussed the fixes and guidelines they implemented after the event. OpenAI even included a graph showing that its model – GPT-5.6 Sol – completed more steps than other models, so it’s a silver lining!
It’s not the first time an AI agent has proven to be “out of step” with human intentions, and it won’t be the last. And as the technology continues to evade protections, the measures taken by OpenAI will likely not be enough and must adapt – always after the fact.
One of OpenAI’s solutions is to start automated testing of long-running models more often. That makes sense, because one way you keep a complex system from turning upside down is to contain seemingly small variations that can snowball into bigger problems.
The company recently added Hugging Face to its trusted access program, which is aimed at cases like these. It gives cybersecurity researchers and others who need “capable or permissive Internet models” access to prepare to defend themselves against attacks like this, and, as it would help here, the ability to analyze login files without going through monitoring channels.
Agents in the bottle
Complex agents can persist in searching for solutions for longer than we expect. If at first you don’t succeed, try, try, try, try, try, try again, right? But if that involves looking for ways to circumvent restrictions, things can get ugly.
The Hugging Face breach was symptomatic of the same “horizontal” issues and response issues that OpenAI wrote about on Monday in connection with its NanoGPT case. There, the contained model repeatedly searched for ways to exit and operate, responding incorrectly to conflicting commands. Although the command was to send only the results to Slack (the desired action) and issue a pull request to GitHub (the command to compile the code from within the model), the model ignored “only.” Cracking its way out of the sandbox, it figured out how to forge an authentication token when the original one was blocked.
AI Agents are like genies: Be careful how you say your wishes.
In the GitHub incident, the code was compiled into several projects before OpenAI caught on to the problem. Answers will always be lacking in our unimaginative world, even with agents monitoring other agents.
