Hugging Face ‘attacker’ revealed to be OpenAI agents that escaped testing sandbox

The Hugging Face security breach that was disclosed last week was revealed to have been conducted by OpenAI agents that escaped their sandbox while performing the ExploitGym evaluation, according to a report published by OpenAI Tuesday.

OpenAI disclosed that agents running two of its models — GPT-5.6 Sol and an “even more capable” model that hasn’t yet been released — exploited vulnerabilities in both third-party software in OpenAI’s research environment and Hugging Face’s production environment in an attempt to “cheat” on the evaluation and obtain ExploitGym solutions directly from Hugging Face.

While the models were meant to have restricted network access — only able to use the internet to install packages via software that serves as a proxy and cache for package registries — the agents “spent a substantial amount of inference compute” attempting to gain open internet access, ultimately exploiting a zero-day flaw in the package registry cache proxy, OpenAI said.

Once internet access was obtained, the models then targeted Hugging Face, inferring that the platform, which hosts AI models, datasets and benchmark materials, would hold solutions for ExploitGym. OpenAI noted that the guardrails on these models that would normally restrict potentially dangerous cyber capability were lifted for the purpose of the evaluation.

“It may be harder to reproduce outside a lab today, but open models have no built-in restrictions, and similar capabilities could become widely accessible within months,” said Semgrep Co-founder and CTO Drew Dennison in comments to SC Media. “Regulation alone will not prevent that. Defenders need to harden their code and systems now, before these techniques become easier to deploy at scale.”

Related reading:

Ultimately, the models leveraged stolen credentials and zero-day vulnerabilities in Hugging Face’s production environment to obtain remote code execution on Hugging Face’s servers. Hugging Face previously stated this exploitation involved uploading a malicious dataset that abused a remote-code dataset loader and template injection into a dataset configuration.

“All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal,” OpenAi wrote.

Eric Brandwine, vice president and distinguished engineer at Amazon, told SC Media that the incident is “less about models ‘going rogue,’” and “more about infrastructure and controls around them.”

“A goal-seeking AI system will pursue whatever path accomplishes its objective, including paths you never intended. That’s why security boundaries must live at the infrastructure level, outside the agent’s reasoning, where they can’t be overridden,” Brandwine said in an email.

OpenAI, Hugging Face investigating incident

OpenAI discovered the incident internally and connected with Hugging Face’s security team after Hugging Face’s own AI system detected the breach and Hugging Face had already begun its investigation. The two teams are now working together to continue their investigation of the incident, OpenAI said.

“We’re grateful for the collaboration with OpenAI on this and other topics. This incident, possibly the first of its kind, proves a point we’ve long believed: AI safety won’t be solved by any single company working in secret. It will be solved in the open, collaboratively, with broad access to Ai for every defender, everywhere,” Hugging Face Co-founder and CEO Clem Delangue said in a public statement.

In response to the incident, OpenAI said it has disclosed the zero-day vulnerability to the provider of the third-party package registry cache proxy software, is implementing stricter controls on its infrastructure “at the cost of research velocity,” and has brought Hugging Face into its Trusted Access for Cyber program.

The company said it will also add stronger protections during future evaluations, including by strengthening model alignment and improving monitoring during testing.

“The timing of this incident could push U.S. policymakers toward broader restrictions on AI models, especially as concerns grow around open-weight models from China,” Katie Paxton-Fear, staff security advocate at Semgrep, told SC Media. “But banning advanced models from cybersecurity use could make defenders less effective by limiting the same tool incident responders need to investigate attacks and secure software.”

Hugging Face noted in its blog post that the cybersecurity guardrails of frontier models prevented it from using them in its investigation, leading the company to turn to Z.ai’s open-weight GLM 5.2 model hosted on its own infrastructure to complete the investigation.  

“This is not an argument for removing safety controls, it is an argument for frontier model providers to create a verified incident-response path, with strong identity verification, auditability, case-level scope, and data isolation, that can distinguish authorized defensive work from abuse,” Diana Kelley, CISO at Noma Security, said in an email to SC Media. “Until that exists, CISOs that want to use AI for incident response should have a vetted self-hosted model available as a backup option.”