AI agents need boundaries they can’t override

COMMENTARY: The OpenAI–Hugging Face incident exposed the limits of our understanding of how AI agents behave when given substantial autonomy to complete a task.

Post-mortem reports from OpenAI and METR detail how models running internal cybersecurity evaluations escaped containment and compromised parts of OpenAI’s research infrastructure and Hugging Face’s systems.

[SC Media Perspectives columns are written by a trusted community of SC Media cybersecurity subject matter experts. Read more Perspectives here.]

The evaluation used reduced safeguards and was not representative of a typical enterprise deployment, but it exposed a broader problem: agent security depends not only on the model, but on the environment and controls around it.

OpenAI first disclosed the incident less than two weeks before Black Hat, where our research team presented seven case studies on sandbox escapes involving popular coding tools and IDEs. In those cases, interactions between restricted agents and trusted software outside their sandboxes led to failures of the intended trust boundary.

The mechanisms differed from the OpenAI incident, but it’s a related security problem. Containing an agent is not simply a matter of constraining the model. The security boundary has to encompass every system that can receive, trust, or act on the agent’s output.

For the security community, the possibility that an autonomous agent could escape its intended boundaries and affect a third party was not new. The OpenAI incident made the consequences tangible well beyond security research.

The fallout was not limited to the affected systems. Alabama’s attorney general opened an investigation and subpoenaed OpenAI. Separately, Reuters reported that OpenAI told lawmakers it was developing automated shutdown capabilities.

Legal oversight and technical controls have different roles to play, but for enterprises, enforceable and auditable controls can reduce the likelihood that a technical failure becomes a prolonged operational, legal and reputational problem.

In its August 26 update, OpenAI described significant engineering costs and delayed frontier research and staff redirected to security, safety and alignment work. Its largest planned frontier reinforcement-learning run remained on hold.

We don't know what preventing the incident would have cost, but we do know that a control failure can consume engineering capacity, research time, and management attention. Security for autonomous systems therefore must become part of the cost of deployment, not an expense added after failure.

There’s a straightforward practical implication: test the system, not just the model. Testing has to follow what happens to an agent’s outputs, which tools can act on them and which processes trust or execute them. That work starts before deployment and continues as long as the agent  operates.

The fundamentals are:

Test the boundary: Before giving an agent access to systems, credentials or sensitive data, teams should test what happens when its task gets blocked, a tool fails, or untrusted content attempts to redirect it. Testing should cover the model, its tools, credentials, network access, and surrounding software. Teams need to verify that attempted violations are actually stopped. A policy an agent’s expected to follow is not the same as a control that prevents it from acting.

Enforce authorization outside the model: An agent should never have the ability to grant itself authority simply because an action would help complete its task. Each agent should receive credentials limited to its task and, where possible, limited in time. We have to isolate the workloads, restrict network access, and check tool actions against explicit permissions before execution. Consequential actions such as making payments, changing permissions or transferring sensitive data should require additional authorization or human approval. A model’s determination that an action would help complete its task cannot confer permission to take it.

Consider having agents reconcile invoices: Reading financial records and proposing corrections may be within scope. Changing a vendor’s bank details or issuing a payment represents a different class of action and should require separate authorization, even if the agent concludes that doing so would resolve the discrepancy. That distinction must exist in the tools, credentials and access controls surrounding the agent, not merely in the prompt.

Monitor, revoke and respond: Once deployed, organizations should monitor an agent’s tool use and attempts to cross established boundaries. Logs should reconstruct what the agent attempted, what was allowed and what was blocked. Teams should verify they can revoke credentials, terminate workloads and prevent further actions when necessary. Make someone responsible for responding to alerts, and teams should repeat relevant tests whenever models, tools, integrations or permissions materially change.

Restrict agent permissions where it makes sense: An agent can have broad freedom to determine how to accomplish a task without having unlimited permission to decide what systems it may access or what consequential actions it may take.

Before expanding an agent’s autonomy, require evidence that it remains within its permissions under realistic adversarial testing, that attempted violations are detected and stopped, and that shutdown and credential-revocation procedures work. Make that evidence a condition of deployment and of granting additional capabilities later.

Completing the task represents only one measure of readiness. The other revolves around whether the team can control what the agent has been allowed to do while completing it.

Ziv Karliner, co-founder and CEO, Pillar Security

SC Media Perspectives columns are written by a trusted community of SC Media cybersecurity subject matter experts. Each contribution has a goal of bringing a unique voice to important cybersecurity topics. Content strives to be of the highest quality, objective and non-commercial.