Defend AI together, or fail alone

COMMENTARY: There are two noteworthy incidents that should give every security team pause. First, an agent hacked its way into Hugging Face’s systems. It took 17,600 actions over 4.5 days in an attempt to find answers to cybersecurity challenges it had been tasked with solving.

Second, Anthropic disclosed that AI agents escaped a testing environment and accessed real systems belonging to three organizations, including a production database. One agent even created a malicious software package that ended up running on 15 real systems.

[SC Media Perspectives columns are written by a trusted community of SC Media cybersecurity subject matter experts. Read more Perspectives here.]

Today, agents can now carry out thousands of actions without stopping, leaving security teams with little or no time to contain the attack or stop it from happening. At the same time, companies have to worry about their own agents, which may have access to sensitive systems and data and can cause damage when they make a mistake or are manipulated.

No company can learn about every possible failure from its own agents alone. As more companies deploy agents, they need a way to learn from attacks and failures happening elsewhere so they can protect themselves before the same thing happens to them.

We need to learn from each other's failures to stay secure

Companies deploy agents built on similar models and connected to similar types of systems, which means an incident at one company can affect many others without even knowing it.

When companies keep those incidents to themselves, everyone has to discover the same problems on their own. That gives attackers more chances to exploit weaknesses that someone else has already found.

Companies need to share what they learn from these incidents so others can protect themselves. That means sharing enough detail about what happened, how it happened, and what caused it so other companies can determine whether they face the same risk and fix it before it becomes an incident.

But to learn from an incident, we need to know exactly what happened

When an AI agent causes a security incident, we need to replay exactly what happened in detail. That requires keeping a detailed record of what the agent saw, what it did, and what people and security systems did in response.

Unfortunately, sometimes evidence lives in outside systems. An agent could respond to a prompt injection embedded in a third-party webpage. If that page later changes or disappears, investigators can lose the original source and struggle to reconstruct what the agent saw.

That’s why it’s critical security keeps a tight record of agent activity and also saves evidence from outside the company. This includes webpages, files and other external content an agent encountered, along with logs showing what tools it used, what actions it took and what it produced.

Security teams also need to look at what happened around the agent. For example, did earlier testing flag the behavior? Did monitoring catch it? Did the team get a warning and, if so, what happened next? That trail helps explain where the safeguards failed.

Then other companies can actually learn from it. Instead of simply telling another company that our agent was compromised by a prompt injection, we can tell them exactly how it happened and which safeguards failed. They can then test whether their agents are also exposed.

We need a standard way to share what goes wrong

Sharing what happened only works if other companies can understand and act on it. If every company describes incidents differently, it becomes much harder to spot when the same problem happens again and again.

That’s where standards can help. Companies need a common way to describe what went wrong, what the agent was doing, which safeguards failed, and what the consequences were. They also need a place to share that information, including anonymously when companies cannot disclose the details publicly.

We already have examples of this in other industries. NASA’s Aviation Safety Reporting System lets people confidentially report safety incidents and near misses so the aviation industry can learn from them. CyberAcuView brings insurers together to share cyber risk data and identify patterns across incidents.

AI security needs the same kind of feedback loop. If companies continue to see agents trying to get around the same safeguard, that’s an immediate red flag and cause for testing. If the same type of attack keeps succeeding, security standards should require safeguards designed to prevent it.

But different parts of the industry see different problems, and bringing that information together gives everyone a better picture of what’s going wrong and what they need to test for.

It takes a village to create standards

Security teams see what happens when agents are deployed inside real companies. They see which safeguards work, which ones fail and what happens when something goes wrong. What they learn should help determine how agents are tested and what security standards require.

Researchers and insurers are equally important in the process. Researchers test agents to find vulnerabilities and to see where safeguards fail.

Insurers see what keeps going wrong across different companies. If claims show that a particular safeguard works, they can encourage more companies to adopt it. If the same weakness keeps leading to claims, they can push companies to fix it.

Bringing all of this knowledge together can give the industry a much better picture of where agents are failing. Standards can then turn what the industry learns into concrete security requirements. If companies repeatedly encounter the same weakness, standards can require others to test whether their agents are vulnerable to it.

This may be easier said than done. Companies are often reluctant to share exactly what went wrong, especially if an incident could expose them to legal liability or reveal sensitive information.

But there are ways to share findings while avoiding risk, like anonymous reporting. Security teams can explain what the agent did, where the safeguards failed and what others should look for, without naming the company or disclosing sensitive information.

That knowledge needs to travel. Otherwise, the next security team may only find the same weakness after it causes another incident.

There are always good reasons to keep some details private. But we should not have to learn the same security lessons company-by-company.

Rune Kvist-Dattani, co-founder and CEO, AIUC; Rajiv Dattani, co-founder, AIUC

SC Media Perspectives columns are written by a trusted community of SC Media cybersecurity subject matter experts. Each contribution has a goal of bringing a unique voice to important cybersecurity topics. Content strives to be of the highest quality, objective and non-commercial.