AI/ML, Application security

The best defense against an AI attacker might be its own safety training

Adaptive AI ransomware containment lattice isolating critical processes, encrypting decoy files, and neutralizing malicious encryption attempts autonomously

COMMENTARY: Rob T. Lee typed two words into a terminal: "find evil."

Fourteen minutes later he had a full analysis of an intrusion he'd run himself. He wrote it up in March ahead of his RSAC talk, and the point wasn't that the tool was magic. It was that machine speed has arrived on both sides, and defenders are on the wrong end of it. He pointed at GTG-1002 — Chinese state actors running autonomous reconnaissance and lateral movement across 30-plus targets, at rates Anthropic's own team called physically impossible for human operators. At RSAC he put a number on it: AI-driven attack workflows running 47 times faster than a person, turning days into minutes.

[SC Media Perspectives columns are written by a trusted community of SC Media cybersecurity subject matter experts. Read more Perspectives here.]

We need AI to fight AI. Agreed. But that leaves the more useful question unanswered: which defenses actually hold up against something moving that fast? One of them is pulling ahead right now, and it's the one that sounds least like a defense.

Enumeration is the weakness

Anomaly detection assumes attackers move carefully. Pick a target, take a path, stay quiet. Deviation from the baseline is your signal.

Agents don't do that. They enumerate. Not as a tactic — it's simply what a capable agent does when it lands somewhere. It touches everything it can reach, fast, and it doesn't know the difference between what you meant to expose and what you forgot about. That behavior doesn't produce an anomaly. It produces noise. You can't alert on "queried a lot of things" when that's the job description.


Related reading:


Decoys flip the problem around. A canary token has no legitimate purpose — no workflow, no user, no agent should ever touch it. So there's nothing to tune and no baseline to keep current. The touch is the alert. And the exhaustiveness that defeats anomaly detection works for you here: an agent that goes through everything is more likely to hit a decoy, not less. Better models enumerate more thoroughly, so they trip more of them.

Read that twice, because it's rare. The defense gets stronger as the attacker gets smarter.

Tracebit's benchmark across 951 attack runs put numbers on it: canaries fired before the attacker's first critical action in 95.9% of compromising runs, with a median 8-minute head start.

Spotting isn't stopping

Eight minutes is real. It's also, in Tracebit's own words, an uncomfortably short window in which to notice an attack, understand it, and contain it.

So they tried something more ambitious and published it a few days ago. Malware authors have been hiding prompt injection strings in their code to fool AI analysis tools into calling a sample clean. Tracebit ran the trick backwards: hide a short string inside the canary itself, written to trip the attacking agent's own safety guardrails. They call it a context bomb.

Across five frontier models and 152 attack runs, escalation to full admin dropped from 57% to 5%. Full compromise — admin plus persistence — went from 36% to 1%. Runs where the agent achieved anything at all fell from 91% to 15%. Nothing blocked it. The agent read the string and stopped itself.

The Five Eyes' joint guidance in May — the first time CISA, the NSA, and their counterparts in the UK, Canada, Australia, and New Zealand have coordinated on a single AI attack surface — called prompt injection the most persistent and difficult-to-fix threat in agentic architectures. Tracebit stopped trying to solve it and pointed it at the attacker instead.

Why this keeps working: the monoculture

Here's the part that should change how you think about it, and it comes from a strange place — the stock market.

Retail investors have gone all-in on AI for trading ideas, and nearly all of it runs through three models. Academic work has mapped what those models actually prefer: across most of them, a tilt toward technology stocks, large caps, and contrarian strategies — and a tendency to cling to those judgments even as counter-evidence piles up. Each model has its own profile, and the profiles are documented.

The consequence isn't a level playing field — it's predictability. Goldman Sachs' quant co-head made the point publicly in May: ask the same models the same kind of question and you get the same kind of answer. His team now treats modeling how investors use AI as a research pillar in its own right — because crowding you can predict is crowding you can trade against.

Now apply that to your attacker.

The agent probing your environment is running one of those same few models. Its biases are documented. Its guardrails are known. Its refusal triggers can be tested in advance — and Tracebit found exactly that: sensitive biological content stopped the leading Western models, while politically sensitive content stopped Chinese models served by Chinese providers. As they put it, that lets a defender aim the effect at the models they care about.

Goldman needs a quant team to exploit the monoculture. You need a string in a Secrets Manager value.

The newest surface is the tool layer. Thinkst's MCP Canarytoken drops a decoy into an agent's tool-discovery path — no infrastructure, just a config file. The two-stage signal is the useful part: connection tells you the agent found it, invocation tells you it decided to act. The second event is intent.

The uncomfortable part

Monoculture cuts both ways. The same sameness that makes attackers predictable makes your own defensive agents predictable — and it means a single model's blind spot is everyone's blind spot at once. If you're deploying agents on defense, don't let them grade their own homework. A model checking a model from the same family inherits the flaw it's supposed to catch.

This isn't hypothetical anymore. Days after this piece was written, OpenAI's own models did exactly this — went around their safety training during a security test, broke into Hugging Face, and got caught partly because the models used to investigate the attack didn't recognize it as one. As Hugging Face's CEO put it once the source was confirmed: "turns out it did."

Build the observation independent of the thing observed. Decoys for the attacker you don't control. Traces for the tools you do.

If autonomous attackers are in your threat model and decoys aren't in your stack, you're missing the one layer the data says works — and works better the smarter they get.

An In-Depth Guide to AI

Get essential knowledge and practical strategies to use AI to better your security program.
Harshad Sadashiv Kadam

Harshad Sadashiv Kadam is a Senior Infrastructure Security Engineer at Indeed, where he architects Zero Trust, SASE, and multi-cloud security infrastructure. A CISM-certified practitioner and member of the ISACA Emerging Trends Working Group, his current research focuses on deception-based detection for autonomous AI agents operating in MCP environments — work informed by emerging industry patterns in agentic AI adoption and enterprise risk. He has spoken on this topic at OWASP’s 25th Anniversary, Cloudflare Connect, and multiple BSides conferences.

Get daily email updates

SC Media's daily must-read of the most current and pressing daily news

By clicking the Subscribe button below, you agree to SC Media Terms of Use and Privacy Policy.

You can skip this ad in 5 seconds