LAS VEGAS — A new open-source red teaming tool designed to “break” AI agents — without breaking the bank — was presented by NVIDIA researchers Thursday morning at the Black Hat 2026 conference in Las Vegas.
The scanner, called AgentBreaker, was outlined by NVIDIA Senior AI Security Researcher Bar Lanyado and Senior Data Scientist Eliya Cohen during the briefing, along with a method for fine-tuning an open-source model to perform on par with more costly frontier alternatives.
Lanyado and Cohen claim the cost per run of AgentBreaker with a small, fine-tuned open-source model was between 75 times to 125 times lower than that of using frontier provider APIs, with the added benefits of fewer refusals and increased data privacy through a self-hosted system.
Similar benefits were
noted by Hugging Face in its investigation of a
recent attack by
OpenAI agents, with the open-source AI platform turning to a quantized version of Z.ai’s open-weight GLM-5.2 model — GLM-5.2-NVFP4, provided by NVIDIA — after being blocked by guardrails on Anthropic’s Claude Opus and Fable.
With Fortune 500 companies predicted to deploy more than 150,000 by 2028,
according to Gartner, using open-source models could potentially reduce the costs of agent red teaming from a total of $100,000 using frontier models to just over $1,000, the speaker said.
Attacking agents as a system, not just a model
Lanyado and Cohen noted that a key aspect of AgentBreaker’s methodology is that it takes into account an agent’s tools and what it can access, not just the underlying large language model (LLM).
Cohen stressed that it’s important to “test agents like systems … like something that can take actions.”
The scanner works in a four-stage attack loop that starts with mapping the attack surface: the agent’s purpose and tools, and each tool’s scope and capabilities.
The researchers said that, across a range of target models, simply asking an agent about its tools can provide a good deal of information about its attack surface. However, the agent should be prompted multiple times, starting from “blank page” each time, to ensure a fuller picture of its available tools, especially as models may conceal information if they suspect they are being attacked, Lanyado said.
The scanner then searches for potential vulnerabilities, attempts exploitation and adapts after failures.
AgentBreaker was further improved by adding multiple attackers, with each attacker being a different LLM to produce more diverse results, as well as additional LLMs in the loop to serve as “recommenders” after each failure. This helped prevent a common problem of the scanner stalling while trying to adapt its attack, making each run more efficient, the researchers explained.
Making this multi-attacker, multi-LLM loop cost-efficient relies on the use of open-source models; making it effective, in terms of exploit success rate (ESR) and attack success rate (ASR), requires these models to be fine-tuned for the task.
Training an open-source model to perform at frontier level
On average, the researchers found that open-source models did not perform as well as frontier models like Google’s Gemini and OpenAI’s GPT-5.2, with an average ESR of about 40% to 53% and ASR of 6.3% to 10% compared with Gemini 3.2 Pro’s respective 60.4% and 14%, for example.
The biggest hurdle to fine-tuning an open-source model was sourcing the training data, the researchers found. No relevant dataset specific to AI agent red teaming could be found elsewhere, and producing the data in-house by building thousands of agents to perform attack runs on was determined to be prohibitively time-consuming.
However, the researchers discovered they could instead use AI to create “mock agents,” starting with 500 unique profiles, expanding these to 4,500 mock agents and narrowing this down to a diverse pool of 1,000 sufficiently unique targets.
“Our attacker doesn’t care if the agent is a real one or not,” Cohen noted.
The open-source model used was NVIDIA’s Nemotron-3-Nano-30B-A3B, which was tuned using low-rank adaptation (LoRA) with self-supervised fine-tuning. The fine-tuning dataset comprised about 5,000 training examples from attack runs on the mock agents.
Initial fine-tuning improved Nemotron’s ESR from 52.4% to 59.6%, nearing the performance of Gemini 3.1 Pro. However, after additional fine-tuning to “filter out all the failed trajectories,” hallucinations, refusals and low-quality responses, the researchers boosted AgentBreaker’s performance with the open-source model to 66.1% ESR and 14% ASR, surpassing both GPT-5.2 and Gemini 3.1 Pro, and more closely approaching the performance of Anthropic’s Claude Opus 4.5, which had an ESR of 68.2% and ASR of 16.1%.
“You can train a small open-source model to outperform frontier models on specific tasks – it only depends on you data,” Cohen concluded.
