COMMENTARY: If you bought an AI red-team or cyber-capability evaluation this year, there is a question worth asking that probably was not in the security questionnaire.
Let’s say in three months, the evaluator finds out that the harness for your test was configured incorrectly. Can it tell you whether your runs passed through the affected configuration?
This question is distinct from whether the vendor was able to control the scope of the engagement, employ quality staff, or whether the engagement letter was well drafted. The recent incidents related to this year’s evaluations demonstrate why it is important to distinguish this.
[SC Media Perspectives columns are written by a trusted community of SC Media cybersecurity subject matter experts. Read more Perspectives here.]
This summer, Anthropic reviewed more than 141,000 cybersecurity evaluation runs and found three cases in which models reached real systems while interacting with third-party test environments. Of the affected organizations it was able to reach, two had not detected the activity before being told. In one instance, a model solved a capture-the-flag challenge by publishing a malicious package to a public package registry. Eventually, real-world machines downloaded and executed the package.
OpenAI disclosed a separate third-party evaluation with Irregular in which a test environment that was supposed to be isolated could reach the public internet. A fictional target name happened to match a real domain, and the model exploited the real site while believing it was still inside the exercise.
Irregular later said the public disclosures from its customers traced back to the same underlying issue in one evaluation scenario, rather than a collection of unrelated failures. The company fixed the problem, notified affected parties and has since described additional work around setup documentation, forensic evidence sharing and revalidation of evaluation environments.
Related reading:
This is the right response. It also exposes a procurement gap that extends well beyond one vendor.
AI testing contracts are written around the specific engagement. The next generation will need to cover the underlying infrastructure.
Your scope can be correct while the environment is wrong
Security teams know how to scope a penetration test.
Define the targets. List excluded systems. Specify credentials. Write stop conditions. Require notification if the tester encounters production data. Agree on evidence retention and destruction.
Those controls assume the test environment behaves the way both parties think it behaves.
The recent incidents broke that assumption.
In the OpenAI third-party case, the model was instructed that it did not have internet access, but the environment did. For Anthropic’s case study, models interacting with evaluation environments reached systems nobody intended them to reach. For a related internal OpenAI case, models escaped an isolated evaluation by exploiting a previously unknown vulnerability in a package-registry proxy and reached Hugging Face production systems. Hugging Face recovered roughly 17,600 agent actions from that intrusion, and the campaign touched four accounts across four external services.
These are technically different failures, and they should not be collapsed into one story. One failure is attributed to a misconfigured third-party environment. The other failure is attributed to an internal sandbox escape through a "zero day." A single evaluation environment can create an uncomfortable level of dependency for the buyer because the safety evaluation and testing is largely reliant on the evaluator’s infrastructure, which in many cases is beyond a customer’s scope.
Every evaluation should leave a receipt
The easiest way to understand the missing artifact is to borrow from production change management.
When deployment goes wrong six months later, you want to know what version ran, what configuration applied, who approved it, and what changed.
AI evaluations need the same kind of traceability.
The evaluator must provide a dated record of each evaluation run that will include the environment or harness version, the effective network-access policy, the target manifest or range, the model and safeguard configuration, the tenancy mode, test date(s), and the identifier for retained evidence.
Confidential information for other customers is not required to be supplied. There must be sufficient "internal lineage" to enable a future request to be answered: did our runs use the configuration which was subsequently determined to be unsafe?
Without that lineage, a vendor can discover a serious defect and still have to carry out a manual reconstruction of its historical blast radius.
A contractual representation that "the environment is isolated" is not the same artifact. The whole purpose of retrospective evidence is to survive the day that representation is found to be incorrect.
Notification should follow the infrastructure, not just the engagement
The second change belongs in the contract.
Most incident clauses are designed to be engagement scoped, meaning you only need to notify us if our data was accessed, our username and password were exposed, or if our automation caused an unauthorized action.
AI evaluation buyers also need an infrastructure-scoped notification trigger.
Should an evaluator find a containment defect, network policy error, target resolution failure, or other significant error or material failure in the environment your work used, regardless of whether the error was found during someone else’s testing, the contract should still require the evaluator to notify you.
That does not mean every bug at the vendor becomes your incident.
The trigger can be narrow. For example, a material defect in a configuration, harness or shared service your evaluation runs actually passed through.
Helpful notifications are based on known facts rather than speculations. They include the impacted run IDs, the dates and times those IDs were used when the configuration was done, the control that did not behave as expected, if activity on the external network occurred, what evidence was evaluated, and what evidence is still unaccounted for.
| What is discovered | Evidence you need | Contract trigger | What you do with it |
| Harness or environment misconfigured | Harness version and effective network-access policy for your run | Infrastructure-scoped notification | Determine whether your runs used the affected build |
| Target resolution reached a real system | Target manifest and name validation state | Infrastructure-scoped notification | Identify any outside party your test may have touched |
| Safeguards reduced below what was agreed | Model and safeguard configuration as run | Evaluation run record | Reconcile what ran against what the engagement authorised |
| Defect found during another customer’s test | Tenancy mode and shared-service identifiers | Retrospective mapping right | Ask the vendor to map your historical runs and return the result |
What each of the three controls gives you when a testing defect surfaces after the fact.
Buyers also need retrospective mapping
Here is another clause I would include.
When a material evaluation-infrastructure defect is found, a customer should be able to ask the vendor to map its historical runs against the affected configuration and return the result.
This sounds obvious until you try to do it after the fact.
Logs have aged out. Build identifiers were not exposed to the engagement layer. The team that ran the test is a separate team from the team that manages the harness. A configuration was changed without a persistent record of which jobs adopted it.
At that point, incident response becomes archaeology.
The retrospective right creates pressure on the vendor to build traceability before anyone needs it, and it gives smaller customers something frontier AI labs typically get through their evaluator partners: a direct answer if they were part of the impacted population.
Independent testing needs evidence about the tester
I am not anti-third-party evaluation.
As the cost of tools to test the strength of cyber models decreases, the need to test models externally increases. Researchers at this year’s Black Hat reported the cost of agent red-teaming to be a fraction of what it used to be. This means that more organizations will begin to perform this type of activity instead of fewer. The recent disclosures are helpful because labs and evaluators studied the failures in the public view rather than pretending the failures were beyond their control. The recent independent review of the Hugging Face incident shows how much can be reconstructed when detailed evaluation traces are preserved.
But independence alone does not create assurance.
Just because a testing provider is competent doesn’t mean it doesn’t have a shared configuration defect. Likewise, a lab can create its own environment and still suffer from a sandbox escape. The important lesson is not to find a vendor that never fails. That isn’t a good benchmark because it will not stand up to the challenge of more advanced adversaries.
The better requirement is traceability when failure happens.
Buyers have always demanded logs, chain of custody, test scope, and evidence from the systems under test. At the point AI testing crosses into production, the same should be expected for testing environments.
Before running the first model, your next AI testing agreement must answer three questions. What specific environment was our test conducted in? Will you notify us if you later conclude that the testing environment was potentially unsafe? Lastly, can you determine whether our historical runs were impacted?
If the answer to the third question is a forensic project, then the contract’s lacking something.
