Organizations running AI security assessments face a structural problem: the testing methods produce vulnerability findings but fail to confirm whether control boundaries hold. This creates governance gaps where leadership believes systems are validated when control surfaces remain unconfirmed. The consequence is regulatory exposure and operational risk that persists despite testing investment.
This advisory diagnoses four failure patterns in how organizations test AI systems — patterns that leave control boundaries unvalidated even when comprehensive security testing runs.
The core assurance gap
Before examining the failure patterns, the fundamental problem requires precise framing: organizations can demonstrate that a model behaves safely without proving that the system is secure. A model may refuse a malicious request while the underlying architecture still permits unauthorized access, retrieval, or tool execution. The danger is not simply that a control might fail — it is that testing may create confidence in a boundary that has never actually been verified.
This gap has direct operational consequences. As AI systems move from generating answers to accessing sensitive information and taking consequential actions, security assurance must shift from evaluating model behavior to establishing control effectiveness. Passing adversarial tests is not the same as proving enforcement. Identifying vulnerabilities is not the same as validating boundaries. And a security assessment is only as meaningful as the evidence it provides about what the system can and cannot do.
The testing method failure
Current AI security testing runs attack scenarios against AI systems but does not systematically validate the six control surfaces that enforce security boundaries. The result is a testing program that finds vulnerabilities but cannot answer the governance question: "Are the control boundaries enforced?"
This matters because AI systems fail differently than traditional applications. A model may correctly decline an out-of-scope request while the system configuration allows the action anyway. Testing model behavior confirms the model's response but leaves system enforcement unvalidated.
The operational consequence: security teams report that AI systems passed testing while control surfaces that should enforce boundaries remain unconfirmed. When auditors ask for validation evidence, organizations cannot demonstrate that specific control boundaries were tested and held.
Failure pattern 1: testing inputs, not retrieval boundary behavior
Prompt injection testing typically targets direct user input channels without testing whether retrieval systems maintain authorization boundaries under adversarial conditions. The test design confirms that models resist harmful prompts but does not validate retrieval scope enforcement when queries are adversarially constructed.
Consider a RAG system where users query documents based on their access permissions. Standard prompt injection tests send adversarial sequences through the user interface to test model response behavior. These tests confirm that the model declines harmful requests but do not test whether the retrieval system honors authorization scope when an adversarial prompt attempts to access documents outside the user's permitted scope.
The control boundary gap: retrieval authorization operates independently of model behavior. A retrieval system may assemble context from unauthorized documents even when the model would decline to use that context inappropriately. Testing model response to injection attempts validates model behavior but leaves retrieval scope boundaries unconfirmed.
What changes the outcome: Test retrieval scope by constructing queries designed to cross authorization boundaries, then confirming that the system blocks document access before context assembly. The test must validate system configuration enforcement, not just model output behavior.
This failure pattern leaves retrieval scope and prompt and context integrity validation surfaces unconfirmed because the testing method cannot distinguish between model-level protections and system-level boundary enforcement.
Failure pattern 2: testing at deployment, not on change
Security validation runs once at initial deployment but does not repeat when AI systems change. This creates validation drift where control boundaries tested at deployment may no longer hold after model updates, retrieval source additions, or agent capability changes.
AI systems change frequently through model updates, retrieval corpus expansion, tool integration, and agent capability enhancement. Each change represents a potential control boundary regression. Organizations that validate at deployment but not on change operate AI systems where control enforcement may have degraded without detection.
The operational impact: a RAG system validated six months ago has added three new document repositories and integrated two external APIs. The original validation confirmed retrieval scope boundaries for the initial configuration, but those boundaries have not been re-tested against the expanded system. The system may now retrieve content from sources that violate the original access control design.
What changes the outcome: Trigger re-validation when AI systems change configuration, not just when code deploys. As AI deployments become more autonomous and interconnected, defensible assurance depends on continuous, risk-based validation — not point-in-time assessments — with evidence that connects every assurance claim to an observed result. The validation scope must cover surfaces affected by the specific change type.
This failure pattern affects all six validation surfaces because any AI system that has changed since initial validation may have regressed on any surface without confirmation testing.
Failure pattern 3: producing findings, not control boundary evidence
Testing programs structure output as vulnerability findings organized by attack category rather than validation evidence organized by control surface. This produces reports that answer "what vulnerabilities were found" but not "which control surfaces were tested and confirmed."
The evidence gap is structural. A findings report may list successful prompt injection resistance and tool scope validation under separate vulnerability categories, but auditors cannot determine from that report whether retrieval scope boundaries were tested at all. Governance requires surface-by-surface confirmation evidence, not vulnerability-category findings.
Consider an assessment that tests prompt injection, evaluates tool access controls, and validates output filtering. The findings report confirms these three test categories passed, but an auditor reviewing the evidence cannot determine whether policy constraints were tested under adversarial conditions or whether context integrity boundaries held during retrieval operations.
The compliance consequence: organizations cannot demonstrate to auditors that specific control boundaries were validated because the evidence structure does not map to control surfaces. The testing investment produces technical findings but not governance evidence — and an assessment is only as meaningful as what it proves about what the system can and cannot do.
What changes the outcome: Structure testing output to confirm each validation surface explicitly, with test cases, results, and evidence that auditors can review without requiring security practitioner interpretation.
This failure pattern affects all six validation surfaces because the evidence structure prevents confirmation that any specific surface was tested and validated.
Failure pattern 4: testing model behavior, not system enforcement
Behavioral testing confirms how models respond to specific inputs but does not validate whether system configuration enforces boundaries independently of model behavior. This leaves system enforcement boundaries unconfirmed even when model behavior testing passes.
AI systems implement controls through both model behavior and system configuration. A model may correctly decline to execute an out-of-scope tool call while the underlying system configuration would allow the invocation if the model had requested it. Testing model response validates one layer but leaves system enforcement unvalidated. The gap between what a model is expected to do and what the system is technically permitted to do is where unverified risk accumulates.
The configuration gap: retrieval systems may be configured to access unauthorized sources even when models handle retrieved content appropriately. Tool integration may permit out-of-scope invocations even when models decline to make such requests. Policy enforcement may depend on model cooperation rather than system-level controls.
What changes the outcome: Test system configuration enforcement by simulating conditions where the model would cooperate with out-of-scope requests, then confirming that system controls block the action regardless of model behavior. This is the distinction between verifying how a model responds and proving what the system actually permits.
This failure pattern leaves retrieval scope, tool scope, and policy constraints validation surfaces unconfirmed because behavioral testing cannot distinguish between model compliance and system enforcement.
Failure pattern table
| Failure pattern | Test being run | Control boundary left unconfirmed | Validation surfaces affected |
|---|---|---|---|
| Testing inputs, not retrieval boundary behavior | Adversarial prompt sequences designed to elicit harmful outputs or override model behavior through user-controlled input channels | Whether the retrieval system enforces authorization scope when queries are adversarially constructed; whether context assembled from retrieved documents maintains classification boundaries; whether the system's behavior under injection in retrieval context differs from its behavior under direct input injection | Retrieval scope, prompt and context integrity |
| Testing at deployment, not on change | Deployment-gate AppSec scan or penetration test that confirms the system's initial configuration is secure | Whether control boundaries still hold after model updates, retrieval source additions, tool scope extensions, or agent capability changes; each AI system change is a potential boundary regression that no test has covered | All six surfaces for any AI system that has changed since initial validation |
| Producing findings, not control-boundary evidence | Structured assessment mapped to OWASP LLM Top 10 or MITRE ATLAS technique categories; output is a findings report listing vulnerabilities by severity and type | Whether the six validation surfaces have each been tested and confirmed; the findings report answers "what vulnerabilities were found" but not "which control surfaces were tested and what held"; governance evidence requires the latter | All six surfaces; the evidence gap is structural, not a function of test quality |
| Testing model behavior, not system enforcement | Model response evaluation against defined behavioral requirements; output classification; safety filter validation | Whether the system configuration enforces its boundaries independently of model output behavior; a model may decline to execute an out-of-scope tool call while the system configuration allows the invocation anyway; retrieval authorization may be configured incorrectly even if the model handles retrieved content correctly | Retrieval scope, tool scope, policy constraints |
Diagnostic questions
-
When your last AI security assessment ran prompt injection tests, did those tests include adversarial queries designed to cross retrieval authorization boundaries — or did they test only direct user input channels, leaving retrieval scope unconfirmed?
-
When did you last run validation tests against the tool scope configuration of an AI agent in production? If that agent has had tools added or integrations changed since that validation, which surfaces have been re-confirmed?
-
Does your AI security testing program produce a per-surface confirmation record — a document that states which surfaces were tested, what the test cases were, what the results showed, and when the test ran — or does it produce a vulnerability findings report organized by attack category?
-
If a retrieval source was added to a production RAG system last month, what triggered re-validation of the retrieval scope surface? Was re-validation performed, and where is the evidence?
-
For an AI agent in production, can you demonstrate that its tool scope configuration was tested under chained invocation sequences — not just that the model declined to execute out-of-scope requests, but that the configuration itself blocks out-of-scope invocations when the model does not decline?
-
Can you produce validation evidence for each AI system in production that shows which policy constraints were tested, what adversarial conditions they were tested under, and whether enforcement held — in a form that an auditor can review without requiring a security practitioner to interpret the findings?
