AI application security explained

Your web application firewall blocks SQL injection attempts, but it won't catch a prompt injection that manipulates an LLM into disclosing customer data. Static analysis tools identify hardcoded secrets in your codebase, but they can't detect when training data contains leaked API keys that the model memorized. Dynamic testing validates input sanitization, but it won't find when an AI agent exceeds its intended permissions and accesses unauthorized systems.

OWASP Top 10 for Large Language Model Applications identifies four attack surface clusters that span AI application vulnerabilities: input and context manipulation (including prompt injection and insecure output handling); supply chain and training data integrity (training data poisoning and third-party model supply chain); agent and plugin behavior (excessive agency, insecure plugin design); and model information disclosure (sensitive information leakage and model extraction). Each cluster maps to a distinct security control domain that existing SAST, DAST, and WAF tooling was not designed to instrument (Source: owasp.org).

What AI application security is

AI application security addresses vulnerabilities that emerge when applications incorporate large language models, machine learning components, or AI agents into their architecture. Unlike traditional application vulnerabilities that typically exploit code logic, memory management, or input validation gaps, AI vulnerabilities often exploit the statistical and probabilistic nature of machine learning models themselves.

The security boundary extends beyond the application's code and configuration to include training datasets, model weights, prompt engineering logic, and agent behavior patterns. This creates attack surfaces that standard application security tools cannot instrument because they operate at the code level, not the model behavior level.

For IAM practitioners, this means non-human identities now include AI agents and model inference services that make decisions about data access and system interactions. Traditional role-based access control assumes deterministic behavior from applications — AI components introduce probabilistic decision-making that can be manipulated through adversarial inputs.

The four attack surfaces

Input and Context Manipulation

Prompt injection attacks manipulate the context or instructions provided to an LLM to bypass intended restrictions or extract unauthorized information. Unlike SQL injection, which exploits parsing errors in database queries, prompt injection exploits the model's training to follow instructions, even when those instructions are embedded in user input.

The downstream consequence: applications that use LLMs for content filtering, data access decisions, or user interaction can be manipulated to behave contrary to their design. Implementing input sanitization that treats all user content as untrusted data and separating system instructions from user content through structured prompting techniques are foundational controls — but they should be paired with cryptographic signing of prompt inputs wherever possible. Treating prompt inputs as signed artifacts, analogous to signed API requests, creates an auditable chain of custody that allows security teams to verify the integrity of what was submitted to the model and detect tampering or injection attempts in the pipeline before they reach inference.

For enterprise cloud environments, this often manifests when AI-powered customer service systems access internal documentation or when content moderation models process user-generated input that contains hidden instructions. Model access to internal documentation is necessary for useful AI applications but also defines the prompt injection blast radius.

Supply Chain and Training Data Integrity

AI models inherit vulnerabilities from their training data and dependency chain. Training data poisoning occurs when attackers inject malicious examples into datasets to create predictable model behavior. Model supply chain attacks target the distribution and deployment of pre-trained models, model weights, or fine-tuning datasets.

A relevant architectural pattern in this space is generative adversarial networks (GANs), where two neural networks — a generator and a discriminator — work in opposition to refine model outputs. While GANs are primarily a training technique, they illustrate how AI model behavior is shaped iteratively by data and adversarial feedback loops, which also describes how attackers approach training data poisoning: introducing examples that gradually shift model behavior in a predictable direction. Understanding this dynamic is relevant context for security teams evaluating how model behavior can be influenced over time through data-level manipulation.

MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) documents adversarial machine learning tactics and techniques targeting AI applications, including adversarial input attacks (perturbing model inputs to cause misclassification or context manipulation), model extraction (reconstructing proprietary model behavior through repeated queries), training data poisoning (injecting adversarial examples to create predictable model behavior), and supply chain compromise of model components. ATLAS extends MITRE ATT&CK to cover AI-specific attack categories that do not map to traditional web application or network vulnerability classes (Source: atlas.mitre.org).

The operational risk: models deployed from third-party sources or trained on externally sourced data can contain embedded vulnerabilities that activate under specific conditions. This creates a gap when organizations apply traditional software composition analysis to AI components without accounting for data-level dependencies.

A practical control that addresses this gap is cryptographic signing of AI supply chain artifacts. Training dataset updates, fine-tuning datasets, and model weight updates should each carry a verifiable signature that confirms provenance and detects unauthorized modification before the artifact is consumed. This mirrors code signing practices in software supply chain security and applies the same principle to data and model components. Organizations that cannot verify the signature of a model update or training dataset injection should treat that artifact with the same caution as unsigned third-party code.

Agent and Plugin Behavior

AI agents that interact with external systems or APIs create excessive agency risks when their behavior exceeds intended boundaries. Plugin ecosystems for LLM applications introduce third-party code execution risks similar to browser extensions or mobile app permissions, but with less mature security models.

The business consequence: AI agents with overprivileged access can perform unauthorized actions that appear legitimate to logging and monitoring systems because the agent's identity is authorized for the action. Implementing least-privilege principles for AI agent identities and monitoring agent decision patterns for anomalous behavior are baseline requirements.

For cloud workload identity management, this means treating AI agents as a distinct class of non-human identity with behavior-based monitoring requirements. The agent's service account should reflect its minimum required permissions, not broad access to the systems it might interact with.

Model Information Disclosure

LLMs can leak sensitive information from their training data through direct memorization or through inference attacks that reconstruct private data. Model extraction attacks use repeated queries to reverse-engineer proprietary model behavior or training data.

The technical risk: models trained on internal documentation, customer data, or proprietary information can be manipulated to disclose that information through carefully crafted prompts. This creates a persistent data exposure risk that doesn't resolve through traditional data loss prevention controls.

Why traditional tools miss AI vulnerabilities

Detection Gap Analysis

Static analysis tools identify code-level vulnerabilities but cannot analyze the behavior of machine learning models during inference. SAST tools scan for hardcoded credentials and input validation errors, but they cannot detect when a model has memorized sensitive information from training data.

Dynamic testing validates application behavior under known input conditions, but AI models exhibit non-deterministic outputs that make traditional test case validation ineffective. Fuzzing techniques designed for memory corruption or parsing errors don't translate to probabilistic model outputs.

Web application firewalls filter malicious requests based on known attack patterns, but prompt injection attacks often use natural language that appears benign to signature-based detection. The attack payload is the semantic meaning, not the syntactic structure.

Monitoring and Logging Limitations

Traditional application logs capture request/response patterns and system interactions, but they don't capture the decision-making process within AI models. When an AI agent performs an unauthorized action, the logs show legitimate API calls from an authorized service account, not the adversarial input that influenced the decision.

NIST AI RMF Map 5.1 requires organizations to identify and document vulnerabilities in AI systems relevant to the deployment context, including adversarial inputs, data poisoning, model output integrity failures, and supply chain risks introduced through third-party AI components. These risk categories define an application security responsibility surface that extends beyond traditional code and API vulnerability management (Source: airc.nist.gov).

Security information and event management systems correlate events based on network traffic, system calls, and authentication patterns. AI-specific attacks often occur within the application logic layer without generating traditional indicators of compromise.

Identity and Access Control Blindspots

Role-based access control assumes applications make deterministic access decisions based on user identity and resource permissions. AI applications introduce probabilistic decision-making where the same user input can produce different access patterns depending on model inference.

Traditional privilege escalation detection looks for unauthorized access to resources or elevation of user permissions. AI agents can exceed their intended behavior without technically violating their assigned permissions, creating a gap between authorized capabilities and intended use.

What changes for AppSec programs

Security Testing Evolution

AppSec teams need to incorporate adversarial testing techniques that probe model behavior rather than code execution paths. This includes red team exercises that attempt prompt injection, data extraction, and agent manipulation attacks.

Testing frameworks should validate model outputs for consistency, safety, and alignment with intended behavior patterns. Comprehensive model behavior testing and development velocity pull in opposite directions — security validation cycles for AI-enabled features run longer than traditional feature testing because model output variation requires statistical sampling rather than deterministic test execution.

Model-specific security testing requires understanding the training data provenance and testing for memorization of sensitive information. Traditional penetration testing skills need to incorporate adversarial machine learning techniques.

Code Review and Architecture Changes

Security code reviews need to evaluate prompt engineering logic, model integration patterns, and agent permission boundaries. Reviewers should validate that user input and system instructions are properly isolated in LLM interactions.

Architecture reviews should assess the blast radius of AI component failures and the privilege scope of AI agents. Treating AI components as untrusted third-party integrations with appropriate isolation and monitoring controls is the correct default posture.

Infrastructure as code templates for AI workloads should enforce least-privilege principles for model inference services and implement monitoring for anomalous behavior patterns. The verification step: confirm that AI service accounts cannot access resources beyond their specific inference requirements.

Architecture reviews should also evaluate whether cryptographic integrity controls are applied consistently across the AI pipeline — covering prompt inputs, training data updates, and model weight updates. The absence of signing and verification at any of these points represents a trust gap that traditional architecture review frameworks were not designed to flag.

Monitoring and Response Integration

Security operations centers need visibility into AI model decision-making processes and agent behavior patterns. This requires logging model inputs, outputs, and decision factors alongside traditional application telemetry.

Incident response procedures should address AI-specific attack scenarios including prompt injection, model extraction, and agent compromise. The operational challenge: distinguishing between model errors and adversarial manipulation without access to model internals.

Threat hunting queries need to identify patterns consistent with AI attacks, such as unusual prompt structures, repeated model queries suggesting extraction attempts, or agent actions that deviate from baseline behavior. Where signed inputs are in use, threat hunting workflows can also look for unsigned or improperly signed prompt submissions as an early indicator of pipeline tampering.