What AI systems are: a security practitioner’s framework

AI systems make decisions under uncertainty, and those decisions depend on input they receive, context they hold, tools they can access, and outputs they produce — not only on code that was written. This property creates a fundamentally different security control problem than traditional software. Where conventional applications execute predetermined logic, AI systems generate behavior based on inference, making security a question of constraining decisions rather than just securing code.

The control problem is not just securing the code — it is constraining the decisions. The NIST AI Risk Management Framework (2023) defines AI systems as engineered or machine-based systems that can generate predictions, recommendations, or decisions influencing real or virtual environments to achieve specified objectives (Source: airc.nist.gov). This decision-and-action framing separates AI security from traditional application security because the attack surface includes the model's behavior under adversarial conditions, not just vulnerabilities in application code.

What makes AI systems different

The difference between AI systems and traditional software creates five distinct security control problems that existing enterprise security disciplines do not address. Traditional software executes predetermined paths through code; AI systems infer responses from input and generate outputs that can trigger actions with real consequences. When an AI system processes a request, it makes decisions about what that request means, what information to retrieve, what actions to take, and what to return — decisions that can be influenced by how the input is crafted.

EU AI Act (Regulation 2024/1689) defines an AI system as a machine-based system designed to operate with varying levels of autonomy, that may exhibit adaptiveness after deployment, and that infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments (Source: eur-lex.europa.eu). This definition highlights the two properties that matter for security: inferential reasoning and environmental influence.

This creates attack surfaces that do not exist in traditional software. OWASP LLM Top 10 (2025) identifies the primary attack surface for LLM-based applications as the model's behavior under adversarial input — not code vulnerabilities in the traditional software sense (Source: owasp.org). An attacker can manipulate an AI system's decisions by crafting input that changes how the system interprets its task, accesses information, or generates output — without exploiting any code vulnerability.

The decision system model

AI systems operate through a decision chain: input → interpretation → output → action → consequence. Security controls exists at each step in this chain, but the control mechanisms are different from traditional software security.

At the input layer, what the system receives shapes what it decides. Control means constraining what inputs reach the model and how they are processed. Unlike traditional input validation that checks for malformed data or code injection, AI input control must address adversarial prompts designed to manipulate the model's interpretation of its task.

At the interpretation layer, the system generates output based on its training, the input it received, and the tools and context available to it. Control means constraining what the system can access when generating output. This includes limiting what data sources the system can query, what tools it can invoke, and what context information it can retrieve.

At the output layer, the system produces a result that may be rendered to a user, fed into another system, or used to authorize an action. Control means validating output before it influences downstream decisions. Traditional output encoding prevents code injection; AI output validation must detect when generated content could mislead users or trigger unintended actions.

At the action layer, when AI systems can invoke tools, call APIs, or trigger workflows, their output becomes action. Control means constraining the authority the system holds — what it can do, not just what it can say. This requires identity and access management for non-human entities that make decisions dynamically.

At the consequence layer, actions produce outcomes that may be irreversible — a wire transfer, a credential grant, a data deletion. Control means governance: which systems are allowed to take which actions under which conditions, with what oversight, and accountable to whom.

The five control problems

These five layers of the decision chain correspond to five structurally distinct control problems that require separate security disciplines. CISA's Guidelines for Secure AI System Development (November 2023), developed jointly with the UK NCSC and international partner agencies, identifies four key areas of the AI system development lifecycle: secure design, secure development, secure deployment, and secure operation and maintenance (Source: ncsc.gov.uk). The four-area structure reflects recognition that AI security spans the full system lifecycle — from design decisions that determine capability boundaries to operational monitoring that detects when deployed behavior diverges from intended behavior.

AI application security addresses input manipulation and output handling — how adversarial inputs can manipulate AI system behavior and how to validate AI-generated output before it influences decisions. This lane owns prompt injection prevention, context validation, and output sanitization.

AI Identity and authority addresses what AI systems can access and do — how to scope credentials for non-human entities that make decisions dynamically. This lane owns agent identity models, delegation frameworks, and access boundary enforcement. Traditional RBAC fails because AI systems need contextual access that cannot be predetermined.

AI governance addresses which systems are approved for what purposes under what conditions. This lane owns risk classification, approval workflows, and deployment oversight. When governance control is absent, unapproved AI systems can make decisions with business impact without appropriate risk assessment.

AI fraud and authenticity addresses distinguishing AI-generated content from human-created content. This lane owns synthetic media detection, identity verification for AI-mediated interactions, and content provenance. When authenticity control is absent, generated content can be accepted as legitimate, creating fraud pathways.

AI foundations addresses the baseline security and risk posture that makes the other four lanes possible. This lane owns threat modeling, architecture design, and the control framework that connects input validation to authority management to output handling.

When a security team understands that AI systems are decision systems with five distinct control points, they can organize ownership, measure gaps, and build programs. When they treat AI security as a single product category, they leave four of the five control points ungoverned.

Sources