AI framework vulnerabilities and model supply chain risk

What you may be missing

Your organization tracks every software dependency through SBOM practices, scans for vulnerabilities in third-party libraries, and maintains version control over code deployments. But if you're using AI models in production — whether through fine-tuned internal models, externally sourced base models, or AI framework libraries — your supply chain visibility has important gaps at the AI layer. Model weights, training datasets, and AI inference frameworks create attack surfaces that standard software composition analysis cannot fully address.

OWASP LLM Top 10 LLM05 (Supply Chain Vulnerabilities) identifies AI-specific supply chain risks, including third-party base model weights, plugins, fine-tuning datasets, and vector store contents as trust boundaries that organizations may not have reviewed before production incorporation. Unlike software dependency supply chain attacks, AI supply chain attacks target model behavior rather than code execution paths — leaving standard SCA and SBOM tooling without detection capability for several of the primary attack vectors. (Source: owasp.org)

To be clear, SCA and SBOM practices remain valuable for the code and library layers of AI deployments — they can surface vulnerabilities in inference engines, serving frameworks, and other software dependencies in the AI stack. The structural gap is elsewhere: your SBOM documents code dependencies but captures nothing about model origins, training data integrity, or behavioral guarantees. An attacker who compromises model weights or fine-tuning datasets can alter your application's decision-making logic without touching a single line of code that your security tools monitor.

Key risk areas

Poisoned base model weights — MITRE ATLAS documents training data poisoning (AML.T0020) and model backdoor attacks as established adversarial ML techniques with documented real-world use cases. Backdoored models produce normal outputs for most inputs but exhibit predictable malicious behavior when specific trigger conditions are present — behavior that standard model evaluation testing may not expose if the trigger condition was not included in test inputs. (Source: atlas.mitre.org)

When you download pre-trained models from external sources, you inherit any backdoors embedded during their training process. Impact: applications make decisions based on compromised logic that appears to function normally until specific conditions trigger malicious behavior.

Malicious fine-tuning data injection — OWASP LLM Top 10 LLM03 (Training Data Poisoning) identifies adversarial manipulation of training or fine-tuning data as a high-impact AI supply chain attack vector — an attack that occurs before model deployment and cannot be detected through runtime monitoring alone. Organizations that incorporate fine-tuning datasets from external sources without integrity verification introduce model behavior risk that no runtime control can remediate after the fact. (Source: owasp.org)

Attackers inject adversarial examples into datasets used to customize models for your environment. Impact: your fine-tuning process unknowingly trains biased or backdoored behavior into models that appear to meet performance benchmarks.

Compromised AI framework libraries — Vulnerabilities in inference engines, serving frameworks, and model quantization tools create code execution risks in the model serving pipeline. This is an area where traditional SCA and SBOM tooling does provide meaningful coverage — these are software dependencies that can be tracked, versioned, and scanned for known vulnerabilities. However, AI framework exploits can also manipulate model outputs directly or enable extraction of model weights through inference API abuse, creating risks that extend beyond what vulnerability scanning alone addresses. Impact: attackers gain execution access to model serving infrastructure or steal proprietary model assets through library-level compromise.

Dependency confusion in model repositories — Model distribution platforms operate like package repositories but often lack the namespace controls and integrity verification that mature software package ecosystems provide. Impact: attackers upload malicious models with names similar to legitimate models, causing organizations to unknowingly download compromised weights through typosquatting or repository confusion.

Supply chain blindness in model updates — Model versioning practices in many organizations do not track behavioral changes between model updates, making it difficult to detect when model behavior shifts unexpectedly after weight updates or framework library changes. Impact: compromised models can enter production through routine update processes without triggering security review or validation.

Governance and compliance implications

CISA's Guidelines for Secure AI System Development, co-authored with the UK's NCSC and international partners, highlight AI-specific supply chain risks that extend beyond traditional software security. The guidance recommends assessing the provenance of foundation models, verifying the security and integrity of training and fine-tuning datasets, and monitoring third-party AI components throughout their lifecycle. These measures address risks involving models, datasets, and external AI services that conventional software bills of materials (SBOMs) and software composition analysis (SCA) tools may not fully capture on their own (Source: CISA/NCSC, Guidelines for Secure AI System Development, 2023).

Current governance frameworks address AI supply chain risks inconsistently. NIST SP 800-53 AC (Access Control) and CM (Configuration Management) control families apply to AI model deployment but do not specify requirements for model provenance verification or training data integrity. CIS Controls 2 (Inventory and Control of Software Assets) captures software components — including AI framework libraries — but provides no guidance for inventorying model weights, training datasets, or AI-specific dependencies that affect application behavior without manifesting as code changes.

SOC 2 Type II logical access controls (CC6.1 and CC6.2) require documentation of system components that affect security-relevant processing, but many organizations exclude model weights and training datasets from this inventory by design. The compliance gap emerges where auditors examine software inventory completeness but find no controls governing the source, integrity, or behavioral validation of AI models that make security-relevant decisions.

ISO/IEC 27001 A.9 (Access Control) requires organizations to restrict access to information and application system functions, but model supply chain governance often lacks clear ownership assignment for model provenance verification, behavioral validation after updates, and incident response procedures specific to model compromise. Policy and practice diverge where organizations have change management procedures for software deployments but no equivalent controls for model weight updates that can alter application logic without code changes.

The enforcement gap widens in third-party risk management, where vendor assessment questionnaires capture software development practices but rarely address model training data sourcing, weight integrity verification, or supply chain security for AI components that vendors incorporate into their platforms.

What to review now

Inventory: Enumerate all base models in production use and identify their source repositories, download methods, and integrity verification status. Document which models lack provenance records showing origin, training methodology, and known data sources.

Validate: Check whether fine-tuning datasets used in model customization have documented sourcing and integrity verification. Identify datasets obtained from external sources without cryptographic verification or content validation.

Escalate: Flag to leadership any production models sourced from repositories that lack namespace controls, integrity verification, or established reputation — particularly models downloaded through manual processes rather than controlled deployment pipelines.

Document: Establish ownership records for each model in production, including who approved its incorporation, who validates behavioral consistency after updates, and who owns the rollback decision if model behavior changes unexpectedly.

Inventory: Map AI framework libraries and inference engines to your software composition analysis tooling. Identify AI-specific dependencies that are not tracked in current SCA processes or vulnerability management workflows. This is an area where existing SCA practices can be extended to provide meaningful coverage.

Validate: Test whether your incident response plan includes procedures specific to AI supply chain compromise, including criteria for triggering model rollbacks and decision authority for removing compromised models from production.

Document: Create behavioral baselines for production models that define expected output characteristics, enabling detection of unexpected behavior changes after model or framework updates.

Decision checklist

Sources