What powers your AI? A fragile mix of models, adapters, and weights
The LLM supply chain is the behind-the-scenes machinery that powers most generative AI: foundation models (the massive, pretrained systems like GPT-3 or LLaMA that everything else is built on), third-party adapters (small tuning files used to customize base models), safetensors (a format used to safely store model weights), and inference pipelines (the systems that actually run the model and produce output).These components are often distributed across the cloud or embedded in edge devices such as smartphones, smart speakers, or AI-powered surveillance cameras.The hidden role of adapters: Small files, big influence
Adapters in this context are small modules commonly used in a process called fine-tuning, adjusting a general-purpose model to specialize in a specific task, like legal writing or fraud detection. Rather than retraining a model from scratch, developers attach adapters that tweak its behavior by influencing its internal “weights,” which are numerical values the model uses to determine how to interpret input and generate output. These weights are the foundation of everything an AI system knows.But unlike traditional software, such as a web browser or operating system, AI systems are rarely distributed with clear security guardrails. There are few cryptographic signatures to prove where a model came from, little visibility into its history, and almost no guarantees that what you downloaded is what you think it is.“Most models are distributed without cryptographic signatures, with no SBOMs, and little visibility into their lineage,” OWASP warns. “This poses serious threats to consumers who rely on the integrity of downloaded models.”That lack of visibility carries real consequences.When Ray breaks: How ShadowRay opened the doors
In early 2024, researchers uncovered the ShadowRay attacks, where thousands of servers using the Ray AI framework were breached. Ray, an open-source tool often used to run AI workloads across cloud clusters, did not include authentication by default. Its creators argued that security was the user’s responsibility. But many organizations, including those in biotech, cryptocurrency, and academia, deployed Ray directly to the internet, leaving their clusters (groups of servers that work together to process large-scale jobs) wide open. Attackers moved in, injecting malicious commands, stealing data, and repurposing servers for their own use.“AI experts are NOT security experts,” warned Oligo Security, speaking to CSO Online in a 2024 article. “That leaves them dangerously unaware of the very real risks posed by AI frameworks.”Around the same time, Trail of Bits uncovered a GPU-based exploit known as LeftoverLocals. The vulnerability allowed adversaries to listen in on another user’s LLM session by reading leftover data from the shared memory of cloud-based graphics processors. This kind of attack is nearly invisible, especially in multi-tenant environments where customers share hardware.
Even trusted tools like Hugging face are vulnerable
Even trusted tools have become vectors. Hugging Face’s SFConvertbot, a utility designed to convert models into the safetensors format, was exploited by attackers who embedded malicious payloads in pull requests. While safetensors are meant to provide a safer way to store and share models — unlike older formats that could execute arbitrary code — they are only as safe as the upload process.“Consumers often trust usernames and star counts,” wrote HiddenLayer in a 2024 blog post. “But trust without verification is a recipe for sabotage.”Enter the AI-BOM: The new standard for trust and traceability
Software Bills of Materials (SBOMs) have become standard practice in cybersecurity. They act like ingredient labels, listing every library or component used in an application. But LLMs aren’t just software — they’re data, weights, parameters, and custom behavior stitched together from countless sources.That’s why experts now advocate for AI-BOMs: structured inventories that document not just code, but the model’s training data, fine-tuning history, adapter lineage, and more. Without these, companies may unknowingly deploy models with corrupted training inputs, conflicting licenses, or backdoors inserted by unknown actors.This gets even riskier with the rise of parameter-efficient fine-tuning (PEFT), which lets developers fine-tune massive models using compact modules like LoRA adapters. These adapters can be swapped in or merged dynamically, changing a model’s output in real time. LoRA (Low-Rank Adaptation) adapters work by altering specific layers of a base model rather than rewriting it entirely, kind of like clipping a new lens onto a camera.“The ability to dynamically recompose model behavior raises both opportunity and risk,” explains Hugging Face engineers.These tools offer speed and efficiency, but they also introduce complexity. How do you know what adapter is being loaded? Who created it? Was it tested? In many cases, you don’t. And with no requirement to track or verify these changes, organizations are flying blind.OWASP: AI supply chain threats are unlike anything we've seen
“Supply chain risks in AI differ from traditional software risks,” OWASP noted. “They're not just about libraries or packages. They're about compromised models, poisoned datasets, and hidden backdoors in seemingly benign tools.”The industry has spent the past few years celebrating the rapid democratization of AI. But as more enterprises adopt open-source models, integrate community-built adapters, and move LLMs to production, the cracks in the foundation are beginning to show.Where we go from here: From blind trust to secure pipelines
The first step is acknowledging that these systems are vulnerable by default and that many developers are unknowingly deploying compromised components.In Part Two, we’ll shift from the warning signs to the playbook. We’ll explore real-world solutions—from AI-BOMs and LoRA attestation to trusted model repositories and cryptographic signing—that aim to bring transparency, accountability, and sanity to the LLM supply chain.Because if we don’t fix this now, the next backdoor won’t be found in source code. It’ll be smiling at you through a chatbot window.[Editor's Note: This is part one of two where SC Media helps unpack OWASP's Top 10 for LLM Applications. Read part two: OWASP’s cure for a sick AI supply chain]
