Hugging Face model namespace reuse poses AI supply chain risk

Hugging Face model namespaces could be reclaimed by malicious actors after a model author deletes their account, posing a significant AI supply chain risk, Palo Alto Networks Unit 42 reported Wednesday.  

Open-source models hosted on Hugging Face are Git repositories identified by namespaces following the format Author/ModelName. Unit 42 researchers found that when an author’s Hugging Face account is deleted, it can be re-registered by someone else, making it possible for namespaces to be recreated and hijacked by malicious actors.

If an attacker registers the Hugging Face username of a deleted author and uploads their own malicious version of one of the old author’s open-source models, projects that are dependent on that model and use its namespace to fetch its Hugging Face library are at risk of running the threat actor’s malicious code.

Unit 42 also found another potential version of this attack involving the way model ownership is transferred on Hugging Face. When a model’s namespace changes due to a change in author (for example, the model is transferred to a new organization after an acquisition), Hugging Face automatically redirects references to the old model namespace to the new one.

However, the researchers found that if the old author account is deleted and registered by someone else who then uploads a model with the same name, the redirect will no longer occur and the new model will fetched instead.

This can similarly be misused by attackers to introduce malicious models into the supply chain and may be less likely to draw suspicion due to a lack of downtime between the original namespace retiring and the hijacked version appearing.

Unit 42 identified several “orphaned” models on Hugging Face whose namespaces could potentially be hijacked and re-registered some of these namespaces to demonstrate how malicious code, such as a reverse shell, could be injected in a supply chain attack.

They also found orphaned models sourced from Hugging Face in Google’s Vertex AI Model Garden and Microsoft’s Azure AI Foundry Model Catalog. Palo Alto Networks reached out to Google, Microsoft and Hugging Face about their findings, and noted that Google has begun scanning models for deleted authors daily, preventing orphaned models from being deployed to Vertex AI.

References to orphaned models were also found in libraries of several open-source projects on GitHub, some of which were noted to be popular and widely used. Developers can avoid falling victim to supply chain attacks from re-registered model name spaces through methods such as version pinning and model cloning.

When writing methods to fetch Hugging Face models such as from_pretrained(“Author/ModelName”), developers can use the “revision” parameter to pin a specific model version, ensuring only that trusted version will be fetched even if the namespace changes hands, the researchers wrote.

Additionally, developers may choose to clone a trusted open-source model repository to a trusted location, such as local storage, to prevent external tampering.

Lastly, Palo Alto Networks recommends scanning code repositories for model references before use and ensure that any models fetched are trusted and not orphaned.

“Scan model references in code repositories and treat model references like any other dependency subject to policy and review,” the researchers wrote. “Scanning should be comprehensive as models can exist in unexpected places, such as default arguments, docstrings and comments. Proactively scanning codebases for model references reduces the risk of supply chain attacks caused by model namespace reuse.”