The AI model supply chain: how a poisoned model reaches production
“We downloaded the model from Hugging Face. It downloaded our credentials to an attacker.”
TL;DR
JFrog found 100+ malicious models on Hugging Face (February 2024). Protect AI scanned 4.47 million model versions: 352,000 unsafe issues across 51,700 models. NullifAI bypassed Picklescan, the primary scanner. Hugging Face added 1 million new models in 2024. The supply chain spans base models, datasets, adapters, plugins, and RAG corpora. Cryptographic model provenance is the structural fix the ecosystem hasn’t adopted. For how backdoors survive downstream use, see the companion post on Weight poisoning and model backdoors.
What is the AI model supply chain?
Every component that goes into a production AI system, from the base model to the last data point in the RAG corpus.
Base models are the foundation. Organizations download pre-trained models from Hugging Face, replicate, or directly from providers. The model’s weights carry everything the pre-training data contained, including any backdoors or biases in the training process.
Fine-tuning datasets shape the model’s behavior for the specific use case. If the fine-tuning data is poisoned (containing adversarial examples, biased samples, or backdoor triggers), the fine-tuned model inherits those properties. Data sourced from the internet, scraped from user interactions, or purchased from data brokers may contain content the organization never reviewed.
Adapters and LoRA weights modify the base model without full retraining. They’re smaller, easier to distribute, and therefore easier to poisoned and harder to audit. A malicious LoRA adapter from an untrusted source can alter model behavior while the base model appears unchanged.
Plugins and tools (MCP servers, function calling integrations) execute code in the model’s context. A malicious plugin can exfiltrate data, modify outputs, or compromise the host system. With 13,000+ MCP servers on GitHub with no verification, the plugin supply chain is the AI equivalent of npm before lockfiles.
RAG corpora provide the model’s retrieval context. Poisoned documents in the vector store influence every query that retrieves them. The RAG supply chain is anyone who can add documents to the indexed data sources.
How do malicious models reach Hugging Face?
JFrog’s February 2024 discovery found over 100 malicious models using Python’s Pickle deserialization as the attack vector. When a model is saved in Pickle format (the default for PyTorch), it can contain a __reduce__ method that executes arbitrary Python code when the model file is loaded. Loading the model = running the attacker’s code. JFrog found models that established reverse shells, granting the attacker full access to the victim’s machine.
Three upload vectors:
Typosquatting. Upload a malicious model with a name similar to a popular model: bert-base-uncasedd instead of bert-base-uncased. A developer in a hurry downloads the wrong version. The model works (it might even be a copy of the legitimate model) but executes the payload on load.
Hijacked accounts. Compromise a legitimate model author’s Hugging Face account and modify their existing models. The model retains the trusted author’s name and download count. In June 2024, Hugging Face disclosed unauthorized access to their Spaces platform exposing authentication tokens. Lasso Security found 1,600+ exposed Hugging Face API tokens in public repositories, affecting 723 organizations including Google, Meta, and Microsoft.
Direct upload. Simply publish a new model with a Pickle payload. There was no mandatory scanning at upload time during JFrog’s initial discovery.
How did nullifAI bypass Picklescan?
Picklescan is Hugging Face’s primary safety tool for detecting malicious Pickle files. ReversingLabs discovered the nullifAI evasion technique in February 2025.
The technique is elegant in its simplicity: instead of compressing the Pickle payload with ZIP (which Picklescan knows how to inspect), the attacker compresses it with 7z. PyTorch’s torch.load() handles both compression formats and loads the model normally. Picklescan only inspects ZIP-compressed archives and fails silently on 7z. The model loads, the payload executes, and the scanner reports “clean.”
Hugging Face removed affected models within 24 hours of ReversingLabs’ report and updated Picklescan. But the incident demonstrates the fragility of relying on a single scanning tool as the primary security boundary. Any file format change, compression variant, or encoding trick that the scanner doesn’t handle creates a bypass.
Protect AI’s broader scans tell the scale story: 4.47 million model versions scanned, 352,000 unsafe or suspicious issues found across 51,700 models. Hugging Face added 1 million new models in 2024 alone. The ratio of scanning throughput to upload velocity is not in the defenders’ favor.
What does model provenance look like?
The structural fix is ensuring every model has a verifiable chain of custody from creation to deployment.
Cryptographic signing. The model creator signs the model files with their private key. Anyone downloading the model can verify the signature against the creator’s public key. If the model was modified after signing (payload injection, weight tampering), the signature fails. This doesn’t prevent a malicious creator from signing a malicious model, but it prevents tampering after the fact.
Hash verification. Every model file should have a published hash (SHA-256 or stronger). Organizations verify the hash of downloaded files against the published hash before loading. This catches tampering in transit and ensures the downloaded file matches what the publisher intended.
SafeTensors format. Hugging Face developed SafeTensors as a Pickle-free model format. SafeTensors stores only tensor data, not arbitrary Python objects. Loading a SafeTensors file cannot execute code because the format doesn’t support it. Migration from Pickle to SafeTensors eliminates the entire class of deserialization attacks. The adoption gap: many models and tools still use Pickle by default.
NSA/CISA guidance (2025) provides procurement-oriented controls: verify model provenance before deployment, maintain model inventories, scan models with multiple tools, run models in sandboxed environments, and implement monitoring for anomalous model behavior in production.
Key takeaways
- 100+ malicious models with code execution payloads found on Hugging Face (JFrog, 2024). 352,000 unsafe issues across 51,700 models (Protect AI)
- NullifAI bypass: changing Pickle compression from ZIP to 7z evaded Picklescan entirely
- Supply chain spans base models, fine-tuning data, adapters, plugins, and RAG corpora. Each is a potential poisoning vector.
- 1 million new models added to Hugging Face in 2024. Scanning can’t keep up.
- Structural fixes: cryptographic signing, hash verification, SafeTensors format (eliminates Pickle deserialization), and multi-tool scanning before deployment
- 1,600+ exposed Hugging Face API tokens found in public repos affecting 723 organizations
FAQ
How do malicious models get onto Hugging Face?
Typosquatting (similar names), hijacked accounts, and direct upload with Pickle payloads. Python’s reduce method executes arbitrary code when a Pickle model loads. No mandatory scanning existed at upload time during JFrog’s initial discovery.
What is nullifAI?
An evasion technique compressing Pickle payloads with 7z instead of ZIP. Picklescan only inspects ZIP archives. The model loads normally in PyTorch but the scanner misses the payload. Fixed within 24 hours of disclosure.
How many Hugging Face models are unsafe?
352,000 unsafe issues across 51,700 models from 4.47 million versions scanned. 100+ with active code execution payloads. 1 million new models added in 2024. The scanning-to-upload ratio is falling behind.
What is the fix?
Cryptographic signing, hash verification, SafeTensors format (no Pickle deserialization), multi-tool scanning, sandboxed execution, and model inventories with provenance chains. NSA/CISA published procurement guidance in 2025.
Want to work together?
I take on projects, advisory roles, and fractional CTO engagements in AI/ML. I also help businesses go AI-native with agentic workflows and agent orchestration.
Get in touch