Full Report
The technique, called nullifAI, allows the models to bypass Hugging Face’s protective measures against malicious AI models
Analysis Summary
# Vulnerability: Malicious ML Model Distribution via Abused Pickle Serialization
## CVE Details
- CVE ID: Not Assigned (Novel distribution technique/platform security flaw)
- CVSS Score: N/A (Relates to platform security enforcement rather than single software component vulnerability)
- CWE: CWE-502: Deserialization of Untrusted Data (Related weakness pattern)
## Affected Systems
- Products: Hugging Face ML Model Platform
- Versions: Prior version of the Picklescan security scanning mechanism.
- Configurations: ML models packaged using 7z compression containing serialized Pickle files with broken structures or malicious opcodes positioned early in the stream.
## Vulnerability Description
Researchers discovered two malicious machine learning models hosted on Hugging Face that successfully bypassed the platform's `Picklescan` security tool. The attack abuses Python's `Pickle` serialization mechanism, which inherently allows arbitrary code execution upon deserialization.
The novel aspect of this exploitation involves two tricks:
1. **File Compression:** Models were stored in PyTorch format but compressed using **7z** instead of the traditional ZIP format. This caused the standard loading mechanism (`_torch.load()`) to fail, potentially bypassing initial checks reliant on standard PyTorch loading procedures.
2. **Broken Pickle Files:** The malicious payloads (malicious opcodes) were strategically placed early in the Pickle stream. Because `Picklescan` validates the file *before* deep scanning, or because the deserialization process interprets opcodes sequentially until it hits a 'broken' instruction, the scanner failed to execute or detect the malicious code embedded in what appeared to be a corrupted file.
## Exploitation
- Status: Proof-of-Concept models detected; not publicly confirmed as exploited in the wild prior to researcher discovery.
- Complexity: Medium (Requires understanding of Pickle internals, opcodes, and bypassing specific platform scanning logic).
- Attack Vector: Network (Uploading malicious assets to the platform).
## Impact
- Confidentiality: High (If successfully executed, the payload could lead to information leakage).
- Integrity: High (Arbitrary code execution can lead to model tampering or data corruption).
- Availability: Medium (Payload execution could potentially disrupt model serving or platform stability).
## Remediation
### Patches
- Hugging Face updated the Picklescan tool to handle and detect threats within "broken" Pickle files. (Specific version updates not detailed in the text).
### Workarounds
- **Trusted Sources Only:** Users should only load ML models from sources they fully trust, especially when those models utilize Pickle serialization.
- **Manual Inspection:** For high-risk environments, manually inspecting model repositories or avoiding Pickle-serialized models altogether may be necessary until full remediation confidence is established.
## Detection
- **Indicators of Compromise:** Models exhibiting unusual file compression formats (e.g., 7z for PyTorch models) or those that fail expected loading routines when using standard functions.
- **Detection Methods and Tools:** The update to Picklescan specifically targets the sequential execution mechanism of Pickle, validating files or scanning for nefarious opcodes even in seemingly corrupted streams.
## References
- Vendor advisories: Hugging Face Security Update (Assumed following RL notification).
- Relevant links - defanged: hXXps://www.reversinglabs.com/blog/rl-identifies-malware-ml-model-hosted-on-hugging-face