Full Report
Wiz researchers discovered architecture risks that may compromise AI-as-a-Service providers and put customer data at risk. Wiz and Hugging Face worked together to mitigate the issue.
Analysis Summary
# Vulnerability: Cross-Tenant Access via Malicious AI Models (Container Escape)
## CVE Details
- CVE ID: *Not specified in the provided text*
- CVSS Score: *Not specified in the provided text*
- CWE: *Not explicitly numbered, but relates to insecure deserialization (Pickle loading) and insufficient isolation/sandboxing.*
## Affected Systems
- Products: Hugging Face Inference API (and potentially other similar AI-as-a-Service platforms)
- Versions: Not specified, generally applies to environments running untrusted models using the 'pickle' format in shared inference infrastructure.
- Configurations: Shared inference infrastructure executing custom, potentially malicious models serialized using the Python `pickle` format.
## Vulnerability Description
Wiz Research discovered risks associated with running untrusted AI models on shared inference infrastructure (like the Hugging Face Inference API). The primary issue stems from the use of the Python `pickle` serialization format for models. A malicious actor can upload a custom, pickle-serialized model containing a Remote Code Execution (RCE) payload. By leveraging container escape techniques, the attacker can break out of their tenant boundary and gain cross-tenant access to compromise the entire service, including accessing other customers' private models and data stored on the shared infrastructure. A secondary risk related to CI/CD pipeline takeover via malicious AI applications was also noted.
## Exploitation
- Status: PoC available (demonstrated by Wiz Research)
- Complexity: Implied Medium/High (Requires deep knowledge of container runtime environments and malicious model creation/payload insertion).
- Attack Vector: Network (via model upload/submission to the Inference API)
## Impact
- Confidentiality: High (Access to other tenants' private models and data)
- Integrity: High (Ability to execute arbitrary code on the host infrastructure)
- Availability: Medium/High (Potential service disruption due to takeover)
## Remediation
### Patches
- Hugging Face has implemented security improvements based on this research (details not listed, but collaboration was completed).
- General recommendations include: Ensuring models are running in strictly sandboxed environments; enforcing container escape prevention; enforcing authentication even for internal container registries.
### Workarounds
- Users should be extremely cautious about running untrusted AI models, especially those serialized using `pickle`.
- Utilize environments that ensure strict tenant separation and sandboxing for any external or untrusted model execution.
## Detection
- Indicators of compromise: Unexpected outbound connections, file system modifications, or privilege escalation attempts originating from the inference containers/workers used for model execution.
- Detection methods and tools: Implementing robust Cloud Security Posture Management (CSPM) and vulnerability scanning (as Hugging Face did) to identify toxic risk combinations. Monitoring for attempts to break out of containerized inference environments.
## References
- Vendor advisory (Hugging Face response): hXXps://huggingface.co/blog/hugging-face-wiz-security-blog
- Research context (Wiz Report): hXXps://www.wiz.io/blog/key-findings-from-the-state-of-ai-in-the-cloud-report-2024