Full Report
A malicious Hugging Face repository managed to take a spot in the platform's trending list by impersonating OpenAI's Privacy Filter open-weight model to deliver a Rust-based information stealer to Windows users. The project, named Open-OSS/privacy-filter, masqueraded as its legitimate counterpart released by OpenAI late last month (openai/privacy-filter), including copying the entire description
Analysis Summary
# Incident Report: Malicious Impersonation of OpenAI Privacy Filter
## Executive Summary
A malicious repository named `Open-OSS/privacy-filter` on Hugging Face impersonated a legitimate OpenAI release to distribute a Rust-based information stealer. By leveraging typosquatting and copying legitimate documentation, the attackers reached the platform's "trending" list to target Windows users. The incident highlights the growing risk of supply chain attacks within AI model registries.
## Incident Details
- **Discovery Date:** Early October 2024 (following the legitimate OpenAI release in late September)
- **Incident Date:** September - October 2024
- **Affected Organization:** Hugging Face users (specifically Windows users downloading AI models)
- **Sector:** Technology / Artificial Intelligence / Research
- **Geography:** Global
## Timeline of Events
### Initial Access
- **Date/Time:** Late September 2024
- **Vector:** Typosquatting / Brand Impersonation
- **Details:** Attackers created the repository `Open-OSS/privacy-filter` shortly after OpenAI released the authentic `openai/privacy-filter`. They performed "Star-jacking" or engagement manipulation to propel the repository into Hugging Face’s trending section.
### Lateral Movement
- **Details:** N/A - This was an external supply chain attack. However, the malware aimed to gain lateral movement capabilities by harvesting credentials from infected workstations.
### Data Exfiltration/Impact
- **Details:** The Rust-based malware targeted sensitive data on Windows systems, including browser credentials, cookies, crypto wallets, and system metadata.
### Detection & Response
- **How it was discovered:** Security researchers and platform monitoring identified the anomalous repository rising in the trending ranks despite being a duplicate.
- **Response actions taken:** Hugging Face removed the malicious repository and the associated user account.
## Attack Methodology
- **Initial Access:** Social Engineering/Masquerading; impersonating legitimate OpenAI open-weight models.
- **Persistence:** Not explicitly detailed, but standard for info-stealers via registry keys or startup folder placement.
- **Privilege Escalation:** Exploitation of local user permissions on Windows workstations.
- **Defense Evasion:** Use of the Rust programming language (which often has lower detection rates in legacy AV) and hosting on a trusted domain (huggingface[.]co).
- **Credential Access:** Harvesting saved passwords and session tokens from web browsers.
- **Discovery:** System profiling of the host Windows environment.
- **Collection:** Gathering browser data, Discord tokens, and cryptocurrency wallet files.
- **Exfiltration:** Data sent to an attacker-controlled Command and Control (C2) server.
- **Impact:** Compromise of user accounts and potential unauthorized access to downstream corporate resources.
## Impact Assessment
- **Financial:** Potential loss from stolen cryptocurrency and costs associated with incident response for affected organizations.
- **Data Breach:** High; theft of personal and professional credentials.
- **Operational:** Minimal disruption to Hugging Face, but high risk to individual developer workstations.
- **Reputational:** Moderate; raises concerns regarding the vetting process of "trending" models on AI repositories.
## Indicators of Compromise
- **Network Indicators:**
- Communications with known Rust-stealer C2 endpoints (specific IPs not provided in summary).
- Downloads from `huggingface[.]co/Open-OSS/privacy-filter`.
- **File Indicators:**
- Malicious `.exe` or library files within the `Open-OSS` repository.
- Rust-based binary execution in temporary directories.
- **Behavioral Indicators:**
- Unexpected outbound connections from Python/AI development environments to unknown external IPs.
## Response Actions
- **Containment:** Removal of the `Open-OSS` repository from the Hugging Face platform.
- **Eradication:** Deletion of the malicious user account associated with the upload.
- **Recovery:** Public disclosure by security researchers to alert users who may have downloaded the model.
## Lessons Learned
- **Metadata Reliability:** Repository descriptions and "trending" status are not indicators of safety; attackers can easily clone documentation.
- **Platform Trust:** Users often implicitly trust repositories hosted on major platforms like Hugging Face, especially when they appear to be affiliated with major entities like OpenAI.
- **Binary Risks:** AI models are not just weights; accompanying scripts (e.g., `setup.py`, `install.sh`) or spiked binaries can carry traditional malware.
## Recommendations
- **Verification:** Always verify the official organization name (e.g., `openai/`) before downloading models.
- **Sandboxing:** Execute and test new models in isolated environments (containers or VMs) without access to sensitive host credentials.
- **Supply Chain Security:** Implement software composition analysis (SCA) tools that scan AI model repositories for known malicious signatures.
- **Platform Vigilance:** Registries should implement stricter "verified" badges for high-profile AI organizations to prevent impersonation.