Full Report
The developers of PyTorch (a popular machine-learning framework) recently identified a malicious dependency confusion attack on the open-source project. Security teams are advised to check for infected resources and rotate any exposed keys.
Analysis Summary
# Incident Report: PyPI Dependency Confusion Attack Targeting PyTorch Nightly Builds
## Executive Summary
An attacker exploited a dependency confusion vulnerability by uploading a malicious package named `torchtriton` to the Python Package Index (PyPI), mimicking an internal package used by PyTorch nightly builds. Users installing PyTorch nightly builds via `pip` between December 25 and December 30, 2022, were unknowingly infected. The malware collected sensitive system and user data, exfiltrating it via DNS tunneling to a malicious domain. The incident was resolved by removing the malicious package and replacing it with a placeholder, followed by security recommendations to uninstall the package and rotate secrets.
## Incident Details
- **Discovery Date:** Shortly after December 30, 2022 (implied by the end date of the infection window).
- **Incident Date:** Between December 25, 2022, and December 30, 2022.
- **Affected Organization:** PyTorch developers/users of PyTorch nightly builds.
- **Sector:** Software Development/Machine Learning Infrastructure.
- **Geography:** Global (affecting anyone using the specified PyPI installation method).
## Timeline of Events
### Initial Access
- **Date/Time:** On or around December 25, 2022.
- **Vector:** Dependency Confusion via PyPI.
- **Details:** An attacker uploaded a package named `torchtriton` to PyPI. When users installed PyTorch nightly builds using `pip` with the `extra-index-url` argument, `pip` prioritized the malicious public PyPI version over the intended internal registry version.
### Lateral Movement
- **Details:** Once executed, the malicious `triton` binary collected existing system information, environment variables, and files from the user's home directory (`$HOME`). Specific file targets included `/etc/hosts`, `/etc/passwd`, `$HOME/.gitconfig`, and `.ssh/` directories.
### Data Exfiltration/Impact
- **Details:** Collected data (including hostnames, usernames, resolutions, and credential/SSH file contents) was exfiltrated using DNS tunneling to the domain `h4ck[.]cfd`, resolving through the DNS server `wheezy[.]io`.
### Detection & Response
- **Details:** The malicious package was eventually identified and removed from PyPI and replaced with a placeholder package named `pytorch-triton`. Security teams were instructed to scan environments, uninstall the package, and rotate any exposed secrets.
## Attack Methodology
- **Initial Access:** Dependency Confusion (exploiting `pip` behavior where public PyPI is prioritized when using `--extra-index-url`).
- **Persistence:** Not explicitly detailed, but malware was installed as part of the dependency.
- **Privilege Escalation:** Not explicitly detailed, likely relying on the privileges of the user running the installation/import.
- **Defense Evasion:** Infiltration via a seemingly legitimate, highly trusted software supply chain source (PyPI).
- **Credential Access:** Directly copied contents from sensitive files like `$HOME/.ssh/` and `$HOME/.gitconfig`.
- **Discovery:** Collected hostname, current working directory, nameservers (`/etc/resolv.conf`), and environment variables.
- **Lateral Movement:** Malware installed a malicious binary within the Python site packages path: `PYTHON_SITE_PACKAGES/triton/runtime/triton`.
- **Collection:** System information scraping and file copying from sensitive user locations.
- **Exfiltration:** Data uploaded via DNS tunneling to `h4ck[.]cfd`.
- **Impact:** Theft of system credentials, keys, and configuration data.
## Impact Assessment
- **Financial:** Not estimated in the source, but likely included remediation costs and potential losses from compromised credentials/secrets.
- **Data Breach:** Sensitive configuration files, SSH keys, `.gitconfig` files, system data (hostname, user, environment variables) from affected machines. Over 3,000 installations occurred.
- **Operational:** Disruption requiring users to uninstall packages, scan resources, and rotate secrets.
- **Reputational:** Negative impact on the trust associated with PyTorch nightly builds and PyPI security.
## Indicators of Compromise
Note: Hashes are provided for reference but should not be used for scanning in a production environment without verification.
- **Network Indicators:**
- Exfiltration Domain (defanged): `h4ck[.]cfd`
- DNS Server (defanged): `wheezy[.]io`
- **File Indicators:**
- Malicious Binary Path: `PYTHON_SITE_PACKAGES/triton/runtime/triton`
- SHA256 Hash: `2385b29489cd9e35f92c072780f903ae2e517ed422eae67246ae50a5cc738a0e` (Use with caution)
- SHA1 Hash: `9c89731a94dd4f2a594a22c1b171cb6c20f55e41` (Use with caution)
- **Behavioral Indicators:**
- Execution of code that imports the `triton` package within the `torchtriton` dependency.
- Outbound DNS queries to the specified malicious domain.
## Response Actions
- **Containment measures:** The malicious `torchtriton` package was removed from PyPI and replaced with a placeholder (`pytorch-triton`).
- **Eradication steps:** Users were advised to uninstall the package if they installed the nightly build during the window.
- **Recovery actions:** Security teams were advised to scan affected resources for secrets and rotate any discovered keys/credentials.
## Lessons Learned
- **Key takeaways:** Dependency confusion remains a potent supply chain attack vector, particularly when internal resource names conflict with public repositories, exploiting default package manager resolution logic.
- **What could have been done better:** PyTorch could have claimed the `torchtriton` namespace on PyPI preemptively, even if only a placeholder was needed, to prevent namespace squatting.
## Recommendations
- **Prevention measures for similar incidents:**
1. Preemptively register all intended internal package names on public registries (like PyPI) to prevent namespace squatting and dependency confusion attacks.
2. Users of internal registries when using public package managers should review configuration (e.g., `extra-index-url`) to ensure internal artifacts are correctly prioritized, or use clearly distinguished namespaces.
3. Immediately uninstall the package if the nightly build was installed between Dec 25 and Dec 30, 2022.
4. Conduct comprehensive secret scanning on any potentially compromised machine and rotate credentials/SSH keys.