Full Report
The incident highlights growing concerns over the security of the open-source software supply chain, where widely-used tools maintained by small teams can provide a gateway into thousands of organizations if compromised.
Analysis Summary
# Incident Report: Supply Chain Compromise of LiteLLM Python Package
## Executive Summary
The widely-used open-source Python package `liteLLM` was targeted in a supply chain attack where malicious versions (1.82.7 and 1.82.8) were published to the Python Package Index (PyPI). The compromised code was designed to steal cloud credentials, API keys, and cryptocurrency wallets while establishing persistence via a downloader. Attributed to the threat group "TeamPCP," the incident potentially impacts thousands of organizations, given that the package is present in an estimated 36% of cloud environments.
## Incident Details
- **Discovery Date:** March 24, 2026
- **Incident Date:** March 24, 2026 (Active for at least two hours)
- **Affected Organization:** Users of `liteLLM` open-source package
- **Sector:** Technology / Software Development / Artificial Intelligence
- **Geography:** Global
## Timeline of Events
### Initial Access
- **Date/Time:** March 24, 2026
- **Vector:** Likely account takeover (ATO) of a package maintainer.
- **Details:** Malicious versions 1.82.7 and 1.82.8 were uploaded to PyPI using valid publishing credentials, allowing the attackers to bypass standard trust barriers.
### Lateral Movement
- **Snowball Effect:** The attackers used the compromised package to gain entry into development and cloud environments. By stealing cloud management credentials, the attackers positioned themselves to move from the application layer into broader cloud infrastructure.
### Data Exfiltration/Impact
- **Theft:** Automated extraction of sensitive environment variables, cloud service provider credentials, AI API keys, and local cryptocurrency wallet files.
- **Persistence:** Installation of a persistent downloader intended for subsequent malware delivery.
### Detection & Response
- **Detection:** Identified by security researchers at Sonatype and Wiz Research.
- **Response Actions:** Public disclosure of the malicious versions; community alerts issued to roll back versions and rotate all secrets exposed in affected environments.
## Attack Methodology
- **Initial Access:** Supply Chain Attack (Compromised PyPI package).
- **Persistence:** Installation of a persistent downloader; maintainers used a heartbeat mechanism (50-minute interval) to check for tasks.
- **Privilege Escalation:** Not explicitly detailed, but implied via the theft of high-privilege cloud credentials.
- **Defense Evasion:** 50-minute execution delay to bypass automated sandboxes; selective payload delivery (sometimes serving YouTube links to researchers instead of malware).
- **Credential Access:** Extraction of API keys and cloud authentication tokens.
- **Discovery:** Automated scanning for `.env` files and credential stores.
- **Lateral Movement:** Cloud-to-cloud movement via stolen management keys.
- **Collection:** Automated gathering of secrets and wallet data.
- **Exfiltration:** Data sent to attacker-controlled command endpoints.
- **Impact:** Potential for total cloud environment takeover and financial theft.
## Impact Assessment
- **Financial:** Risk of cryptocurrency theft and unauthorized cloud resource consumption; long-term costs of incident response and credential rotation.
- **Data Breach:** High risk; includes API keys and cloud access tokens.
- **Operational:** Disruption to development pipelines as teams scramble to audit and purge the malicious package.
- **Reputational:** Significant; affects trust in the `liteLLM` project and highlights vulnerabilities in the AI software ecosystem.
## Indicators of Compromise
- **File indicators:**
- `liteLLM` versions 1.82.7
- `liteLLM` versions 1.82.8
- **Behavioral indicators:**
- Network requests to external endpoints every 50 minutes.
- Unexpected outbound traffic from development environments to unknown command-and-control (C2) servers.
## Response Actions
- **Containment:** Removal of malicious versions from PyPI (standard procedure for such incidents).
- **Eradication:** Users advised to uninstall affected versions and audit all systems for the downloader.
- **Recovery:** Mandatory rotation of all credentials (AWS, Azure, GCP keys, etc.) that existed in environments where the compromised package was executed.
## Lessons Learned
- **Dependency Proliferation:** Small teams maintaining high-impact tools represent a critical single point of failure.
- **Sandbox Limitations:** Attackers are increasingly aware of sandbox timeout windows (typically <10 minutes) and are using long delays to evade detection.
- **Need for MFA:** The suspected account takeover emphasizes the need for Mandatory Multi-Factor Authentication (MFA) for all PyPI and GitHub maintainers.
## Recommendations
- **Version Pinning:** Pin requirements to specific, vetted versions rather than allowing automatic updates to the latest release.
- **Secret Scanning:** Implement automated tools to detect if environment variables or credentials have been accessed by unauthorized processes.
- **Zero Trust Architecture:** Limit the scope of API keys and cloud credentials used in development environments.
- **Software Composition Analysis (SCA):** Utilize SCA tools to monitor for "known-malicious" versions of dependencies in real-time.