Full Report
The Grafana data breach was caused by a single GitHub workflow token that slipped through the rotation process following the TanStack npm supply-chain attack last week. [...]
Analysis Summary
# Incident Report: Grafana Source Code and Operational Data Breach
## Executive Summary
The Grafana data breach was a supply-chain-originated incident caused by the execution of a malicious npm package (TanStack) within Grafana’s CI/CD environment. Although the company initiated token rotations after the initial discovery, a single missed GitHub workflow token allowed attackers to maintain access and exfiltrate private source code and business contact information. No customer production systems or cloud platforms were compromised, and no unauthorized code modifications were detected.
## Incident Details
- **Discovery Date:** May 1, 2026
- **Incident Date:** Late April to early May 2026
- **Affected Organization:** Grafana Labs
- **Sector:** Technology / Software Development
- **Geography:** Global
## Timeline of Events
### Initial Access
- **Date/Time:** Late April 2026 (Following the TanStack npm supply-chain attack)
- **Vector:** Supply Chain Compromise (npm)
- **Details:** Grafana’s CI/CD workflow consumed a malicious version of a TanStack npm package infected with "Shai-Hulud" malware. The malware executed in the GitHub environment and exfiltrated workflow tokens to the attackers.
### Lateral Movement
- **Mechanism:** Using the stolen GitHub workflow token, the threat actors (TeamPCP) gained unauthorized access to Grafana’s private GitHub repositories.
### Data Exfiltration/Impact
- **Assets Stolen:**
- Private source code.
- Operational information and business details.
- Business contact names and email addresses.
### Detection & Response
- **May 1, 2026:** Grafana detected malicious activity related to the TanStack packages and initiated incident response.
- **Initial Response:** Rotation of a "significant number" of GitHub tokens; however, one token was missed.
- **Subsequent Discovery:** Further review confirmed a specific workflow (initially deemed safe) was compromised via the missed token.
## Attack Methodology
- **Initial Access:** Supply Chain Attack via malicious npm package (TanStack).
- **Persistence:** Use of a valid, non-rotated GitHub workflow token.
- **Privilege Escalation:** Not explicitly detailed, but involved access to private repository resources via CI/CD service accounts.
- **Defense Evasion:** Malware embedded in legitimate library updates; use of valid credentials (tokens) for repository access.
- **Credential Access:** Credential-stealing code ("Shai-Hulud" malware) executed within the CI/CD environment to capture tokens.
- **Discovery:** Automated exfiltration of environment variables/secrets.
- **Lateral Movement:** Movement from CI/CD pipeline environment to private GitHub repositories.
- **Collection:** Gathering source code and business operational data.
- **Exfiltration:** Data sent to attacker-controlled infrastructure.
- **Impact:** Source code theft and exposure of business contact information.
## Impact Assessment
- **Financial:** No ransom paid; costs associated with incident response and remediation.
- **Data Breach:** Source code and business contact lists; no customer production data or Grafana Cloud data affected.
- **Operational:** Disruption to development workflows; emergency token rotation and auditing.
- **Reputational:** Public disclosure of a security breach resulting from a missed step in the remediation process.
## Indicators of Compromise
- **Network indicators:** Traffic to known TeamPCP / Shai-Hulud C2 infrastructure (not specified in article).
- **File indicators:** Malicious `tanstack` or `mistral` npm packages published during the campaign.
- **Behavioral indicators:** Unauthorized GitHub repository clones/access from unexpected IP addresses or through workflow tokens outside of normal CI/CD windows.
## Response Actions
- **Containment measures:** Deployment of incident response plan immediately upon detection.
- **Eradication steps:** Mass rotation of GitHub workflow tokens.
- **Recovery actions:** Forensic analysis of code repositories to ensure no malicious code was injected (verified clean).
- **Communication:** Public security updates and direct notifications to stakeholders.
## Lessons Learned
- **The "Validation Gap":** Assets initially deemed "not impacted" during an incident must be re-verified with high-fidelity logs.
- **Remediation Completeness:** A single missed credential in a rotation process can negate the security benefits of rotating hundreds of others.
- **Supply Chain Vulnerability:** Heavily used development dependencies (npm) remain a high-risk vector for CI/CD environments.
## Recommendations
- **Automated Token Rotation:** Implement automated, programmatic rotation of CI/CD secrets to minimize human error.
- **Dependency Pinning & Auditing:** Lock down dependency versions (e.g., via `package-lock.json`) and use automated tools to block high-risk or newly published suspicious packages.
- **Least Privilege for Tokens:** Ensure GitHub workflow tokens have the minimum necessary permissions (e.g., read-only where possible) and are restricted by network or environment where feasible.
- **Secret Scanning:** Implement continuous monitoring to detect if internal tokens are being used from unauthorized locations.