Full Report
On December 29, 2022, CircleCI's security team were alerted to suspicious activity on one of their customer's GitHub OAuth tokens. The team then rotated all GitHub OAuth tokens on December 31, 2022 as a precautionary measure. By January 4, 2023, CircleCI's internal investigati...
Analysis Summary
# Incident Report: CircleCI Platform Compromise (Dec 2022 - Jan 2023)
## Executive Summary
In late December 2022, CircleCI experienced a significant security breach originating from a malware infection on an engineer's laptop. The threat actor stole session tokens to bypass Multi-Factor Authentication (MFA) and access internal production systems, leading to the unauthorized exfiltration of customer secrets and environment variables. CircleCI responded by rotating all GitHub OAuth tokens and advising customers to rotate their internal credentials to mitigate risk.
## Incident Details
- **Discovery Date:** December 29, 2022
- **Incident Date:** December 16, 2022 – January 4, 2023
- **Affected Organization:** CircleCI
- **Sector:** Software Development / DevOps (SaaS)
- **Geography:** Global
## Timeline of Events
### Initial Access
- **Date/Time:** December 16, 2022
- **Vector:** Malware (Infostealer)
- **Details:** An engineer’s workstation was compromised by malware that bypassed antivirus software. This allowed the attacker to steal a valid, active SSO session token.
### Lateral Movement
- **Date/Time:** Mid-to-late December 2022
- **Details:** Using the stolen session token, the attacker impersonated the engineer. Because the session was already authenticated, the attacker bypassed MFA requirements and gained access to CircleCI’s internal development and production environments.
### Data Exfiltration/Impact
- **Date/Time:** Ongoing until January 4, 2023
- **Details:** The attacker accessed internal databases and exfiltrated customer secrets, including environment variables, SSH keys, and GitHub OAuth tokens stored in the platform.
### Detection & Response
- **December 29, 2022:** CircleCI notified of suspicious GitHub OAuth token activity by a customer.
- **December 31, 2022:** CircleCI proactively rotated all GitHub OAuth tokens.
- **January 4, 2023:** CircleCI internal investigation confirmed a broad system compromise; a public advisory was issued urging customers to rotate all secrets.
- **January 5, 2023:** Full platform remediation and system rebuilding began.
## Attack Methodology
- **Initial Access:** Malware infection on a remote employee's laptop.
- **Persistence:** High-privilege session token theft; the attacker did not need to maintain a persistent backdoor if they held "live" tokens.
- **Privilege Escalation:** Use of administrative/engineer session tokens to access sensitive production databases.
- **Defense Evasion:** Use of legitimate session tokens to appear as an authorized user; malware was reportedly undetected by existing endpoint protection.
- **Credential Access:** Extraction of data from CircleCI’s encrypted data stores.
- **Discovery:** Reconnaissance of internal production databases and secret management systems.
- **Lateral Movement:** Transitioning from an employee workstation to cloud production environments via SSO session hijacking.
- **Collection:** Automated harvesting of customer configuration data and secrets.
- **Exfiltration:** Data transferred to attacker-controlled infrastructure.
- **Impact:** Significant customer-side risk requiring mass rotation of global development secrets.
## Impact Assessment
- **Financial:** Significant costs associated with incident response, forensic auditing, and legal counsel.
- **Data Breach:** Compromise of an undisclosed volume of customer environment variables, API keys, and private keys.
- **Operational:** Disruption of CI/CD pipelines as customers were forced to pause deployments to rotate credentials.
- **Reputational:** High; widespread concern across the DevOps community regarding the security of third-party supply chain tools.
## Indicators of Compromise
- **Network indicators:** Activity from unauthorized IP addresses [defanged]: `162[.]247[.]74[.]201`, `162[.]247[.]74[.]206`.
- **File indicators:** Presence of unidentified infostealer malware on the compromised workstation.
- **Behavioral indicators:** Unusual database queries and high-volume data access patterns from a single engineer's session.
## Response Actions
- **Containment:** Revoked all active sessions and rotated all internal/customer GitHub OAuth tokens.
- **Eradication:** Isolated the infected workstation and wiped/rebuilt compromised production hosts.
- **Recovery:** Collaborated with law enforcement and third-party forensics teams to validate the environment was clean before resuming normal operations.
## Lessons Learned
- **MFA Vulnerability:** Standard MFA is insufficient if session tokens can be stolen after the authentication handshake.
- **Workstation Security:** Endpoint protection must be capable of detecting modern "stealer" malware that targets browser data.
- **Secret Management:** Sensitive data like environment variables should be stored with additional layers of encryption or hardware-backed security where possible.
## Recommendations
- **Implement Hardware MFA:** Move toward FIDO2/WebAuthn (e.g., YubiKeys) to prevent session hijacking and phishing.
- **Shorten Session Lifespans:** Reduce the "Time to Live" (TTL) for administrative session tokens to minimize the window of opportunity for stolen tokens.
- **Enhanced Monitoring:** Implement behavioral analytics to flag instances where a single user session accesses an anomalous number of secrets or database records.