Full Report
Wiz Research reveals the data behind Shai-Hulud's 2.0 long tail, the massive gap in cloud credential rotation, a potential link to the Trust Wallet incident, and how we finally "snipped the tail" on a month of ongoing infections.
Analysis Summary
# Incident Report: Shai-Hulud 2.0 Long Tail Infection
## Executive Summary
The Shai-Hulud 2.0 (sha1-hulud) worm established a persistent "long tail" of infections following initial high-volume activity. Persistence was primarily achieved through overlooked dependencies in private artifact caches and a specific, un-updatable IDE extension (`asyncapi-preview v1.0.1`). Wiz Research intervened by coordinating a clean update, successfully "snipping the tail" of daily infections, though risks remain due to unrotated cloud credentials.
## Incident Details
- **Discovery Date:** Tracking initiated post-peak outbreak (after November 24th).
- **Incident Date:** Ongoing infection activity observed from November 25th to December 24th (Long Tail period). Initial outbreak peak was November 24th.
- **Affected Organization:** Over ⅓ of the Fortune 100, among hundreds of other organizations.
- **Sector:** Not explicitly disclosed, assumed to be wide-ranging given the prevalence among large enterprises leveraging cloud ecosystems and software supply chains.
- **Geography:** Global (implied by widespread cloud usage and dependency on public registries like npm/OpenVSX).
## Timeline of Events
### Initial Access
- **Date/Time:** Unknown start date; peak activity recorded on November 24th (13,686 new compromised repositories).
- **Vector:** Supply chain compromise via malicious packages on public registries (implied from prior reporting context).
- **Details:** Initial infections were aggressively curtailed by ecosystem partners (npm team).
### Lateral Movement / Persistence
- **Date/Time:** November 25th to December 24th (Long Tail).
- **Vector:** Continued serving of poisoned packages from private instances/caches, and the continued execution of a non-updating IDE extension.
- **Details:** Approximately 100-200 new compromised repositories per day were observed during this month-long persistence phase, driven by:
1. **Private Registries (5%):** Registries failed to purge revoked malicious packages, serving them to internal developers.
2. **Local Caches (Performance Flags):** CI/CD environments using `--offline` or `--prefer-offline` flags continued using locally cached malicious versions.
3. **OpenVSX "Zombie" Extension (90%+):** The malicious `asyncapi-preview v1.0.1` extension lacked a new version number to trigger updates, keeping the infection vector active on developers' machines.
### Data Exfiltration/Impact
- **Date/Time:** Ongoing during the long tail period.
- **Vector:** Data theft associated with artifacts left by the worm.
- **Details:** The article notes a potential link between residual indicators found by the worm and a **\$7M Trust Wallet incident**. The scope of data exfiltrated is not quantified but involves data linked to over 1/3 of the Fortune 100.
### Detection & Response
- **Date/Time:** December 24th (Tail Snip Action).
- **Vector:** Wiz Research intervention.
- **Details:** Wiz Research coordinated with the AsyncAPI team to publish a clean version (**v1.1.0**) of the OpenVSX extension. This forced an automatic update across developer machines, leading to new infections plummeting by December 29th.
## Attack Methodology
- **Initial Access:** Supply chain poisoning targeting software development dependencies (npm, OpenVSX).
- **Persistence:** Exploitation of synchronization gaps in private artifact mirrors and the inability of a specific OpenVSX extension (`v1.0.1`) to self-update.
- **Privilege Escalation:** Not detailed in the summary, but implied to be successful based on the subsequent data exfiltration/leakage.
- **Defense Evasion:** Evading mitigation by hiding within development environment caches and registry mirrors that were not immediately synchronized with the public purge.
- **Credential Access:** The incident highlights a "massive gap in cloud credential rotation," indicating credentials leaked by the worm remained valid. Potential link to the Trust Wallet breach suggests significant credential access occurred.
- **Discovery:** Not detailed.
- **Lateral Movement:** Not detailed outside of persistence mechanisms.
- **Collection:** Not detailed directly, but tied to activities that resulted in leaked credentials and data associated with the Trust Wallet incident.
- **Exfiltration:** Data linked to exfiltration was recorded by GHArchive during the outbreak.
- **Impact:** Leakage of sensitive data implied by the link to the $7M exploit; persistent unauthorized access due to unrotated cloud secrets.
## Impact Assessment
- **Financial:** Potential linkage to a \$7M exploit (Trust Wallet incident). Details on internal financial costs for compromised organizations are not available.
- **Data Breach:** Sensitive data associated with over 1/3 of Fortune 100 organizations was potentially exposed or accessible during the compromise window.
- **Operational:** Intermittent operational risk maintained for a month due to lingering infections, particularly within high-velocity CI/CD pipelines.
- **Reputational:** Medium to High, affecting numerous major global enterprises.
## Indicators of Compromise
- **Network Indicators:** Not explicitly listed (defanged).
- **File Indicators:** Artifacts pointing to the malicious fork: `"_PACOTE_NO_PREPARE_": "git+ssh://[email protected]/asyncapi/cli.git#2efa4dff59bc3d3cecdf897ccf178f99b115d63d"`
- **Behavioral Indicators:** Continuous serving of revoked packages from internal artifact repositories (e.g., Nexus, Artifactory). Use of `--offline` or `--prefer-offline` flags coinciding with package usage.
## Response Actions
- **Containment:** Aggressive containment triggered by npm team and ecosystem partners leading to an initial crash in infections (late November).
- **Eradication:** Coordinated release of a clean version (**v1.1.0**) of the `asyncapi-preview` extension on December 24th to force updates and neutralize the primary persistence vector.
- **Recovery:** Not fully detailed, but critical next steps involve comprehensive secret rotation addressing the stated "massive gap."
## Lessons Learned
- **Supply Chain Gaps are Deep:** Relying solely on public registry purges is insufficient; private caches and local build environments can perpetuate threats long after the source is cleaned.
- **IDE Extensions are Persistence Vectors:** Minor software components (like niche IDE extensions) can become the most critical persistence mechanism if they lack auto-update triggers or version control discipline.
- **Secret Sprawl is a High Risk:** The existence of long-lived, unrotated cloud credentials (which were not cleaned up by the researchers' efforts) demonstrates a critical, ongoing risk inherited from the initial compromise.
## Recommendations
- **Own Your Private Registry Governance:** Implement strict policies, auditing, and synchronization mechanisms to ensure private artifact caches immediately reflect public revocations, mitigating desynchronization risk.
- **Neutralize the Cache:** Review CI/CD configurations to limit the reliance on `--offline` or `--prefer-offline` flags, or establish mandatory validation checks even when operating locally.
- **Comprehensive Secret Rotation:** Immediately audit and enforce rotation policies for all cloud credentials exposed during the compromise window, as these secrets may still be valid months later.
- **Beware Orphaned Malicious Versions:** Continuously monitor third-party dependency consumers (like IDE extensions and toolsets) for orphaned malicious versions that can persist silently on developer desktops.