Full Report
Hear a tale about the time the BHIS SOC team conducted a 14-hour overnight incident response... from the Wild West Hackin' Fest conference in Deadwood, South Dakota. The post When the SOC Goes to Deadwood: A Night to Remember appeared first on Black Hills Information Security, Inc..
Analysis Summary
# Incident Report: Overnight Ransomware Response during Conference
## Executive Summary
During an organization's fall conference in Deadwood, South Dakota, the Black Hills Information Security (BHIS) Security Operations Center (SOC) team detected and responded to an active ransomware precursor event that started around 6:00 PM. Due to suboptimal customer configurations allowing malicious execution, the attackers began deleting Volume Shadow Copy Service (VSS) shadows. The presence of almost the entire SOC team on-site facilitated an extremely rapid, 14-hour overnight response that successfully contained the threat, prevented the ransomware from damaging backups, and allowed the customer to avoid a ransom payment.
## Incident Details
- Discovery Date: Incident started around 6:00 PM (date undisclosed, occurring during the Wild West Hackin' Fest conference).
- Incident Date: Occurred overnight, lasting approximately 14 hours.
- Affected Organization: A BHIS Customer (Specifics not disclosed).
- Sector: Undisclosed.
- Geography: Deadwood, South Dakota (where the SOC team was gathered).
## Timeline of Events
### Initial Access
- **Date/Time**: Around 6:00 PM.
- **Vector**: Exploitation arising from suboptimal customer configurations that safelisted items, inadvertently reducing EDR protection, allowing the execution of malicious code.
- **Details**: Alerts began triggering, and an analyst investigating noticed indicators consistent with a ransomware precursor event, specifically the deletion of VSS shadows.
### Lateral Movement
- **Details**: Risk-based alerts rapidly started triggering for a number of different users, indicating substantial scope and movement within the environment.
### Data Exfiltration/Impact
- **Details**: The primary immediate threat was the encryption/damage associated with the ransomware payload. The impact was mitigated before the ransomware reached customer backups.
### Detection & Response
- **Detection**: Detected by an analyst (Tom) reviewing baseline alerts around 6:00 PM, shortly after the team had gathered for dinner.
- **Response Actions**: Dinner plans were immediately dropped. The SOC team assembled in a conference room, shifted to an emergency response footing, and engaged the BHIS IR lead (Patterson) remotely. The team focused on tight control, narrowing the response group to maintain speed and clear ownership. The response continued through the night until morning.
## Attack Methodology
*Note: Specific technical steps for the attack payload were not detailed, but the precursors and observed malicious behavior are noted.*
- **Initial Access**: Successful execution of malicious code due to lax EDR configuration (safelisted items).
- **Persistence**: Not explicitly detailed.
- **Privilege Escalation**: Not explicitly detailed, but implied by widespread alerts across multiple users.
- **Defense Evasion**: The customer's EDR was bypassed or rendered ineffective due to previous suboptimal configurations.
- **Credential Access**: Not explicitly detailed.
- **Discovery**: Implied, indicated by widespread alerts across different users.
- **Lateral Movement**: Indicated by alerts triggering for "a number of different users."
- **Collection**: Not explicitly detailed.
- **Exfiltration**: Not explicitly detailed.
- **Impact**: Deletion of VSS shadows (a known ransomware precursor) and potential file damage/encryption.
## Impact Assessment
- **Financial**: The customer avoided a ransom payment. (Specific costs for incident handling not detailed).
- **Data Breach**: Not detailed, but file integrity was at risk.
- **Operational**: Significant team mobilization and 14 hours of continuous incident response effort. Business operations were maintained, though the response team was heavily engaged.
- **Reputational**: None reported, as the incident was contained successfully.
## Indicators of Compromise
- **Network indicators**: Deletion of VSS shadows (e.g., `vssadmin.exe delete shadows /all /quiet`).
- **File indicators**: Not detailed.
- **Behavioral indicators**: Risk-based alerts triggering across multiple different user accounts, indicative of substantial malicious activity.
## Response Actions
- **Containment measures**: Immediate lockdown of the response effort to a small core team (Hayden, Tom, Ethan, Patterson, Phil) to ensure faster decision-making. The ransomware execution was stopped before it reached backups.
- **Eradication steps**: Not explicitly detailed, but implied subsequent to containment.
- **Recovery actions**: Recovery guidance was initiated for the affected environment by morning.
## Lessons Learned
- The BHIS non-traditional SOC model (end-to-end analyst ownership, real-time collaboration) proved highly effective under extreme pressure, especially given the entire team was colocated coincidentally.
- Having the entire SOC team physically present allowed for immediate, high-bandwidth collaboration, eliminating context loss associated with remote handoffs.
- The incident highlighted the need for continuous vigilance, even during organizational functions like conferences.
- Team exhaustion began setting in around 3:00 AM, demonstrating the human endurance limits in a high-stakes, protracted response.
## Recommendations
- Review and remediate suboptimal customer configurations, particularly EDR safelisting exceptions, to ensure full security control is maintained.
- Ensure configuration management strictly enforces policies to prevent the deletion of VSS shadows as a primary defense against ransomware.
- Maintain appropriate staffing levels and clearly defined procedures for mandatory downtime, even when analysts are gathered for events.