Full Report
A Ukrainian cloud provider said it had restored services after a power outage disrupted operations for customers including government agencies and major companies over the weekend.
Analysis Summary
# Incident Report: De Novo Cloud Provider Power Outage
## Executive Summary
A significant power outage occurred at Ukrainian cloud provider De Novo over the weekend, severely disrupting services for major government agencies and commercial entities, including the Diia app and major banks. This outage was attributed to an internal failure of the automatic power switching system, preventing backup battery and generator activation. Services were restored after approximately six hours, emphasizing the fragility of critical infrastructure dependent on a single point of failure, even when attempting to mitigate physical risks.
## Incident Details
- **Discovery Date:** During the weekend power disruption.
- **Incident Date:** Occurred over a weekend (specific date not provided, but recent).
- **Affected Organization:** De Novo (Cloud Provider).
- **Sector:** Cloud Services, Critical Infrastructure (Government, Banking, Payments).
- **Geography:** Ukraine (Kyiv area mentioned).
## Timeline of Events
### Initial Access
- **Date/Time:** During the weekend disruption.
- **Vector:** Internal Technical Failure.
- **Details:** An "unexpected malfunction" in the automatic power switching system occurred.
### Lateral Movement
Not applicable (This was an infrastructure/power failure, not a cyber intrusion).
### Data Exfiltration/Impact
- **Impact:** Services went offline for approximately 15 minutes initially, followed by a multi-hour restoration period. Affected services included the Diia government app, banks, Nova Post, and payment systems (Apple Pay, Google Pay).
### Detection & Response
- **Detection:** Discovery of the facility losing power due to the switching system failure.
- **Response Actions:** De Novo personnel worked to restore power. Services were fully restored in nearly six hours.
## Attack Methodology
*(Note: Since this incident was reported as a technical power failure and not a cyberattack, the following sections reflect the nature of the incident as reported.)*
- **Initial Access:** Internal automatic power switching system malfunction.
- **Persistence:** N/A.
- **Privilege Escalation:** N/A.
- **Defense Evasion:** N/A.
- **Credential Access:** N/A.
- **Discovery:** N/A.
- **Lateral Movement:** N/A.
- **Collection:** N/A.
- **Exfiltration:** N/A.
- **Impact:** Complete loss of electrical power to the data center premises, bypassing redundant power sources (batteries and generators).
## Impact Assessment
- **Financial:** Not quantified, but significant due to disruption of commercial and government services.
- **Data Breach:** No data breach reported; impact was operational and availability-based.
- **Operational:** Major disruption to critical services in Ukraine, including government services, banking, and national logistics (Nova Post). Contactless payment systems and metro access for Kyiv residents were impacted.
- **Reputational:** Negative impact on De Novo and users relying on them for failover reliability.
## Indicators of Compromise
- **Network indicators:** Power loss event (Internal infrastructure failure).
- **File indicators:** N/A.
- **Behavioral indicators:** Failure of automatic auxiliary power systems to engage.
## Response Actions
- **Containment measures:** Immediate localized power restoration efforts.
- **Eradication steps:** Investigation into the root cause of the automated power switching malfunction.
- **Recovery actions:** Restoration of services took nearly six hours for all clients.
## Lessons Learned
- **Key takeaways:** Critical infrastructure reliance on cloud providers proved vulnerable to single points of failure, even when the failure mechanism was technical (power system malfunction) rather than external cyberattack. Redundant power systems failed to activate.
- **What could have been done better:** Improved maintenance and testing of automatic power switching/failover systems to ensure battery/generator activation upon primary failure.
## Recommendations
- **Prevention measures for similar incidents:** Implement rigorous, frequent, and realistic testing of all Uninterruptible Power Supply (UPS) and generator failover systems, simulating a cascading failure where batteries and generators fail to engage sequentially. Organizations should continue to diversify cloud providers (multi-cloud/hybrid strategies) to mitigate the impact of failures at any single provider.