Full Report
Microsoft is investigating an ongoing outage preventing Outlook on the web users from accessing their Exchange Online mailboxes. [...]
Analysis Summary
# Incident Report: Exchange Online Service Disruption via Buggy Update
## Executive Summary
This incident involved service disruptions affecting Microsoft Exchange Online users, primarily impacting Outlook on the web search functionality due to a buggy code deployment. A separate ongoing incident, tracked under EX1030895, is also causing intermittent email delays, failures, and malformed calendar invites (winmail.dat attachments) due to a residual code issue. Microsoft is actively mitigating the search issue by reverting the faulty update and addressing the message delivery problems.
## Incident Details
- **Discovery Date:** Not explicitly stated, but coinciding with the deployment of a recent update causing search failures. (Secondary issue noted as ongoing, previously tracked under EX1027675, now EX1030895).
- **Incident Date:** Dates of service impairment are implied by the reporting period surrounding the buggy update deployment and mitigation attempts.
- **Affected Organization:** Microsoft Exchange Online customers accessing services via Outlook on the web or the new Outlook client.
- **Sector:** Technology / Cloud Services (SaaS)
- **Geography:** Global (Affecting Microsoft 365 service users worldwide)
## Timeline of Events
### Initial Access
- **Date/Time:** Coinciding with the deployment of a recent (unspecified) update.
- **Vector:** Deployment of faulty code/update into the production environment.
- **Details:** The code error specifically prevented some Exchange Online users from initiating successful searches using Outlook on the web or the new Outlook client, resulting in the error message: "We didn't find anything, try a different keyword".
### Lateral Movement
*Not applicable for this type of service incident, which appears rooted in a faulty internal update/deployment rather than external threat actor activity.*
### Data Exfiltration/Impact
- **Search Functionality Impairment:** Users were unable to effectively search their mailboxes ("We didn't find anything...").
- **Email/Calendar Disruption (Secondary Issue):** An ongoing issue (EX1030895) is causing delays/failures in sending/receiving messages, plus rendering plain text calendar invites with `winmail.dat` attachments for a subset of messages.
### Detection & Response
- **Detection:** Users reporting search failures; Microsoft observing service health telemetry. The secondary issue was tracked previously under EX1027675 and then escalated to EX1030895.
- **Response Actions:**
1. Microsoft acknowledged the issue and provided a temporary workaround for the search problem (using filters in addition to a string query).
2. Microsoft initiated the reversion of the problematic code change.
3. Service health telemetry showed improvements following the rollback as of March 19, 15:37 EDT.
## Attack Methodology
*This section primarily describes a service reliability failure caused by a deployment error, not a typical cyberattack. ATT&CK terms are adapted or marked N/A.*
- **Initial Access:** Faulty Software Deployment/Update.
- **Persistence:** N/A
- **Privilege Escalation:** N/A
- **Defense Evasion:** N/A
- **Credential Access:** N/A
- **Discovery:** N/A
- **Lateral Movement:** N/A
- **Collection:** N/A
- **Exfiltration:** N/A
- **Impact:** Functionality impairment (Search failure, message delivery issues).
## Impact Assessment
- **Financial:** Not explicitly quantified, but costs associated with incident response and customer support are implied.
- **Data Breach:** No data breach or unauthorized data access was reported; the impact was service availability and functionality related.
- **Operational:** Significant disruption to user productivity due to failed mailbox searching and intermittent email failures/delays.
- **Reputational:** Negative impact due to repeated service outages/issues affecting core email functionality in a short period (the article references several near-term outages).
## Indicators of Compromise
*As this incident resulted from a deployment error, traditional IOCs are not relevant. Failures were tied to specific internal identifiers:*
- **Network indicators:** N/A
- **File indicators:** N/A
- **Behavioral indicators:** Search returning "We didn't find anything, try a different keyword"; NDR failures; Intermittent plain text calendar invites with winmail.dat attachments.
- **Internal Tracking IDs:** EX1027675 (Previous issue), EX1030895 (Ongoing issue).
## Response Actions
- **Containment measures:** Identification and isolation (or marking for reversion) of the faulty code deployment.
- **Eradication steps:** Reverting the problematic code change.
- **Recovery actions:** Monitoring service health telemetry to confirm recovery progress and expedite full restoration.
## Lessons Learned
- The necessity of robust pre-deployment testing, especially for updates impacting core functionality like search indexing and message transport/rendering.
- The risk of cascading/related service issues; the search problem followed closely on the heels of other (weekend) outages linked to code issues.
## Recommendations
- Implement more stringent quality assurance and staged rollouts for critical service updates affecting high-use features like search.
- Improve tooling and monitoring thresholds to rapidly detect and automatically trigger rollbacks for widespread functional degradation observed after deployment.