Hidden July Software Bug Triggered Global Microsoft 365 Outage
More than 80 percent of servers behind Microsoft’s networking gateway were unresponsive at the peak of last week’s global outage of Outlook and other Microsoft 365 services. The company attributed this disruption to a hidden software defect that was introduced in July. Three factors collectively contributed to the outage of the networking gateway that serves as a front for numerous Microsoft 365 services, as stated by Charles Lamanna, Microsoft’s Executive Vise President of Copilot, Agents and Platform, in a post on X on Monday. “We are deeply sorry for the impact to all our customers and recognise the impact that this has on their normal business and operations,” he said. The initial factor was a latent defect that led to servers becoming overwhelmed and unresponsive under specific load conditions. It originated from a fix designed specifically for Microsoft 365 networking gateway servers and is not applicable to wider Windows deployments, Lamanna stated.
The initial change commenced its rollout in July and remained in effect globally for over three months without any discernible impact, he stated. The company’s load testing failed to identify the issue as the tests were conducted on less powerful hardware, whereas the defect is more pronounced on newer machines equipped with additional processing cores. Approximately 15 hours prior to the outage, Microsoft Teams transitioned from an alternative gateway to the primary gateway, which additionally supports Exchange Online, the company’s email platform. This heightened the burden on that gateway and altered its traffic dynamics. Lamanna stated that such a shift constitutes a standard aspect of operations. Third, nearly 14 percent of the gateway’s server capacity was out of service at the time due to an unrelated issue from the previous day.
In summary, the increased load coupled with diminished capacity led the gateway to encounter the latent bug identified in July, resulting in a backlog of requests until the servers ceased to respond. This occurred during the peak period of traffic in the United States, he stated. “When servers lose connectivity, our traffic management systems automatically route users to other healthy endpoints. This happens often and is critical to avoiding any interruption due to hardware, software, or network defects and is normally what makes the service resilient.” And “However, the redirected traffic increased the likelihood for the same problem to occur on the remaining endpoints. This caused a cascading failure from server to server as the same amount of load was spread across fewer servers. In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80 per cent of servers were unresponsive,” Lamanna said.
The safeguards that typically delineate various components of the gateway failed to prevent the incident due to an unrelated issue, which facilitated the global spread of the impact, he added. Services including Teams, OneDrive, SharePoint, and Office applications successfully transitioned away from the malfunctioning gateway and were reinstated earlier, with Teams chats and meetings functioning for the majority of the outage period. Exchange Online cannot do so, Lamanna stated. Recovery extended beyond the typical duration as servers that resumed service were swiftly inundated by the backlog of traffic, resulting in a return to unresponsiveness. The company’s automated recovery processes were not equipped to handle such an extensive failure, and given that the defect had been present for several months, its standard methods were unable to swiftly rectify the situation, he stated. Engineers subsequently implemented an update that restored the code to its former condition. Groups of servers were taken out of service, changes were applied, and they were restarted before being brought back online. A final post-incident review with additional details will be made available later this week, Lamanna stated.
The outage, which commenced late on Thursday, impacted Indian airlines, financial services firms, and various businesses. Domestic airlines IndiGo, Air India, and Akasa Air have encountered disruptions in their email-based customer support systems, which has hindered their capacity to receive or respond to certain passenger enquiries. Air India has temporarily redirected its email services to maintain customer communication with the airline, while there have been no reports of any impact on flight operations. Financial services firms encountered challenges in obtaining research reports and engaging with clients. The disruption primarily impacted Microsoft 365 functionalities that engage with Exchange Online, the firm’s cloud-based email service. Users were unable to access mailboxes through Outlook on the web and desktop applications, and email delivery was delayed. At its zenith, the outage elicited over 130,000 reports of Outlook issues on the outage-tracking platform Downdetector, with disruptions noted across 72 to 74 nations.








