How would we learn about status in case of auth-adjacent incident?
There are logged in pages for people to raise tickets, see specific services/regions they have access or restricted ones(Gov/Sovereign Cloud only available to some users) and also other features only logged in users would need.
Yeah it will not work under an auth downtime or the service application downtime, but it is not designed for that.
It is for for the items which are localized at service/DC/region level. The service app pages on azure covers planned maintenance, retirement of old services, security advisories and so on.
The advisories are not disclosure, they are public already, it is more like you running this version of insecure service you have to apply this patch or we will do this automatically for you by this date.
The login redirect probably stops a lot of the traffic, and from closer to the edge of their network.
Or in other words: all hammering on a status page should have near zero effect on services currently affected.
Maybe it's doing all sorts of queries internally.
Issue ID: MO1550821 Affected services: Exchange Online, Microsoft 365 suite, Microsoft Copilot (Microsoft 365), Microsoft Defender XDR, Microsoft Teams, SharePoint Online Status: Service degradation Issue type: Incident
User impact Users may be unable to connect to multiple Microsoft 365 services.
More info More info: Potentially impacted services include: - Exchange Online - Microsoft Teams - SharePoint Online - Microsoft Copilot 365 - Microsoft Defender XDR - Microsoft Planner - Microsoft To Do - Microsoft Purview: Multiple Purview services including Authorization, Auto Labelling, Data Loss Prevention (DLP), Communication Compliance, and Data Classification are impacted.
Scope of impact This issue may affect users attempting to access the affected Microsoft 365 services.
Current status We've validated an alternative health check method on a subset of affected infrastructure, which allowed the servers in that subset to successfully return to service, and we're now applying this change more broadly. We're also preparing an automated recovery process to continually detect and restart unresponsive servers, to help keep them in service while this change progresses. Additionally, our review of recent changes has identified a small number of configuration updates that we're investigating as potential contributors to the issue.
History of updates
- While we haven't identified any recent changes that correlate with the start of impact, our investigation indicates the failed health checks may originate from a specific component on the affected servers that's responsible for processing them. We're deploying a targeted change to bypass this component on a subset of servers to validate whether this restores availability, and we've taken steps to prevent additional servers from being removed from service while we work to return the affected servers to service. We're also investigating why affected servers are experiencing elevated processor usage while handling requests.
- We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure. We're investigating what's causing these servers to become unavailable while we work to safely return them to service.
- We've completed redistributing connection traffic across our worldwide infrastructure, and we're moving additional traffic off the affected infrastructure to further reduce load. We're also restarting affected servers to return them to service. In parallel, we're investigating whether the elevated traffic reflects a genuine increase in requests or a shift in how traffic is being routed across our infrastructure.
- We’re investigating an issue impacting multiple Microsoft 365 services and we've determined the affected infrastructure is experiencing elevated load across multiple regions, which is likely contributing to the connectivity errors. We're restarting and repairing impacted portions of this infrastructure to restore capacity, and reviewing an unexpected increase in a specific type of service traffic to determine whether it's contributing to the elevated load. In parallel, we're evaluating options to redirect traffic to alternate infrastructure to expedite relief.