165 pointsby player854 hours ago28 comments
  • vld_chk2 minutes ago
    This is arguably one of the biggest outages in modern history. AWS region level outage was more visible, but email is a backbone of the modern world.

    BBC is running live coverage of it at the level of government and ministers.

  • magnioan hour ago
    Apparently this single outage means Office 365 uptime this year drops to 3 nines (2 nines for this month). You can claim SLA credit worth 50% service fee by applying before December 31.
    • igleria31 minutes ago
      Thanks for the tip!
    • tyingqan hour ago
      "Office 364" is apparently a trending term.
    • JumpCrisscrossan hour ago
      I couldn't specifically say since my admin.microsoft.com fails pukes on a wildcard certificate for azureedge.net.

      Also, multiple bankers I'm speaking to about Anthropic's IPO can't send or receive work e-mails. So that's a thing.

  • BLKNSLVR2 hours ago
    How many Microsoft employees are working on this? It's concerning that one of the biggest, most valuable companies on the planet, that runs services internationally, can have one of its primary services have gone down for this long.

    6 hours ago they seemed to not exactly understand what the problem was, and that was 6 hours into the outage.

    This is concerning.

    • pjmlpan hour ago
      Let me put this way, the Visual C++ standard library has officially one FTE working on it, everything else are open source contributions.

      WinUI 3.0, WinAppSDK, and related CsWinRT, C++/WinRT are in a ridiculous position six years after Project Reunion announcement, due to ever meagre resources.

      XBox app keeps under delivering user experience, and bugs on patch Tuesdays are getting a joke by now.

      All this in a four trillion valued company.

      • ctmlran hour ago
        Maybe acting like jerk-offs got them to four trillion but there are consequences to burning all your social capital. A large camp of software developers have genuinely hostile feelings towards Microsoft, I would bet that this affects the quality of talent they can hire and these are downstream affects.

        Lately it seems like more non-techies are souring on the company.

        • sterlindan hour ago
          talent they can hire? large swaths of the company have been in a hiring freeze. they just sent 25% of the company voluntary severance offers.
          • rsalus35 minutes ago
            they've still been hiring, just on an attritional basis - headcount has remained flat for a few yrs now. it is pretty clear though that they want to operate even leaner.
        • mihaaly5 minutes ago
          Jerk-offs can skim the profit of going big yesterday while the consequences affect everyone, but only in the future of yesterday. Who does such things, one might ask? Jerk-offs, off course. And PE too (likely not a disjoint set).
    • InsideOutSantaan hour ago
      The ones who actually knew how the system works were too expensive to retain, so now it's just some interns typing frantically into Claude Code.
      • digikata5 minutes ago
        "Too expensive" as measured by bean counters that don't understand the actual risk profile - or do understand it but have such a short window of strategic concern it leads to this. It would be an interesting analysis to stack up costs to Msoft vs cost of labor cut as this outage evolves.
      • StingyJellyan hour ago
        Company policy is to use copilot but copilot is down so they are just hitting F5 and hoping
        • SillyUsername22 minutes ago
          This should be the top comment, my eyes are watering to this one
        • aucisson_masque35 minutes ago
          Haha this is ridiculous and yet there is some truth in it.
    • wooger11 minutes ago
      And which made a load of staff redundant in the last month.
    • xyst2 minutes ago
      the vibe coding group in charge of this product has decided to delete the production database and backups
    • boudin9 minutes ago
      /s Their internal AI is full steam on it. The issue is just that their internal AI infrastructure is down too, so AI just need to fix itself first. Then it will focus on fixing the issue. Oh, wait...
    • micw2 hours ago
      > How many Microsoft employees are working on this?

      all 3!

      • digikata4 minutes ago
        They're more effective than ever because with this outage, ever manager above them can't email to ask for status every 5 min
    • karlkloss2 hours ago
      How may Microsoft employees are left anyway?
      • wolfi116 minutes ago
        I guess the old light bulb jokes about MS will not be applying anymore
      • phoronixrlyan hour ago
        The lights are on but there's no one home
        • rdtsc12 minutes ago
          The wheel is still spinning but the hamster is dead.
    • surgical_fire9 minutes ago
      This is HN. We should be asking why frontier models couldn't still dolve this! How many parallel agents does MS have working on this?
    • DarkCrusader2an hour ago
      I don't understand why they haven't asked copilot to fix it yet, make no mistakes.
      • mint5an hour ago
        Don’t be silly, they tried that but copilot is/was down!

        That’s why it’s taken over 12 hours now, no one’s left that knows without copilot’s help.

      • cokeandpepsian hour ago
        [dead]
    • 2 hours ago
      undefined
    • krigean hour ago
      Maybe Nadella is still trying to figure out which of the 8000+ sacked employees knew how the system works /s
      • sterlindan hour ago
        he was such a nice and effective CEO the first few years. he's really changed.
        • cyberax3 minutes ago
          Not really. He killed the QA department immediately and consciously put MS onto the "slide into mediocrity" path.
        • pjmlp13 minutes ago
          Even with some of the Microsoft <3 Open Source, I do miss the Steve Balmer era in regards to Windows development experience.
        • bell-cot22 minutes ago
          Executive compensation corrupts, chief executive compensation corrupts absolutely.
  • pitabas106an hour ago
    I have always felt it's less about email redundancy and more about having a backup plan. If outlook goes down tomorrow, does everyone know how to keep work and client communication moving? that is something worth thinking about.
    • SubiculumCode7 minutes ago
      For the amount of uneeded emails about future emails about other emails, probably no real work was lost.
  • karlkloss2 hours ago
    Why would anyone introduce a single point of failure out of their control into their company?

    We're doing anything to avoid single points of failure in our products, but our IT is doing the opposite.

    • baliexan hour ago
      Outside of tech and disaster recovery, people _really_ don't understand that some redundancy--in this context--is a good thing.

      The idea of paying for a backup system, or even having one on standby for situations like this is just not on people's minds, or if it is, it's simply not affordable because it would cost you too much compared to your competition.

      Razor thin margins and generally getting away with it is what makes this work. Until it doesn't.

    • Arainachan hour ago
      The difference between a distributed system and a monolith is that in a monolith you have some idea what your single point of failure is, but in a distributed system you don't know your dozen single points of failure until they happen in production.
      • cowlevelan hour ago
        [dead]
      • adzman hour ago
        I love the dependency cycles that prevent things from being started up again from scratch.
  • eddythompson80an hour ago
    Having friends who work at Microsoft, this is not really surprising at all. The company has always had deep structural problems. The AI change has really just exposed it and is aggressively accelerating the inevitable implosion. It's truly in a free fall that is planned and orchestrated perfectly by their upper management. Their CEO and VPs are convinced you don't need engineering know-hows any more because of AI, and they got all the management layers to sincerely agree with them. Not in a malicious or subverting way, but in a "you can't get a man to understand something his salary depends on him not understanding it" way.

    Microsoft has a moat, but their moat is having billions of dollars to spend on data centers. It's a failure everywhere else. It'll continue to rake in billions in revenue because people need compute now, but it's rotten at the top.

  • enopod_an hour ago
    Good morning and happy Friday the 13th everyone
  • bob102939 minutes ago
    All 3 of my outlook accounts are still dead.

    I am looking at other current events and beginning to wonder if there is some kind of correlation.

  • MisterMunchkin37 minutes ago
    This is going to cause absolute chaos. Basically every government organisation and business in the UK runs on outlook. The whole day is going to be a write off. It’s like a bank holiday!
    • erdoc6931 minutes ago
      Just the UK? Lol, you can add many, if not most government and private companies and educational institutions in the US and countless other countries to that list. The government and corporate world is grinding to a halt.
    • byronsharman28 minutes ago
      This observation applies to many other countries, too. Mmmm centralization is pretty fun!
  • DavCreator43 minutes ago
    • cornfieldlabs28 minutes ago
      The amount of racist memes targeting Indians on the replies.
      • 18 minutes ago
        undefined
  • SubiculumCode5 minutes ago
    ..but why is it so centralized? It makes these events so final.

    MY University's whole email system is Outlook/365, and that also seems kind of dumb to me.

  • fuglede_11 minutes ago
    Self-host your email.
  • 36 minutes ago
    undefined
  • ReDressan hour ago
    [dead]
  • sedan_baklazhanan hour ago
    26 Oct: Microsoft offers voluntary buyouts to 60,000 employees as AI leads to efficiency push

    It's just a wild theory, but maybe there are still too much employees left, standing in the way of AI? I would try firing another 60,000 first.

    • gb2d_hnan hour ago
      No one knows if this is AI related, but it's easy to imagine that where changes written by AI fail in production, if AI can't find the solution (which will be common as AI access to production systems is typically reduced), then the devs won't either, or at least it will take them some time.
      • karmakurtisaanian hour ago
        Perhaps the velocity is the problem, not that AI inherently produces worse code (you just need one bad or just inexperienced engineer have their PR slip through to have bad human written code in you code base). When the velocity is high, there's less time to battle test each change and new code piles up on the old one. When the velocity is low, shitty code fails in production, but people are aware of what was recently changed and can find and fix the problems.
        • _kulangan hour ago
          Same thing, now, just faster: one bad engineer can now ship a million lines
          • karmakurtisaani5 minutes ago
            Well, yes. My point was more that even if the quality was better, the speed of change is a problem. In particular in mature systems, I should add.
      • lkramer23 minutes ago
        The bigger issue with having your systems written by AI, is that nobody knows how the system works anymore and can't pinpoint where the likely failure points are.
      • gbalduzzian hour ago
        Maybe the error itself isn't event AI related, but the engineers that knew how to fix it were let go
      • solid_fuelan hour ago
        The status updates have a pretty strong "LLM-driven agent failing to debug a complex system" feel to them. I hope it's not the case, but I could imagine a scenario in which inexperienced engineers - or experienced ones who have lost touch with the codebase - are deferring to LLMs to drive the troubleshooting and fix here.
        • koyotean hour ago
          They sound exactly like frustrating conversations I'd have with a cheap model and an overextended context window.
    • Zecc40 minutes ago
      Ugh. My sarcasm detector is down. Shouldn't have hosted on MS servers.
  • whatever1an hour ago
    Who needs senior engineers. We are going to have LLMs troubleshoot billion-scale systems!

    What could go wrong, said all of the hyperscalers. Enjoy your multi day downtimes.

    • gb2d_hnan hour ago
      I do actually believe that LLMs and process around them will continue get better and become the accepted, but that there's going to be a 'fossil record' of a lot of problems having been seeded in these early years of AI that will take some time repair.
  • pelagicAustral41 minutes ago
    If Teams is down I take that as a service upgrade
  • aucisson_masque37 minutes ago
    > We're aggressively pursuing all pathways to reduce impact

    I love this kind of wording. It makes everything suddenly more serious lol.

    • byronsharman25 minutes ago
      What a fancy way to say, "we highkey have no idea why it's broken"
  • danielsoto3172 hours ago
    The repeated comments on their twitter feed are weird, is it an LLM writing it? why can't they just wait until its fully fixed?
    • Arainachan hour ago
      Regular communication is an important part of outage/crisis communication, even if that communication is "nothing new to share".
    • InsideOutSantaan hour ago
      It's actually nice to have some updates on what they're doing that go further than just "we're investigating."
    • a77ec820a40f6an hour ago
      [flagged]
  • keyle40 minutes ago
    At least Microsoft 360 still has a nice sound to it.

    No Outlook, they should make this a public holiday!

  • ieie336612 minutes ago
    Just by looking at the O365 UI changes they have been doing tells me there’s a massive competency crisis within that team.

    It would not surprise me if the entire project is complete and utter (human-written) slop

  • harvey926 minutes ago
    This is very serious: I cannot access Viva Engage.
    • asimovDev2 minutes ago
      TIL that Yammer turned into that
  • vachinaan hour ago
    Funny, my self hosted email has a better uptime than this pos
    • cowlevelan hour ago
      Email is actually designed with intermittent connectivity in mind. The sending server buffers outgoing emails until they're accepted by the receiving server. Typical settings are to retry every 4 hours, with a permanent failure being counted after 3-7 days.
    • gb2d_hnan hour ago
      I would take 99.0pct self hosted over having to use outlook!
  • redrix29 minutes ago
    “Co-pilot, fix the outage. Make no mistakes.”
  • 2 hours ago
    undefined
  • aktuelan hour ago
    Welcome to Outlook/364
    • mint5an hour ago
      If only this year were a leap year Microsoft would be in the clear
  • devsdaan hour ago
    Their current engineers must have stored their KT notes from their laid off and leaving senior engineers in OneNote on their Mac, and might have been working with copilot to find the root cause /s
  • 2 hours ago
    undefined