177 pointsby mstolpm5 hours ago44 comments
  • fron3 hours ago
    Woke up to a billing alarm email. Thought I had leaked my AWS keys accidentally and somehow run up 437 billion dollars of charges. Joke's on them though, I don't have 437 billion dollars

    Anyways I didn't need coffee. That produced an adrenaline release unlike any I've experienced before. Thanks AWS

    • lijok3 hours ago
      If you owe AWS 437k bucks, that’s a big problem for you

      If you owe AWS 437B bucks, that’s a big problem for AWS

      • 7952an hour ago
        Far less scary than a smaller amount
    • binaryturtle3 hours ago
      I wonder how many people may have gotten an actual heart infarct because of that. There may be a person out there that may be dead as a result.

      It's entirely irresponsible of Amazon to even display such values to the user.

    • chrismarlow9an hour ago
      Same story for $500 million. I was shaking so bad I couldn't type my password.
    • trial32 hours ago
      huge irl laugh at “i don’t have 437 billion dollars”
    • logicallee3 hours ago
      Worth a shot to give them a call and explain that. They can probably adjust it down to 100 billion.
    • tcp_handshakeran hour ago
      Clearly the Agentic AI is running free on AWS. Matt does it again...what a success...
  • astonex3 hours ago
    From their status page

    >The second path involves rolling back a recent change to the billing computation subsystem.

    Want to bet AI code was involved?

    • Cthulhu_2 hours ago
      I'm not a betting person but I am looking forward to the postmortem, whether or not AI was involved, and what their code to production verification stuff looks like now. This kind of thing should have been caught by automated tests.
    • Xunjinan hour ago
      User: "I want to improve the billing system to be more efficient and só we can earn more money"

      AI: "No problem, let me change how we bill and fix the tests for the new increase in value"

    • CarRamrod2 hours ago
      >Spawning Sub-Agent: "Dr. Evil"
    • a0123 hours ago
      Is it still a bet if you have 99.999% chance to win?
    • phyzome2 hours ago
      They'll never admit it.
      • devin-20302 hours ago
        Why would they? Changing colors on the status dashboard needs VP approval.
    • verzali2 hours ago
      Can I bet you 437 billion dollars?
  • elondaits3 hours ago
    Woke up to a 100 billion dollars in S3, which is above the USD 4 alarm threshold I had set (I pay $0.55 monthly) . Some AI decided to prune the most impressionable of us.
  • AngryKitten3 hours ago
    You folks are completely irresponsible with your finances. I only spent $2.4 million last night. You've got to learn to manage your money.
    • Cthulhu_2 hours ago
      Yeah, buy less cappuchinos and avocado toast so you can pay your AWS bills people!
  • jwik5 hours ago
    Yes, there is a known issue with cost estimation.

    https://health.aws.amazon.com/health/status

    • mysterydip3 hours ago
      Bit of an understatement: “The displayed billing estimates do not reflect actual usage and charges.”
      • Xunjinan hour ago
        "There are no customer actions required at this time."

        Of course, you provided heart stress failure tests for free.

    • AngryKitten3 hours ago
      Whatever you do, AWS, don't post a related service health alert site wide on the console. Heck, don't even post one in the billing module. We wouldn't want to overdo the alerts, especially when we already have one being displayed to market the new FinOps Agent in Public Preview.
    • mstolpm4 hours ago
      Thank you. Have seen this after I posted.
      • coffee_is_nom3 hours ago
        Thanks, I panicked logged in and could not find the root cause of the bill.

        Thanks AWS, no caffeine needed this morning!

  • hoppyluke4 hours ago
    My estimate was only $21M (vs ~$0.01 average bill). Wish I had checked status sooner and saved myself the panic!

    My process went: verify email is not phishing (it was), login to console and check dashboard (same amount), attempt to understand cost (cost management kept contradicting itself), try to log support ticket and only on that part did I notice the status notification. At least I can breathe again now!

    • zengineer3 hours ago
      same process, same near heart attack - especially some days ago I had malicious activity on my platform and thought that now they actually found something.

      Apparently you can trigger an Action (e.g. prevent uploads) when the billing alert triggers, but then my platform wouldn't work anymore, just because AWS had an issue. Also insane that Amazon still hasn't send an email to clarify.

    • thinkindie2 hours ago
      A friend of mine went through the same - he got an alert for bill over budget, he logged in and boom 107M$.

      I was on the phone with him and we checked that he didn’t leak any APIs keys but no traffic at all. I even thought of a breach at a vendor he uses for some s3 stuff before I found this thread on HN.

    • 35 minutes ago
      undefined
  • WalterGR44 minutes ago
    > Anyone else seeing something like this?

    You can use the search box at the bottom of every page to search for previous posts.

    This was posted an hour before you posted: https://news.ycombinator.com/item?id=48945241

  • carraan hour ago
    Several comments here talk about "nearly" having a heart attack. But I wonder: since it's happened to so many people, chances are someone had a heart attack for real. Can they legally be made responsible for that?
    • justusthanean hour ago
      When someone says "I nearly had a heart attack," it's _highly_ unlikely that they actually nearly had a heart attack. I don't think the chances are good that anyone actually had a heart attack.
  • throwaway_57534 hours ago
    Scared me even though it was obviously a bug once I stopped to consider the magnitude ($bn). Very unfriendly that they don't allow for hard spend caps; closed my mostly dormant personal account as a result.
    • largbae2 hours ago
      If they did allow hard spend caps, it sounds like today would be a global outage.
      • beAbU2 hours ago
        Well then hopefully next time they'll be a bit more careful when shipping billing code updates!
  • cma25621 minutes ago
    In moments like these I'm reminded of all the people who have committed suicide due to billing errors. This is completely unacceptable. These sorts of errors must _never_ happen.
  • orbasker4 minutes ago
    Yes seeing the same, so far no response from AWS support
  • throwatdem123113 hours ago
    Hey man AI makes mistakes sometimes that’s why you need to double check the output.
  • graemep2 hours ago
    Someone I know woke up this morning to over 3 trillion dollars.

    Love to see how hyperscalers make your life easier and less worrying.

  • fuorilegge5 hours ago
    I have just received a similar alert for $ 5b

    AWS on their support data is reporting this:

    Inaccurate Estimated Billing Data

    Jul 17 3:03 AM PDT We continue to work to resolve the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console. We have identified the root cause as an issue with unit pricing within the estimated billing computation subsystem and we are working on a mitigation. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. Once the issue has been mitigated, we expect full resolution to take multiple hours as we work through recomputing the estimated billing data. We will provide another update by 4:00 AM PDT or sooner if more information becomes available.

    Jul 17 2:07 AM PDT Beginning on July 16 7:38 PM PDT, we began displaying incorrect estimated billing data in the Billing and Cost Management Console. Our engineering teams are engaged and investigating root cause. We will provide another update by 3:00 AM PDT or sooner if more information becomes available.

    Jul 17 1:33 AM PDT We are investigating issues with Cost Explorer reflecting inaccurate estimated billing data.

  • boesboes43 minutes ago
    I seem to have spend >35trilion on rds today, sooo yeah, going great at AWS
  • craigmoliver2 hours ago
    Ditto on the heart attack. My cost estimate for the month is currently $223,509,270,216.17. My girlfriend suggested contacting Elon for help. Glad I found this thread. Maybe I should create new keys anyway, this stuff freaks me out.
  • tokioyoyoan hour ago
    Thank you so much! I just woke up, and saw budget alert email for a dormant account to use $434,896.90. I haven't gotten so awake so fast in such a long time.
    • tcp_handshakeran hour ago
      I would not be so relaxed...Your estimate is so low that is likely to be real :-) You should only be relaxed if its in the Trillions...+
  • _joel3 hours ago
    I've had a mysterious Neptune cluster appear on my billing. Never used it, no API key access (or IAM instance profile access, OIDC etc), nothing in my console shows I've ever had one in any region. Raised a case with support, they ignored it.
  • devin-20302 hours ago
    At some point my role was to reduce our startup’s AWS bill. I managed to keep 7 figures on our books instead of handing it to AWS. But a message like that would have given me a heart attack in those days.

    Long story short: it saved the company from irrelevance. “Well-architected” is for the hyperscalers’ balance sheet, not yours.

  • noisy_boyan hour ago
    If I can expect to be penalized for not paying my legitimate bills, companies should also be penalized for failing to implement common-sense reasonable safeguards that prevent them from slapping their consumers with such absurdities.
  • csunbirdan hour ago
    @dang could you merge these threads?

    https://news.ycombinator.com/item?id=48945233

    OT: $286 million for me, almost had a heart attack

  • MacCopperan hour ago
    My Budget is 10$. The month forecasted cost associated with this budget is $182,278,249,263.06.

    Even though I new they could not collect the whole amount, I wondered whether I was hacked. I closed the account, it was an old testing account anyways.

  • dpcx3 hours ago
    My estimate was over 2T. Talk about waking up quickly...
    • Cthulhu_2 hours ago
      For science I hope people are trying to figure out now how they could manage to rack up a 2T bill in a month.
      • dpcx27 minutes ago
        My immediate thought was that someone managed to get an access key, stuffed a ton of data in there, and then was doing a ridiculous amount of egress. That was the only way I could conceivably coming up with anywhere near that much cost.
  • Group_B2 hours ago
    Yeah nearly had a heart attack this morning. Thought keys were leaked for a sec.
  • alfiedotwtf3 hours ago
    Vibe Billing
  • sailfast2 hours ago
    Who is going to compensate us for the years taken off our lives when we received the alerts?
  • segmondy22 minutes ago
    vibe coding for the win.
  • HarHarVeryFunny3 hours ago
    They should have added "make no mistakes" to the prompt.
  • tedk-423 hours ago
    far out it's 10pm here and I was just about to sleep when my wife nugged me about a billing alert from AWS.

    $151 billion the number for me.

  • lordleft3 hours ago
    Got a message that I owe 37 million on an account that I haven't used in probably...6 years?
  • themgt3 hours ago
    "If you owe AWS a hundred thousand dollars, that's your problem. If you owe AWS three billion dollars, that's Amazon's problem."
  • ownagefool3 hours ago
    Mines was $190,594,974,587,761.20 :)
  • dvh3 hours ago
    Prompt: bill our aws customers, make no mistakes.
    • vel0city2 hours ago
      Prompt: investigate new ways to fund additional AI datacenters.
    • zengineer3 hours ago
      or: Increase revenue!
      • Cthulhu_2 hours ago
        I mean just overcharging is one approach to achieving that goal I suppose.

        But so is imprisoning or exterminating all humans for their own good, as most AI dystopias end up as.

  • gioazzi3 hours ago
    Heard of somebody who got 19 quadrillion dollars - I thought they meant Zimbabwean dollars
    • elashri2 hours ago
      one USD is about 362 Zimbabwean dollars. So it would still be about ~53 trillion dollars which is more than the nominal GDP of US and China combined.
    • Waterluvian3 hours ago
      That would have been a great deal!
  • ohnoooooooooo3 hours ago
    Did it recover for you folks? I still see billions of dollars!
  • jdw642 hours ago
    I almost had a heart attack because of this. I was like, did I mess up my API management? Why didn't I just use Lightsail? Those were the thoughts running through my head.

    My personal website is on Lightsail, but those alerts started popping up from some test services I had set up while I was studying AI. I swear my heart nearly stopped and I cried. I really think AWS should have spending limits in place.

    My bank account barely has enough for next month's rent.

  • an hour ago
    undefined
  • steve1977an hour ago
    I mean 3 billion USD is clearly too big to fail, so I wouldn't worry too much
  • tcp_handshakeran hour ago
    Three billion dollars sounds about right for a free week of Kiro in the default Agentic mode. We usually see slightly higher numbers, so I wouldn’t be too concerned.
  • th3o6a1d3 hours ago
    131 billion for me
  • ohnoooooooooo3 hours ago
    did it recover for you? I still see billions
  • lightedman2 hours ago
    Literal basic fucking math, Amazon.

    You don't need hours to recalculate billing. You need to go back to basic algebra.

    Anyone using Amazon and dealing with this should be moving away from their services because something this basic going wrong means the correct people are not at the helm of the ship.

    • tcp_handshakeran hour ago
      >> Literal basic fucking math, Amazon

      LLMs are notoriously bad at it...

  • gib4443 hours ago
    Maybe they accidentally used the Argentine peso ;)
    • _joel3 hours ago
      Less hyper-scaler, more hyper-inflation
  • supersoftware2 hours ago
    [dead]