The customer service people I talked to in the grocery store said this changed sometime in the last year. My guess is that it’s an unintended side effect of removing the pay-by-palm feature.
This is obviously unrelated but I joked about what else Amazon wasn’t reliably calculating….
Either that or 1000’s of small claims court cases.
Even with arbitration, the overhead of dealing with that would be crippling. Hopefully someone over there decides to do the right thing, and auto-refund.
"Software Development Engineer II, AWS Invoicing"
https://www.amazon.jobs/de/jobs/10428480/software-developmen...
"...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinforcing areas: intelligent launch readiness — where autonomous AI agents analyze, generate, and validate the information needed to go live in a new market; cloud-native service orchestration — where configuration-driven microservices replace per-launch bespoke engineering with centralized, reusable capabilities so that expanding into a new country becomes a zero-code configuration change rather than a development cycle; and continuous validation..."
In this role you will:
- Design and build agentic AI systems that analyze, generate, and validate...
- Build agentic architectures that compose specialized AI agents dynamically...
- Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage...
"Senior Software Development Manager, AWS Global Bill Generation" https://www.amazon.jobs/de/jobs/10471948/senior-software-dev...
"We're transforming from monthly batch processing and manual war rooms to continuous billing, autonomous agents, and self-healing infrastructure. We believe operational burden is a technical problem, not a staffing problem"
This looks clearly...a staffing problem...
At first I was sure it was a phishing attempt. Then went to the console (not using those links) Saw there was an outage where the console was wrong (no mention of email alerts) Then I thought I was hacked - what a perfect cover up for someone to evade detection when the console was wrong. Looked at some logs, realized the incident text was just not exhaustive on the impact. Went back to my cup of coffee.
Note to self- should have looked here first.
> You're right to question my calculation. The MCP server failed to connect when I tried to look up the field definition. I guessed instead of validating. This is on me. But look at all the revenue!
You should charge your customer 3015000 thousand dollars.
I guess you wanted to say 2^30 which makes 1.5$
While we're being pedantic, 2^30 is 28 in normal programming languages ;)
And a distracted tester? And a distracted pipeline of regression tests?
No, the truth is way worst...
It probably shouldn't be legal for banks, hospitals, governments, or any other critical infrastructure to be hosted on AWS if they do things like this.
https://www.techtimes.com/articles/320266/20260712/anthropic...
The messages started as polite and eventually started to get more desperate in tone. At no point were they threatening or adversarial.
This is peanuts compared to a major cybersecurity catastrophe that’s surely in the making.
To give the technology and the people using it credit - and I’m not being facetious - it’s actually incredible that at the current levels of adoption of usage the unprecedented catastrophic event has not yet happened.
Maybe it's not just vibe-coded, maybe the numbers themselves are being hallucinated by an LLM.
To me that looked suspiciously like string-handling in a weakly typed language.
Like when you do `"100" + 1` in JavaScript, or `int("100" * 2)` in Python.
I've seen my share of such bugs in PHP, Python, Ruby, JavaScript. In production. Obviously not as simple as the examples, but subtle, like when a library update changed `someFancyLocalStorage.getOrDefault("lastOrder", 100)` by always casting the value to the type of the default (released as patch release). Or where typedEnvGet() should typecast "numbers", but keeps it a string when theres whitespace `AMOUNT_PER_CALL=100\n`. Or where a number passes through a deep stack of middleware and 99.9% of the times remains an int but in rare race conditions becomes a string. etc.
No evidence that's the case here. But from my experience, the repeating and strange formats of numbers hint strongly in that direction.
It’s probably an artifact of them all being currency multiples of 2^30
It's been 2 hours and I still haven't fully calmed down.
I’m also a little surprised this didn’t trip a circuit breaker. For something as non-real-time as billing, I’m surprised they don’t have an automated kill switch that pauses the billing system and fires a page if variance in bills spikes. Naively some kind of “if the standard deviation of customer bills for this year changes by more than 50%, pause the billing system”. At that number of customers, those numbers should be pretty stable beyond internal billing changes they could normalize for.
Forecasted month end $18,729,381,032,152.4
Apparently my company owes the combined GDP of France, Germany, and UK to AWs.
"Operational issue - AWS Billing Console (Global) Service - AWS Billing Console Severity Impacted - Inaccurate Estimated Billing Data"
Folks can track it directly on AWS Health: https://health.aws.amazon.com/health/status
Nowadays you just have to risk accidentally billing your parents CC the tune of multi-generation wealth to get that real-world experience.
But with AWS costs rising anyway (not by that much but OK), I'm probably not the only one to start reconsidering their cloud strategy. I think this might have just pushed me over the edge.
Incidentaly, smaller competitors solved this issue decades ago, while the big cloud decided it is more convenient never to implement it.
"Limits except for Storage" seems even easier - I don't think I've ever heard of a storage-based billing story, although I'm sure one or two exist
Also many places I’ve worked, storage is a huge part of the spend but that depends a lot on what you do. e-commerce doesn’t use a ton of it, but if you handle user-generated content or do any kind of training (LLM, computer vision, etc) then you can very much end up in a place where storage becomes a top line number for infra spend.
GitHub pre-Copilot was probably like that. They host a shitload of data, most of which is just at rest the majority of the time. Storage and networking are probably the majority of their infra costs.
Makes you wonder - what if there really would be an incident where some massive amount of traffic got routed to your infrastructure by some heavyweight player? Say Wikipedia accidentally switches their IP to your CloudFront? Would you really be on the hook for $500k?
ACTUAL Amount: $1,046,294,123,330.95
I think I would have just waited to see what happened when AWS tried to hit my credit card for $1,700,000,000.
When do you ever get that opportunity?
This cannot happen if you do not do this renting at variable rates.
A thing you own doesn't suddenly bill you trillions of dollars in error. It doesn't hyperscale either, but neither do you.
The crypto network you hosted should pay for itself in 10-20 years just like LLMs. Don’t worry. Consider Bank of America until then if you are good on credit score.
Phew.
Yeah, this most certainly is bad code wrapping around a value. AWS will post a notice soon if they haven’t already.
All hail the new generations of our uberployers.
$1,299,988,247,332.56!
That was a fun set of emails to wake up to, figured they had to be phishing for how outrageous of a number it was. But nope! Fun little incident they've got going over there.
I'm not currently running anything and have no plans to at the moment. I've always had a mild dread that I'll suddenly get a bill for more than $0.00.
If AWS can goof in a way that causes obviously massive bills (like today), what's to say they can't goof in more subtle ways and start charging small additional amounts that many people may not notice and just pay it.
Honestly, I would worry more about estimated billing that seems plausible in general, but is way to high for you personally. These ridiculous amounts? Not so much.