https://x.com/tamarajtran/status/1867342095033258466
There's a really good reason that I don't link to my apps on this site (or most, for that matter). They are free apps, running on a shoestring budget.
But that whole account looks a little dodgy. The posts make it seem like one of those "I made $30K/mo, from my living room! You can too!" outfits.
But what's odd to me is that I don't often read about other people having the same experience. So... are all of you seriously with hosts that don't shut off access to your servers?
I'd rather that happen than suddenly get a bill I can't afford.
The structure should look like:
"My App LLC" has contract with "My Host LLC" for hosting services. "My Host LLC" provides those services via a cloud provider. If "My Host LLC" racks up a 2 million dollar bill with the cloud provider and goes out of business then I just move to "New Host LLC" and carry on.
It's not as if the patients of a Doctor who fails to pay his office rent would become liable.
This is the entire purpose of LLCs.
i like this one. the ultimate law to subjugate everybody once and for all!
Firstly, because cloud contracts generally prohibit renting out the cloud services as-is - you can build a product on top of it but not act as a reseller of their platform.
Secondly, and perhaps most importantly because setting up separate LLCs with the goal of avoiding lawful debts is the demonstrates fraudulent intent and will get your LLCs veil-pierced.
You need to be a billionaire, and then you need to threaten to sue anyone who dares to ask you to pay.
Doesn't have to be as-is - a simple infra on top of it will do
> setting up separate LLCs with the goal of avoiding lawful debts
If normal bills are paid as usual and only this kind of crap is not paid, Amazon would have a hard time proving in court that the whole point of the daughter LLC was to avoid paying bills.
"don't worry, all the work has warranted by New Quality Roofing Services IV, LLC"
although, there's very little about that structure that's unique to an LLC. it can be done the same with a corporation.
i'm certain this only works when the amount of debt being ripped off (ahem in dispute) is not a lot more than the cost to sic a really good law firm on you.
It’s not. A lot of weight should be put on the concept of “legitimate business”. Let’s say you decide to open up a car dealership and you use an LLC to buy cars then transfer them to another LLC and sell them. Then declare bankruptcy on the one containing the bills. You won’t be shielded, and you’ll likely go to jail for fraud.
The construction is to allow a business to go bankrupt with limited liability not to allow the expense account of a business to go bankrupt.
When a dealer has a floor plan (to finance their inventory), the cars that are being bought by the dealer are collateral and have liens on them.
The lien has to be satisfied (bank paid with interest) to transfer the title to a buyer or to your theoretical “another LLC.”
Just like my car would have to be, for me to get a clear title to sign over to you.
Not everyone uses a floor plan, an unsecured line of credit would be the one situation where what you’re proposing would “work”
Floor plans are one of the reasons there’s only a few (used) dealerships in town I could walk into, plunk down cash, and walk out with a signed title that day. Because they’re transferring the title to themselves, and then to me, using the chain of ownership boxes on the back,and then sending that to dmv.
I didn’t know a lot about this other than the owner at my previous company had attempted to branch into car sales before he passed away. TLDR, we couldn’t even sell or dispose of the cars with good intentions and they all got repossessed.
Amazing service but its a scary one. Every time a social-media-worthy accident happens, they cancel the bill but I wonder how many are burned without recourse.
Maybe if your rack up something modest like 5K accidentally, its better to push it as high as you can and get it declined on your CC and increase your social media prospects :)
Why does Firebase insist on hanging the sword of Damocles over all of its customers? I’ve read so many stories before and experienced these fears the first time I was setting up Firebase…this has been going on for years
No affiliation, just happy to use a provider with actual caps.
https://firebase.google.com/docs/projects/billing/firebase-p...
I am not sure how they can do that, but cannot let people set their own limits on their paid plans.
Because it's a free plan, the delay between 'limits reached' and actual shutdown only incurs the cost of providing the service during that brief period, not the potential liability of overcharging that might exist on a paid plan.
Or write a disclaimer that the billing cap doesn't necessarily cut off at exactly that amount and that there might be an overcharge.
I am pretty sure most people would be okay with either of these options, we didn't need a perfect system, just one that works well enough
That's fine. The major LLM providers work like this. If you're out of credit, or hit your monthly recharge limit, it stops working, bringing down prod with it if your product relies on it. Not heard anyone complain about the concept.
If it's really a problem for you, you can be all enterprisey and contact sales, then they'll be very excited to offer you extremely high limits and post billing.
This way everyone gets what they wants.
So you deploy an advanced technology known as a radio button to toggle which they want, throw a bunch of ToS & consent agreements about data loss / deletion at the ones opting for hardcaps....and done.
Also reminder that Azure has hard caps for certain account types. This is not a technical problem. They can do this, they just don't want to.
Also authorisation revocation is relatively uncommon, which means you can have a fast-path for approval, and then push only the revoked key IDs to just frontend servers.
When you've exhausted your billing cap, what else could it do? Either shutting off services or blowing past the cap seem like the only serious options.
For a lot of small businesses, I suspect that an outage is often better than risking a surprise 6 figure bill. But it depends on what your software does.
Also if the system shut down automatically when the budget got exhausted, there's a risk that a runaway backup process or something might accidentally eat through your regular budget and get the site shut down. For that, it might make sense to assign different resources into different budgetary buckets or something.
I'm surprised firebase doesn't implement something like that.
They don't have a problem implementing caps on a free tier. No one's asking for perfect, but they don't seem to care about even getting to the ballpark.
I set up automatic recharge of $20. A small amount because not much traffic. A bad actor got ahold of our api that didn’t have rate limit yet and started spamming Africa.
Twilio had zero issue charging my credit card every second. Literally I was getting a hundred emails and bank notifications a minute. Brex didn’t stop anything.
Twilio responded that it was my fault. Yeah. I sure 100% probably should have put in that cloudflare rate limit first. But…
How easy would it be for twilio to prevent this on any level? I need rate limits? How about you rate limit credit card charges. Putting $20 recharge limit should mean $20/day or $20/hr not literal unmetered right to charge as much as possible in 20 increments.
Twilio support sent me all this info about protecting myself from African spammers who use the technique to make money from SMS charges. You know what’s more responsible than informing me of this? How about blocking sending sms to country codes known for this from the get-go and optin to send to them.
it was clear the perverse incentives that encourage twilio to massively benefit from being insecure and easily exploitable by spammers.
Ended up costing almost $3k after bill adjustment when our usual spend was $5/mo. not bankruptcy level so after fighting with support just took it as is and learned my lesson. But twilio made *50 years* of revenue in about 10minutes from their own negligence.
These folk can't even get a stable billing process; the coming surprises will be awesome.
I could easily imagine a 1,000x hour over hour growth as the social media grows.
If I was LinkedIn, I’d be very upset if Firebase pulled the cord when everyone was looking at the new launch.
I pay for storage. Does this mean they delete my production database? They delete my s3 buckets? They delete my server logs? My message queue?
Then yes! I would be extremely upset. How do you expect the cloud provider to magically know what is safe to "pull the plug" and what is not safe?
[ ] continue billing me [ ] throttle bandwidth [ ] gracefully shut down servers (data is deleted after 30 days of non-payment)
This is so painfully obvious even for someone that never deals with cloud vendors, that I just don't understand why you would pretend otherwise.
This isn’t a pre-paid gas pump use, but that could be one way to present it. We all want to fill as fast as possible. And if your fill spout can handle top rates, you get top fill rates, until you close in on the hard limit. Then it trickles down to the metered drop. Then stops precisely where it needs to.
By accepting/requesting a hard cap, the provider can make clear that in order to be precise, soft caps will go into affect earlier and induce progressive throttling where applicable. If the throttle doesn’t catch the final milliliter or two of gasoline, before the pump shuts off, the provider can and should just let it go. It’s a loss, but comparatively a figurative drop in the bucket.
The other obvious route is predictive where prior usage guide the guardrails. Ordering two eggs is typical for a single meal. Ordering twelve is not. Ordering three or four is unusual for most but if you are a regular diner your habits will be observable.
Any of this predicated on the provider to want to do something. They seem to lack incentives at this point for making it easy. It is stories like op that I avoid well known problematic providers like Firebase who don’t respect and foster long term relationships.
It's easy to accidentally make a bunch of information world-readable, and if a malicious user gets hold of the right details they can simply read all your content out, or even start writing bad content in, racking up a bill in the process.
Whether this is what happened here or not I have no idea. And I don't think this is even really a problem with Firebase, just an indicator of the sort of developer who ends up using Firebase and their background.
Especially on Firestore. For example, the access rules that you define on your properties are not filters(they are rules on what you can ask the system to do) even if they look like filters at first glance and the way it accessing data is billed is based on what has had to be processed to return the results and not on what the actual results were returned. This makes it very easy to create very expensive and compromised apps if you slip.
Also, unlike more traditional system, doesn't produce smoke so it passes all the smell tests and you learn about your mistakes once things go very bad.
Over on Reddit someone got stuck with a 450k one.
Struggling to find the third thread again though. Maybe they deleted it
Also seeing an uptick in people suggesting insurance as the solution. Which seems insane to me but what do I know
Are you buying insurance against having a successful business? No one is going to sell that. Are you buying insurance against being hacked? That makes sense and does exist, but only solves one very specific version of this and isn't really about cloud billing, plus it's likely not accessible to the size of business (or individual use) who get bitten by this sort of thing. Are you buying insurance against incompetence using the services? You can get professional indemnity insurance, and I guess your own LLC could probably sue your own indemnity insurance in this sort of case, but you'd need to defend the decisions as reasonable at the time and not negligent.
I have not been thrilled with Upwork, but that’s another story.
In general, I have found that you get what you pay for.
Why would I agree to pay up to $20k on a $50k cloud bill, when you could just choose to spend more on your cloud bill. I would be giving away free money and you'd get a discount.
You have to tie it to either incorrect billing, valid billing but caused by a malicious third party (i.e. hacking), or valid billing caused by negligence, and in the latter case you would need to prove negligence otherwise it's just a business deciding to use more resources and not pay for them. In the same way, fire insurance covers your house burning down, but not if you burn it down yourself.
A lot easier to simply not use these cloud services.
Eliminates risk of a denial of wallet like attack. That does leave risk of footguns & mistakes on my part, but that's a bit more manageable of a risk.
The answer is probably quota management, where a limit on the number of VMs or size of database or something, caps the worst-case scenario, and where it's arguably easier to monitor an approach to that quota as it's more granular than billing.
Personally I think cloud providers could have an explicit "hobby mode" that limits certain things in such a way that the spend can't run away like this, with the trade-off that they're not really production grade in a sense, but then again those accounts are probably worth anything so I understand not building that out. That said, whenever I've seen one of these things happen, it always ends with "FooCloud said that as a one-time gesture of good will they would write off this accidental usage", so while briefly scary, maybe this is the system working fine overall?
https://learn.microsoft.com/en-us/azure/cost-management-bill...
I'm sure people would accept a best-effort system where setting a billing limit for $100 means you may be billed $140 because your spending overshot the limit before the system noticed. It still beats the alternative of waking up to a $20,000 bill out of nowhere.
Let's say the answer is yes, you just delete all the data as I can no longer afford it... delete operations are billable, so do you charge the user for those deletes or not?
Let's say the answer is yes, and you bill the deletes. What happens if too many deletes are required and suddenly you're at 2x the bill cap? Now you can't document the bill cap as being able to go over by up to 1.5x. This may be unlikely, but customers use cloud services in weird and wonderful ways.
This is just one resource type, there are many different resource types on a typical cloud provider, each with multiple axes of billing, each of which has hard decisions to not just be made, but documented and communicated to customers in such a way that they understand the impact. Oh and also it's the "I just put my credit card in and go" crowd who you have to explain it to, who aren't engaging in sales conversations, not those on business contracts who might actually listen or read the documentation.
It's not at all obvious to me that this is preferable to just having someone look at these incidents on a case by case basis and seeing who should be refunded.
It is possible to combine all the billing axes for things like this so that, but when you do you get Dropbox, or Google Drive. The explicit value proposition of cloud hosting is paying for what you use, and generally granular services are lower margin and more commoditised than higher level services.
How much does the problem change if you remove the word 'exactly' from here, though?
Like, I don't mind if I end up paying a couple of extra bucks. Or even tens of bucks! Some people might not mind hundreds or thousands, or even more depending on their scale.
But blowing out several orders of magnitude past my usual monthly spend is the problem I'd like to avoid.
This seems unlikely to me. What is the technical reason for this?
This is not too hard if the billable event is, say, creating a VM. Creating the VM takes a few seconds, it's easy to add a quick API call to check billing. But what about the next hour, when you charge for the VM again? You now have the number of VMs checking in every hour (worst case, at the same time), and you need to handle all those hourly checkins consistently per customer.
That's still probably easy enough, but what if it's not a VM hour, but an API gateway request? Now on every single request you have to check in with the billing service. API gateways need to be very low latency, but you've just added a request to that process that possibly needs to go across the world to check in with the billing service running on another continent.
What if the billable resource is "database query seconds", and now you need to know how many seconds a query is going to take before you start it? Oh, and add the check in time to every database query. What if the billable resource is streaming video, do you check in on every packet you send out? What if it's CDN downloads, do you have every one of thousands of points of presence around the world all check in, even though the point of the product is to be faster than having a single far away delivery node?
There are bad workarounds for each of these, but they all mean either the cloud provider losing money (which, assuming a certain scale of customer, is too expensive), the customer over-spending (which assuming a certain scale, could still be waaay over their budget), or slowing down most services to the point that they functionally don't work anymore.
Additionally, it feels hollow to not have billing cutoff at the same rate as authorization would cutoff if they shut off my account.
The other main issue is documenting this. Google Adwords I believe has an overspend concept, i.e. if you limit your billing to $100, they might still go over it. The problem is that it's limited to 2x your bill, which still bites people. I only know about this from reading HN and Reddit posts complaining about it!
API gateways are similarly sending metrics somewhere. The coordinator can be the place to ingest that data and send the aggregated info to billing. If it gets back over budget, start halting endpoints. etc.
Or do it within the billing service, but fire off a shutdown notification to the coordinator of whatever service created a billing record if over budget. Same idea.
Basically, batch, amortize and cache work. Same as every computer problem. You establish some SLO for how much time your services can continue running after an overage has occurred, and if that's a couple minutes or whatever that will cut out like 99.99% of the impact in these stories.
Solving this across all resource types and billing axes however is a different problem. You can't cache the notion than a VM is under the billing cap for an hour if there's another service that push spend over the cap within that hour.
You're right that you could establish SLOs across everything and minimise the amount of monetary loss, in theory, but at scale (as some resource types necessarily bill infrequently, as customers are spending more per hour), I suspect even this breaks down.
Then there's still the issue of billing at rest. Do you shut off VMs? That might be an easy question to answer. Do you delete their storage though? Harder. Do you delete blob storage? Probably not, but you've got to swallow the cost of that until the customer decides to pay again.
A $1k overrun past your billing cap is still way better than a $50k overrun - the cloud vendor is more likely to get paid in the end, and the customer is more likely to come away from the experience looking at it as an 'oops' instead of a catastrophic, potentially-finances-ruining 'i'm never touching this service again' incident.
There are plenty of really challenging problems in computer science and we solve them with compromises every day while hitting demanding targets. If a SSL certificate expires we expect it to stop working, and if it's revoked we expect the revocation to take effect eventually. But it becomes a situation where these guarantees benefit small companies and independent developers but we suddenly can't solve similar problems?
Fundamentally speaking if you can't afford to check against the billing cap every request, check every 10 requests. If 10 is too often, every 100. If 100 is too often, every 1000. Or check once per minute, or once per hour. Or compute a random number and check if it exceeds a threshold. The check can even be asynchronous to avoid adding intermittent latency to requests.
Any of these are better than nothing and would stop the bleeding of a runaway service incident. It's unrealistic to expect small companies and independent developers to have someone on-call 24/7 and it's also unrealistic to expect that if you sell them $100k worth of stuff they can't pay for that they'll actually pay you somehow.
This is where I question the risk of serverless. Although, now that I think about it, while my one client’s EC2 instances are essentially capped in terms of capacity and spend, we also use S3 etc. I suppose it would be entirely possible to accidentally write a huge amount of data to S3. But again I would rather get warnings from the app that writing to S3 has failed due to limits than get a huge bill!
I don't disagree on wanting this feature, but it's just not something that's possible to implement in totality when you dig into the details.
Or, alternatively, just let me set the size of the volume. Treat it like a hard drive?
In terms of your point about the data at rest, part of the issue is that we get a bill once per month, and that’s probably when we would notice it. Of course there is probably a Cloudwatch alarm or something we could set (I assume) but there’s so many damn services…
This all trades off though against the possibility of bringing down one of your customers when they are hitting peak sales on their website, which is a very bad look.
I suspect that if they tried to sue you over an unpaid bill, they'd lose over issues of proper notice to you. You can't actually bill someone for a service they didn't want and didn't ask for; that's why the wash-your-car-in-traffic people are considered scammers.
[1]: https://firebase.google.com/docs/projects/billing/avoid-surp...
It's not an automatic hard-stop so you could still screw yourself over pretty badly with runaway spending.
> We don't turn off services and usage because although you might have a bug in your app causing an increase in spend, you might just be experiencing unexpected positive growth of your app. You don't want your app to shut down unexpectedly when you need it to work the most.
Frankly, I don't see anything on that page that would actually prevent a surprise bill.
We have a multi-organization OpenAI account and I had set a $4k/month limit on one of the child orgs that was being used for a R&D project. Got billed ~$20k for the project one month and complained that it clearly allowed us to exceed our soft and hard limits. We were told that the limits (which you can set and act like they are a real thing) don't do anything if you have a child organization set up. They refunded us after some persuading and I know it's just normal growing pains for an organization that is undergoing rapid growth and maturity, but was still a little surprising that even the "hard limit" didn't do anything for us!
(Note that this was last year, so this bug is probably long fixed as they have redesigned their portal multiple times now)
For Firebase, their costs are probably pretty marginal.
Hard Spending Limit
If you configure hard spending limits, then upon the sum of money being reached, the service will be stopped until manual user action is taken. This can result in service outages.
Spending limit: ______ $
I have read and agree with the terms: [X]
[Confirm]
Which is likely to happen never. That's why I mostly use traditional VPSes.Correct. Because it would change the dynamic with enterprise customers.
Right now CEO/CFO is forced to accept unlimited open ended billing. If there was a way to set a hard limit companies would utilize it as part of their annual budgeting, not just hobbyists. That would be bad for big cloud's profits.
Not what would be the consumer friendly choice, which would be to charge “same as before”, i. e $0.
Our API helps apps take control of their SMS verification costs by proactively blocking fraudulent and suspicious verification attempts before they ever trigger an SMS. On top of that, we let users set daily limits as an additional safety net, so surprises like this don’t happen.
Arguably worse than an average 2025 LLM.
I've found it pretty nerve-wracking, then, to attempt to get some hobby projects online. I don't like writing blank checks for my hobbies..
Had a good experience with them.
Though I wish taking backups on firestore was easier.
I still have a couple of apps running on Firebase. I can't wait to get rid of those.
In 2016 Firebase seemed the holy grail for frontend devs and I deeply regret ever using it.
Probably because of this discussion.
but anyway, this is partially why I spin up an LLC for every app i make... just declare bankruptcy and kill it
In particular, do you use something like Stripe Atlas for that? And if so, is there any impact to your account with them when you declare a bankruptcy?
https://www.microsoft.com/en-us/startups/blog/trusted-partne...
Doing a search/replace on the operating agreement for the new entity name, filing it away somewhere, etc. typically takes more time.
Also, assume I already have an LLC then how do I indicate this account is an LLC? Can you sign up as an organization on Firebase?
I’d only feel comfortable with a new account at Firebase explicitly in the LLC’s name, and keep my personal name/identity far away from it.
It’s not hard to get an EIN (a few clicks online and free), open a bank account (my credit union turns this around in a day or two), fund it, use debit card as billing method.
To Firebase: either you must automatically condone such mistakes or implement the requested feature.
To anybody else: Does Supabase or other platforms offer this?
also, billing caps really don’t make any sense. a competitor could easily exploit that to take your service down. best to you know, architect your stuff correctly… speaking of architecture, creating a paas where you have a hard billing cap would pretty much obliterate performance as you would need round trips each time you do anything in order to confirm you’re not over the amount.
So it entirely depends on what it is you are doing.
(I'd look this up myself but replies on Xitter seem to be Muskwalled)
Even if your card declines or you used a prepaid card...you're still legally liable for the full amount.