Meta shouldn't be doing this, they need to be more careful, but...

I used to work on a site with basic caching, a big web framework, every page dynamic, and 3 frontend webservers plus a database primary and replica. Super basic infra, and a bill close to this user.

We would never have noticed 3 to 4 requests per second like this. And we weren't being that smart about it, we were rendering every page not serving from cache (we mostly cached DB results). We were also conscious of not accidentally building SEO bot traps that would cause them to go around in loops, not because of the traffic generated, but because it was bad for SEO!

This just strikes me as bad engineering on both sides. Yes Meta is the one with the big budgets and they should sort this out, but also you can't pay 10-100x for your infra and get annoyed when you have a big bill. On the web people and bots are going to make requests and you just have to design for that.

Obviously horrendous but why isn’t this person monitoring his site?

Also, why do people use vercel nowadays? I’m sure there are reasons, but I moved over to railway (you can insert alternative provider here) and I no long f* around trying to fix page load time due to cold starts, I have predictable pricing, and my sites on railway are fast so much faster. Plus, if cost is a factor, railway offers serverless. It’s not as shiny as vercel, but nextjs works perfectly on it.

It astounds me that vercel has positioned themselves as a sanctuary city for normies and yet, the city is littered with landmines and booby traps.

That's like 4 requests per second, hardly seems excessive at all. We're not on dial-up anymore.

undefined

Crazy to me that someone would run a website where you pay for every request you receive, instead of a fixed monthly rate. It’s an obvious recipe for disaster - crossing the wrong guy would cost you dearly. Or just a crawler running amok.

I have the same problem. I have 6M URL’s per domain. 8 different domains. 80% of search traffic is long tail.

If I don’t block, 95% of my resources will be spent on feeding bots.

I had to block all “official” AI useragents and entire countries like Singapore and China. But there are so many unofficial bots which spread their work over dozes of IP addresses that it seems impossible to block on the reverse proxy level. How do you block those?