Something happened, we won't say what, but it was severe enough to notify law enforcement. What floors me is the only actionable advice is to "review environment variables". What should a customer even do with that advice? Make sure the variable are still there? How would you know if any of them were exposed or leaked?
The advice should be to IMMEDIATELY rotate all passwords, access tokens, and any sensitive information shared with Vercel. And then begin to audit access logs, customer data, etc, for unusual activity.
The only reason to dramatically overpay for the hosting resources they provide is because you expect them to expertly manage security and stability.
I know there is a huge fog of uncertainly in the early stages of an incident, but it spooks me how intentionally vague they seem to be here about what happened and who has been impacted.
> Environment variables marked as "sensitive" in Vercel are stored in a manner that prevents them from being read, and we currently do not have evidence that those values were accessed. However, if any of your environment variables contain secrets (API keys, tokens, database credentials, signing keys) that were not marked as sensitive, those values should be treated as potentially exposed and rotated as a priority.
https://vercel.com/kb/bulletin/vercel-april-2026-security-in... as of 4:22p ET
https://vercel.com/docs/environment-variables/sensitive-envi...
So they are harder to introspect and review once set.
It’s probably good practice to put non-secret-material in non-sensitive variables.
(Pure speculation, I’ve never used Vercel)
There are cases where I want env variables to be considered non-secure and fine to be read later, I have one in a current project that defines the email address used as the From address for automated emails for example.
In my opinion the lack of security should be opt-in rather than opt-out though. Meaning it should be considered secure by default with an option to make it readable.
Oh and the owner likes to proudly remind people about his work on Google AMP, a product that has done major damage to the open web.
This is who they are: a bunch of incompetent engineers that play with pension funds + gulf money.
This and because it's so convenient to click some buttons and have your application running. I've stopped being lazy, though. Moved everything from Render to linode. I was paying render $50+/month. Now I'm paying $3-5.
I would never use one of those hosting providers again.
The point is, I used to just throw everything up on a PaaS. Heroku/Render, etc. and pay way more than I needed to, even if I had 0 users, lol.
The only possibility for that not being a reasonable starting point is if they think the malicious actors still have access and will just exfiltrate rotated secrets as well. Otherwise this is deflection in an attempt to salvage credibility.
While a different kind of incident (in hindsight), the other week Webflow had a serious operational incident.
Sites across the globe going down (no clue if all or just a part of them). They posted plenty of messages, I think for about 12 hours, but mostly with the same content/message: "working on fixing this with an upstream provider" (paraphrased). No meaningful info about what was the actual problem or impact.
Only the next day did somebody write about what happened. Essentially a database running out of storage space. How that became a single point of failure, to at least plenty of customers: no clue. Sounds like bad architecture to me though. But what personally rubbed me the wrong way most of all, was the insistence on their "dashboard" having indicated anything wrong with their database deployment, as it allegedly had misrepresented the used/allocated storage. I don't who this upstream service provider of Webflow is, but I know plenty about server maintenance.
Either that upstream provider didn't provide a crucial metric (on-disk storage use) on their "dashboard", or Webflow was throwing this provider under the bus for what may have been their own ignorant/incompetent database server management. I guess it all depends to which extend this database was a managed service or something Webflow had more direct control over. Either way, with any clue about the provider or service missing from their post-mortem, customers can only guess as to who was to blame for the outage.
I have a feeling that we probably aren't the only customer they lost over this. Which in our case would probably not have happened, if they had communicated things in a different way. For context: I personally would never need nor recommend something like Webflow, but I do understand why it might be the right fit for people in a different position. That is, as long as it doesn't break down like it did. I still can't quite wrap my head around that apparent single point of failure for a company the size of Webflow though.
/anecdote
> Indicators of compromise (IOCs)
> Our investigation has revealed that the incident originated from a third-party AI tool whose Google Workspace OAuth app was the subject of a broader compromise, potentially affecting hundreds of its users across many organizations.
> We are publishing the following IOC to support the wider community in the investigation and vetting of potential malicious activity in their environments. We recommend that Google Workspace Administrators and Google Account owners check for usage of this app immediately.
> OAuth App: 110671459871-30f1spbu0hptbs60cb4vsmv79i7bbvqj.apps.googleusercontent.com
https://vercel.com/kb/bulletin/vercel-april-2026-security-in...
This feels like a natural consequence of the direction web development has been going for the last decade, where it's normalised to wire up many third party solutions together rather than building from more stable foundations. So many moving parts, so many potential points of failure, and as this incident has shown, you are only as secure as your weakest link. Putting your business in the hands of a third party AI tool (which is surely vibe-coded) carries risks.
Is this the direction we want to continue in? Is it really necessary? How much more complex do things need to be before we course-correct?
We need a different hosting model.
And my own kernel. Can't trust some shit written by a Finnish dude 30 years ago.
And my own UEFI firmware. Definitely can't trust some shit written by my hardware vendor ever.
The AI maximalists would argue that the only way is through more AI. Vibe code the app, then ask an LLM to security review it, then vibe code the security fixes, then ask the LLM to review the fixes and app again, rinse and repeat in an endless loop. Same with regressions, performance, features, etc. stick the LLM in endless loops for every vertical you care about.
Pointing to failed experiments like the browser or compiler ones somehow don’t seem to deter AI maximalists. They would simply claim they needed better models/skills/harness/tools/etc. the goalpost is always one foot away.
You can write good and bad code with and without AI, on a managed service, self-hosted, or something in between.
And the comment I was replying to said something about not trusting something written in Akron, OH 2 years ago, which makes no sense and is barely an argument, and I was mostly pointing out how silly that comment sounds.
It’s interesting how they all use LLMs to write their Reddit posts, too. Some of them could have drawn in some people if they took 5 minutes to type an announcement post in their own words, but they all have the same LLM style announcement post, too. I wonder if they’re conversing with the LLM and it told them to post it to Reddit for traction?
Protection money from Vercel.
"Pay us 10% of revenue or we switch to generating Netlify code."
So I told something like, "don't use anything node at all", and it immediately rewrote it as a Python backend, and it volunteered that it was minimizing dependencies in how it did that.
[1] only vibe coding as an exercise for a throwaway artifact; I'm not endorsing vibe coding
You don't have to live like this.
I've heard others had similar results with .NET/C#
I once made a golang multi-person pomodoro app by vibe coding with gemini 3.1 pro (when it had first launched first day) and I asked it to basically only have one outside dependency of gorrilla websockets and everything else from standard library and then I deployed it to hugging face spaces for free.
I definitely recommend golang as a language if you wish to vibe code. Some people recommend rust but Golang compiles fast, its cross compilation and portable and is really awesome with its standard library
(Anecdotally I also feel like there is some chances that the models are being diluted cuz like this thing then has become my benchmark test and others have performed somewhat worse or not the same as this to be honest and its only been a few days since I am now using hackernews less frequently and I am/was already seeing suspicions like these about claude and other models on the front page iirc. I don't know enough about claude opus 4.7 but I just read simon's comment on it, so it would be cool if someone can give me a gist of what is happening for the past few days.)
Switch to vibe coding Rust backends and freeze your supply chain.
Super strong types. Immaculate error handling. Clear and easy to read code. Rock solid performance. Minimal dependencies.
Vibe code Rust for web work. You don't even need to know Rust. You'll osmose it over a few months using it. It's not hard at all. The "Rust is hard" memes are bullshit, and the "difficult to refactor" was (1) never true and (2) not even applicable with tools like Claude Code.
Edit: people hate this (-3), but it's where the alpha is. Don't blindly dismiss this. Serializing business logic to Rust is a smart move. The language is very clean, easy to read, handles errors in a first class fashion, and fast. If the code compiles, then 50% of your error classes are already dealt with.
Python, Typescript, and Go are less satisfactory on one or more of these dimensions. If you generate code, generate Rust.
But you're also correct in that Rust is actually possible to write in a more high level way, especially for web where you have very little shared state and the state that is shared can just be wrapped in Arc<> and put in the web frameworks context. It's actually dead easy to spin up web services in Rust, and they have a great set of ORM's if thats your vibe too. Rust is expressive enough to make schema-as-code work well.
On the dependencies, if you're concerned about the possibility of future supply chain attacks (because Rust doesn't have a history like Node) you can vendor your deps and bypass future problems. `cargo vendor` and you're done, Node has no such ergonomic path to vendoring, which imo is a better solution than anything else besides maybe Go (another great option for web services!). Saying "don't use deps" doesn't work for any other language other than something like Go (and you can run `go vendor` as well).
But yeah, in today's economy where compute and especially memory is becoming more constrained thanks to AI, I really like the peace of mind knowing my unoptimised high level Rust web services run with minimal memory and compute requirements, and further optimisation doesn't require a rewrite to a different language.
Idk mate, I used to be a big Rust hater but once I gave the language a serious try I find it more pleasant to write compared to both Typescript and Go. And it's very amiable to AI if that's your vibe(coding), since the static guarantees of the type system make it easier for AI to generate correct code, and the diagnostics messages allow it to reroute it's course during the session.
However it is less clear on how to do this, people mostly take the easiest path.
> b. (Recommended) Do something that works now, you can always make it better later
The internet does that but it feels different with this
That's a funny way of saying "race to the bottom."
> The internet does that but it feels different with this
How does "the internet do that?" What force on the internet naturally brings about mediocrity? Or have we confused rapacious and monopolistic corporations with the internet at large?
It is true that "more diversity in code" probably means less turnkey spray-and-pray compromises, sure. Probably.
It also means that the models themselves become targets. If your models start building the same generated code with the same vulnerability, how're you gonna patch that?
This situation is pretty funny to me. Some of my friends who arent technical tried vibe coding and showed me what they built and asked for feedback
I noticed they were using Supabase by default, pointed out that their database was completely open with no RLS
So I told them not to use Supabase in that way, and they asked the AI (various diff LLMs) to fix it. One example prompt I saw was: please remove Supabase because of the insecure data access and make a proper secure way.
Keep in mind, these ppl dont have a technical background and do not know what supabase or node or python is. They let the llm install docker, install node, etc and just hit approve on "Do you want to continue? bash(brew install ..)"
Whats interesting is that this happened multiple times with different AI models. Instead of fixing the problem the way a developer normally would like moving the database logic to the server or creating proper API endpoints it tried to recreate an emulation of Supabase, specifically PostgREST in a much worse and less secure way.
The result was an API endpoint that looked like: /api/query?q=SELECT * FROM table WHERE x
In one example GLM later bolted on a huge "security" regular expression that blocked , admin, updateadmin, ^delete* lol
That lack of diversity also makes patches more universal, and the surface area more limited.
These libraries/frameworks are not insecure because of bad design and dependency bloat. No! It's because a mythical LLM is so powerful that it's impossible to defend against! There was nothing that could be done.
Both have been changing as people realize it's rarely the right tool for the job, and as LLMs also become more intelligent and better at suggesting other, better options depending on what is asked for (especially Claude Opus).
My impression is Next started becoming popular mostly as a reaction against create-react-app.
nextjs is also powerful due to AI. But the value is a robust interactive front-end, easily iterated, with maybe SSR backing, nothing specific to nextjs (it's routing semantics + React).
So much complexity has gone into SSR. I hate 5MB client runtime just to read text as much as anyone, but not if the tradeoff is isomorphic env with magic file first-line incantations.
Recent Claude models do well with it, especially after adding the official skill.
I have only recently started using it, so would love to hear about anyone else's experience.
I guess they should have put some of that marketing money into hiring someone to manage the security of their systems. It's pretty telling that they had to hire an "incident response provider" just to figure out what happened and clean up after the hack. If you treat security like something you don't have to worry about until after you've been hacked you're probably going to get hacked.
Plenty to criticize them for, but that's totally standard and not something to ding them for. Probably something their cyber insurance has in their contract.
Forensics is its own set of skills, different from appsec and general blue team duties. You really want to make sure no backdoors got left in.
Everything runs fine locally until you try to deploy it, and bam you need 4g ram machine to run the thing.
So you host it on Vercel for free cause it's easy!
Then you want to check for more than 30 seconds of analytics, and it's pay time.
But the argument is if you’re using Vercel for production, you’re paying 5-10x what you’d pay for a VM, with 4gb.
So then what’s the rationale? You can’t be a hobbyist but also “it’s pay time” for production?
I’m still planning to move elsewhere though, the vendor lock-in is not worth it and I’d like to keep our infra in the EU.
That's for the free plan.
Limits are documented here:
https://developers.cloudflare.com/workers/platform/limits/#w...
Good work on workers though, maybe the next generation of sandstorm will be built on CloudFlare in a decade or so after all the bugs have been hammered out.
Knowing how to operate a basic server is perceived as hard and dangerous by many, especially the generation that didn’t have a chance to play with Linux for fun when growing up
I am always feeling like I'm doing something wrong running bare metal based on modern advice, but it's low latency, simple, and reliable.
Probably because I've been using linux since Slackware in the 90s so it's second nature. And now with the CLI-based coding tools, I have a co-sysadmin to help me keep things tidy and secure. It's great and I highly recommend more people try it.
They have a free tier plan for non-commercial usage and a very very good UX for just deploying your website.
Many companies start using Vercel for the convenience and, as they grow, they continue paying for it because migrating to a cheaper provider is inconvenient.
They regularly try to get me to join an enterprise plan but no service cutoff threats yet.
Meaning since 2015, you’ve got an 8.2% chance of having someone walk out with that box. Hopefully there’s nothing precious on it.
Thieves probably look for small stuff like jewelry, cash, laptops, not some big old server.
The chance of being burglarized is not the same as the chance that when you are hit, they decide to take your webserver. Think it through.
> Indicators of compromise (IOCs)
> Our investigation has revealed that the incident originated from a third-party AI tool whose Google Workspace OAuth app was the subject of a broader compromise, potentially affecting hundreds of its users across many organizations.
> We are publishing the following IOC to support the wider community in the investigation and vetting of potential malicious activity in their environments. We recommend that Google Workspace Administrators and Google Account owners check for usage of this app immediately.
> OAuth App: 110671459871-30f1spbu0hptbs60cb4vsmv79i7bbvqj.apps.googleusercontent.com
https://vercel.com/kb/bulletin/vercel-april-2026-security-in...
I’m no security engineer, but this is flatly unacceptable, right? This feels like Vercel is covering its own ass in favor of helping its customers understand the impact of this incident.
Clicking the Vercel logo at the top left of the page hard crashes my Chrome app. Like, immediate crash.
What an interesting bug.
I find it fun we're all reading a story how Vercel likely is compromised somehow, and managed to reproduce a crash on their webpage, so now we all give it a try. Surely could never backfire :)
Chrome Version 147.0.7727.101 (Official Build) (64-bit). Windows 11 Pro.
Video: https://imgur.com/a/pq6P4si
I use uBlock Origin Lite. Maybe it blocks some crash causing script? edit: still no crash when I disabled UBO.
No crash.
Now I don't want to click that "Finish update" button.
https://x.com/theo/status/2045862972342313374
> I have reason to believe this is credible.
https://x.com/theo/status/2045870216555499636
> Env vars marked as sensitive are safe. Ones NOT marked as sensitive should be rolled out of precaution
https://x.com/theo/status/2045871215705747965
> Everything I know about this hack suggests it could happen to any host
https://x.com/DiffeKey/status/2045813085408051670
> Vercel has reportedly been breached by ShinyHunters.
if it's not marked as sensitive (because it is not sensitive) there is no reason to roll them. if you must roll a insensitive env var it should've been sensitive in the first place, no?
I can imagine the reason why an env variable would be sensitive, but need to be re-read at some point. But overwhelmingly it makes sense for the default to be set, and never access again (i.e. Fly env values, GCP secret manager etc)
I feel for the team; security incidents suck. I know they are working hard, I hope they start to communicate more openly and transparently.
So they use third-party for incident management? They are de-risking by spending more, which is a loose-loose for the customers.
He also suggests in another post that Linear and GitHub could also be pwned?
Either way, hugops to all the SRE/DevOps out there, seems like it's going to be a busy Sunday for many.
> Here’s what I’ve managed to get from my sources:
>3. The method of compromise was likely used to hit multiple companies other than Vercel.
https://x.com/theo/status/2045870216555499636
To be fair journalists often do this too, eg. "[company] was breached, people within the company claim"
Theo's content boils down to the same boring formula. 1. Whatever buzzword headline is trending at the time 2. Immediate sponsored ad that is supposed to make you sympathize with Theo cause he "vets" his sponsors. 3. The man makes you listen to a "that totally happened" story that he somehow always involved himself personally. 4. Man serves you up an ad for his t3.chat and how it's the greatest thing in the world and how he should be paid more for his infinite wisdom. 5. A rag on Claude or OpenAI (whichever is leading at the time) 6. 5-10 minutes of paraphrasing an article without critical thought or analysis on the video topic.
I used to enjoy his content when he was still in his Ping era, but it's clear hes drunken the YT marketer kool-aid. I've moved on, his content gets recommend now and again, but I can't entertain his non-sense anymore.
However, since the LLM Cambria explosion, he has become very clickbaity, and his content has become shallow. I don't watch his videos anymore.
When you're putting the bar that low, sure.
He's about as knowledgeable as the junior you hired last week, except that he speaks from a position of authority and gets retweeted by the entire JS slop sphere. He's LinkedIn slop for Gen Z.
I don't think he's a bad guy or that he's trying to be misleading. I suspect he wants his content to actually carry value, but he produces too much for that to be possible. Primarily he's a performer, not a technologist.
> @ErdalToprak: "And use your own vps or k3s cluster there’s no reason in 2026 to delegate your infra to a middle man except if you’re at AWS level needs"
> @theo: "This is still a stupid take"
lol, okay. Thanks for the insight, Theo, whoever you are.
I see Vercel is hosted on AWS? Are they hosting every one on a single AWS account with no tenant isolating? Something this dumb could never happen on a real AWS account. Yes I know the internal controls that AWS has (former employee).
Anyone who is hosting a real business on Vercel should have known better.
I have used v0 to build a few admin sites. But I downloaded the artifacts, put in a Docker container and hosted everything in Lambda myself where I controlled the tenant isolation via separate AWS accounts, secrets in Secret Manager and tightly scoped IAM roles, etc.
It doesn’t make sense for a random employee who mistakenly uses a third party app to compromise all of its users it’s a poor security architecture.
It’s about as insecure as having one Apache Server serving multiple customer’s accounts. No one who is concerned about security should ever use Vercel.
Hey, I’m with you - I think social media needs to die specifically for this reason. I’m reminded of the term “snake oil” - it’s like the dawn of newspapers again.
Next.js is the new PHP, but worse, since unlike PHP you don't really know what's server side and what's client side anymore. It's all just commingled and handled magically.
https://aws.amazon.com/security/security-bulletins/rss/aws-2...
Wasn't unheard of back in the day, that you leaked things via PHP templates, like serializing and adding the whole user object including private details in a Twig template or whatever, it just happened the other way around kind of. This was before "fat frontend, thin backend" was the prevalent architecture, many built their "frontends" from templates with just sprinkles of JavaScript back then.
But there are more people trying to secure this framework and the underlying tools than there would be on some obscure framework or something the average company built themselves.
Also "pay a real provider", what does that mean? Are you again implying that the average company should be responsible for _more_ of their own security in their hosting stack, not less?
Most companies have _zero_ security engineers.. Using a vertically-integrated hosting company like Vercel (or other similar companies, perhaps with different tech stacks - this opinion has nothing to do with Next or Node) is very likely their best and most secure option based on what they are able to invest in that area.
PHP was so simple and easy to understand that anyone with a text editor and some cheap shared hosting could pick it up, but also low level enough that almost nothing was magically done for you. The result was many inexperienced developers making really basic mistakes while implementing essential features that we now take for granted.
Frameworks like Next.js take the complete opposite approach, they are insanely complex but hide that complexity behind layers and layers of magic, actively discouraging developers from looking behind the curtain, and the result is that even experienced developers end up shooting themselves in the foot by using the magical incantations wrong.
What’s worse is vercel corrupted the react devs and convinced them that RSC was a good idea. It’s not like react was strictly in good hands at Facebook but at least the team there were good shepherds and trying to foster the ecosystem.
I’m not exactly surprised, but it seems like the unserious, ill-informed and lazy are taking over. There is absolutely zero reason why a large, essential public service should be overspending and running on an unnecessary managed service like vercel… yet, here we are.
What's your agenda here?
Also LLMs will be used to attack only, no one will be smart to integrate it into CI flows, because everyone is that dumb. No security tools will pop up.
Let that be the end of Microsoft. Was forced to use their shitty products for years, by corporate inertia and their free Teams and Azure licenses, first-dose-is-free, curse.
AI agents have the benefit of working at scale, probably "better" used for mass targeting.
But I get your point.