I've been part of a response team on a security incident and I really feel for them. However, this initial communication is terrible.
Something happened, we won't say what, but it was severe enough to notify law enforcement. What floors me is the only actionable advice is to "review environment variables". What should a customer even do with that advice? Make sure the variable are still there? How would you know if any of them were exposed or leaked?
The advice should be to IMMEDIATELY rotate all passwords, access tokens, and any sensitive information shared with Vercel. And then begin to audit access logs, customer data, etc, for unusual activity.
The only reason to dramatically overpay for the hosting resources they provide is because you expect them to expertly manage security and stability.
I know there is a huge fog of uncertainly in the early stages of an incident, but it spooks me how intentionally vague they seem to be here about what happened and who has been impacted.
> Environment variables marked as "sensitive" in Vercel are stored in a manner that prevents them from being read, and we currently do not have evidence that those values were accessed. However, if any of your environment variables contain secrets (API keys, tokens, database credentials, signing keys) that were not marked as sensitive, those values should be treated as potentially exposed and rotated as a priority.
There are cases where I want env variables to be considered non-secure and fine to be read later, I have one in a current project that defines the email address used as the From address for automated emails for example.
In my opinion the lack of security should be opt-in rather than opt-out though. Meaning it should be considered secure by default with an option to make it readable.
How does the app read the variable if it can't be read after you input it? Or do they mean you can't view it after providing the variable value to the UI?
Seriously. Why am I reading about this here and not via an email? I've been a paying customer for over a year now. My online news aggregator informs me before the actual company itself does?
Please remember that this is the same company that couldn't figure out how to authorize 3rd party middleware and had, with what should be a company ending, critical vulnerability .
Oh and the owner likes to proudly remind people about his work on Google AMP, a product that has done major damage to the open web.
This is who they are: a bunch of incompetent engineers that play with pension funds + gulf money.
I just deleted my account. Their laid-back notice just is not worth it anymore. I will hold them accountable using my cash. You can get out with me. Let their apologies hit your spam filter. They need to be better prepared to react to the storm of insanity that comes with a breach or they lose my info (lose it twice, I guess..)
The Oracle that published an announcement that said "we didn't get hacked" when the hackers had private customer info?
The Oracle that does not allow you to do any security testing on their software unless you use one of their approved vendors?
The Oracle that one of my customers uses where they have to turn off the HR portal for 2 weeks before annual performance evaluations because there is no way to prevent people from seeing things?
The only reason Oracle isn't having nightmarish security problems published every other week is because they threaten to sue anyone that does find an issue.
Oracle is a joke in every conceivable way and I despise them on a personal level.
> The only reason to dramatically overpay for the hosting resources they provide is because you expect them to expertly manage security and stability.
This and because it's so convenient to click some buttons and have your application running. I've stopped being lazy, though. Moved everything from Render to linode. I was paying render $50+/month. Now I'm paying $3-5.
I would never use one of those hosting providers again.
It could be $0 on Render too, but then there's going to be a 3 minute load time for a landing page to become visible, lol. So if you don't want your server to sleep, you're going to have to pay $20/month.
Yeah, given there insane pricing I think the expectations can be higher. Although I know it is impossible to provide 100% secure system, but if something like that happens, then the communication should at least be better. Don’t wait until you have talked to the lawyers... inform your customers first, ideally without this cooperate BS speak, most vercel customers are probably developers, so they understand that incidents like this can happen, just be transparent about it
Completely agreed. At minimum they should be advising secret rotation.
The only possibility for that not being a reasonable starting point is if they think the malicious actors still have access and will just exfiltrate rotated secrets as well. Otherwise this is deflection in an attempt to salvage credibility.
While a different kind of incident (in hindsight), the other week Webflow had a serious operational incident.
Sites across the globe going down (no clue if all or just a part of them). They posted plenty of messages, I think for about 12 hours, but mostly with the same content/message: "working on fixing this with an upstream provider" (paraphrased). No meaningful info about what was the actual problem or impact.
Only the next day did somebody write about what happened. Essentially a database running out of storage space. How that became a single point of failure, to at least plenty of customers: no clue. Sounds like bad architecture to me though. But what personally rubbed me the wrong way most of all, was the insistence on their "dashboard" having indicated anything wrong with their database deployment, as it allegedly had misrepresented the used/allocated storage. I don't who this upstream service provider of Webflow is, but I know plenty about server maintenance.
Either that upstream provider didn't provide a crucial metric (on-disk storage use) on their "dashboard", or Webflow was throwing this provider under the bus for what may have been their own ignorant/incompetent database server management. I guess it all depends to which extend this database was a managed service or something Webflow had more direct control over. Either way, with any clue about the provider or service missing from their post-mortem, customers can only guess as to who was to blame for the outage.
I have a feeling that we probably aren't the only customer they lost over this. Which in our case would probably not have happened, if they had communicated things in a different way. For context: I personally would never need nor recommend something like Webflow, but I do understand why it might be the right fit for people in a different position. That is, as long as it doesn't break down like it did. I still can't quite wrap my head around that apparent single point of failure for a company the size of Webflow though.
> Our investigation has revealed that the incident originated from a third-party AI tool whose Google Workspace OAuth app was the subject of a broader compromise, potentially affecting hundreds of its users across many organizations.
> We are publishing the following IOC to support the wider community in the investigation and vetting of potential malicious activity in their environments. We recommend that Google Workspace Administrators and Google Account owners check for usage of this app immediately.
Huh, curiously; I'm on Arch Linux, crash happens in Google Chrome (147.0.7727.101) for me too, but not in Firefox (149.0.2) nor even in Chromium (147.0.7727.101).
I find it fun we're all reading a story how Vercel likely is compromised somehow, and managed to reproduce a crash on their webpage, so now we all give it a try. Surely could never backfire :)
if it does so happen that the crash originates from a browser exploit, you should expect to be more at risk due to the absence of a crash on an older version, not less
> Ones NOT marked as sensitive should be rolled out of precaution
if it's not marked as sensitive (because it is not sensitive) there is no reason to roll them. if you must roll a insensitive env var it should've been sensitive in the first place, no?
There's a difference between sensitive, private and public. If public (i.e. NEXT_PUBLIC_) then yeah likely not a reason to roll. Private keys that aren't explicitly sensitive probably are still sensitive. It doesn't seem to be the default to have things "sensitive" and I can't tell if that's a new classification or has always been there.
I can imagine the reason why an env variable would be sensitive, but need to be re-read at some point. But overwhelmingly it makes sense for the default to be set, and never access again (i.e. Fly env values, GCP secret manager etc)
He’s a streamer who talks about tech. Previously had a sponsorship relationship with Vercel so is theoretically more well connected than average on the topic. He’s also very divisive because he does a lot of ragebait, grievance reporting, and contrarian takes but famously has blind spots for a few companies and technologies that he’s favored in past videos or been sponsored by. I have friends who watch a lot of his videos but I’ve never been able to get into it.
Incidents like this are a good reminder of how concentrated our single points of failure have become in the modern web ecosystem. I appreciate the transparency in their disclosure so far, but it definitely makes you re-evaluate the risk profile of leaning entirely on fully managed PaaS solutions.
Conjecture, but the wording "limited subset" rarely turns out to be good news. Usually a provider will say "less than 1% of our users" or some specific number when they can to ease concerns. My guess is they don't have the visibility or they don't like the number.
I feel for the team; security incidents suck. I know they are working hard, I hope they start to communicate more openly and transparently.
“Less than 1% of our users” means 10k affected users if you have 1 million users. 10k victims is a lot! Imagine “air travel is safe, only a subset of 1% of travellers die”
The lack of details itself is telling enough. Whatever comes out will be no doubt PR sanitised and some bigger clumps of truth won't make it through the PR process.
This announcement in its current form is quite useless and not actionable. As least people won’t be able to say “why didn’t you say something sooner?” They said _something_
So, the Vercel post says a number of customers were impacted, but not everyone, and they will contact the people that were impacted.
I wasn't contacted so does that mean I'm safe?
We run on Vercel and I wonder if / how long before we're alerted about a leak. Quick look online suggests environment variables marked as sensitive are ok, but to which extent I wonder.
Hey, I’m with you - I think social media needs to die specifically for this reason. I’m reminded of the term “snake oil” - it’s like the dawn of newspapers again.
Vercel acknowledges a security incident, which nobody is claiming doesn't exist. What they don't acknowledge are this person's vague implications about impact elsewhere.
Ah, Theo with his vast insights and connections into everything. That man gets around, and his content is worth it's cost.
Theo's content boils down to the same boring formula.
1. Whatever buzzword headline is trending at the time
2. Immediate sponsored ad that is supposed to make you sympathize with Theo cause he "vets" his sponsors.
3. The man makes you listen to a "that totally happened" story that he somehow always involved himself personally.
4. Man serves you up an ad for his t3.chat and how it's the greatest thing in the world and how he should be paid more for his infinite wisdom.
5. A rag on Claude or OpenAI (whichever is leading at the time)
6. 5-10 minutes of paraphrasing an article without critical thought or analysis on the video topic.
I used to enjoy his content when he was still in his Ping era, but it's clear hes drunken the YT marketer kool-aid. I've moved on, his content gets recommend now and again, but I can't entertain his non-sense anymore.
I just wanted to chime in and say I think he is knowledgeable; he's not a con. I know you didn't say that, but people might have the impression he doesn't know what he's talking about. He does know, and I've learned quite a lot from him in the past.
However, since the LLM Cambria explosion, he has become very clickbaity, and his content has become shallow. I don't watch his videos anymore.
Not that I ever had confidence in his technical knowledge, but it went to zero when he confidently asserted that there was no possible way a single server could handle the massive traffic some NextJS app he had made was serving. He then posted the bill - which was about $5K IIRC - and I was able to determine from the billed runtime and memory that a modestly-spec’d RPi could in fact handle it.
He's about as knowledgeable as the junior you hired last week, except that he speaks from a position of authority and gets retweeted by the entire JS slop sphere. He's LinkedIn slop for Gen Z.
I don't watch his content, but I felt comfortable posting his link as I believe he's generally considered a reputable guy? His tweets sometimes come up in my for you tab and he seems reasonable and knowledgable generally? Maybe I'm wrong and shouldn't have linked to him as a source.
He's kind of like an LLM in that his content has the surface texture of something substantial, and sometimes it's backed by substance, yet it's often half-true or totally off the mark too. You'll notice if you're previously acquainted with what he's talking about, otherwise he seems to be as you described.
I don't think he's a bad guy or that he's trying to be misleading. I suspect he wants his content to actually carry value, but he produces too much for that to be possible. Primarily he's a performer, not a technologist.
> @theo: "I have reason to believe this is credible. If you are using Vercel, it’s a good idea to roll your secrets and env vars."
> @ErdalToprak: "And use your own vps or k3s cluster there’s no reason in 2026 to delegate your infra to a middle man except if you’re at AWS level needs"
> @theo: "This is still a stupid take"
lol, okay. Thanks for the insight, Theo, whoever you are.
You'll have to ask @ErdalToprak on Twitter on that one. I just thought it was funny that this slopfluencer, who's taken money to advertise Vercel, ostensibly believes that using a VPS/k3s is "a stupid take."
This is why you pay a real provider for serious business needs, not an AWS reseller. Next.js is a fundamentally insecure framework, as server components are an anti-pattern full of magic leading to stuff like the below. Given their standards for framework security, it's not hard to believe their business' control plane is just as insecure (and probably built using the same insecure framework).
Next.js is the new PHP, but worse, since unlike PHP you don't really know what's server side and what's client side anymore. It's all just commingled and handled magically.
> Next.js is the new PHP, but worse, since unlike PHP you don't really know what's server side and what's client side anymore. It's all just commingled and handled magically.
Wasn't unheard of back in the day, that you leaked things via PHP templates, like serializing and adding the whole user object including private details in a Twig template or whatever, it just happened the other way around kind of. This was before "fat frontend, thin backend" was the prevalent architecture, many built their "frontends" from templates with just sprinkles of JavaScript back then.
People say "Next.js is the new PHP" because it's the most popular and prominent tooling out there, and so by sheer number of available targets it's the one that comes up the most when things go wrong like this.
But there are more people trying to secure this framework and the underlying tools than there would be on some obscure framework or something the average company built themselves.
Also "pay a real provider", what does that mean? Are you again implying that the average company should be responsible for _more_ of their own security in their hosting stack, not less?
Most companies have _zero_ security engineers.. Using a vertically-integrated hosting company like Vercel (or other similar companies, perhaps with different tech stacks - this opinion has nothing to do with Next or Node) is very likely their best and most secure option based on what they are able to invest in that area.
PHP was so simple and easy to understand that anyone with a text editor and some cheap shared hosting could pick it up, but also low level enough that almost nothing was magically done for you. The result was many inexperienced developers making really basic mistakes while implementing essential features that we now take for granted.
Frameworks like Next.js take the complete opposite approach, they are insanely complex but hide that complexity behind layers and layers of magic, actively discouraging developers from looking behind the curtain, and the result is that even experienced developers end up shooting themselves in the foot by using the magical incantations wrong.
Totally agree. Nextjs is a vehicle to sell their PaaS, every other feature is a coincidence.
What’s worse is vercel corrupted the react devs and convinced them that RSC was a good idea. It’s not like react was strictly in good hands at Facebook but at least the team there were good shepherds and trying to foster the ecosystem.
Why does anyone running a third party tool have access to all of their clients’ accounts? I can’t imagine something this stupid happening with a real service provider.
I see Vercel is hosted on AWS? Are they hosting every one on a single AWS account with no tenant isolating? Something this dumb could never happen on a real AWS account. Yes I know the internal controls that AWS has (former employee).
Anyone who is hosting a real business on Vercel should have known better.
I have used v0 to build a few admin sites. But I downloaded the artifacts, put in a Docker container and hosted everything in Lambda myself where I controlled the tenant isolation via separate AWS accounts, secrets in Secret Manager and tightly scoped IAM roles, etc.
Is AWS security boundary the AWS account? Are you expecting Vercel to provision and manage an AWS account per user? That doesn’t make any sense man, though makes sense if you’re a former AWS employee.
It doesn’t make sense for a random employee who mistakenly uses a third party app to compromise all of its users it’s a poor security architecture.
It’s about as insecure as having one Apache Server serving multiple customer’s accounts. No one who is concerned about security should ever use Vercel.
I recently got hit by a car on my bike. While I was starting the claim filing process the web portal for ICBC (British Columbia insurance) was acting a little funky / stalling / and then gave me a weird access error. Down at the bottom of the error page was a little grey underlined link that said “vercel”.
I’m not exactly surprised, but it seems like the unserious, ill-informed and lazy are taking over. There is absolutely zero reason why a large, essential public service should be overspending and running on an unnecessary managed service like vercel… yet, here we are.
Much as I want to rip on vercel, its clear that ai is going to lead to mass security breaches. The attack surface is so large, and ai agents are working around the clock. This is a new normal. Open source software is going to change, companies wont be running random repos off github anymore
Most of recent issues, including this incident, happened not due to smart superintelligent "agents" taking over the world - chatbots and other text generators are about as intelligent amd powerful as a dead starfish - but due to the combined stupidity of the said chatbots amd lazy idiots who use them to hide their own incompetence and thus produce such embarassing mistakes. A few years ago, they would be fired for exposing secrets in plain text, but since their manager wanted an AI-Workflow...
LOL. Attackers will run these agents but the thousands of maintainers will be so dumb to sit idly and get hammered with exploits. I wonder what the ratio of attackers to maintainers must be, 1:1000 is a fair assessment i take it.
Also LLMs will be used to attack only, no one will be smart to integrate it into CI flows, because everyone is that dumb. No security tools will pop up.
Let that be the end of Microsoft. Was forced to use their shitty products for years, by corporate inertia and their free Teams and Azure licenses, first-dose-is-free, curse.
this like is saying email marketing is done better if you hand write every email. Thats true, but the hit rate is so low, that you are better off generating 1 million hyper personalized emails and firing them off into the ether
As someone who did the former for a couple years, “better off” is subjective and dependent on your business model, particularly for B2B. It’s a trade off like anything else. You may get more leads, but they may convert at a lower rate. Sending at that scale also increases your risk of email deliverability problems. Trashing your domain has more impacts than you’d think. In smaller, targeted markets it even can damage your business reputation and hurt future sales if done poorly; word gets around.
I disagree. Many humans are phishing in a different language than their native tongue, and LLMs are way better at sounding legit/professional than many of them. The best spear-phishing will still be humans, but AI definitely raises the bar.
I've been part of a response team on a security incident and I really feel for them. However, this initial communication is terrible.
Something happened, we won't say what, but it was severe enough to notify law enforcement. What floors me is the only actionable advice is to "review environment variables". What should a customer even do with that advice? Make sure the variable are still there? How would you know if any of them were exposed or leaked?
The advice should be to IMMEDIATELY rotate all passwords, access tokens, and any sensitive information shared with Vercel. And then begin to audit access logs, customer data, etc, for unusual activity.
The only reason to dramatically overpay for the hosting resources they provide is because you expect them to expertly manage security and stability.
I know there is a huge fog of uncertainly in the early stages of an incident, but it spooks me how intentionally vague they seem to be here about what happened and who has been impacted.
Via the incident page:
> Environment variables marked as "sensitive" in Vercel are stored in a manner that prevents them from being read, and we currently do not have evidence that those values were accessed. However, if any of your environment variables contain secrets (API keys, tokens, database credentials, signing keys) that were not marked as sensitive, those values should be treated as potentially exposed and rotated as a priority.
https://vercel.com/kb/bulletin/vercel-april-2026-security-in... as of 4:22p ET
The “sensitive” toggle is off by default. I’m curious about the rationale, what's the benefit of this default for users and/or Vercel?
https://vercel.com/docs/environment-variables/sensitive-envi...
Sensitive environment variables are environment variables whose values are non-readable once created.
So they are harder to introspect and review once set.
It’s probably good practice to put non-secret-material in non-sensitive variables.
(Pure speculation, I’ve never used Vercel)
I have used Vercel though prefer other hosts.
There are cases where I want env variables to be considered non-secure and fine to be read later, I have one in a current project that defines the email address used as the From address for automated emails for example.
In my opinion the lack of security should be opt-in rather than opt-out though. Meaning it should be considered secure by default with an option to make it readable.
Simpler for vibe coders.
How does the app read the variable if it can't be read after you input it? Or do they mean you can't view it after providing the variable value to the UI?
Seriously. Why am I reading about this here and not via an email? I've been a paying customer for over a year now. My online news aggregator informs me before the actual company itself does?
Says they emailed affected customers...
Please remember that this is the same company that couldn't figure out how to authorize 3rd party middleware and had, with what should be a company ending, critical vulnerability .
Oh and the owner likes to proudly remind people about his work on Google AMP, a product that has done major damage to the open web.
This is who they are: a bunch of incompetent engineers that play with pension funds + gulf money.
I just deleted my account. Their laid-back notice just is not worth it anymore. I will hold them accountable using my cash. You can get out with me. Let their apologies hit your spam filter. They need to be better prepared to react to the storm of insanity that comes with a breach or they lose my info (lose it twice, I guess..)
Last year Vercel bungled the security response to a vulnerability in Next's middleware. This is nothing new.
https://news.ycombinator.com/item?id=43448723
https://xcancel.com/javasquip/status/1903480443158298994
Security is hard and there are only three vendors I trust: AWS, Google and IBM ( yes IBM ). Anything else is just asking for trouble.
Oracle too
Oracle? Oracle?
The Oracle that published an announcement that said "we didn't get hacked" when the hackers had private customer info?
The Oracle that does not allow you to do any security testing on their software unless you use one of their approved vendors?
The Oracle that one of my customers uses where they have to turn off the HR portal for 2 weeks before annual performance evaluations because there is no way to prevent people from seeing things?
The only reason Oracle isn't having nightmarish security problems published every other week is because they threaten to sue anyone that does find an issue.
Oracle is a joke in every conceivable way and I despise them on a personal level.
I love a good cathartic rant
Having worked both public and private, I can agree with this.
Google in particular has been staggeringly good, and don't sleep on IBM when they Actually Care.
> The only reason to dramatically overpay for the hosting resources they provide is because you expect them to expertly manage security and stability.
This and because it's so convenient to click some buttons and have your application running. I've stopped being lazy, though. Moved everything from Render to linode. I was paying render $50+/month. Now I'm paying $3-5.
I would never use one of those hosting providers again.
If you're only paying $3-5 on Linode then your level of usage would probably be comfortably at $0 on Vercel.
It could be $0 on Render too, but then there's going to be a 3 minute load time for a landing page to become visible, lol. So if you don't want your server to sleep, you're going to have to pay $20/month.
Does Vercel do the same?
No, I run several small websites on Vercel for free for years, always served static pages very quickly
Makes sense considering the quality of Vercel's security response and customer communication.
Looking at linode, those prices get you an instance with 1Gb of ram and a mediocre CPU. So you are running all of your applications on that?
Personal projects/MVPs/small projects? Absolutely. For what I'm running, there's no reason to need anything beyond that.
The point is, I used to just throw everything up on a PaaS. Heroku/Render, etc. and pay way more than I needed to, even if I had 0 users, lol.
For $3.5, Hetzner gives 2 vCPU, 4GB RAM, 40 GB SSD, and 10 TB of bandwidth.
how much work should the GP do to migrate if Linode is good enough, to potentially save up to $1.50/month (or spend 50 cents more)?
exactly people paid the premium so somebody else's OAuth screwup wouldn't become their Sunday. and here we are.
Yeah, given there insane pricing I think the expectations can be higher. Although I know it is impossible to provide 100% secure system, but if something like that happens, then the communication should at least be better. Don’t wait until you have talked to the lawyers... inform your customers first, ideally without this cooperate BS speak, most vercel customers are probably developers, so they understand that incidents like this can happen, just be transparent about it
Completely agreed. At minimum they should be advising secret rotation.
The only possibility for that not being a reasonable starting point is if they think the malicious actors still have access and will just exfiltrate rotated secrets as well. Otherwise this is deflection in an attempt to salvage credibility.
Welcome to the show.
While a different kind of incident (in hindsight), the other week Webflow had a serious operational incident.
Sites across the globe going down (no clue if all or just a part of them). They posted plenty of messages, I think for about 12 hours, but mostly with the same content/message: "working on fixing this with an upstream provider" (paraphrased). No meaningful info about what was the actual problem or impact.
Only the next day did somebody write about what happened. Essentially a database running out of storage space. How that became a single point of failure, to at least plenty of customers: no clue. Sounds like bad architecture to me though. But what personally rubbed me the wrong way most of all, was the insistence on their "dashboard" having indicated anything wrong with their database deployment, as it allegedly had misrepresented the used/allocated storage. I don't who this upstream service provider of Webflow is, but I know plenty about server maintenance.
Either that upstream provider didn't provide a crucial metric (on-disk storage use) on their "dashboard", or Webflow was throwing this provider under the bus for what may have been their own ignorant/incompetent database server management. I guess it all depends to which extend this database was a managed service or something Webflow had more direct control over. Either way, with any clue about the provider or service missing from their post-mortem, customers can only guess as to who was to blame for the outage.
I have a feeling that we probably aren't the only customer they lost over this. Which in our case would probably not have happened, if they had communicated things in a different way. For context: I personally would never need nor recommend something like Webflow, but I do understand why it might be the right fit for people in a different position. That is, as long as it doesn't break down like it did. I still can't quite wrap my head around that apparent single point of failure for a company the size of Webflow though.
/anecdote
They just added more details:
> Indicators of compromise (IOCs)
> Our investigation has revealed that the incident originated from a third-party AI tool whose Google Workspace OAuth app was the subject of a broader compromise, potentially affecting hundreds of its users across many organizations.
> We are publishing the following IOC to support the wider community in the investigation and vetting of potential malicious activity in their environments. We recommend that Google Workspace Administrators and Google Account owners check for usage of this app immediately.
> OAuth App: 110671459871-30f1spbu0hptbs60cb4vsmv79i7bbvqj.apps.googleusercontent.com
https://vercel.com/kb/bulletin/vercel-april-2026-security-in...
I wonder which tool that is
I'm on a macbook pro, Google Chrome 147.0.7727.56.
Clicking the Vercel logo at the top left of the page hard crashes my Chrome app. Like, immediate crash.
What an interesting bug.
Huh, curiously; I'm on Arch Linux, crash happens in Google Chrome (147.0.7727.101) for me too, but not in Firefox (149.0.2) nor even in Chromium (147.0.7727.101).
I find it fun we're all reading a story how Vercel likely is compromised somehow, and managed to reproduce a crash on their webpage, so now we all give it a try. Surely could never backfire :)
Works in Safari too. Sounds like a Google Chrome thing.
Following since I just reproduced the crash on my own system (Chrome on Ubuntu)
Sadly I coudn't make Chrome crash here. Would be fun.
Chrome Version 147.0.7727.101 (Official Build) (64-bit). Windows 11 Pro.
Video: https://imgur.com/a/pq6P4si
I use uBlock Origin Lite. Maybe it blocks some crash causing script? edit: still no crash when I disabled UBO.
Reminds me of circa 2021 Chromium bug where opening the dropdown menu on GitHub would crash the entire system on Linux. At some point, it got fixed.
Same with Chrome on Windows 11. I opened the vercel home page using the url once after which it stopped crashing when clicking on the logo.
MBP - M4 Max - Chrome 146.0.7680.178.
No crash.
Now I don't want to click that "Finish update" button.
if it does so happen that the crash originates from a browser exploit, you should expect to be more at risk due to the absence of a crash on an older version, not less
I'm running 147.0.7727.57 and this doesn't happen. Macbook Air M5. VERY interesting.
Same hard crash on Chrome Windows 11
Do you have a chrome://crashes/ entry ?
it did add an entry - windows 11, chrome
Related: https://news.ycombinator.com/item?id=47824426
https://x.com/theo/status/2045862972342313374
> I have reason to believe this is credible.
https://x.com/theo/status/2045870216555499636
> Env vars marked as sensitive are safe. Ones NOT marked as sensitive should be rolled out of precaution
https://x.com/theo/status/2045871215705747965
> Everything I know about this hack suggests it could happen to any host
https://x.com/DiffeKey/status/2045813085408051670
> Vercel has reportedly been breached by ShinyHunters.
> Ones NOT marked as sensitive should be rolled out of precaution
if it's not marked as sensitive (because it is not sensitive) there is no reason to roll them. if you must roll a insensitive env var it should've been sensitive in the first place, no?
There's a difference between sensitive, private and public. If public (i.e. NEXT_PUBLIC_) then yeah likely not a reason to roll. Private keys that aren't explicitly sensitive probably are still sensitive. It doesn't seem to be the default to have things "sensitive" and I can't tell if that's a new classification or has always been there.
I can imagine the reason why an env variable would be sensitive, but need to be re-read at some point. But overwhelmingly it makes sense for the default to be set, and never access again (i.e. Fly env values, GCP secret manager etc)
Who is this “theo” person and why are multiple people quoting him? He seems to have little to say that’s substantive at this point.
He’s a tech influencer, probably getting quoted here because he has the biggest reach of people covering this so far.
He’s a streamer who talks about tech. Previously had a sponsorship relationship with Vercel so is theoretically more well connected than average on the topic. He’s also very divisive because he does a lot of ragebait, grievance reporting, and contrarian takes but famously has blind spots for a few companies and technologies that he’s favored in past videos or been sponsored by. I have friends who watch a lot of his videos but I’ve never been able to get into it.
Theo Browne is a reasonably well known YouTuber & YC founder.
https://t3.gg/
He is a paid Vercel shill (literally, he does sponsored content for them on his YouTube channel)
Not in a few years.
He literally doesn't. https://x.com/theo/status/1832228209573949947
YT tech vlogger
Incidents like this are a good reminder of how concentrated our single points of failure have become in the modern web ecosystem. I appreciate the transparency in their disclosure so far, but it definitely makes you re-evaluate the risk profile of leaning entirely on fully managed PaaS solutions.
Is this one of those situations where _a lot_ of customers are affected and the “subset” are just the bigger ones they can’t afford to lose?
Conjecture, but the wording "limited subset" rarely turns out to be good news. Usually a provider will say "less than 1% of our users" or some specific number when they can to ease concerns. My guess is they don't have the visibility or they don't like the number.
I feel for the team; security incidents suck. I know they are working hard, I hope they start to communicate more openly and transparently.
“Less than 1% of our users” means 10k affected users if you have 1 million users. 10k victims is a lot! Imagine “air travel is safe, only a subset of 1% of travellers die”
The lack of details makes me wonder how large this "subset" of users really is
I remember working support and being told "always say 'subset' unless you absolutely know it's exactly 100% of customers" lol
The lack of details itself is telling enough. Whatever comes out will be no doubt PR sanitised and some bigger clumps of truth won't make it through the PR process.
This announcement in its current form is quite useless and not actionable. As least people won’t be able to say “why didn’t you say something sooner?” They said _something_
Use VPS, nowadays with the help of AI it's a lot easier to set everything up, you don't need Versel at all. And of course way cheaper
So, the Vercel post says a number of customers were impacted, but not everyone, and they will contact the people that were impacted. I wasn't contacted so does that mean I'm safe?
I don't get why everything is not marked as sensitive in env vars by default instead.
We run on Vercel and I wonder if / how long before we're alerted about a leak. Quick look online suggests environment variables marked as sensitive are ok, but to which extent I wonder.
> incident response provider
So they use third-party for incident management? They are de-risking by spending more, which is a loose-loose for the customers.
The point I am taking away here is to never use Vercel's environment variables to store secrets.
Time to ipo
https://news.ycombinator.com/item?id=45416353
Oy vey: https://x.com/theo/status/2045862972342313374?s=46
He doesn't work at Vercel but he is the type to never pass up any opportunity to chase clout.
Almost like that’s his job.
Hey, I’m with you - I think social media needs to die specifically for this reason. I’m reminded of the term “snake oil” - it’s like the dawn of newspapers again.
Media as a whole needs to die
Including books and the internet?
Reminder the Vercel CEO is a genocide supporter, if you need more reasons to move away from it.
You forgot the source to backup your claim
https://x.com/rauchg/status/1972669025525158031
https://x.com/theo/status/2045871215705747965 - "Everything I know about this hack suggests it could happen to any host"
He also suggests in another post that Linear and GitHub could also be pwned?
Either way, hugops to all the SRE/DevOps out there, seems like it's going to be a busy Sunday for many.
I don't know if I'd trust some random programmer-streamer-influencer on anything other than the topic of streamer-influencing.
The link at the top of the page it to vercel acknowledging it...
Vercel acknowledges a security incident, which nobody is claiming doesn't exist. What they don't acknowledge are this person's vague implications about impact elsewhere.
Based on what, "feels like it"? Claiming that Cloudflare is affected by the same hack has to come from somewhere, but where is that coming from?
from his "sources".
> Here’s what I’ve managed to get from my sources:
>3. The method of compromise was likely used to hit multiple companies other than Vercel.
https://x.com/theo/status/2045870216555499636
To be fair journalists often do this too, eg. "[company] was breached, people within the company claim"
Isn’t he a Vercel evangelist though?
He quite publicly is not anymore.
He is "whatever gives me short-term boost in popularity". Including doing 180 turns on whatever he's evangelizing or bashing.
Fair enough. That’s probably a better description from what I’ve seen from him. I remember that arc browser shilling.
Good for the content but would sponsors be on board long term?
Let's see. Roasting vercel is more popular than defending but his posts so far he seems to be defending and arguing in the replies.
Ah, Theo with his vast insights and connections into everything. That man gets around, and his content is worth it's cost.
Theo's content boils down to the same boring formula. 1. Whatever buzzword headline is trending at the time 2. Immediate sponsored ad that is supposed to make you sympathize with Theo cause he "vets" his sponsors. 3. The man makes you listen to a "that totally happened" story that he somehow always involved himself personally. 4. Man serves you up an ad for his t3.chat and how it's the greatest thing in the world and how he should be paid more for his infinite wisdom. 5. A rag on Claude or OpenAI (whichever is leading at the time) 6. 5-10 minutes of paraphrasing an article without critical thought or analysis on the video topic.
I used to enjoy his content when he was still in his Ping era, but it's clear hes drunken the YT marketer kool-aid. I've moved on, his content gets recommend now and again, but I can't entertain his non-sense anymore.
I just wanted to chime in and say I think he is knowledgeable; he's not a con. I know you didn't say that, but people might have the impression he doesn't know what he's talking about. He does know, and I've learned quite a lot from him in the past.
However, since the LLM Cambria explosion, he has become very clickbaity, and his content has become shallow. I don't watch his videos anymore.
Not that I ever had confidence in his technical knowledge, but it went to zero when he confidently asserted that there was no possible way a single server could handle the massive traffic some NextJS app he had made was serving. He then posted the bill - which was about $5K IIRC - and I was able to determine from the billed runtime and memory that a modestly-spec’d RPi could in fact handle it.
> he's not a con.
When you're putting the bar that low, sure.
He's about as knowledgeable as the junior you hired last week, except that he speaks from a position of authority and gets retweeted by the entire JS slop sphere. He's LinkedIn slop for Gen Z.
I don't watch his content, but I felt comfortable posting his link as I believe he's generally considered a reputable guy? His tweets sometimes come up in my for you tab and he seems reasonable and knowledgable generally? Maybe I'm wrong and shouldn't have linked to him as a source.
He's kind of like an LLM in that his content has the surface texture of something substantial, and sometimes it's backed by substance, yet it's often half-true or totally off the mark too. You'll notice if you're previously acquainted with what he's talking about, otherwise he seems to be as you described.
I don't think he's a bad guy or that he's trying to be misleading. I suspect he wants his content to actually carry value, but he produces too much for that to be possible. Primarily he's a performer, not a technologist.
I agree with this comment. YouTube's summarize this video feature has been a godsend when it comes to Theo's videos.
Nothing on x.com is reputable at this point.
”Any host” of what? That’s such a non-descriptive statement and clearly not true at face value.
I do remember that OpenAI did use Vercel a year ago. They might have likely moved off of it to something better.
OpenAI owns Contexts.ai, doesn't it?
> @theo: "I have reason to believe this is credible. If you are using Vercel, it’s a good idea to roll your secrets and env vars."
> @ErdalToprak: "And use your own vps or k3s cluster there’s no reason in 2026 to delegate your infra to a middle man except if you’re at AWS level needs"
> @theo: "This is still a stupid take"
lol, okay. Thanks for the insight, Theo, whoever you are.
What is AWS level needs?
Hell doing this with fixed price AWS Lightsale based services would be better.
You'll have to ask @ErdalToprak on Twitter on that one. I just thought it was funny that this slopfluencer, who's taken money to advertise Vercel, ostensibly believes that using a VPS/k3s is "a stupid take."
Hmmm, the dashboard 404 I got 6 hours ago now makes a bit more sense..
Looks like their rampant vibe coding is starting to catch up to them. Expect to see many pre vulns like this in the future.
This is why you pay a real provider for serious business needs, not an AWS reseller. Next.js is a fundamentally insecure framework, as server components are an anti-pattern full of magic leading to stuff like the below. Given their standards for framework security, it's not hard to believe their business' control plane is just as insecure (and probably built using the same insecure framework).
Next.js is the new PHP, but worse, since unlike PHP you don't really know what's server side and what's client side anymore. It's all just commingled and handled magically.
https://aws.amazon.com/security/security-bulletins/rss/aws-2...
> Next.js is the new PHP, but worse, since unlike PHP you don't really know what's server side and what's client side anymore. It's all just commingled and handled magically.
Wasn't unheard of back in the day, that you leaked things via PHP templates, like serializing and adding the whole user object including private details in a Twig template or whatever, it just happened the other way around kind of. This was before "fat frontend, thin backend" was the prevalent architecture, many built their "frontends" from templates with just sprinkles of JavaScript back then.
People say "Next.js is the new PHP" because it's the most popular and prominent tooling out there, and so by sheer number of available targets it's the one that comes up the most when things go wrong like this.
But there are more people trying to secure this framework and the underlying tools than there would be on some obscure framework or something the average company built themselves.
Also "pay a real provider", what does that mean? Are you again implying that the average company should be responsible for _more_ of their own security in their hosting stack, not less?
Most companies have _zero_ security engineers.. Using a vertically-integrated hosting company like Vercel (or other similar companies, perhaps with different tech stacks - this opinion has nothing to do with Next or Node) is very likely their best and most secure option based on what they are able to invest in that area.
Next.js is the polar opposite of PHP, in a way.
PHP was so simple and easy to understand that anyone with a text editor and some cheap shared hosting could pick it up, but also low level enough that almost nothing was magically done for you. The result was many inexperienced developers making really basic mistakes while implementing essential features that we now take for granted.
Frameworks like Next.js take the complete opposite approach, they are insanely complex but hide that complexity behind layers and layers of magic, actively discouraging developers from looking behind the curtain, and the result is that even experienced developers end up shooting themselves in the foot by using the magical incantations wrong.
Totally agree. Nextjs is a vehicle to sell their PaaS, every other feature is a coincidence.
What’s worse is vercel corrupted the react devs and convinced them that RSC was a good idea. It’s not like react was strictly in good hands at Facebook but at least the team there were good shepherds and trying to foster the ecosystem.
Why does anyone running a third party tool have access to all of their clients’ accounts? I can’t imagine something this stupid happening with a real service provider.
I see Vercel is hosted on AWS? Are they hosting every one on a single AWS account with no tenant isolating? Something this dumb could never happen on a real AWS account. Yes I know the internal controls that AWS has (former employee).
Anyone who is hosting a real business on Vercel should have known better.
I have used v0 to build a few admin sites. But I downloaded the artifacts, put in a Docker container and hosted everything in Lambda myself where I controlled the tenant isolation via separate AWS accounts, secrets in Secret Manager and tightly scoped IAM roles, etc.
Is AWS security boundary the AWS account? Are you expecting Vercel to provision and manage an AWS account per user? That doesn’t make any sense man, though makes sense if you’re a former AWS employee.
Yes the security boundary is the AWS account.
It doesn’t make sense for a random employee who mistakenly uses a third party app to compromise all of its users it’s a poor security architecture.
It’s about as insecure as having one Apache Server serving multiple customer’s accounts. No one who is concerned about security should ever use Vercel.
what's the cause of the breach?
There is no serious reason to use Vercel, other than for those being locked into the NextJs ecosystem and demo projects.
I recently got hit by a car on my bike. While I was starting the claim filing process the web portal for ICBC (British Columbia insurance) was acting a little funky / stalling / and then gave me a weird access error. Down at the bottom of the error page was a little grey underlined link that said “vercel”.
I’m not exactly surprised, but it seems like the unserious, ill-informed and lazy are taking over. There is absolutely zero reason why a large, essential public service should be overspending and running on an unnecessary managed service like vercel… yet, here we are.
Much as I want to rip on vercel, its clear that ai is going to lead to mass security breaches. The attack surface is so large, and ai agents are working around the clock. This is a new normal. Open source software is going to change, companies wont be running random repos off github anymore
Your entire recent posting history is "software engineering is over, AI has won."
What's your agenda here?
The guy has like 10 thousand comments boosting AI and 600 karma, whatever his agenda is people aren’t buying it.
Paid by a Sama minion, I bet.
how many recent security breaches have we seen?
Most of recent issues, including this incident, happened not due to smart superintelligent "agents" taking over the world - chatbots and other text generators are about as intelligent amd powerful as a dead starfish - but due to the combined stupidity of the said chatbots amd lazy idiots who use them to hide their own incompetence and thus produce such embarassing mistakes. A few years ago, they would be fired for exposing secrets in plain text, but since their manager wanted an AI-Workflow...
How many can unequivocally be attributed to malicious AI?
LOL. Attackers will run these agents but the thousands of maintainers will be so dumb to sit idly and get hammered with exploits. I wonder what the ratio of attackers to maintainers must be, 1:1000 is a fair assessment i take it.
Also LLMs will be used to attack only, no one will be smart to integrate it into CI flows, because everyone is that dumb. No security tools will pop up.
Slop coding and makeshift sites being thrown up with abandon at breakneck speeds is going to buy me a lot of minivans.
>> ai is going to lead to mass security breaches.
Let that be the end of Microsoft. Was forced to use their shitty products for years, by corporate inertia and their free Teams and Azure licenses, first-dose-is-free, curse.
ShinyHunters are a phishing group. What does this have to do with AI agents?
Run ai agents around the clock to do hyper targeted fishing
I feel like humans would be better at hyper targeting.
AI agents have the benefit of working at scale, probably "better" used for mass targeting.
this like is saying email marketing is done better if you hand write every email. Thats true, but the hit rate is so low, that you are better off generating 1 million hyper personalized emails and firing them off into the ether
As someone who did the former for a couple years, “better off” is subjective and dependent on your business model, particularly for B2B. It’s a trade off like anything else. You may get more leads, but they may convert at a lower rate. Sending at that scale also increases your risk of email deliverability problems. Trashing your domain has more impacts than you’d think. In smaller, targeted markets it even can damage your business reputation and hurt future sales if done poorly; word gets around.
If you’re targeting a million people, I wouldn’t consider that a hyper targeted attack.
But I get your point.
I disagree. Many humans are phishing in a different language than their native tongue, and LLMs are way better at sounding legit/professional than many of them. The best spear-phishing will still be humans, but AI definitely raises the bar.