It's an especially awkward situation because Railway is a competitor of Google Cloud, with many third parties involved. So, I just think it will take them a little more time to figure out how to message things.
To me, what it sounds like is that a Google Cloud system identified Railway as a misbehaving customer. Spam, hackers, that sort of thing. Often this happens for "platform as a service" companies, because Railway themselves probably do host some spammers and hackers, and they have their own systems for dealing with it.
So, it's quite possible that according to the Google team, Railway violated the terms of something or other, and according to the Railway team, they did not, and now everyone has to argue about it.
But who knows, this is just me guessing based on some experience running a PaaS that itself was running on top of AWS.
Railway's system for dealing with hackers is a $5/mo fee gate.
Railway plays around vibecoding as they go along, and their tech practices don't inspire confidence either. Unlike other PaaS like Render or Heroku, I doubt Railway has adequate rate-limiting to stop bad apples.
Yeah, and rate-limiting is only one of the things a PaaS needs to handle, to avoid looking like a bad actor to the underlying platform. The trickiest thing to handle is people using your PaaS to host malware, because:
1. There may be no simple rule of thumb like "suddenly using tons of bandwidth"
2. Bad actors can open up so many accounts, you have to close them automatically
3. Malware can infect a good actor, who is unaware or struggling to deal with it
I think so. I'm a GCP user and I'm afraid of hosting workloads there now. I've heard too many nightmare stories, and I thought Google would be proper and thus not be infested with these kind of problems that cheaper providers are known for.
Maybe AWS is the only player in town now? I don't know. Google doesn't instill confidence with these incidents, same with those cases of insurmountable bills caused by simple mistakes where there should be a way for smaller customers to cap usage.
> I thought Google would be proper and thus not be infested with these kind of problems that cheaper providers are known for.
These sorts of things have happened before with Google and the other expensive providers.
Are cheaper providers known for doing this? I would have thought they would be less lively to, as they are smaller and therefore every customer is relatively more important to them, and they are therefore more likely to check before turning services off.
> Are cheaper providers known for doing this? I would have thought they would be less lively to, as they are smaller and therefore every customer is relatively more important to them, and they are therefore more likely to check before turning services off.
In my experience, the reason they're cheaper is because they offer fewer features (cut down on ongoing expenses) and because they aggressively enforce TOS (the margins are thinner, so you're less able to afford people using more resources than allocated).
The very cheapest are on thin margins, so that seems to be as expected, but there are a lot of providers between the very cheapest and the high prices charged by the hyperscalers. Pricing somewhere in between and offering some actual customer service?
if it's some automated behavior, yes. I'd like to know what it is, why it exists, how it works (how to avoid triggering it) and what they're doing to make sure it never happens again
Your statement was one of those that made me chuckle because it started as a joke...and then reality set in the more i thought about it, and then it made me feel sad, nervous, etc.
I really don't see how they can. The business and usage details of their clients are confidential. We have their word that ToS where violated, I don't really think they should say more. This needs to go to arbitration.
There's a reason we used to have courts with public records. Public records and transparency are a good thing. Binding arbitration has destroyed all of that.
There is something fundmentally undemocratic and unlawful about contracts that basically preclude the ability of the parties to resolve disputes in open courts.
Arbitration is an inefficient and unproductive process. I suppose that may be what the parties chose, but the public will likely never know what happened and the problem will be allowed to reoccur again and again. Things like these are better resolved in courts, with res judicata, precedent, and visibility so that legislators can fix statutes where necessary.
If the ToS was actually violated and the suspension was legitimate, they wouldn't have reinstated Railway's account after five hours or whatever it was. Their behaviour is already a mea culpa.
Given that Railway publicly boasted about running their own infra, revealing that they were completely and totally dependent on GCP is massively damaging to their PR, far more damaging than admitting they had a problem in their own infra. Their customers are using them under the pretense of not using GCP, so if they're just a proxy for GCP, they have no reason to exist. I seriously doubt they concocted such a self-damaging lie.
GCP is basically just a toy. If you're hosting something important there you should probably not do that. It works as a Plan B, but by no means should Google be your Plan A.
I don't know what happened in this case, there's a chance it wasn't Google's fault but it doesn't matter, Google already lost all benefit of the doubt long ago.
Yeah this is the main argument I have for AWS. Is it the best tech-wise? Arguably not, but they have real support you can contact on a moment's notice, they'll even reach out ahead of time if they suspect something is seriously wrong.
Shopify has $11 billion dollars in revenue. I assume that buys them at least an email address of a human at Google.
Nobody doubts Google's engineering ability. It's their soft skills that are lacking. Google is culturally unable to understand the concept of customers.
Xoogler here. This hits hard; right on the kidney.
That is a huge problem at Google: "we're google; we know what you want and need better than you do." That and an unshakable confidence in their own infallibility.
When I was there, I remember saying, "we do some cool stuff, but always remember that hubris is the death of empires."
I don't even mind about that. GCP is a perfectly good platform technically, it's ahead of AWS in some areas e.g. for k8s it's widely regarded to be the best.
The product is great. However I have GCP blacklisted because it's basically a vending machine. If your drink gets stuck in a vending machine all you can do is hit it hopelessly.
If I have a problem on AWS I'm fairly sure I can talk to a human and resolve it. As GP mentioned AWS support is pro-active and will even reach out to you of their own initiative. This gives me a warm fuzzy feeling.
Amazon understands that cloud services are a trust-based business. Their customers trust them with the power of life or death over their business.
yes, especially since this didn't seem to be exactly "private" where it was anything specific, it was just some kind of automated system without a human in the loop
if it wasn't something specific to their setup, it should be disclosed publicly, because this is a catastrophic incident that makes you think it could happen to you as well, and there's no way to know what could trigger it
They're a web host; it could be any number of plausible mundane things that triggered automated action. This is a big recurring problem for any shared hosting provider.
yes and yes, inaccessible where they had to be recovered..
there's like 0% chance there was domething super damaging going on where they couldn't get anybody on the phone yet within 10 mins or so were able to get the restoration process going with their account managers
I dont see what could be going on where an automated process would have to step in except for something like suddenly provisioning infinite amounts of resources, but quota limits should hit first so...
If it were google's bug as Railway has certainly at least insinuated, then yes. I would also be fine with them saying "blah blah blah abuse was detected; we're working through it with our customer and we apologize to those impacted."
I'd also expect a story around how it is this happened w/o a human spending at least an hour working his/her way through a call list to reach someone at Railway. Starting with ops and escalating to the ceo if necessary.
This is actually scary. If Google can suspend a company like Railway without warning, what chance does a smaller startup have? The lack of any human escalation path at Google Cloud has been a known problem for years. You'd think enterprise customers paying real money would at least get a phone call before getting shut down
Then you should be scared about every single online account you use, since they all have this same capability of suspension of your account. That's inherent in any service policy.
Google seems to be the only one that actually does it, regularly, with no warning or comment, to other people. Google also spreads bans to accounts it deems related, so your personal Google account is likely to be deleted at the same time as your GCP account. That includes your Gmail address and your phone login and cloud storage.
And vice versa. Company accounts have been deleted because an employee's personal account tripped Google's random number generator. Again, only Google does this.
Railway can only share their side of the story,while Google stays silent due to policy constraints.The assymmetry makes it hard to evaluate what actually happened,even if both parties acted correctly.
Being an advocate for GCP all these years, I can only say the earlier you get out of it, the safer it is for your business. All it takes is for their automated system to go haywire, and you can say bye bye to all your goodwill and customers. Go look at twitter how many customers are blaming railway. Founder had history getting screwed by GCP, yet still choose to depend on them.
You can't rely your business on GCP. Honestly, this is the most silly way to kill your own business.
For context, copied from my post 3 years ago.
March 10, 2023 | hide | past | favorite
As a 4 years customer, our production severs have been suspended by Google Cloud because we didn't fill up some information on-time. Contacted support but they expect us to wait for 24-48 hours to get it resolved while all our servers are down.
Anyone linked with someone powerful in google cloud can help?
======
- Running production on google cloud for 4 yrs with my startup. 100% legit SaaS business.
- Always pay bills on-time no issue. Good customer never open tickets, ask for help or what just quietly pay my bills each month.
- Our servers was abruptly suspended yesterday midnight and my whole business is now down for > 10hrs.
- We run a SaaS business that other ecommerce stores rely on and have hundreds of paying merchants.
- My customers have been grilling me and I don't feel gcloud's trust and safety team understand/care how urgent the issue is.
======
Why were our servers suspended? Because we didn't fill up information in time?
- Apparently they dropped us an email 10 days back that I missed out
- Titled "Important Information Regarding Your Google Account" with no indication of suspension or what in title.
- Given the number of subprocessor "Important" emails they send it's too easy to miss out the email.
- 10 days gone by and our servers were abruptly shutdown with zero suspension notification or what.
- We've been paying $400-$700/mo for the past 4 yrs consistently and they shut us down because we didn't fill up some information?
When I tried to ask them to at least temporarily get our servers back while the verification is ongoing, I didn't get any answers.
Google Cloud have zero empathy for customers.
It's not like my account got suspended for fradulent issue or what. It's suspended because I didn't fill up some information on-time and they don't even allow me to temporarily reactivate my services or what. Especially when I had to wait for hours to get their team to verify my details before I can get my servers back.
You can't trust them with your business. Don't run any production stuffs with Google Cloud, ever.
For privacy, B2B providers often won't even acknowledge that any given company has an account, let alone publish information about that account's standing.
> Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action. This action extended to many accounts within Google Cloud. As this was a platform-wide action, there was no proactive outreach to individual customers prior to the restriction.
I'm interpreting that bit from Railway's blog to mean it wasn't just them that was impacted.
Totally agreed the news like this makes me nervous too. For a high profile customer it feels like Google should give a clear explanation of what happened.
I don't suspect anyone here is demanding laws be written requiring every single player in any SaaS outage to make a public statement immediately following an event like this. That's an odd thing to state your preference on.
But, given that this incident unjustly caused real damages to another company, I am pretty certain that Google will be required to make some sort of response to this, and if it ends up in the courts, it will be public.
It's entirely plausible Google won't tell Railway without an NDA to prevent them disclosing exactly what set it all off.
The bigger point though is Google really need to flag any business account as not subject to these suspensions until checked into by several humans. Back when I had a team that used a lot of App Engine they would even call us when we caused all their pagers to go off, and then conspire to keep the lights on while things got fixed. It's sad they have ended up like this.
> It's entirely plausible Google won't tell Railway without an NDA to prevent them disclosing exactly what set it all off.
That case is called:
> or Railway can tell us that Google is refusing to tell them.
I'm not paying you (which let's face it, that's what an NDA is) just to find out why you messed up about as severely as one can imagine. In theory, there was a contract here: $ for cloud services, and the rug got pulled. One should get a very clear, and very apologetic explanation as to why, with no strings attached, or one should be voting with their wallet.
Now, whether Railway will do any of that, who knows.
"We take full responsibility for the architectural decisions that allowed a single upstream provider action to cascade into a platform-wide outage, and detail below what happened, how we recovered, and the changes we are making to prevent this from happening again."
My guess is they will be switching away from Google Cloud. Because anything else would be nuts.
I know hating GCP is hip, but why do only a minority entertains the possibility that Railway did really something off to trigger some alarms?
The calibration of the alarms might be off, and that's acceptable, but in the end if something can be held wrong, somebody will hold it wrong.
I did the same thing in the past, albeit in a much smaller scale. There's no shame in being wrong and admitting as long as it results in progress, so this stance of "we do nothing wrong" from both parties is getting a bit old now.
Of course it doesn’t work this way but why shouldn’t it? It doesn’t because Google is a massive company and could kill dozens of Railways before they notice an issue to their bottom line. However in a world where companies care I’d expect them to make a statement.
I took this a different way which was that to google railway is their customer and out of a variety of professional and security considerations want the communications to come from their customer and not them.
I think there’s been far too many Google/GCP ‘suspensions with no human in the loop’ that Google does need to put out a statement about their practices.
The company I work for uses GCP and we preciously had intermittent CloudSQL connection errors for a few hours. We reached out and they resolved it after a day or so and said there was a minor incident but I don’t think it was ever publically reported.
Yup, I think so. Makes one think about how dependent we are on cloud infra for core pieces versus supporting pieces of the architecture. They've probably negotiated some kind of private settlement.
I really think Google underestimates the damage they've done to their reputation with these. These incidents are rare, but they're common enough where you can't trust them reliably anymore, and if that's the case, why would you pick them over other vendors?
it's not like they're the only provider, or even the #1
I’m surprised when I hear about anyone depending on GCP.
Why would you do that?
Seriously, what are some good reasons to choose GCP over vendors that have demonstrated to be much more reliable? Are they much cheaper than the alternatives? Do they have unbeatable features?
This makes me so mad, Google is not the first i've seen doing this (AWS, Upwork, Twilio) - and the cheek to say precisely nothing about why. If you're a buyer of cloud services right, i bet old-school on-prem is looking as tempting as an actress to a teenager rn.
Every regulated firm running on GCP is going to spend Monday explaining to their board how their resilience plan accounts for a hyperscaler that operates this opaquely. The compliance paperwork is the easy bit - the honest answer is we trusted that a hyperscaler would behave like a utility and they didn't.
So yes - they owe a statement. The whole point of paying hyperscaler prices is the assumption you won't wake up suspended with no explanation.
Google has given a public statement about this category of incident (to wit: cloud provider imperils customer's operations by way of automated decision deliberately designed to withhold recourse).
We already know the explanation. It's fundamental to their business model and continued existence to automate everything, false positives be damned, and they don't care about all the people who roll snake eyes on a given day. Because everyone just stays and keeps using them.
Yep. Google doing Google things. Not everyone stays and just keeps using them though, we're actively planning to remove GCP from our primary workloads now and will cut our spend to about 1/10 of current as we keep them in the stack as a cold multi-cloud failover target only.
I don't understand how you see what Google did to Railway and think you know what I'll build my business on GCP.
#1. you will most likely never be as big a customer as railway. if they did it them, you're fucked.
#2. if shit hits the fan, even a company as large as railway can get no human being on the phone. that's insane.
how is gcp still a thing after this event? if you're in charge of tech at your company, how do you stomach choosing this? how do you defend your choice to your CEO?
Related ongoing thread:
Incident Report: May 19, 2026 – GCP Account Suspension - https://news.ycombinator.com/item?id=48204770 - May 2026 (144 comments)
Previously:
Incident Report: Railway Blocked by Google Cloud [resolved] - https://news.ycombinator.com/item?id=48201484 - May 2026 (344 comments)
It's an especially awkward situation because Railway is a competitor of Google Cloud, with many third parties involved. So, I just think it will take them a little more time to figure out how to message things.
To me, what it sounds like is that a Google Cloud system identified Railway as a misbehaving customer. Spam, hackers, that sort of thing. Often this happens for "platform as a service" companies, because Railway themselves probably do host some spammers and hackers, and they have their own systems for dealing with it.
So, it's quite possible that according to the Google team, Railway violated the terms of something or other, and according to the Railway team, they did not, and now everyone has to argue about it.
But who knows, this is just me guessing based on some experience running a PaaS that itself was running on top of AWS.
If they're a PaaS, isn't it insane for them to run their own infrastructure and the customer workloads under the same cloud account?
Railway's system for dealing with hackers is a $5/mo fee gate.
Railway plays around vibecoding as they go along, and their tech practices don't inspire confidence either. Unlike other PaaS like Render or Heroku, I doubt Railway has adequate rate-limiting to stop bad apples.
Yeah, and rate-limiting is only one of the things a PaaS needs to handle, to avoid looking like a bad actor to the underlying platform. The trickiest thing to handle is people using your PaaS to host malware, because:
1. There may be no simple rule of thumb like "suddenly using tons of bandwidth"
2. Bad actors can open up so many accounts, you have to close them automatically
3. Malware can infect a good actor, who is unaware or struggling to deal with it
I think so. I'm a GCP user and I'm afraid of hosting workloads there now. I've heard too many nightmare stories, and I thought Google would be proper and thus not be infested with these kind of problems that cheaper providers are known for.
Maybe AWS is the only player in town now? I don't know. Google doesn't instill confidence with these incidents, same with those cases of insurmountable bills caused by simple mistakes where there should be a way for smaller customers to cap usage.
> I thought Google would be proper and thus not be infested with these kind of problems that cheaper providers are known for.
These sorts of things have happened before with Google and the other expensive providers.
Are cheaper providers known for doing this? I would have thought they would be less lively to, as they are smaller and therefore every customer is relatively more important to them, and they are therefore more likely to check before turning services off.
> Are cheaper providers known for doing this? I would have thought they would be less lively to, as they are smaller and therefore every customer is relatively more important to them, and they are therefore more likely to check before turning services off.
In my experience, the reason they're cheaper is because they offer fewer features (cut down on ongoing expenses) and because they aggressively enforce TOS (the margins are thinner, so you're less able to afford people using more resources than allocated).
The very cheapest are on thin margins, so that seems to be as expected, but there are a lot of providers between the very cheapest and the high prices charged by the hyperscalers. Pricing somewhere in between and offering some actual customer service?
Would you be okay with GCP making public issues you encounter with your account?
if it's some automated behavior, yes. I'd like to know what it is, why it exists, how it works (how to avoid triggering it) and what they're doing to make sure it never happens again
Well, I would host workloads on GCP... provided I could easily move them elsewhere and I just treat them as disposable.
What about Azure?
Azure is a toy platform, not a real cloud. Too many changes, far from being reliable.
Unless imposed by employer to use, run a mile.
I'd rather go back to bare metal than use Azure.
Do you care about uptime and security?
https://damrnelson.github.io/github-historical-uptime/
https://isolveproblems.substack.com/p/how-microsoft-vaporize...
HN discussion: https://news.ycombinator.com/item?id=47616242
I would rather raise goats, than use Azure.
Honestly the best counterattack Iran could make rn would be somehow convincing the Pentagon to move everything to Azure
Your statement was one of those that made me chuckle because it started as a joke...and then reality set in the more i thought about it, and then it made me feel sad, nervous, etc.
I really don't see how they can. The business and usage details of their clients are confidential. We have their word that ToS where violated, I don't really think they should say more. This needs to go to arbitration.
Is there a statement somewhere from Google saying the ToS was violated? I hadn't seen that and can't find one right now.
There's a reason we used to have courts with public records. Public records and transparency are a good thing. Binding arbitration has destroyed all of that.
Yes, court would be better, and totally justified here, but that also seems extremely unlikely given the contracts.
There is something fundmentally undemocratic and unlawful about contracts that basically preclude the ability of the parties to resolve disputes in open courts.
Arbitration is an inefficient and unproductive process. I suppose that may be what the parties chose, but the public will likely never know what happened and the problem will be allowed to reoccur again and again. Things like these are better resolved in courts, with res judicata, precedent, and visibility so that legislators can fix statutes where necessary.
> and the problem will be allowed to reoccur again and again
You're already placing blame here.
We don't know the details other than what Railway has said.
I think that's true regardless of what happened.
If Railway did something wrong, then letting that be known may help other customers avoid the same ~mistake.
If the ToS was actually violated and the suspension was legitimate, they wouldn't have reinstated Railway's account after five hours or whatever it was. Their behaviour is already a mea culpa.
I disagree. They could have done this to themselves and instead of fessing up, are placing the blame on an entity as a scapegoat.
Is that possible? Absolutely.
Has Google also done similar things before? Absolutely.
I certainly wouldn't bet my house on either possiblity.
Given that Railway publicly boasted about running their own infra, revealing that they were completely and totally dependent on GCP is massively damaging to their PR, far more damaging than admitting they had a problem in their own infra. Their customers are using them under the pretense of not using GCP, so if they're just a proxy for GCP, they have no reason to exist. I seriously doubt they concocted such a self-damaging lie.
GCP is basically just a toy. If you're hosting something important there you should probably not do that. It works as a Plan B, but by no means should Google be your Plan A.
I don't know what happened in this case, there's a chance it wasn't Google's fault but it doesn't matter, Google already lost all benefit of the doubt long ago.
Yeah this is the main argument I have for AWS. Is it the best tech-wise? Arguably not, but they have real support you can contact on a moment's notice, they'll even reach out ahead of time if they suspect something is seriously wrong.
Shopify is all on GCP and works well.
Shopify has $11 billion dollars in revenue. I assume that buys them at least an email address of a human at Google.
Nobody doubts Google's engineering ability. It's their soft skills that are lacking. Google is culturally unable to understand the concept of customers.
Xoogler here. This hits hard; right on the kidney.
That is a huge problem at Google: "we're google; we know what you want and need better than you do." That and an unshakable confidence in their own infallibility.
When I was there, I remember saying, "we do some cool stuff, but always remember that hubris is the death of empires."
I don't even mind about that. GCP is a perfectly good platform technically, it's ahead of AWS in some areas e.g. for k8s it's widely regarded to be the best.
The product is great. However I have GCP blacklisted because it's basically a vending machine. If your drink gets stuck in a vending machine all you can do is hit it hopelessly.
If I have a problem on AWS I'm fairly sure I can talk to a human and resolve it. As GP mentioned AWS support is pro-active and will even reach out to you of their own initiative. This gives me a warm fuzzy feeling.
Amazon understands that cloud services are a trust-based business. Their customers trust them with the power of life or death over their business.
Tbh, even if "you know what somebody wants better than they do" it's a moot point since people pay for what they _think_ they want.
Would you want your vendors publicly disclosing potentially private reasons for an outage?
With consent, yes.
Is there any indication Railway has consented to such disclosure?
But Railway has been blaming GCP for the outage. Shouldn't GCP be given an opportunity to defend itself?
Railway hasn't placed any blame that I've seen. They've posted a timeline of what happened, without any speculation on causes.
Is that what you'd want your vendors to do?
No, that’s what the money is for.
yes, especially since this didn't seem to be exactly "private" where it was anything specific, it was just some kind of automated system without a human in the loop
But shouldn't that be disclosed to Railway, and not the public? If they had someone running a botnet on compromised accounts there, for example.
If Railway isn't satisfied with the explanation, they're able to say so publicly, yes?
if it wasn't something specific to their setup, it should be disclosed publicly, because this is a catastrophic incident that makes you think it could happen to you as well, and there's no way to know what could trigger it
> if it wasn't something specific to their setup
They're a web host; it could be any number of plausible mundane things that triggered automated action. This is a big recurring problem for any shared hosting provider.
a huge account like theirs should not be subject to automated actions like that.
an entire gcp project deleted along with its persistent disks.
how does that make any sense? nobody thought to call them or anything
> a huge account like theirs should not be subject to automated actions like that.
No matter how damaging the behavior?
> an entire gcp project deleted along with its persistent disks.
Railway doesn't say that - "persistent disks inaccessible", followed by "persistent disks restored to ready state". It was a suspension, not a wipe.
yes and yes, inaccessible where they had to be recovered..
there's like 0% chance there was domething super damaging going on where they couldn't get anybody on the phone yet within 10 mins or so were able to get the restoration process going with their account managers
I dont see what could be going on where an automated process would have to step in except for something like suddenly provisioning infinite amounts of resources, but quota limits should hit first so...
If it were google's bug as Railway has certainly at least insinuated, then yes. I would also be fine with them saying "blah blah blah abuse was detected; we're working through it with our customer and we apologize to those impacted."
I'd also expect a story around how it is this happened w/o a human spending at least an hour working his/her way through a call list to reach someone at Railway. Starting with ops and escalating to the ceo if necessary.
This is actually scary. If Google can suspend a company like Railway without warning, what chance does a smaller startup have? The lack of any human escalation path at Google Cloud has been a known problem for years. You'd think enterprise customers paying real money would at least get a phone call before getting shut down
Then you should be scared about every single online account you use, since they all have this same capability of suspension of your account. That's inherent in any service policy.
Google seems to be the only one that actually does it, regularly, with no warning or comment, to other people. Google also spreads bans to accounts it deems related, so your personal Google account is likely to be deleted at the same time as your GCP account. That includes your Gmail address and your phone login and cloud storage.
And vice versa. Company accounts have been deleted because an employee's personal account tripped Google's random number generator. Again, only Google does this.
Does Google have any customer big enough for it to give a fuck about?
Google really should publish a flow diagram for how they decide to turn off someone's business.
It would just be a control flow graph of rand(3)
Railway can only share their side of the story,while Google stays silent due to policy constraints.The assymmetry makes it hard to evaluate what actually happened,even if both parties acted correctly.
No warning, no email, nothing. Just you find out your business is down the same way your users do. Google needs to explain this.
Being an advocate for GCP all these years, I can only say the earlier you get out of it, the safer it is for your business. All it takes is for their automated system to go haywire, and you can say bye bye to all your goodwill and customers. Go look at twitter how many customers are blaming railway. Founder had history getting screwed by GCP, yet still choose to depend on them.
You can't rely your business on GCP. Honestly, this is the most silly way to kill your own business.
For context, copied from my post 3 years ago.
March 10, 2023 | hide | past | favorite As a 4 years customer, our production severs have been suspended by Google Cloud because we didn't fill up some information on-time. Contacted support but they expect us to wait for 24-48 hours to get it resolved while all our servers are down. Anyone linked with someone powerful in google cloud can help?
======
- Running production on google cloud for 4 yrs with my startup. 100% legit SaaS business.
- Always pay bills on-time no issue. Good customer never open tickets, ask for help or what just quietly pay my bills each month.
- Our servers was abruptly suspended yesterday midnight and my whole business is now down for > 10hrs.
- We run a SaaS business that other ecommerce stores rely on and have hundreds of paying merchants.
- My customers have been grilling me and I don't feel gcloud's trust and safety team understand/care how urgent the issue is.
======
Why were our servers suspended? Because we didn't fill up information in time?
- See https://imgur.com/a/x0Y3RJl
- Apparently they dropped us an email 10 days back that I missed out
- Titled "Important Information Regarding Your Google Account" with no indication of suspension or what in title.
- Given the number of subprocessor "Important" emails they send it's too easy to miss out the email.
- 10 days gone by and our servers were abruptly shutdown with zero suspension notification or what.
- We've been paying $400-$700/mo for the past 4 yrs consistently and they shut us down because we didn't fill up some information?
When I tried to ask them to at least temporarily get our servers back while the verification is ongoing, I didn't get any answers.
Google Cloud have zero empathy for customers.
It's not like my account got suspended for fradulent issue or what. It's suspended because I didn't fill up some information on-time and they don't even allow me to temporarily reactivate my services or what. Especially when I had to wait for hours to get their team to verify my details before I can get my servers back.
You can't trust them with your business. Don't run any production stuffs with Google Cloud, ever.
For privacy, B2B providers often won't even acknowledge that any given company has an account, let alone publish information about that account's standing.
> Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action. This action extended to many accounts within Google Cloud. As this was a platform-wide action, there was no proactive outreach to individual customers prior to the restriction.
I'm interpreting that bit from Railway's blog to mean it wasn't just them that was impacted.
Totally agreed the news like this makes me nervous too. For a high profile customer it feels like Google should give a clear explanation of what happened.
> For a high profile customer
Are they really "high profile"?
I've never heard of them until this incident or maybe the other one with the database.
They just seem... Loud, publicly. And always about failure.
In this sort of community, fairly well-known yes.
https://hn.algolia.com/?dateEnd=1779148800&dateRange=custom&...
They publish some stats at https://railway.com/stats
72M deploys from 3M customers across 10M services in the last month.
It depends on what you mean by ‘need’. If you mean they should for PR purposes, I probably agree.
If you are saying they should be required to by law, then no I disagree.
I don't suspect anyone here is demanding laws be written requiring every single player in any SaaS outage to make a public statement immediately following an event like this. That's an odd thing to state your preference on.
But, given that this incident unjustly caused real damages to another company, I am pretty certain that Google will be required to make some sort of response to this, and if it ends up in the courts, it will be public.
I can't believe that readers of HN actually think that that is how it does or should work.
Google/GCP can only make very general statements and in this case we want more than that.
They need to tell Railway and Railway needs to tell us, or Railway can tell us that Google is refusing to tell them.
Either way, we need to hear about this from Railway.
It's entirely plausible Google won't tell Railway without an NDA to prevent them disclosing exactly what set it all off.
The bigger point though is Google really need to flag any business account as not subject to these suspensions until checked into by several humans. Back when I had a team that used a lot of App Engine they would even call us when we caused all their pagers to go off, and then conspire to keep the lights on while things got fixed. It's sad they have ended up like this.
> It's entirely plausible Google won't tell Railway without an NDA to prevent them disclosing exactly what set it all off.
That case is called:
> or Railway can tell us that Google is refusing to tell them.
I'm not paying you (which let's face it, that's what an NDA is) just to find out why you messed up about as severely as one can imagine. In theory, there was a contract here: $ for cloud services, and the rug got pulled. One should get a very clear, and very apologetic explanation as to why, with no strings attached, or one should be voting with their wallet.
Now, whether Railway will do any of that, who knows.
They did:
"We take full responsibility for the architectural decisions that allowed a single upstream provider action to cascade into a platform-wide outage, and detail below what happened, how we recovered, and the changes we are making to prevent this from happening again."
My guess is they will be switching away from Google Cloud. Because anything else would be nuts.
I know hating GCP is hip, but why do only a minority entertains the possibility that Railway did really something off to trigger some alarms?
The calibration of the alarms might be off, and that's acceptable, but in the end if something can be held wrong, somebody will hold it wrong.
I did the same thing in the past, albeit in a much smaller scale. There's no shame in being wrong and admitting as long as it results in progress, so this stance of "we do nothing wrong" from both parties is getting a bit old now.
Of course it doesn’t work this way but why shouldn’t it? It doesn’t because Google is a massive company and could kill dozens of Railways before they notice an issue to their bottom line. However in a world where companies care I’d expect them to make a statement.
I took this a different way which was that to google railway is their customer and out of a variety of professional and security considerations want the communications to come from their customer and not them.
I think there’s been far too many Google/GCP ‘suspensions with no human in the loop’ that Google does need to put out a statement about their practices.
The only way is for PR to think they should open this
Google fucked up. They should convince people that it won't happen again.
> Railway's GCP account manager engaged directly
This would have been an amazing conversation to witness.
The company I work for uses GCP and we preciously had intermittent CloudSQL connection errors for a few hours. We reached out and they resolved it after a day or so and said there was a minor incident but I don’t think it was ever publically reported.
Yup, I think so. Makes one think about how dependent we are on cloud infra for core pieces versus supporting pieces of the architecture. They've probably negotiated some kind of private settlement.
I've directly asked our account manager about it. It's pretty scary that we don't know what automated mechanisms could just cut us off.
I really think Google underestimates the damage they've done to their reputation with these. These incidents are rare, but they're common enough where you can't trust them reliably anymore, and if that's the case, why would you pick them over other vendors?
it's not like they're the only provider, or even the #1
> why would you pick them over other vendors?
Their security track record is pretty exceptional compared to the other two.
Maybe if it was a US Gov site going dark, and Congress got involved. Maybe then. Maybe not.
I’m surprised when I hear about anyone depending on GCP.
Why would you do that?
Seriously, what are some good reasons to choose GCP over vendors that have demonstrated to be much more reliable? Are they much cheaper than the alternatives? Do they have unbeatable features?
They offer $250k credit for startups, that's the only reason I can think of. You use them for a start, and stuck there.
This is exactly why people get nervous about platform risk
This makes me so mad, Google is not the first i've seen doing this (AWS, Upwork, Twilio) - and the cheek to say precisely nothing about why. If you're a buyer of cloud services right, i bet old-school on-prem is looking as tempting as an actress to a teenager rn.
Every regulated firm running on GCP is going to spend Monday explaining to their board how their resilience plan accounts for a hyperscaler that operates this opaquely. The compliance paperwork is the easy bit - the honest answer is we trusted that a hyperscaler would behave like a utility and they didn't.
So yes - they owe a statement. The whole point of paying hyperscaler prices is the assumption you won't wake up suspended with no explanation.
Google has given a public statement about this category of incident (to wit: cloud provider imperils customer's operations by way of automated decision deliberately designed to withhold recourse).
That statement is the last 15 or so years.
> Since this is a relatively high profile customer standards
This is a small unknown startup.
Should they give a statement? Maybe. Should they be required to? Not if they don't want to. Should they be expected to? Historically, nope.
Railway can simply move to other service. We all know Google in unreliable, so why should google give public statement?
Thanks.
We already know the explanation. It's fundamental to their business model and continued existence to automate everything, false positives be damned, and they don't care about all the people who roll snake eyes on a given day. Because everyone just stays and keeps using them.
Yep. Google doing Google things. Not everyone stays and just keeps using them though, we're actively planning to remove GCP from our primary workloads now and will cut our spend to about 1/10 of current as we keep them in the stack as a cold multi-cloud failover target only.
I don't understand how you see what Google did to Railway and think you know what I'll build my business on GCP.
#1. you will most likely never be as big a customer as railway. if they did it them, you're fucked.
#2. if shit hits the fan, even a company as large as railway can get no human being on the phone. that's insane.
how is gcp still a thing after this event? if you're in charge of tech at your company, how do you stomach choosing this? how do you defend your choice to your CEO?
Google is a lawnmower, just like Larry Ellison. You don't expect the lawnmower to give a public statement about why it chopped your hand off.
Post your ask in the discussion: https://news.ycombinator.com/item?id=48204770
https://news.ycombinator.com/item?id=48201484
Stop calling Google Cloud, “Google”.
Completely different group of people, culture and incentives.
Google Cloud is another company under Alphabet.
They brand it Google, trade it under the ticker GOOG, and are run by the same CEO.