Identified
Google Cloud has blocked our account, making some Railway services unavailable. We have escalated this directly with Google. The Railway Platform team has since confirmed access to Google Cloud and is working on restoring access to all workloads. We have access to some of our Google Cloud–hosted infrastructure and are working to restore the rest of the service. We apologize for the disruption.
I believe the problem was always the process that Google has where they tend to want to automate everything from the start and make it very difficult to reach a human and explain the situation. The service itself seems solid but if you make it difficult to address any problems when(not if) they occur then I won't be comfortable doing business with you. I stay away from their services for anything serious for this reason and always recommend to others to do the same.
It's going to be interesting to watch this unfold. Google's automation vs LLM agents, no humans from either side.
You should publish some of this, as if this is true and AWS and Azure are banning large organisations without realistic recourse on the same scale it should destroy the cloud service providers.
Usually they don't ban large organizations without some prior consultations. SMEs are victims usually. AWS/Azure and sorts know there will be no public recourse so they can cut without any notice. It's same with banks - if you are big enough they won't suspend you, but if you are SME they can suspend in a moment without any legitimate reason. They can suspend and say their "risk assessment team is looking at it" which is BS excuse knowing how banks work. At same time your contractors couldn't be paid and you can't receive money yourself.
I don't have an answer on how to solve that besides putting in law that any service provider vital for company operations (bank, telco, etc) shouldn't suspend or limit service rendering without court order. This is rather unrealistic and won't hit legislation because of same big organization lobby.
Railway had a similar reliability issue two weeks ago when an AI agent deleted a customer's production database via their API — no confirmation step, no environment scoping. Now this. Both incidents suggest the same pattern: infrastructure decisions made without thinking through failure modes, fixed reactively after damage is done.
Agree. The author of that article took 0 responsbility and despite the warnings of "Hey, AI with power in prod is a bad idea" thought "This wouldn't happen to me!" and then when it does "HOW COULD IT DO THIS?!"
Identified Google Cloud has blocked our account, making some Railway services unavailable. We have escalated this directly with Google. The Railway Platform team has since confirmed access to Google Cloud and is working on restoring access to all workloads. We have access to some of our Google Cloud–hosted infrastructure and are working to restore the rest of the service. We apologize for the disruption.
So the solution is the same as it has been for over a decade. Don’t do business with Google.
I believe the problem was always the process that Google has where they tend to want to automate everything from the start and make it very difficult to reach a human and explain the situation. The service itself seems solid but if you make it difficult to address any problems when(not if) they occur then I won't be comfortable doing business with you. I stay away from their services for anything serious for this reason and always recommend to others to do the same.
It's going to be interesting to watch this unfold. Google's automation vs LLM agents, no humans from either side.
> Don’t do business with Google.
If you think other big tech or bank won't do it for you if you "don't do anything wrong" - you might be delusional.
The only way to fight it is redundancy. Dont vendor lock in all you stuff in aws/google/cf/whatever. They can (and will) fuck you up at some point.
It is a weekly occurrence for large named organisations and Google.
It isn't for other big tech/banks. It's just a risk assessment, and Google are a high risk partner.
Just because you don't see it, it doesn't meant it doesn't happen.
I deal with it daily. If i would start writing, I would spend half day just writing.
It's in no way to say Google is any good - it's just equally bad as any reasonably big organisation. No more and no less.
You should publish some of this, as if this is true and AWS and Azure are banning large organisations without realistic recourse on the same scale it should destroy the cloud service providers.
There is no point in publishing this.
Usually they don't ban large organizations without some prior consultations. SMEs are victims usually. AWS/Azure and sorts know there will be no public recourse so they can cut without any notice. It's same with banks - if you are big enough they won't suspend you, but if you are SME they can suspend in a moment without any legitimate reason. They can suspend and say their "risk assessment team is looking at it" which is BS excuse knowing how banks work. At same time your contractors couldn't be paid and you can't receive money yourself.
I don't have an answer on how to solve that besides putting in law that any service provider vital for company operations (bank, telco, etc) shouldn't suspend or limit service rendering without court order. This is rather unrealistic and won't hit legislation because of same big organization lobby.
Some discussion started an hour ago (12 points, 3 comments) https://news.ycombinator.com/item?id=48200827
We'll rapidly deploy a reliable backup website with proper hosting to get Railway back online within 48 hours
The fix isn't don't use GCP. It's never letting one cloud account suspension take down your control plane and customer workloads simultaneously
Exactly. Whatever happened to cloud just being "someone else's computer" and "cheap scaling on speed dial"?
Reply at hello@fourfivestudio.in
Railway had a similar reliability issue two weeks ago when an AI agent deleted a customer's production database via their API — no confirmation step, no environment scoping. Now this. Both incidents suggest the same pattern: infrastructure decisions made without thinking through failure modes, fixed reactively after damage is done.
I wouldn't blame that incident on Railway.. you can delete your prod database on AWS just as easily with their API.
That incident wasn't Railways fault at all. Don't use AI in your staging and prod tools.
Agree. The author of that article took 0 responsbility and despite the warnings of "Hey, AI with power in prod is a bad idea" thought "This wouldn't happen to me!" and then when it does "HOW COULD IT DO THIS?!"
If everyone can delete your prod database via API or by any other means - you need to sack CTO without severance package.