I recently moved all my projects to a self-hosted forgejo instance and have found it quite satisfactory so far. And it's fast! If you're in the market for a github alternative, take a look - there are options.
It’s not fashionable anymore, but I feel that Phabricator deserves an honorable mention as a self-hostable GH alternative too. Actually its “dated” UI is kind of a plus considering how bad everything is now.
I always balked at GitHub, but was impressed with git very soon after I was first introduced; I migrated from an old Gitea instance to Forgejo for my personal projects and have been very happy with it.
This is a real business continuity issue for us. We’re kinda stuck with GitHub Enterprise but we may need to move from cloud to on-premises if this keeps up.
From a user perspective this makes sense. But if you’re MSFT or GitHub this number is pretty embarrassing.
They would love if everyone on the platform used all of the features and had massive lock-in right? So if some part of that is always broken, it’s not a confidence booster for users to adopt more of the feature set.
Sure the more things you use the more likely it is that one has an issue but clearly stability isn’t a goal for these type of companies anymore.
On the other hand when you have a reasonably complex deployment it's easy to get swamped with dashboards showing CPU, Memory, I/O, application-metrics, signups, active users/sessions, etc.
Instead it's nice to think about how you can express the state of a complete system as a single number. It might be you divide active user sessions by database-connections, and then scale by memory capacity.
But as a single digit you can then get used to normal ranges, and have it always visible somewhere obvious. A single number won't show details, but when it changes you can go look at the specific metrics. It's a cute shorthand, and it can work well as a basic "are we normal" check.
It’s obviously a meme website, the meme is more funny when the number isn’t high. Anyone looking for actual accurate info would go to the real status page.
Ironically I’ve never found official status pages to be all that accurate either since companies love to exclude all kinds of outages from counting towards uptime. Anthropic is hilariously egregious about that as a recent example I can think of, but I assume GitHub does the same since it’s so common in the industry.
splitting the status page like they do, to the point where it is only a bit of humourous exaggeration to say that they track broken `git push` and `git pull` separately, is a sleight of hand / accounting / SLA-fudging that we should not excuse
there is a subset of the site that pretty much everyone uses — git, issues, pull requests, actions — and if any part of that is broken then the site is broken and the status page should indicate how often this happens
> splitting the status page like they do, to the point where it is only a bit of humourous exaggeration to say that they track broken `git push` and `git pull` separately, is a sleight of hand / accounting / SLA-fudging that we should not excuse
This is a pretty ungenerous take. You could look at it the other way: if I don't use actions then it's useful for me to know that only actions are broken, and I can continue in my normal usage. If you bundle everything up then the status page is reporting an unhelpful false positive for me.
If S3, EC2, EKS and RDB alone had a similar uptime as all of Github right now, we'd all know.
No one cares that much if repo wikis, commit stats or gist had these issues. It's the combination of inter-dependent services that are used in combination, like PRs, actions, discussions, etc.
If one were to build a single percentage for each of these components of both systems, github would still lose. Maybe it's a few days without outages more but this isn't a comparison.
I think the correct middle ground is a site that lets you select the parts of the platform you rely on and ignore the others. For example, GitHub is "down" for me when I can't push, process PRs, or release packages, but I don't care about Actions or AI features.
You’re kind of an outlier - nobody wants AI but Actions are core for tons of workflows and deployment pipelines. Everyone bought into the “only robots can deploy” mantra (correctly IMO, it’s a huge time and friction saver) only to be bit in the ass by the platform being so u reliable they can be stuck for days without deploys.
I'm currently setting up a self-hosted "Knot" for use on tangled.org.
Mainly doing it because I think AtProto is cool and self-hosting is fun, but also because owning the infrastructure that hosts my projects is definitely the direction I want to move in.
Tangled's Knot system feels like a really strong abstraction for this. I host the data in an AtProto Repository, but can rely on a third party to host/manage the AtProto Application that presents it to the rest of the world. If Tangled goes under, I can happily take my AtProto login to a different platform and point it at my Knot without changing a thing about my hosting setup.
Much more convenient that hosting an entire, siloed webapp on my own corner of the internet.
Lots of apologia for Github here. Aside from the fact that defending a billion-dollar company is a bit strange; especially one that is steward to the the overwhelming majority of open-source software.
Maybe that's good-will doing the work? For me it's always been a sour pill to swallow that I have to buy in to a large companies internal politics and practices in order to work on projects I love. I don't feel like I owe them anything.
Especially if they can't hold up their end of the deal.
Unfettered access to the world's software repositories, for the princely sum of a bucketload of Azure credits.
Let me ask the question in reverse: what do you have against them such that the fellow human beings struggling to maintain their operations don’t deserve even a modicum of kindness, respect, and good will? Are you unable to separate the business from the hard working people behind it?
It’s not like they don’t know that people like us are counting on them: they recognize that their service is the “dial tone” for much of the world’s software development capability. They are keenly aware of the impact.
What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?
If I to hire a contractor to redo my roof, and that roof leaks, whether they worked hard or not is immaterial. They did not do the task in they were paid to do. I'm not going to buy their services again just because their shingles guy was particularly charming.
MS has talented engineers, but that's a complete misdirection. Github is a service in decline: there is nothing wrong with criticizing them.
A corporation is not a person. If your organization cannot handle the load, then you need to adjust your practices. The organization needs to prioritize their paying users. The organization needs to shift people from new features to keeping the lights on. And maybe the organization needs to find another strategy to manage its azure transition.
A corporation is made of people. GitHub cannot exist but for the people who continue to work for it. And they’ve already said, multiple times, that restoring availability is their top priority.
I think it's possible to be simultaneously: gracious and supportive towards the developers and ops staff who have been struggling to maintain reasonable uptime on the extremely important piece of shared internet infrastructure that everyone commenting probably relies on (either directly or indirectly) on a daily basis; and spiteful and cruel towards the massive (and, historically speaking, ethically fraught) corporation whose cynical acquisition and subsequent mismanagement of that same resource got us here in the first place.
>What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?
Would you feel the same way about a colleague who kept causing downtime in your product again and again, seemingly without making any progress in addressing whatever issue was causing their repeated mistakes?
There are web applications out there that are far more complex than GitHub but have much less downtime. It's not like they're facing an unsolvable problem.
You don’t know that it was “their mistake.” Unless you’ve personally successfully scaled a suite of nontrivial services equivalent to GitHub’s to accommodate an unexpected 14x increase in traffic, you respectfully have no basis for such an assertion.
You could argue the scales are different, but computers are also faster now.
So, argument to credentialism out of the way... What should we do as consumers if a provider that is a defacto monopoly due to network effects stops functioning?
> You could argue the scales are different, but computers are also faster now.
Scale is everything and a faster computer doesn’t always help. Vertical scaling has limits, and complex distributed systems are complex.
Since you seem to possess a diagnosis and remedy with a reasonable amount of certainty, I’m sure they’d love to hear from you and have you fix all their problems for them. Especially if you can do it while not making the problem worse in any dimension.
The link in my previous comment answers the credentials question in detail- including specific technical post-mortems on horizontally scaled stateful systems. Vertical scaling wasn't the topic.
You’re missing the point: a doctor doesn’t diagnose and practice medicine on a patient he hasn’t thoroughly evaluated himself. This is the sort of wisdom that a staff engineer and CTO is expected to have earned.
I skimmed your profile. Working on the infrastructure for a couple mid-tier video games is a cool accomplishment, but equating this to having solved GitHub level scale rings hollow.
GitHub has a couple orders of magnitude more daily active visitors than the games you worked on had at their peak.
You can make valid criticisms of GitHub without trying to reduce their scale or inflate your credentials to create a false equivalence.
"false equivalence" needs an equivalence claim to be false.
I didn't make one. The sentence after "I have" was literally "you could argue the scales are different."
GitHub spent a decade asking the world to host its code with them. They got what they asked for. You don't get to beg everyone to run services for you for ten years and then have "scaling is hard" be the answer. They should be improving, not regressing over time, and they have some of the worlds best engineers and a trillion dollar corporation behind them, they don't need my sympathy.
The original question is still open and nobody's engaging with it.
I'm not sure that resorting to personal attacks against the parent commenter for making a legitimate critique is the right, fair, sensible, or mature approach here.
Discarding legitimate criticism based on some self-determined criteria of intellectual superiority isn't a good look. It smacks of elitism and isn't something conducive to a productive and positive community discussion.
It is unhelpful, rude, condescending, and completely fails to address the underlying problem.
The commenter inserted his own personal bona fides (as a proxy for skill, experience, and knowledge) and use them to bolster his conclusion of culpability and incompetence of the GitHub team. If you take that risk, you should expect to be challenged if those skills are not up to par.
Put more simply: if you get into the ring, you’d better be prepared to take a punch.
Yeah, they should be testing for that, right? I think there's a lot of people reading comments like yours and thinking, is this person a paid shill or what?
The earn bucket loads of money, they should be planning for exactly that. And testing for it via load testing every day.
Perhaps you've forgotten the days of GitHub presenting themselves of software engineering thought leaders.
I’ve worked at some very well-endowed organizations. Having money is no guarantee of a particular outcome. There is a lot of money chasing a limited supply of talent. Moreover, distributed systems that were built long ago with certain assumptions can’t be refactored as quickly as the HN populace might believe. The Mythical Man-Month is a popular book for a reason.
#hugops is to your coworkers, not to the nameless big-corps who can't maintain a service for paying customers. You should be raising a shitstorm when things you pay for aren't reliable or unusable.
Hot take, if it's traffic is causing issues, throttle your free-tier, pause signups, or stop giving out free things (like runner time).
There are two options, either they are lousy at their jobs, or they are incapable of pushing back against unrealistic demands. Neither is a good indicator of their skill and talent as engineers.
I know I am speaking from a position of some privilege, but I have previously left workplaces that did not allow me to practice good engineering, and I do expect others to do so.
In a SWE job market like this, do you really want to be seen as the "conscientious objector"?
There are literally thousands of people who are ready to ride up the totem pole, it would not be a difficult decision for a bad manager to swing his axe and replace the new head
Talented engineers shouldn’t have much problem finding another position even in this market (of course they should find one before leaving I’m not discounting family responsibilities and whatnot), so if your argument is they’re not able to leave and find another job then you’re essentially agreeing with the person you’re replying to.
Or, they've been given crap primitives to work with. There's only so much lipstick you can put on a pig. I don't know what database they're using or what their pub sub and streaming looks like, or even what their system diagram actually looks like. But, well, you don't see Google having these kinds of problems. Other ones, sure, but between Chubby and Spanner, if Google had bought GitHub we wouldn't be having these problems.
But it wasn't a pig. It was a reliable system, and then it increasingly became an unreliable one, in a way that is not explainable by the mere increase in demand. Whatever rearchitecture was performed, it was done and is apparently being perpetuated by software engineers who should be held accountable. Not necessarily guilty, or even directly at fault, but accountable nevertheless. "I am just an employee of a bad company" is not a valid excuse for an engineer.
If you pay someone full price to do a job, they know they can't fulfil the terms up front, accept the work, deliver less than the agreed upon terms and still charge you full price, you'd probably call that transaction fraudulent.
GitHub is promising service they know they cannot meet, not telling you that, and still charging you full price. What's more, one can argue quite convincingly that they're lying about their level of delivered service by not reflecting the actual level of uptime on their status page.
To give benefit of the doubt requires that the other party is not blatantly and overtly acting in bad faith. When they are, you're just apologizing for fraudulent behavior.
Fraud is a serious civil and criminal accusation that’s not to be taken or given lightly. Can you detail the fraud that’s being committed? What is the specific promise they made that you’re being deprived of? Remember, the four corners of your agreement with them are controlling.
I think it depends if you pay them money. If you do, then you should indeed have strong expectations towards them and hold them accountable. If they provide a free service to you, then it's still reasonable to feel upset, but at the same time you get what you pay for.
Does this logic still applies if the company is getting other benefits from having me as a user? (Genuine question, I can see arguments for both sides)
For example, if I am using the free tier of a service and "paying" by seeing ads, should I have similar expectations?
I'm not saying that's how users pay for github - in that case it's more subtle, for example by giving up control of some of their stack and bolstering github already near monopolistic network effect.
I'm surprised at how little the perception of GitHub changed post-acquisition. Coupled with WSL, it almost balanced things for a lot of people and put Microsoft back in the "benefit of the doubt" column. This is undoing a lot of that, on top of the operational costs. Suddenly the bad press is more noticeable and harder to ignore.
I think its the fact that people have used the software for so long that they feel emotional to it (Hashimoto crying tears of sadness when he decided to move ghostty away from github) and there is completely nothing wrong about it as we are emotional human beings.
But, you are right in the sense that, Github has failed to accept its part of the deal which is actually to just be a usable place. People HAVE previously tolerated so much AI slop and slowness in github's UI just because of its reliability but this downtime is like the Github's achilles heel.
At some point, I recommend people to accept this and move to more healthier alternatives, there is also an momentum. For example, the only reason I joined github was that I wanted to join codeberg but so many of projects used github and involved sign in with github that I finally gave in into github and I had thought that codeberg is so good but nobody is gonna come here because of the network effects but the tide is turning and I hope more people look into codeberg and healthier alternatives.
> Aside from the fact that defending a billion-dollar company is a bit strange
More than a bit strange. This is an HNism that I'll never get. Why would you go to the comment section anywhere to passionately try to defend the honor of a trillion dollar company, unless 1. you're being paid to astroturf or 2. you own that company's stock? Satya Nadella isn't going to read a post here and say, "Gosh, how nice of that commenter! I'm going to send him some Microsoft stock as a show of appreciation for him defending us online!" I don't think I'll ever understand company-fanboys.
1. Telling that you think the only possible motivations are financial (getting paid, stockholder, or foolish expectations of a gift from Satya).
2. Maybe you know a bunch of people who work there, could be ex-colleagues etc. and you think overall it’s mostly good well-intentioned people there. Therefore you want to see them succeed, and also you might disbelieve that the company is deliberately being awful.
I don’t have any specifically warm feelings about a corporate legal entity, but I know people who work at various companies and partly for that reason I am not rooting for those companies to fail and I also don’t believe the least charitable explanations for all their failings.
My free, open-source, bare-bones, caching-free, dependency-free, authentication- and authorization-free pure PHP raw Git viewer. I developed it because GitList blew out my shared host's drive space and memory (due to a caching bug) and to consolidate my GitHub, BitBucket, and GitLab repos. There's something rewarding about self-hosting and not being beholden to the whims of third parties.
A vibe coded app that most likely contributed to the onslaught of vibe coded apps that are causing Github to go down. I feel bad for the people working at Github who are basically trying to keep a sinking ship afloat and Microsoft doing everything they can to sink their own ship.
I'm pretty sure we all took down a production enterprise system once or twice. At InVision we had an incident every week, despite all the SOPs and safety nets. And that way waaay before vibe coding..
14x is insane, especially since the quality and quantity of IRL software has barely budged.
One could hope that we'd use these newfound agentic coding powers to actually realize value, improve quality, etc. Instead I see enshittification and stagnation. What are we even doing with all these tokens?
If it can't provide the service, it should stop selling until it can.
This is like the AOL dialup busy signal fiasco of the mid-90's all over again. Except this time, instead of getting mad, people are making excuses for the poor, beleaguered trillion-dollar company.
I really don't understand people saying that this is due to AI commits and it is all the volume's fault.
A volume increase that is a single order of magnitude (which 14x is) should not result in this level of failures.
When I compare what Github does and the volumes vs social media companies, payment companies, video platforms, etc, it just doesn't make sense that it is just a volume problem.
It looks a lot more like a platform that already has baseline issues that are compounded by increased volume.
With Github going up and down and Ubuntu going up and (mostly) down, there's a lot of time for intra-office sword fighting or whatever, lately. If somebody takes down Claude, everybody's going to have to just go home for the day. (https://xkcd.com/303/)
It's not great. Just talked to a hubber last week. They said everyone inside feels pretty dejected right now, and these posts don't help.
I feel for them -- with AI coders submitting 25 PRs within an hour of an issue being filed, GitHub bears the brunt of that along with the maintainers. That's a lot of work that gets done with each PR.
But the amount of compute needed to serve is not very high. It's all text. The amount of bandwidth and compute needed to serve a Netflix or YouTube is far far harder and they managed just fine.
Netflix and YouTube both built custom CDNs. Netflix uses AWS for control plane only.
Also, respectfully, you have no idea what you're talking about. "Just text" doesn't make it easy to solve. GitHub Actions aren't just text and take a lot of compute.
I don't believe they pay top of market, but even if they did, it's possible to make a lot of money and still feel bad when you have a sense of ownership and responsibility to the users of your service.
The comment I responded to said they felt bad for GH employees. I was saying I don't feel all that bad given they are well-compensated white collar workers (like many of us here).
Life is pretty good if one's biggest concern is work stuff and you're not personally in danger or actively being harmed. That's all I'm saying.
That being said, 300k TC for E4 is still pretty good. Plus the RSUs have gone up like 60% in the last several years so that 300k package from a few years ago is maybe 350k or more by now.
My point is that they are compensated well. They should be feeling pressure to get this stuff right when their product is core infrastructure for a majority of the digital products that exist today.
I just dont really buy the explanation though. It seems so solvable to hack a throttle or something in place, especially for non-paid plans. The cracks were also showing before AI hit the scene.
Im not saying this is the end-game solution but absolutely they could have put temporary safeguards in place while they "figure it out" if it _really_ is just AI driven slop setting their computers on fire.
The whole "anyone can submit a PR" thing has been a UX issue from day one. That probably needs to go away, and I doubt anyone would really miss it. Where Github could help is by providing a means to build trust that doesn't involve random unknown people slinging code at projects.
I think you are taking an excessive interpretation of what was suggested.
Let's level-set on the issue: Of late, GH has suffered a continuous stream of noteworthy outages. It is hypothesized the underlying cause of the instability has been the dramatic rise in submissions from coding agents ("AI"). The open question is how (or whether) GH can get load at a manageable level, with the proposal being, 'don't immediately allocate build/compute resources against any and all submissions.'
I don't see why that is equivalent to rampant disenfranchisement in the open source community. I believe what people have in mind is closer to, "don't immediately trigger an expensive build process as soon as someone submits a pull request."
That's a hard problem! I don't know. But when we select colleagues we build trust before we let them in the building by interviewing them, looking at their work, checking their references. So maybe there's some sort of analogous process that isn't just "here's a big PR, look at it" that would be useful? If there was such a process, maybe that kid could go through it and become trusted.
EDIT: from Github's selfish perspective, this would gatekeep their CI load. I assume (I have no idea, it's just a guess) that mostly serving source code and handling commits is not primarily the scale problem. Instead (again just guessing) probably the vast majority of the compute load due to PRs is running all the CI checks. Nontrivial projects can spawn a hell of a lot of compute per PR, and on every subsequent commit pushed while the PR is open.
I recently moved all my projects to a self-hosted forgejo instance and have found it quite satisfactory so far. And it's fast! If you're in the market for a github alternative, take a look - there are options.
It’s not fashionable anymore, but I feel that Phabricator deserves an honorable mention as a self-hostable GH alternative too. Actually its “dated” UI is kind of a plus considering how bad everything is now.
It's not "unfashionable". Phabricator has been unmaintained since 2021.
There is Phorge which is a community fork.
It seems that Phorge is the community-run fork of it that's still worked on.
Looks good!
I unironically love the aesthetics of Phabricator.
I also like stacked PRs (which is mercurials default).. Maybe it's worth a shot tbh.
I always balked at GitHub, but was impressed with git very soon after I was first introduced; I migrated from an old Gitea instance to Forgejo for my personal projects and have been very happy with it.
This is a real business continuity issue for us. We’re kinda stuck with GitHub Enterprise but we may need to move from cloud to on-premises if this keeps up.
I don't think aggregating the whole platform into one number is fair. It's like adding the whole aws into one number
From a user perspective this makes sense. But if you’re MSFT or GitHub this number is pretty embarrassing.
They would love if everyone on the platform used all of the features and had massive lock-in right? So if some part of that is always broken, it’s not a confidence booster for users to adopt more of the feature set.
Sure the more things you use the more likely it is that one has an issue but clearly stability isn’t a goal for these type of companies anymore.
On the other hand when you have a reasonably complex deployment it's easy to get swamped with dashboards showing CPU, Memory, I/O, application-metrics, signups, active users/sessions, etc.
Instead it's nice to think about how you can express the state of a complete system as a single number. It might be you divide active user sessions by database-connections, and then scale by memory capacity.
But as a single digit you can then get used to normal ranges, and have it always visible somewhere obvious. A single number won't show details, but when it changes you can go look at the specific metrics. It's a cute shorthand, and it can work well as a basic "are we normal" check.
It’s obviously a meme website, the meme is more funny when the number isn’t high. Anyone looking for actual accurate info would go to the real status page.
Ironically I’ve never found official status pages to be all that accurate either since companies love to exclude all kinds of outages from counting towards uptime. Anthropic is hilariously egregious about that as a recent example I can think of, but I assume GitHub does the same since it’s so common in the industry.
splitting the status page like they do, to the point where it is only a bit of humourous exaggeration to say that they track broken `git push` and `git pull` separately, is a sleight of hand / accounting / SLA-fudging that we should not excuse
there is a subset of the site that pretty much everyone uses — git, issues, pull requests, actions — and if any part of that is broken then the site is broken and the status page should indicate how often this happens
> splitting the status page like they do, to the point where it is only a bit of humourous exaggeration to say that they track broken `git push` and `git pull` separately, is a sleight of hand / accounting / SLA-fudging that we should not excuse
This is a pretty ungenerous take. You could look at it the other way: if I don't use actions then it's useful for me to know that only actions are broken, and I can continue in my normal usage. If you bundle everything up then the status page is reporting an unhelpful false positive for me.
If S3, EC2, EKS and RDB alone had a similar uptime as all of Github right now, we'd all know.
No one cares that much if repo wikis, commit stats or gist had these issues. It's the combination of inter-dependent services that are used in combination, like PRs, actions, discussions, etc.
If one were to build a single percentage for each of these components of both systems, github would still lose. Maybe it's a few days without outages more but this isn't a comparison.
Github has far less services and regions that AWS.
I think the correct middle ground is a site that lets you select the parts of the platform you rely on and ignore the others. For example, GitHub is "down" for me when I can't push, process PRs, or release packages, but I don't care about Actions or AI features.
You’re kind of an outlier - nobody wants AI but Actions are core for tons of workflows and deployment pipelines. Everyone bought into the “only robots can deploy” mantra (correctly IMO, it’s a huge time and friction saver) only to be bit in the ass by the platform being so u reliable they can be stuck for days without deploys.
Thats kind of my point, everyone has a different set of GitHub features they rely on. Some people even want the AI bits.
I'm currently setting up a self-hosted "Knot" for use on tangled.org.
Mainly doing it because I think AtProto is cool and self-hosting is fun, but also because owning the infrastructure that hosts my projects is definitely the direction I want to move in.
Tangled's Knot system feels like a really strong abstraction for this. I host the data in an AtProto Repository, but can rely on a third party to host/manage the AtProto Application that presents it to the rest of the world. If Tangled goes under, I can happily take my AtProto login to a different platform and point it at my Knot without changing a thing about my hosting setup.
Much more convenient that hosting an entire, siloed webapp on my own corner of the internet.
Lots of apologia for Github here. Aside from the fact that defending a billion-dollar company is a bit strange; especially one that is steward to the the overwhelming majority of open-source software.
Maybe that's good-will doing the work? For me it's always been a sour pill to swallow that I have to buy in to a large companies internal politics and practices in order to work on projects I love. I don't feel like I owe them anything.
Especially if they can't hold up their end of the deal.
Unfettered access to the world's software repositories, for the princely sum of a bucketload of Azure credits.
Let me ask the question in reverse: what do you have against them such that the fellow human beings struggling to maintain their operations don’t deserve even a modicum of kindness, respect, and good will? Are you unable to separate the business from the hard working people behind it?
It’s not like they don’t know that people like us are counting on them: they recognize that their service is the “dial tone” for much of the world’s software development capability. They are keenly aware of the impact.
What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?
When did OP blame the people involved personally?
If I to hire a contractor to redo my roof, and that roof leaks, whether they worked hard or not is immaterial. They did not do the task in they were paid to do. I'm not going to buy their services again just because their shingles guy was particularly charming.
MS has talented engineers, but that's a complete misdirection. Github is a service in decline: there is nothing wrong with criticizing them.
I have all the empathy for people in the world.
A corporation is not a person. If your organization cannot handle the load, then you need to adjust your practices. The organization needs to prioritize their paying users. The organization needs to shift people from new features to keeping the lights on. And maybe the organization needs to find another strategy to manage its azure transition.
A corporation is made of people. GitHub cannot exist but for the people who continue to work for it. And they’ve already said, multiple times, that restoring availability is their top priority.
I think it's possible to be simultaneously: gracious and supportive towards the developers and ops staff who have been struggling to maintain reasonable uptime on the extremely important piece of shared internet infrastructure that everyone commenting probably relies on (either directly or indirectly) on a daily basis; and spiteful and cruel towards the massive (and, historically speaking, ethically fraught) corporation whose cynical acquisition and subsequent mismanagement of that same resource got us here in the first place.
I agree 100%! But this important distinction and nuance seems to be lost here.
OP didn’t blame the staff. His focus is on the company.
Invoking individual workers well-being to defend a billion dollar company is also very strange.
A company is made of individual workers. That doesn’t change because there are a lot of them or that their employer has a lot of money.
If anything your argument against MS’s uniqueness makes OPs case stronger.
Tell us how.
>What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?
Would you feel the same way about a colleague who kept causing downtime in your product again and again, seemingly without making any progress in addressing whatever issue was causing their repeated mistakes?
There are web applications out there that are far more complex than GitHub but have much less downtime. It's not like they're facing an unsolvable problem.
You don’t know that it was “their mistake.” Unless you’ve personally successfully scaled a suite of nontrivial services equivalent to GitHub’s to accommodate an unexpected 14x increase in traffic, you respectfully have no basis for such an assertion.
I have.
You could argue the scales are different, but computers are also faster now.
So, argument to credentialism out of the way... What should we do as consumers if a provider that is a defacto monopoly due to network effects stops functioning?
https://news.ycombinator.com/item?id=47947719
https://www.linkedin.com/in/jharasym/
> You could argue the scales are different, but computers are also faster now.
Scale is everything and a faster computer doesn’t always help. Vertical scaling has limits, and complex distributed systems are complex.
Since you seem to possess a diagnosis and remedy with a reasonable amount of certainty, I’m sure they’d love to hear from you and have you fix all their problems for them. Especially if you can do it while not making the problem worse in any dimension.
The link in my previous comment answers the credentials question in detail- including specific technical post-mortems on horizontally scaled stateful systems. Vertical scaling wasn't the topic.
You’re missing the point: a doctor doesn’t diagnose and practice medicine on a patient he hasn’t thoroughly evaluated himself. This is the sort of wisdom that a staff engineer and CTO is expected to have earned.
> I have.
I skimmed your profile. Working on the infrastructure for a couple mid-tier video games is a cool accomplishment, but equating this to having solved GitHub level scale rings hollow.
GitHub has a couple orders of magnitude more daily active visitors than the games you worked on had at their peak.
You can make valid criticisms of GitHub without trying to reduce their scale or inflate your credentials to create a false equivalence.
"false equivalence" needs an equivalence claim to be false.
I didn't make one. The sentence after "I have" was literally "you could argue the scales are different."
GitHub spent a decade asking the world to host its code with them. They got what they asked for. You don't get to beg everyone to run services for you for ten years and then have "scaling is hard" be the answer. They should be improving, not regressing over time, and they have some of the worlds best engineers and a trillion dollar corporation behind them, they don't need my sympathy.
The original question is still open and nobody's engaging with it.
> I didn't make one. The sentence after "I have" was literally "you could argue the scales are different."
Don't you at least see how it's misleading to respond "I have" in response to a question about scaling GitHub-scale services?
Trying to caveat it with "the scales are different" misses the point. The parent commenter was talking about scale.
I'm not sure that resorting to personal attacks against the parent commenter for making a legitimate critique is the right, fair, sensible, or mature approach here.
Discarding legitimate criticism based on some self-determined criteria of intellectual superiority isn't a good look. It smacks of elitism and isn't something conducive to a productive and positive community discussion.
It is unhelpful, rude, condescending, and completely fails to address the underlying problem.
The commenter inserted his own personal bona fides (as a proxy for skill, experience, and knowledge) and use them to bolster his conclusion of culpability and incompetence of the GitHub team. If you take that risk, you should expect to be challenged if those skills are not up to par.
Put more simply: if you get into the ring, you’d better be prepared to take a punch.
Not a personal attack to fact check someone's claims.
I didn't bring their credentials into the conversation. They did.
Yeah, they should be testing for that, right? I think there's a lot of people reading comments like yours and thinking, is this person a paid shill or what?
The earn bucket loads of money, they should be planning for exactly that. And testing for it via load testing every day.
Perhaps you've forgotten the days of GitHub presenting themselves of software engineering thought leaders.
I’ve worked at some very well-endowed organizations. Having money is no guarantee of a particular outcome. There is a lot of money chasing a limited supply of talent. Moreover, distributed systems that were built long ago with certain assumptions can’t be refactored as quickly as the HN populace might believe. The Mythical Man-Month is a popular book for a reason.
> Perhaps you've forgotten the days of GitHub presenting themselves of software engineering thought leaders
Genuinely could use a refresher here.
#hugops is to your coworkers, not to the nameless big-corps who can't maintain a service for paying customers. You should be raising a shitstorm when things you pay for aren't reliable or unusable.
Hot take, if it's traffic is causing issues, throttle your free-tier, pause signups, or stop giving out free things (like runner time).
Who is “maintain[ing] the service”? The workers, of course!
Executives have made a choice to not pay for top talent at Microsoft Azure and Github.
Would you consider telling this to the people working at GitHub directly? I’m sure they’d appreciate your evaluation of their skills and talent.
There are two options, either they are lousy at their jobs, or they are incapable of pushing back against unrealistic demands. Neither is a good indicator of their skill and talent as engineers.
I know I am speaking from a position of some privilege, but I have previously left workplaces that did not allow me to practice good engineering, and I do expect others to do so.
In a SWE job market like this, do you really want to be seen as the "conscientious objector"?
There are literally thousands of people who are ready to ride up the totem pole, it would not be a difficult decision for a bad manager to swing his axe and replace the new head
Talented engineers shouldn’t have much problem finding another position even in this market (of course they should find one before leaving I’m not discounting family responsibilities and whatnot), so if your argument is they’re not able to leave and find another job then you’re essentially agreeing with the person you’re replying to.
Or, they've been given crap primitives to work with. There's only so much lipstick you can put on a pig. I don't know what database they're using or what their pub sub and streaming looks like, or even what their system diagram actually looks like. But, well, you don't see Google having these kinds of problems. Other ones, sure, but between Chubby and Spanner, if Google had bought GitHub we wouldn't be having these problems.
But it wasn't a pig. It was a reliable system, and then it increasingly became an unreliable one, in a way that is not explainable by the mere increase in demand. Whatever rearchitecture was performed, it was done and is apparently being perpetuated by software engineers who should be held accountable. Not necessarily guilty, or even directly at fault, but accountable nevertheless. "I am just an employee of a bad company" is not a valid excuse for an engineer.
Really? Only two possibilities?
yes I would tell them "you're underpaid, if you can, come to a company that appreciates your talents more".
What you said came across as an adverse judgment of their skill and talent. Is that not what you meant?
Are you hiring?
Yes google cloud is actively hiring.
If you pay someone full price to do a job, they know they can't fulfil the terms up front, accept the work, deliver less than the agreed upon terms and still charge you full price, you'd probably call that transaction fraudulent.
GitHub is promising service they know they cannot meet, not telling you that, and still charging you full price. What's more, one can argue quite convincingly that they're lying about their level of delivered service by not reflecting the actual level of uptime on their status page.
To give benefit of the doubt requires that the other party is not blatantly and overtly acting in bad faith. When they are, you're just apologizing for fraudulent behavior.
Fraud is a serious civil and criminal accusation that’s not to be taken or given lightly. Can you detail the fraud that’s being committed? What is the specific promise they made that you’re being deprived of? Remember, the four corners of your agreement with them are controlling.
Defending a multi-trillion dollar company you mean (Microsoft).
I think it depends if you pay them money. If you do, then you should indeed have strong expectations towards them and hold them accountable. If they provide a free service to you, then it's still reasonable to feel upset, but at the same time you get what you pay for.
Does this logic still applies if the company is getting other benefits from having me as a user? (Genuine question, I can see arguments for both sides)
For example, if I am using the free tier of a service and "paying" by seeing ads, should I have similar expectations?
I'm not saying that's how users pay for github - in that case it's more subtle, for example by giving up control of some of their stack and bolstering github already near monopolistic network effect.
I'm surprised at how little the perception of GitHub changed post-acquisition. Coupled with WSL, it almost balanced things for a lot of people and put Microsoft back in the "benefit of the doubt" column. This is undoing a lot of that, on top of the operational costs. Suddenly the bad press is more noticeable and harder to ignore.
As far as I'm concerned, any benefit of the doubt I might have had for Microsoft is gone after this debaucle: https://news.ycombinator.com/item?id=47989883
You must've been fuming at email clients for the last 20 years hijacking people's signatures eh?
I don't use any email clients that mess with signatures, so I think I'm fairly consistent here, yes.
there are two groups of people willing to die defending [billion-dollar company]: HN users and Nintendo fans
Apple, clothing brands, even some Microsoft.
> Maybe that's good-will doing the work?
Of course. GitHub has been an enormous gift to the open source community. Arguably more than Git itself. They deserve a lot of good will.
they are not the non-profit. they make money of it and devs expect certain kind of service in return. GH failed to deliver on the service expectation.
What money do GH make off open source projects on the free tier? I haven’t seen ads, micropayments to clone repos, etc?
It's the marketing budget. People only pay for it because they've used it for free.
https://xkcd.com/1150/
Oversimplification.
You're right, but that GitHub is dead.
Also, the former stewards of that open source goodness sold it to Microsoft for a cheap buck.
Any goodwill they earned has been spent.
Using "apologia" here is pretty embarrassing.
I think its the fact that people have used the software for so long that they feel emotional to it (Hashimoto crying tears of sadness when he decided to move ghostty away from github) and there is completely nothing wrong about it as we are emotional human beings.
But, you are right in the sense that, Github has failed to accept its part of the deal which is actually to just be a usable place. People HAVE previously tolerated so much AI slop and slowness in github's UI just because of its reliability but this downtime is like the Github's achilles heel.
At some point, I recommend people to accept this and move to more healthier alternatives, there is also an momentum. For example, the only reason I joined github was that I wanted to join codeberg but so many of projects used github and involved sign in with github that I finally gave in into github and I had thought that codeberg is so good but nobody is gonna come here because of the network effects but the tide is turning and I hope more people look into codeberg and healthier alternatives.
> Aside from the fact that defending a billion-dollar company is a bit strange
More than a bit strange. This is an HNism that I'll never get. Why would you go to the comment section anywhere to passionately try to defend the honor of a trillion dollar company, unless 1. you're being paid to astroturf or 2. you own that company's stock? Satya Nadella isn't going to read a post here and say, "Gosh, how nice of that commenter! I'm going to send him some Microsoft stock as a show of appreciation for him defending us online!" I don't think I'll ever understand company-fanboys.
1. Telling that you think the only possible motivations are financial (getting paid, stockholder, or foolish expectations of a gift from Satya).
2. Maybe you know a bunch of people who work there, could be ex-colleagues etc. and you think overall it’s mostly good well-intentioned people there. Therefore you want to see them succeed, and also you might disbelieve that the company is deliberately being awful.
I don’t have any specifically warm feelings about a corporate legal entity, but I know people who work at various companies and partly for that reason I am not rooting for those companies to fail and I also don’t believe the least charitable explanations for all their failings.
https://repo.autonoma.ca/treetrek
My free, open-source, bare-bones, caching-free, dependency-free, authentication- and authorization-free pure PHP raw Git viewer. I developed it because GitList blew out my shared host's drive space and memory (due to a caching bug) and to consolidate my GitHub, BitBucket, and GitLab repos. There's something rewarding about self-hosting and not being beholden to the whims of third parties.
quite literally kicking github while it's down!
The joke lands because everyone has quietly accepted a lot of concentration risk for the sake of convenience.
A vibe coded app that most likely contributed to the onslaught of vibe coded apps that are causing Github to go down. I feel bad for the people working at Github who are basically trying to keep a sinking ship afloat and Microsoft doing everything they can to sink their own ship.
I'm pretty sure we all took down a production enterprise system once or twice. At InVision we had an incident every week, despite all the SOPs and safety nets. And that way waaay before vibe coding..
That purple to blue gradient is the emdash of css.
Feature request: can you make it look like a chalk board from an old manufacturing plant in the appropriate Green? :)
Supposedly commits on GitHub are up 14x YoY.
They are getting spammed by AI agents?
Yes. There’s no other explanation for 14x, that’s nuts.
Is it spam when they’ve been pushing for this shit and putting AI prompt everywhere fir a year or more?
Commits or pushes? Commits aren't really a worthwhile source of measurement in terms of load.
14x is insane, especially since the quality and quantity of IRL software has barely budged.
One could hope that we'd use these newfound agentic coding powers to actually realize value, improve quality, etc. Instead I see enshittification and stagnation. What are we even doing with all these tokens?
The same thing we’ve done with every other productivity increase in a world based on unfettered growth: garbage.
Supposedly commits on GitHub are up 14x YoY.
So?
If Microsoft can't scale, who can?
If it can't provide the service, it should stop selling until it can.
This is like the AOL dialup busy signal fiasco of the mid-90's all over again. Except this time, instead of getting mad, people are making excuses for the poor, beleaguered trillion-dollar company.
>If it can't provide the service, it should stop selling until it can.
You literally cannot buy GitHub Copilot right now [1].
1: https://github.com/features/copilot/plans
Precisely.
If Microsoft can't scale something like Git 14x, then the problem is with Microsoft.
I really don't understand people saying that this is due to AI commits and it is all the volume's fault.
A volume increase that is a single order of magnitude (which 14x is) should not result in this level of failures.
When I compare what Github does and the volumes vs social media companies, payment companies, video platforms, etc, it just doesn't make sense that it is just a volume problem.
It looks a lot more like a platform that already has baseline issues that are compounded by increased volume.
What happens at your job if there's suddenly 14 times as much load?
> What happens at your job if there's suddenly 14 times as much load?
You mean like every startup ever that has been successful?
And for a service that is heavily text bound? A 14x increase would not be a big deal.
Not buying that untill they use a picture from the Simpsons
With Github going up and down and Ubuntu going up and (mostly) down, there's a lot of time for intra-office sword fighting or whatever, lately. If somebody takes down Claude, everybody's going to have to just go home for the day. (https://xkcd.com/303/)
I wonder what morale is like at github. This is like gamer level hating
It's not great. Just talked to a hubber last week. They said everyone inside feels pretty dejected right now, and these posts don't help.
I feel for them -- with AI coders submitting 25 PRs within an hour of an issue being filed, GitHub bears the brunt of that along with the maintainers. That's a lot of work that gets done with each PR.
But they need to make some changes quickly.
But the amount of compute needed to serve is not very high. It's all text. The amount of bandwidth and compute needed to serve a Netflix or YouTube is far far harder and they managed just fine.
they also aren't using azure. IDK what youtube is on, but netflix has actually faced their problems and found solutions (freebsd, mostly)
Netflix and YouTube both built custom CDNs. Netflix uses AWS for control plane only.
Also, respectfully, you have no idea what you're talking about. "Just text" doesn't make it easy to solve. GitHub Actions aren't just text and take a lot of compute.
They should migrate to AWS.
Its webscale
I wouldn't feel too bad for them with their top-of-market comp and valuable RSU packages.
I don't believe they pay top of market, but even if they did, it's possible to make a lot of money and still feel bad when you have a sense of ownership and responsibility to the users of your service.
You missed my point.
Apparently so did everyone else. What was your point?
The comment I responded to said they felt bad for GH employees. I was saying I don't feel all that bad given they are well-compensated white collar workers (like many of us here).
Life is pretty good if one's biggest concern is work stuff and you're not personally in danger or actively being harmed. That's all I'm saying.
GitHub doesn't pay top of market.
You're right.
That being said, 300k TC for E4 is still pretty good. Plus the RSUs have gone up like 60% in the last several years so that 300k package from a few years ago is maybe 350k or more by now.
My point is that they are compensated well. They should be feeling pressure to get this stuff right when their product is core infrastructure for a majority of the digital products that exist today.
I just dont really buy the explanation though. It seems so solvable to hack a throttle or something in place, especially for non-paid plans. The cracks were also showing before AI hit the scene.
Im not saying this is the end-game solution but absolutely they could have put temporary safeguards in place while they "figure it out" if it _really_ is just AI driven slop setting their computers on fire.
"AI coders submitting 25 PRs within an hour of an issue being filed, GitHub bears the brunt of that....".
What "brunt"? These are not large numbers.
Before AI coding, a GitHub issue might get one or two PRs after six months.
AI coding has made this orders of magnitude bigger.
The individual numbers are small, but they add up quickly.
Maybe I am really dense, but a single issue getting 2 vs 25 PRs seems to be no practical difference.
Well two in six months vs 25 in one hour. So that's a 54,000x increase.
But also, each PR kicks off a bunch of CI work, often in GitHub Actions.
The whole "anyone can submit a PR" thing has been a UX issue from day one. That probably needs to go away, and I doubt anyone would really miss it. Where Github could help is by providing a means to build trust that doesn't involve random unknown people slinging code at projects.
Any sort of trust requirement would break the entire model and cause some serious inequality.
How would a random kid in a 3rd world country ever get noticed enough to enter a trust circle, for example?
> would break the entire model
The "model" - GH effectively allowing an overload of their infra - is already broken
> How would a random kid in a 3rd world country ever get noticed enough to enter a trust circle
By submitting a quality change with a clear description, preferably with unit tests? Is that no longer considered an acceptable hurdle?
> By submitting a quality change with a clear description, preferably with unit tests? Is that no longer considered an acceptable hurdle?
But the proposal is to specifically disallow that unless the person is already known.
That is the model today, the one that people want to get rid of.
I think you are taking an excessive interpretation of what was suggested.
Let's level-set on the issue: Of late, GH has suffered a continuous stream of noteworthy outages. It is hypothesized the underlying cause of the instability has been the dramatic rise in submissions from coding agents ("AI"). The open question is how (or whether) GH can get load at a manageable level, with the proposal being, 'don't immediately allocate build/compute resources against any and all submissions.'
I don't see why that is equivalent to rampant disenfranchisement in the open source community. I believe what people have in mind is closer to, "don't immediately trigger an expensive build process as soon as someone submits a pull request."
That's a hard problem! I don't know. But when we select colleagues we build trust before we let them in the building by interviewing them, looking at their work, checking their references. So maybe there's some sort of analogous process that isn't just "here's a big PR, look at it" that would be useful? If there was such a process, maybe that kid could go through it and become trusted.
EDIT: from Github's selfish perspective, this would gatekeep their CI load. I assume (I have no idea, it's just a guess) that mostly serving source code and handling commits is not primarily the scale problem. Instead (again just guessing) probably the vast majority of the compute load due to PRs is running all the CI checks. Nontrivial projects can spawn a hell of a lot of compute per PR, and on every subsequent commit pushed while the PR is open.
not joking, is there a github repo for this project?
No - I made this as a joke for work. Didn't think anyone would look at it!
Meanwhile, my local Gitlab install just hums along no issues.
Becoming a joke is the one think that could end the GitHub monopoly.
GitHub is not a monopoly. It never has been. You've always been able to self-host or you can use gogs, gitea, gitlab, bitbucket, you get the idea
Love. Hope Github is a relic of the past inside 12 months
Won't happen. Stars are money.
Should be 0 today AFAIK
EDIT: I’m a moron, lol.
it already is. re-read?
Yes, it is, my bad. I was on my way to delete my comment actually! Oh well, too late now… (:
Holy bootlickers Batman.
Microsoft is causing Github incidents when Azure data-centers are too hot and they need to make room for Palantir's workload.
This wouldn't surprise me at all, but I'd appreciate evidence to that effect.