GitHub's response time on malicious repositories is often lackluster. However, from conversations with folks at GitHub, my suspicion is that this is because they're being starved for resources, not incompetency or maliciousness.
This (IMO) points to a perverse reality: things need to get worse before they can get better. In other words, Microsoft probably needs to feel more pain (in the form of negative revenue pressure) before they take their own platform responsibilities (vis a vis not distributing malware) seriously.
We say this play out recently with improvements to GitHub Actions security, I expect we'll see the same here.
(Edit: to be absolutely clear, I have first-hand experience that GitHub's security folks work extremely hard, and are often doing the kinds of invisible, thankless "deck-swabbing" work that nobody even thinks about. They're just under-resourced.)
Whilst I personally don't have first hand experience with GitHub's security team - your experiences and suppositions perfectly match my own general observations with our sector.
I've had the pleasure of working with absolutely outstanding security professionals, doing exactly what you see - relatively thankless work.
Due to the alignment of incentives, it can in fact be worse than merely thankless and face internal opposition - so the perverse reality of Microsoft/GitHub needing to feel more pain is sadly likely true.
This article's author is wonderfully honest about having not worked at a larger company and is right to call out the issues that are all about organisational alignment/ownership.. or lack thereof.
You could never pay me enough money to be responsible for security at GitHub though, that's for sure.
Is this happening? GitHub status page is a Christmas tree for 3 quarters and comically had an incident 10 minutes after the "we're working hard on fixing stability!" blog post published.
The free market has responded by buying GitHub Enterprise to avoid the incidents. So we've effectively rewarded GitHub's lack of serious engineering with more money. Doesn't seem like they have any real pressure to improve.
Reminds me of when Windows security problems hit critical mass in the early-2000s and Microsoft went all-in on addressing it (this started in the XP days IIRC).
The post-release XP service packs, to be precise. XP's release was a perfect storm of events: Suddenly, millions of home users receiced an OS that had dozens of networked RPC services running in the background, and received DSL modems around the same time that exposed their computer directly to the internet. A slightly suboptimal combination at a time when software firewalls were a separate product you had to buy for a hundred dollars a year, if you even knew they were a thing.
See also: Steve Gibson advocating for doing away with raw sockets and the nerds of that era losing their minds over such a supposedly preposterous call.
This is like US healthcare. Almost nobody in US healthcare starts of evil. Almost none of the processes start with a clear evil intent. The problem is that the only real reward is money so the processes and people that generate money stay while the processes and people that merely made things better, but didn't increase revenue, slowly fade away.
Does GH get money for taking these actions? Do they have any monetary incentive other than 'their reputation', which clearly isn't changing usage, to improve? Maybe the best question here is why is it that after a lot of black eyes on data usage, reliability and monitoring aren't people switching. What keeps you using GH after stories like this?
I have a suggestion. PyPi and similar package managers should start publishing security warnings about the hosts of projects. That in turn can eventually lead to bans of packages from generally insecure places. Maybe if we start seeing 'WARNING: projects from github.com may contain malware!' after doing pip install XXX MS will start listening.
> PyPi and similar package managers should start publishing security warnings about the hosts of projects. That in turn can eventually lead to bans of packages from generally insecure places. Maybe if we start seeing 'WARNING: projects from github.com may contain malware!' after doing pip install XXX MS will start listening.
PyPI goes out of its way to not be an arbiter of package quality or security. PyPI really doesn't want people assuming those things based on presence, since it's (1) an open index, and (2) the resources needed to make those kinds of determinations at PyPI's scale are several orders of magnitude greater than what PyPI actually has access to.
(This is different from PyPI removing malware based on user reports, which does happen. But that's a reactive task and not one that comes with any sort of blanket guarantee.)
GitHub today isn't GitHub from the early-mid 2010's. Today it is a Microsoft side-gig, something they bought simply because nobody wanted theirs. But, GitHub makes good money, and turns out they're a great source of training material.
They're just limping along. In reality, I think Microsoft wants to kill the GitHub brand, they're just doing it slowly, feeding poison.
I generally believe in hanlons razor (assume stupidity rather than maliciousness), it's likely that github saw the easiest possible solution rather than diving deeper into the cause of the problem to fix it permanently
Interesting article but doesn't seem to understand how security teams work or even how to address a threat outside of an brittle easily bypassed response.
The initial report was investigated and resolved, with the removal of specific repositories. GitHub likely lacks ability to prevent the attacker from repeating the attack (this is potentially a hard technical problem to solve depending on the infrastructure, available telemetry and how the threat actor is performing their attack).
Once that initial report was resolved, the ticket was closed, archived, and the team moved on. No one is going to sit and re-check a blog post to see if it is updated. That just doesn't happen anywhere, not just at GitHub. Likely a separate ticket was created though to see if they could identify the root cause (compromised accounts, spam attack, etc) and potentially address it. That analysis takes time though and AI doesn't provide much value in scaling here (yet).
> All they needed to do was allocate a few days for any regular employee so they could use Copilot to find all these repositories and block them.
That statement in particular shows a poor understanding of malicious repositories. Blocking on a string signature is almost always worthless as that is a brittle signature and very easy for an actor to identify (that the actor is being blocked by said string).
If you're going to block malicious uploads you need something much more concrete than a string. Having a bot search periodically via regular expressions to remove things on a brittle signature runs the risk of blocking a non-malicious repository. Even if the regular expression is perfect, the threat actor would realize relatively quickly what is going on.
If the threat actor is infecting repositories via compromising GitHub accounts, blocking the repository also does little to resolve the problem as the account remains compromised.
Preventing spam or malware problems at scale in any organization the size of GitHub to date not solved well. Social media platforms (using these as an example as they have comparable scale) still struggle with spam and they have more telemetry and experience dealing with such things.
"security" is a cost-center, it does not generate revenue. Incidents from lack of "security" need to have a greater impact on revenue before the typical corporate entity spends money.
Exec Bonuses being able to be clawed back for Security issues is just about the only fix that will have real effects. Anything else will have MBAs believing it's a "next guy" problem, and a obstacle to hitting their objectives.
Though right now the US thinks it's winning the Security Vulnerability Stockpile war, so it won't change the state quo.
Europe is implementing a law that requires software made for profit to hit at least industry security standards. Punishments include the purchaser being able to sue the seller as well as jail time for execs.
At some point, we need to push back against the reality in the US that we have effectively no way to stop mass harvesting (and then breaching) of our PII -- and there's basically zero downside to companies when it happens.
And how do you enact exec bonuses being clawed back? They're often the most connected people in the company to those setting the rules of the company, the board.
Hell, some companies have a CEO that has an absolute majority of voting power, meaning they cannot be held accountable and made to implement changes like the one you suggest.
The only reason they deleted the static list of identified repositories was because of the negative PR of hitting HN. Although I have no affiliation with GitHub, I would imagine that no one with any resource allocation power is incentivized to take action to police public repos in general. The reality is that as soon as they start, it will become an arms race, and they will have to deal with false positives and the community headache that comes with that. They also have bigger fish to fry with the effect the influx of AI slop is having to their open source bread and butter. The two problems are not entirely disjoint, and I suspect solutions won't come until the problem gets much worse, and the agentic coding space stabilizes a bit more.
Simple: further action was not requested in the Jira ticket, and what is not in a Jira ticket is not getting done because the team responsible for doing the needful is already overloaded with other crap.
Or maybe because the higher-up whose authorization is needed to go on a few days worth of deep dive other than doing exactly what is asked and accounted for in tickets doesn't have the time for a few minutes to explain to them why it is needed, or the higher-up needs authorization from finance or legal first.
Obvious /s, but I wouldn't be surprised at all if this is exactly what happened. If I were to guess, the "legal" is my biggest suspicion - if a provider reacts on notice of illegal/harmful content, they're just fulfilling legal obligations. But if they go and actively wade through the archives to find more incriminating content, that might be construed as Github doing active moderation of their own, leading to a loss of pure content hoster legal protections.
So far this year 40% of my contracts have been a recovery from a Shai Hulud like attack. That’s in the area of 750k profit and more spent by the companies.
The instability and security issues that come of relying on the software supply chain has been pointed out for around 30 years. Seriously, go look. Multiple articles have pointed out the problems we’re seeing.
I used to ask the same question on behalf of clients but after multiple non-answers and silence my response now is just put something between you and GitHub that you can control.
I use this account as an outlet not tied to my actual consulting work so unfortunately I can’t point you to that.
Also, the way I do business is more along the lines of “word of mouth” if that makes sense.
I can tell you though that it’s nothing special. I lead IR operations through compromise to recovery and also train. It’s a “let’s get you out of this and make sure it doesn’t happen again” approach.
GitHub's response time on malicious repositories is often lackluster. However, from conversations with folks at GitHub, my suspicion is that this is because they're being starved for resources, not incompetency or maliciousness.
This (IMO) points to a perverse reality: things need to get worse before they can get better. In other words, Microsoft probably needs to feel more pain (in the form of negative revenue pressure) before they take their own platform responsibilities (vis a vis not distributing malware) seriously.
We say this play out recently with improvements to GitHub Actions security, I expect we'll see the same here.
(Edit: to be absolutely clear, I have first-hand experience that GitHub's security folks work extremely hard, and are often doing the kinds of invisible, thankless "deck-swabbing" work that nobody even thinks about. They're just under-resourced.)
Whilst I personally don't have first hand experience with GitHub's security team - your experiences and suppositions perfectly match my own general observations with our sector.
I've had the pleasure of working with absolutely outstanding security professionals, doing exactly what you see - relatively thankless work.
Due to the alignment of incentives, it can in fact be worse than merely thankless and face internal opposition - so the perverse reality of Microsoft/GitHub needing to feel more pain is sadly likely true.
This article's author is wonderfully honest about having not worked at a larger company and is right to call out the issues that are all about organisational alignment/ownership.. or lack thereof.
You could never pay me enough money to be responsible for security at GitHub though, that's for sure.
> in the form of negative revenue pressure
Is this happening? GitHub status page is a Christmas tree for 3 quarters and comically had an incident 10 minutes after the "we're working hard on fixing stability!" blog post published.
The free market has responded by buying GitHub Enterprise to avoid the incidents. So we've effectively rewarded GitHub's lack of serious engineering with more money. Doesn't seem like they have any real pressure to improve.
It's happening for Actions security, at least. I don't know about their other services.
Reminds me of when Windows security problems hit critical mass in the early-2000s and Microsoft went all-in on addressing it (this started in the XP days IIRC).
The post-release XP service packs, to be precise. XP's release was a perfect storm of events: Suddenly, millions of home users receiced an OS that had dozens of networked RPC services running in the background, and received DSL modems around the same time that exposed their computer directly to the internet. A slightly suboptimal combination at a time when software firewalls were a separate product you had to buy for a hundred dollars a year, if you even knew they were a thing.
See also: Steve Gibson advocating for doing away with raw sockets and the nerds of that era losing their minds over such a supposedly preposterous call.
This is like US healthcare. Almost nobody in US healthcare starts of evil. Almost none of the processes start with a clear evil intent. The problem is that the only real reward is money so the processes and people that generate money stay while the processes and people that merely made things better, but didn't increase revenue, slowly fade away.
Does GH get money for taking these actions? Do they have any monetary incentive other than 'their reputation', which clearly isn't changing usage, to improve? Maybe the best question here is why is it that after a lot of black eyes on data usage, reliability and monitoring aren't people switching. What keeps you using GH after stories like this?
I have a suggestion. PyPi and similar package managers should start publishing security warnings about the hosts of projects. That in turn can eventually lead to bans of packages from generally insecure places. Maybe if we start seeing 'WARNING: projects from github.com may contain malware!' after doing pip install XXX MS will start listening.
> PyPi and similar package managers should start publishing security warnings about the hosts of projects. That in turn can eventually lead to bans of packages from generally insecure places. Maybe if we start seeing 'WARNING: projects from github.com may contain malware!' after doing pip install XXX MS will start listening.
PyPI goes out of its way to not be an arbiter of package quality or security. PyPI really doesn't want people assuming those things based on presence, since it's (1) an open index, and (2) the resources needed to make those kinds of determinations at PyPI's scale are several orders of magnitude greater than what PyPI actually has access to.
(This is different from PyPI removing malware based on user reports, which does happen. But that's a reactive task and not one that comes with any sort of blanket guarantee.)
Agreed. It is a hard job. Decisions like this make sense but likely need to be reevaluated regularly based on how the world is changing.
GitHub today isn't GitHub from the early-mid 2010's. Today it is a Microsoft side-gig, something they bought simply because nobody wanted theirs. But, GitHub makes good money, and turns out they're a great source of training material.
They're just limping along. In reality, I think Microsoft wants to kill the GitHub brand, they're just doing it slowly, feeding poison.
> Today it is a Microsoft side-gig, something they bought simply because nobody wanted theirs.
Theirs? Perforce, TeamServer aren't really competing with GitHub.
And embrace extend extinguish lives on
I generally believe in hanlons razor (assume stupidity rather than maliciousness), it's likely that github saw the easiest possible solution rather than diving deeper into the cause of the problem to fix it permanently
Interesting article but doesn't seem to understand how security teams work or even how to address a threat outside of an brittle easily bypassed response.
The initial report was investigated and resolved, with the removal of specific repositories. GitHub likely lacks ability to prevent the attacker from repeating the attack (this is potentially a hard technical problem to solve depending on the infrastructure, available telemetry and how the threat actor is performing their attack).
Once that initial report was resolved, the ticket was closed, archived, and the team moved on. No one is going to sit and re-check a blog post to see if it is updated. That just doesn't happen anywhere, not just at GitHub. Likely a separate ticket was created though to see if they could identify the root cause (compromised accounts, spam attack, etc) and potentially address it. That analysis takes time though and AI doesn't provide much value in scaling here (yet).
> All they needed to do was allocate a few days for any regular employee so they could use Copilot to find all these repositories and block them.
That statement in particular shows a poor understanding of malicious repositories. Blocking on a string signature is almost always worthless as that is a brittle signature and very easy for an actor to identify (that the actor is being blocked by said string).
If you're going to block malicious uploads you need something much more concrete than a string. Having a bot search periodically via regular expressions to remove things on a brittle signature runs the risk of blocking a non-malicious repository. Even if the regular expression is perfect, the threat actor would realize relatively quickly what is going on.
If the threat actor is infecting repositories via compromising GitHub accounts, blocking the repository also does little to resolve the problem as the account remains compromised.
Preventing spam or malware problems at scale in any organization the size of GitHub to date not solved well. Social media platforms (using these as an example as they have comparable scale) still struggle with spam and they have more telemetry and experience dealing with such things.
"security" is a cost-center, it does not generate revenue. Incidents from lack of "security" need to have a greater impact on revenue before the typical corporate entity spends money.
Exec Bonuses being able to be clawed back for Security issues is just about the only fix that will have real effects. Anything else will have MBAs believing it's a "next guy" problem, and a obstacle to hitting their objectives.
Though right now the US thinks it's winning the Security Vulnerability Stockpile war, so it won't change the state quo.
Europe is implementing a law that requires software made for profit to hit at least industry security standards. Punishments include the purchaser being able to sue the seller as well as jail time for execs.
At some point, we need to push back against the reality in the US that we have effectively no way to stop mass harvesting (and then breaching) of our PII -- and there's basically zero downside to companies when it happens.
And how do you enact exec bonuses being clawed back? They're often the most connected people in the company to those setting the rules of the company, the board.
Hell, some companies have a CEO that has an absolute majority of voting power, meaning they cannot be held accountable and made to implement changes like the one you suggest.
You’ll often find that problems aren’t solved by the people who would find themselves out of work if they solved them.
> Why did they stop and take no further action?
It costs them nothing to blame the user for clicking links that they don't trust, even though they're hosting the links...
The only reason they deleted the static list of identified repositories was because of the negative PR of hitting HN. Although I have no affiliation with GitHub, I would imagine that no one with any resource allocation power is incentivized to take action to police public repos in general. The reality is that as soon as they start, it will become an arms race, and they will have to deal with false positives and the community headache that comes with that. They also have bigger fish to fry with the effect the influx of AI slop is having to their open source bread and butter. The two problems are not entirely disjoint, and I suspect solutions won't come until the problem gets much worse, and the agentic coding space stabilizes a bit more.
They do stuff perfect for gorgeous looks that can be pushed asap.
> Why did they stop and take no further action?
Simple: further action was not requested in the Jira ticket, and what is not in a Jira ticket is not getting done because the team responsible for doing the needful is already overloaded with other crap.
Or maybe because the higher-up whose authorization is needed to go on a few days worth of deep dive other than doing exactly what is asked and accounted for in tickets doesn't have the time for a few minutes to explain to them why it is needed, or the higher-up needs authorization from finance or legal first.
Obvious /s, but I wouldn't be surprised at all if this is exactly what happened. If I were to guess, the "legal" is my biggest suspicion - if a provider reacts on notice of illegal/harmful content, they're just fulfilling legal obligations. But if they go and actively wade through the archives to find more incriminating content, that might be construed as Github doing active moderation of their own, leading to a loss of pure content hoster legal protections.
So far this year 40% of my contracts have been a recovery from a Shai Hulud like attack. That’s in the area of 750k profit and more spent by the companies.
The instability and security issues that come of relying on the software supply chain has been pointed out for around 30 years. Seriously, go look. Multiple articles have pointed out the problems we’re seeing.
I used to ask the same question on behalf of clients but after multiple non-answers and silence my response now is just put something between you and GitHub that you can control.
Hi ofjcihen, where can I learn more about your professional services?
I use this account as an outlet not tied to my actual consulting work so unfortunately I can’t point you to that.
Also, the way I do business is more along the lines of “word of mouth” if that makes sense.
I can tell you though that it’s nothing special. I lead IR operations through compromise to recovery and also train. It’s a “let’s get you out of this and make sure it doesn’t happen again” approach.
Thanks!
I would enjoy a brief exchange, can you hit me up at <username> at <username> dotnet?
Oh this one is so easy: nobody cares, especially investors.