At this point lifecycle script should be disabled by default in NPM.
It's a convenience feature that provides built-in Arbitrary Code Execution (even for transient dependencies), and every one of these widespread NPM worm style attacks has propagated through it, because of the default setting. Also enabling it for one command shouldn't automatically permit all transient dependencies to run lifecycle scripts, it should be required to explicitly mark each dependency to limit it to where it's absolutely necessary.
The vast majority of NPM packages do not depend on these scripts, and you should disable them globally if you haven't already.
That's helpful to prevent individual secret harvesting, however it does not protect you from potential silent downstream effects in packages. i.e. inclusion of malicious code in libraries you depend on, this could be as simple as inserting a cryptominer, or as severe as data exfil in a front end package.
This default can affect all consumers of NPM packages, regardless of whether you use yarn, pnpm or npm itself, because most package maintainers use NPM. This is why it's NPM's responsibility to change this default in order to prevent spread of malware in packages.
Yeah. Or they should run in a sandbox. I would have no problem with a post install script which ran arbitrary commands in the context of the installed package itself. But arbitrary scripts + user level permissions is a recipe for disaster.
That said, packages could still just run whatever junk they want when they first get imported in a program.
Is there anything about npm that makes it particularly susceptible to these attacks, other than the fact that it's the most popular package manager of all?
There's not a word in the English language that really expresses how absolutely stupid the npm ecosystem is and the developers that perpetuate it by importing a package rather than writing 5 lines of code.
I'm far from an expert, but this feels like an oversimplification.
Python packages traditionally use setup.py to install code, and setup.py is all executable code under the installed package's control.
Native Ruby Gems execute arbitrary code via extconf.rb.
Pre .NET Core, NuGet packages could ship scripts like `install.ps1`. That's been removed, but they can still ship `.targets` and `.props` files that are incorporated into your build (and so can run code at build time).
PHP Composer packages can ship install scripts or configure themselves as Composer plugins.
The venerable .tar.gz approach to packaging, covering decades of C and C++ code, is all about executing code during installation.
There are measures that can help (e.g., PHP Composer doesn't run install scripts of _transitive_ dependencies) but the JS space is adopting measures that can help too (like pnpm's approve-builds).
But nowadays prefer pyproject.toml, and most people use pre-built distributions (wheels) for their architecture from PyPI, so don't execute arbitrary code to install packages.
> PHP Composer packages can ship install scripts
Which requires the user to say yes to running them, but they can also say they only want a specific package to run scripts with something like "composer -n config allow-plugins.foo/bar true && composer -n require foo/bar"
> The venerable .tar.gz approach to packaging
Which most people don't install directly, but have already had built for them by their distro.
As more and more languages get "package managers", there's an expectation that installing what should just be inert package/library code should not run commands. Sometimes generated files are needed, and the direction seems to be that these package managers should be like distro package managers, where they take the risk of running the build instructions and generate those files for you, serving up os/architecture-specific builds.
This is the direction npm ought to take, and furthermore shouldn't allow things like electron being a small bundle of javascript code that fetches large lumps of binary code from somewhere else on the internet to install. It should all be uploaded to, and sourced from, NPM.
> But nowadays prefer pyproject.toml, and most people use pre-built distributions (wheels) for their architecture from PyPI, so don't execute arbitrary code to install packages.
Yes, and these are positive changes. But they aren't security boundaries, and they don't mean that pip won't execute arbitrary code: a malicious update could ship an update with sdist instead of wheels, a malicious pyproject.toml could provide an arbitrary-code `build-backend`, etc., and pip would still function as designed.
I appreciate the clarifications/corrections on PHP.
> Which most people don't install directly, but have already had built for them by their distro.
Yes, but the original claim was that npm is "particularly susceptible to these attacks" because "npm can execute code after install and most package managers don't do that." I don't think that's accurate: we've seen hundreds of NPM packages compromised in multiple high-profile attacks over the last several months, while .tar.gz was used for decades with nowhere near the same number of compromises.
Rather, I suspect it's a combination of factors: Early JS had a relatively anemic standard library in the early days, and NPM made code reuse dramatically simpler than before. This normalized the use of large and deep dependency trees among JS projects. And the extreme popularity of JS, the centralization of NPM + GitHub, and increased usage of automation makes attacks more practical and more lucrative.
Taking a step back from that particular debate, I'm very much in favor of changes like what you describe.
Taking still another step back, I'm not sure that even those will be enough. If I download a package, it's because I intend to run its code at some point: if it's malicious, I may be less automatically hosed than if its postinstall script runs, but I'm still hosed at execution time. I trust my distro packages, not because they don't execute arbitrary code on installation (RPMs and .debs both do), but because I _trust my distro_. NPM et al. simply cannot vouch for every package they host.
> Early JS had a relatively anemic standard library in the early days, and NPM made code reuse dramatically simpler than before. This normalized the use of large and deep dependency trees among JS projects.
This always seems like a very convenient excuse. C also have a very small standard library. And unless you're doing system programming, you often have to find utility library. It's just that those libraries tries to solve their domain instead of splitting themselves into molecules. Before npm, we had good js libraries too like jQuery as a fundamental one, backbone.js, dropzone.js,... where you import a few files (and vendor them in your project) and be done with it.
The issue with NPM is that it led to the creation of weird ecosystem like babel, webpack, eslint,... where instead of having a good enough solution, it was plugins ad infinitum. And other maintainers started doing the same thing, splitting their libraries, and writing libraries where a gist or a blog post would be enough[0]. Cargo is suffering the same[1]
Couldn't you accomplish the same thing by adding a malicious [build-system] to a pyproject.toml file? You can pull in arbitrary code by providing exact URLs for requirements:
That's a very visible Ken Thompson style attack. The modern expectation is that PyPI would be evaluating this build-system section and would only accept build-systems that they trust to turn package distributions into wheels, and the end users only need the wheels. If you need a specific version of hatchling that they know of, that's fine. If you need something they haven't heard of, they should say no.
> most people use pre-built distributions (wheels) for their architecture from PyPI, so don't execute arbitrary code to install packages
Technically true, but wheels can include a `.pth` which will run arbitrary code as soon as Python is started, which is only marginally less dangerous. Recently exploited in the LiteLLM attack.
That appears to be an exploitable feature of the language, not the package manager per se.
We could then add the philosophical question of asking what's the difference between:
1. Adding malicious code to a package's .pth file that's evaluated automatically on every python invocation
2. Adding malicious code to the package itself that's evaluated automatically on every python invocation _that uses that package_
Packaging systems that don't run arbitrary code when you install a package are more trustworthy than ones that do, but there's still the essential trust you have to place in all code you're installing, directly and indirectly.
/me looks at the `build.rs` file in my Rust's project's `Cargo.toml` and laughs nervously...
(For non Rustaceans: "Placing a file named build.rs in the root of a package will cause Cargo to compile that script and execute it just before building the package.")
aaand many Cargo packages do use it, right now. It became an inseparable part of Rust ecosystem. You cannot build any embedded program or something slightly complex (omg Protobufs!) without build.rs. Not just your own program but all of your dependencies' build.rs is also executed.
I get the need for simple ways to make ecosystem inviting to the new developers. However, I think Cargo was completely mis-designed for simplicity only, where a system language like Rust should shine in its ability to control complexity. With the bad initial design, they invented hacky solutions like build.rs which speaks a string-based language to talk with Cargo!
On top of that crates.io is completely ripe for typosquatting and package overtakes. I think the ecosystem should be completely revamped to a Maven-style namespaced repos and it should require strong GPG signatures.
You're installing code from npm anyway, presumably to execute it at some point on some machine. If it's will be executed from an install script or later during app execution should not really make much of a difference for most devs.
2. Then five more packages appear so you don't have to write that one terrifying line of JavaScript yourself.
3. Then someone writes a wrapper around those five packages.
4. Then someone writes a "modern, lightweight, zero-config" wrapper around the wrapper.
5. Then a framework adopts it, a build tool requires it, and suddenly your todo app has a dependency graph that looks like international diplomacy.
6. Out of 100 devs building the same product, there are now 300 different dependency combinations, all somehow involving 'left-pad' spiritually if not literally.
7. Half the packages are maintained by one person, unpaid, at 2 a.m., after getting yelled at in GitHub issues.
8. The other half were abandoned three years ago but still have 40 million weekly downloads because removing them would break civilization.
9. Pinning dependencies sounds nice until the ecosystem tells you, "sorry, this package only works with Node 22, this plugin needs Node 18, and this transitive dependency has discovered ESM enlightenment."
10. So everyone lives on the bleeding edge, except nobody agrees where the edge is, and the bleeding part is very real.
So yeah, npm is not uniquely cursed because JavaScript devs are worse. It's cursed because it turned code reuse into a lifestyle, dependency trees into rainforests, and 'npm install' into an act of faith.
It was.. but not in the way people generally think. Im not a native english speaker. Therefore, I use chatgpt to fix my comment sometimes. This was done the same way.
I get it and I'm not trying to get down on you, but I've seen people around say this and it bugs me.
I don't know Japanese at all. Sometimes I use LLMs to translate discord messages into Japanese so I can communicate with Japanese people. Then as verification I translate the messages back (with different LLMs) and they usually come out as a near-verbatim version of what I wanted to say. In other words, they don't come out in the chatgpt style of writing.
If I'm able to do that, then chatgpt should be able to fix your English without chatgpt-ifying the whole comment.
At a certain point, is it better to just turn off Dependabot and freeze all NPM packages (minor/patch version and all), rather than continuously update? Particularly for frontend packages, meaningful security fixes seem less likely than supply chain attacks these days.
It's a sad state of affairs, for sure - but is there a reason we can't just switch our frontends to static BOMs, and trust that NPM at least gets their "you can't republish to an old version" bare-minimum constraint right?
> At a certain point, is it better to just turn off Dependabot and freeze all NPM packages (minor/patch version and all), rather than continuously update?
But then the compliance team gets annoyed because some CVE with a CVSS score of 3.1 that has a patch available sits unfixed.
I wonder if the only thing that will solve this is an insurer or regulator saying that: "A system that automatically pulls updates for dependencies without human review, where said updates are not protected by multi-factor authentication by their respective maintainers, shall not be considered secure."
That would wake NPM up at least to the notion that it's absolutely reasonable to require OSS maintainers to press a button on their phones when releases go out, and that's a good thing not a bad thing.
Enforce a “seasoning” period, for example don’t let any pull pull versions newer that 30 days. Perhaps with an exception for versions that address known CVEs.
Yes, and the problem here is that most projects have automated systems that automatically update those lockfiles on every upstream release of a library, under an assumption that minor releases are either security patches or bugfixes that would immediately be useful to the consuming project.
IMO this is built on a pre-ShaiHulud, pre-AI set of assumptions, and should be evaluated from first principles with today's security situation.
My point was the "with hashes" part. You aren't in fact "trusting" NPM to ensure that old versions aren't replaced if the package installer is verifying the hash.
The situation is getting crazy ... personally I have already uninstalled node, python and all package managers from my machine and instead only use them in devcontainers / VMs.
But even if the dev community comes up with super hardened security, I fear in at least a year the models will be good enough in social engineering that we are still running a losing game.
> the models will be good enough in social engineering
Never mind questions of how good the models will/can get. I'm confused why people expect that, in principle, models getting really good at social engineering would have such huge impact. Seems to me like it has diminishing returns and is severely bottlenecked by the fact that the target operates at human speed.
The amount of effort involved in the XZ hack, for example, was immense, and it couldn't have been accelerated because it worked specifically by wearing the existing maintainer down over time. You could generate and send all the necessary nastygrams in seconds and it wouldn't speed up the human consuming them (and in fact, having all of them arrive at once would raise suspicion).
And there is a limit to how persuasive that input can be. Take any of the random nastygrams directed at the XZ maintainer; maybe they could have been made more nasty, more pointed, more aware of the maintainer's personal weaknesses and fears — but would that have actually been overall more effective? I think not, or at most only a tiny bit.
I'm imagining an influence campaign where someone weaponizes targeted advertising and recommendation engines against Mr. Trusted Maintainer, so that it's a The Truman Show effect, where the background of their entire digital bubble pushing the idea that they need retire and delegate.
how do containers solve the problem? if they are connected to the internet (and they are) you have got the same problem, if the credentials can be read by the container, at least to my understanding
Things on my workstation that the container does not have access to: browser cookies, 1Password cli, SSH keys (even if I allow the SSH agent socket), cargo publish tokens (unless it’s a rust project), npm tokens (unless it’s an npm project), and not to mention anything relating to my other clients (don’t compromise my employer when I vibe install some dep for a random side project).
That wouldn't help in that case as exfiltrated data is committed to public GitHub repositories. Unless you have to accept every time an app posts or requests data from known hosts?
Personally I don't allow outbound connections from almost any app, except web browsers to port 80/443. So nodejs, pip, ruby, curl, wget, etc, opening unexpected outbound connections is a big red flag for me.
In some cases, maybe you need to allow permanently git to open outbound resquests to github.com (or gitlab, etc), but at least in my case, I'm okey allowing these connections manually.
> preinstall script: bun run index.js
> Dual exfiltration:
> stolen data is committed as Git objects to public GitHub repositories (api.github.com)
> and sent as RSA+AES encrypted HTTPS POSTs to hxxps://t.m-kosche[.]com/api/public/otel/v1/traces (disguised as OpenTelemetry traces)
> The Bun installer command (command -v bun >/dev/null 2>&1 || (curl -fsSL https://bun.sh/install | bash && export PATH=$HOME/.bun/bin:$PATH)) prepends every injected hook to guarantee Bun availability
> A separate gh-token-monitor daemon (decrypted from J7, deployed by class so) installs to ~/.local/bin/gh-token-monitor.sh with its own systemd service and LaunchAgent. It polls stolen GitHub tokens at 60-second intervals with a 24-hour TTL
This attack in particular would have caused OpenSnitch to go crazy, giving you the opportunity to review what's going on.
1) write a well crafted exfil payload to mozilla or chrome directory (there are sqlite databases and files that store eg. indexeddb content)
2) trigger a tab open to attacker's website, website takes the exfil data from indexeddb and posts it to the server (have something inocuous looking on that website - like a fake npm homepage or whatever, so you don't close it fast enough)
from one step process, this will become universally usable two step process
be sure not to use extra cli parameters like "firefox --new-tab <url>", because if the rule is filtering by process path + cmdline it'll trigger a pop-up to allow the outbound request.
> Personally I don't allow outbound connections from almost any app, except web browsers to port 80/443. So nodejs, pip, ruby, curl, wget, etc, opening unexpected outbound connections is a big red flag for me.
Yep, exactly. Reject by default, with reasonably judicious always-allow rules.
> That wouldn't help in that case as exfiltrated data is committed to public GitHub repositories
Correct in general that it doesn't protect against stuff like that. But this whitelisting is done per-command (in this case, the whitelisting is scoped to the node executable). I've had no need to allow node access to Git in the first place, so no problem there.
> Unless you have to accept every time an app posts or requests data from known hosts?
OpenSnitch doesn't have access to application-level information, so it has no concept of "post" or "request." It's got DNS names, layer 3 info, layer 4 info, and other such things that are visible to the kernel. Your rules get matched to network traffic based on these various properties.
btw, this analysis of a node linux malware with OpenSnitch and other tools was published on reddit a year ago (a malicious linkedin interview targeting web3/crypto devs that resulted in a system compromise):
Excellent example, thank you. This is the kind of stuff that skeeves me out and is entirely within the model of threats that I want to guard against. Sandboxing + OpenSnitch is good stuff. And, ofc, npm bad.
We need to prevent direct connections to internet for containers... once you have a proxy, predefined credentials (api keys) can maybe be added there (per container/target).
the model most people are talking about is in the cloud. for the harness to do useful work, it needs to talk to the cloud
the trouble is, we need protocols that are software determined that force AI interaxtions into limited scope but currently theyre all just bash adjacent and inherit your tools.
You need to use full isolated VM with its own kernel. But then again, I've read somewhere that this malware is also trying to escape the VM isolation as well...
Watertight subdivision in a ship doesn't promise: "there'll never ever be water in this ship". It says: "If there's water in this ship due to one hole, it'll stay in one compartment". Note that I said one hole: you have the titanic, many compartment gets holes, that one ship is still going to sink.
(btw that the Titanic sunk is not an excuse not to secure other ships. And it did save a great many other ships to have watertight subdivision.)
So... Although there are exploits escaping containers and VMs and then bad guys doing lateral moves across machines, you still want defense in depth.
One thing could be more clear, is that the Titanic's bulkheads didn't go all the way to the top deck. They did not seal completely and were not watertight.
I want to go all in on zed now that it's 1.0 but it's security model is all or nothing AFAIK - either I allow it to download and install unknown to me NPM packages whenever it wants with no notice to me, or I turn off all LSP functionality. And then I keep seeing news like this.
> In regards to the whole ecosystem: TC39 should take a look into adding a better standard library to JS itself, which would reduce the amount of one-liner packages.
I concur, the best part of working with Deno way back was its standard library [0] and overall complete dev environment. It is just so damn obvious that a runtime comes with an integrated test runner and assertion library.
To be fair, Node has `node:test` [0] and `node:assert/strict` [1] modules out of the box for several LTS versions now. `node --test` can easily replace Mocha and `node:assert/strict` is fine, but `chai` is still sometimes nicer (include ergonomics like `expect`). (Deno's @std includes an `expect` style assertion library.)
Of course, the trouble is that there are so many test runners in the Node ecosystem and many of them cannot be as easily replaced as Mocha, so the shift to the out-of-the-box test harness and assertion library will of course be painfully slow. People like the over-complicated nature of both Jest and Vitest for all sorts of reasons. Major companies thought Karma was a good idea. (I still don't understand why more developers didn't cringe at "Yo dog, I heard you like V8 for your unit tests, so we're gonna spin up a second, different copy of V8 from your existing V8 environment.")
Much came after Deno added them. I don't know what will happen to Deno, but I am glad they showed the Node world than life can be a little more pleasant
(Specifically `Temporal.Now.instant().since(somePastInstant)` returns a Temporal.Duration that you can relatively easily determine the highest granularity you want and pass that to an Intl.RelativeTimeFormat instance. Also Intl.DurationFormat which is what a Temporal.Duration's `toLocaleString()` uses may also be good enough in many "x hours ago" type situations, though it is over-precise for them.)
So, back to the GP post - TC39 should make a bigger stdlib.
Lets say timeago.js is warranted (as a polyfill and terser API) AND TC39 is taking action.
On slice.js, TC39 took action AND usage is unwarranted since the functionality is widely available. Maybe a stride.js would be needed.
There are 2 modules where npm's culture of "tiny modules because the stdlib is impoverished" holds - but the issue isn't TC39 really. There are 312 modules that aren't related to npm's culture of "tiny modules because the stdlib is impoverished".
> Lets say timeago.js is warranted (as a polyfill and terser API)
Though it may be useful to point out that timeago.js specifically is not a wrapper for Intl.RelativeTimeFormat. It implements its own unique internationalization beyond/instead of the platform capability. (Similarly, so do Moment, Luxon, and date-fns.)
I would argue that it is unwarranted to use such libraries, because we can do better in Vanilla with platform features now. Though yes, Safari still doesn't implement enough of Temporal today for it to be considered Baseline Widely Available. (There also are direct Temporal polyfills. Surprisingly I've found vanilla polyfilling with Date by hand isn't terrible enough to warrant a Temporal polyfill, but also my most complex Temporal math is usually server-side in Deno.)
But yeah, that's mostly quibbling outside of the point being made.
> There are 2 modules where npm's culture of "tiny modules because the stdlib is impoverished" holds - but the issue isn't TC39 really. There are 312 modules that aren't related to npm's culture of "tiny modules because the stdlib is impoverished".
Ah, yeah, I think that is a fair call and I mostly agree with it. "The stdlib is impoverished" argument has changed a lot since the "leftpad" days. So much so it has started to feel more like "the stdlib was impoverished for too long" (and/or "Node was too slow to adopt Browser platform stdlib features and accidentally forked the stdlib for too long"). Most of the damage happened in the past, but a lot of those libraries remain in the ecosystem as developers are slow to adopt new platform features or switch away from old "comfortable" libraries (timeago.js or slice.js versus more vanilla approaches; the quibbling above wasn't entirely tangential, I suppose).
But also the proliferation of modules in npm goes far beyond that. Of course there are a lot of blockchain libraries that will never be stdlib. Of course there are client modules to proprietary APIs that will never be stdlib. The giant size of the npm ecosystem includes a lot more than just "stdlib" style libraries.
I even think the "leftpad" debacle itself did a lot to accelerate npm out of the "tiny modules" approach in general. It also may have itself been a sign that that attitude was already dying. (It was caused by one of the earliest and most prolific "tiny modules" authors leaving the ecosystem in a huff because the ecosystem was changing.)
Despite what some people will tell you (including many in the security indistry), Docker is not a strong security boundary, and it should not be treated as one. It shares a kernel with the running system.
It reminds me of the good old days when people would hand out low privilege Linux accounts and rely on the kernel to prevent privilige escalation. Docker is literally the same thing, just with extra steps. Especially today with new kernel LPE'S dropping every 5 minutes.
Yes, Podman is a bit better because you arent handing the attacker root, but... why hand them an account at all? Just use a grown up VM.
> Despite what some people will tell you (including many in the security indistry), Docker is not a strong security boundary, and it should not be treated as one. It shares a kernel with the running system.
Solaris Zones and FreeBSD Jails (their inspiration) also share a kernel with the running system and do not seem to have as many escape vulnerabilities.
(Though partly because there may not be as much scrutiny of course.)
Yeah, I'm reminded of 15 years ago being told Linux was super secure because people were popping Windows all the time. Turns out it was mostly just a function of effort pointed at the target, and I don't have any reason to believe that's not the case here too.
I've been toying with using Jart's Blink, the "tiniest x86-64-linux emulator". If it were hardened against attacks by a smart community, it seems really attractive...
Then I've been toying with using WASM, with a strictly limited API surface... (API stands for what, "Attack, Please, Infiltrators"?)
I'd really love to make a service which can run "untrusted" code... I want this to exist. I want it to feel like the Sandstorm.io marketplace, and to use Tailscale or something like it... and then to make "Facebook but only for friends or friends-of-friends." No public figures. No random people or ads in my feeds. Just friends. And then other similar apps - mail, discord, etc. Don't let it make outbound HTTP connections. Don't let it access local filesystem, other than assets that were bundled with it. Give it a SQLite API... And let it handle inbound HTTP, REST, WebSocket requests... And use gRPC over Tailscale to only talk directly to approved contacts...
Thoughts? How to run untrusted code safely?
I tried using Firecracker and gVisor... But I'm dissatisfied with them. I was trying to make it easy for someone to download and run my service... But Firecracker and gVisor don't at all feel like "libraries that install cleanly." They feel more like, if you're enterprise scale, you can hire someone to make sure Firecracker and gVisor work the way you want... Maybe I'm just not trying hard enough... Maybe using WSL2 on Windows 11 is just asking for trouble. I tried several different ways to run Linux on Windows, and none of them seemed very happy, when I tried to add Firecracker / gVisor.
Docker is a lot more than just an unprivileged user. In particular, it comes with a seccomp filter. A lot of LPEs are blocked by that filter. Docker is actually a quite decent security boundary - in this case the attackers did not attempt to subvert docker by attacking the kernel attack surface, they attacked a weak configuration via the docker socket being mounted.
The reason you don't use a "grown up VM" is because it's significantly more difficult. Which VM? Firecracker requires KVM and a guest operating system - so how are you getting things in and out in a way that doesn't violate security? That's real work.
gVisor is great and my recommendation, certainly, but the difference between "nothing" and "docker" is actually pretty huge imo.
> the difference between "nothing" and "docker" is actually pretty huge
You've got me there, but it's not really saying much.
Seccomp, for example, is nice, but... It blocks ~44 system calls, and leaves 300 plus exposed. Any memory corruption issue in those remaining calls remain wide open. So better than nothing? Absolutely, but it leads to a false sense of security. I know actual security researchers that intentionally run malware inside Docker and think they're safe. They're not.
Then we can talk about docker itself. It had something like 6 public CVE's related to full escapes last year. If your patch cycle takes 30 days then you spend about half the year with a full, public, escape known. Even if you patched those all on day one you spent most of the year vulnerable to one of the many Kernel vulns that it doesn't stop. On any given calendar day it's statistically likely there is at LEAST one way to escape publicly known and unpatched.
So, yeah. It beats nothing by a huge margin, but it's WORSE than nothing if you think it's safe to run arbitrary untrusted code in. That was never what it was for, it's just what people treat it as. It's not a VM, wasn't designed to be, and people need to stop pretending it is.
Only works if the Docker socket is mounted which is pretty stupid thing to do and also not something you would do in a "frontend" container. Essentially is like having a password-less ssh
That doesn't actually do anything, connect(2) doesn't need write access to connect to a socket. If you think about it, if that did work then a socket with read-only permissions would be basically useless -- Docker uses HTTP for its API, how would the request for "read-only data" be sent without the ability to send messages?
I wrote a comment ~8 years about this[1], I'm kinda sad people still do this and seem to misunderstand just how big of a security hole they are opening...
Just don't do it. If you absolutely must then you can configure some very restrictive AuthZ plugin (but those are incredibly fickle and are almost certainly security theatre because they are basically just an application firewall).
I really wish we would’ve gotten something more like jails or zones. Or better yet put the containers in a jail or zone. Is there a comprehensive sandbox for Linux like the bsds have?
Worth noting that cgroups (kernel feature underlying containers) is "something more like jails or zones". Actually it is "something exactly like jails or zones".
I'm not sure I agree/understand. If you've somehow bypassed AppArmor and cgroup mechanisms then any UID/GID remapping is irrelevant. At this point you're in a position to directly manage memory.
I'm enjoying how nobody in this thread seems to know what a container actually is, and folks may be surprised to learn kernel namespace underpins both docker and lxc.
> If you've somehow bypassed AppArmor and cgroup mechanisms then any UID/GID remapping is irrelevant. At this point you're in a position to directly manage memory.
Not really, user namespaces (despite all of the issues that unprivileged user namespaces have caused) provide an additional layer of protection as lots of privilege checks are either based on kuid/kgid or are userns-aware. These are some of the deepest security models that Linux has (in the sense that every codepath that involves operating on kernel objects involves a kuid/kgid check and possibly a capability check), so making full use of them is fairly prudent. The vast majority of container breakouts reported over the past decade were complete non-issues or very limited impact if you used user namespaces.
AppArmor is not a particularly strong security boundary (it's better than nothing, but there are all sorts of issues with having path-based policies and so they mostly act as final layer of defence against dumb attacks). cgroups are mostly just resource limits, but the devices cgroup (and devices eBPF filter) are are security barrier that prevent obviously bad stuff like direct write access to your host drive. However, those are not the only kinds of protections we need or use in containers, and breaking just those is not enough to "directly manage memory" (but /dev/kmem is inaccessible to user namespaced processes so if that is something you're worried about, user namespaces are another good layer of defence ;)).
It should also be noted that LXC is not the only runtime to support this, the OCI ecosystem supports this too and has for quite a long time now (and the latest release of Kubernetes officially supports isolated user namespaces). Most of my container runtime talks in the past decade have had a slide telling people to use user namespaces but sadly they are not widely used yet.
On the topic of whether containers are a security boundary, I consider them to be fairly secure these days if you use reasonable defaults (user namespaces, reasonable seccomp rules, ideally non-root inside the container). The main reason we struggle in ways that BSD Jails and Solaris Zones do not is because containers on Linux require putting together a lot of disparate components and while this does mean that you can harden non-container programs using them, it opens the door to more bugs. If we had a way to consolidate more operations and information in-kernel things would be much easier to secure (one perennial issue is that of the inatomicity of switching to the container security zone).
Awesome, thanks for the explanation. I didn't know that kuid/kgid existed! That also explains why Proxmox manages the re-mapping in the `LXC.conf` rather than the AppArmor profile. The cascade of AppArmor configs seemed to focus quite a bit on access to `/proc` and `/sys` so I think I mixed that up cgroups with my comments about memory access.
> The cascade of AppArmor configs seemed to focus quite a bit on access to `/proc` and `/sys` so I think I mixed that up cgroups with my comments about memory access.
Funnily enough that is a good example of how fickle AppArmor's protections are -- if you give containers mount privileges (needed to enable container nesting and most system container usecases) you can bypass most (if not all) AppArmor path rules because you can create alternative mounts that don't match the ones in the rules. With the fsopen(2) and open_tree(2) mount APIs, it's even easier -- AppArmor uses d_path to compute the path for policy purposes but detached mounts for procfs do not have a /proc prefix in their d_path form!
My general impression is SELinux is better as it applies to objects directly, but we've had security issues with it too.
Namespaces look dangerous to me because they break lot of assumptions software was built on before. For example, sudo relies on /etc/sudoers being accessible only to root. But with unprivileged containers one can easily create a filesystem namespace where /etc/sudoers would contain arbitrary data. I think, SUID bit won't work in container, but there might be other ways to confuse privileged software using containers. Or not?
Also, if the container has access to dbus, one can try to exploit multiple services listening on dbus.
I dunno, why own a car when you could ride your bike instead? They’re just different things and people may choose to use one over the other for a variety of reasons.
The docker CLI tool is normally executed with user privileges, but there's dockerd, a daemon running as root that actually does the container execution.
im not sure people understand the security vectors. a user with docker permissions effectively has root permissions.
often, docker in docker is used to manage docker orchestration. putinng a user in a docker and peoviding docker access is security through obscurity.
on the flip side, i see people blindly installing tools and skills not understanding they are pushing context and capabilities without any significant security features.
Imagine mythos is actually exceptional hacker. if you give it a well crafted malicious prompt, its going to even more insecure.
the double edged sword is really fascinating to think about
This is how docker is best installed on Linux, and there's a convenience script for it as well (https://get.docker.com/rootless). I am surprised that's not how people are using docker.
I rely on podman for my "devcontainers": https://github.com/evertheylen/probox. If anyone can point me to the weak points in my setup I'd appreciate it!
i wish opencode would have a protocol that puts real guardrails around its agents. rather that gaving to try and transplant weve had ssh for decades, surely you can wire a xomms pathway that cant deciate.
This list is incomplete - at least one other package (nx-console VS code extension; 2.2M downloads) was compromised yesterday with this worm: if suitably qualified/connected people are reading this, it could be worth following that dependency chain too in case there are more. See here:
At this point I would very much like to get off Mr Bones' Wild Ride but I fear this is going to continue to happen because, from my own exploration at least, a large number of commercial detection strategies are directed at the repo/device/developer level when loading/using a package.
This seems analogous to how we tackle email spam and general malware. It means that there is almost always a target valuable enough for bad actors to continue trying. However, unlike email (mostly...), package managers are centralised authorities (and anything out-of-band is surely the developers problem?).
My ill-informed feeling is that we might need to change the culture of lazy versioning with rapid releases and focus on stable, deeply scanned versions at registries. There will be some effect of volume and scale so I could be off, but it still seems telling that this impacts high-churn languages more often.
I don't know, I would love a comprehensive article that explores the landscape right now.
Wondering about Mr Bones' Wild Ride and suspecting it might be a reference to the 1991 movie Nothing But Trouble I took a look, and found I had remembered it wrong.
As for the comparison with spam, there we kind of settled on making people accept spam by vacuuming up their email addresses in pretty much every commercial and social computer network setting, giving it a veneer of legitimacy. I think it is likely to happen in this area too, perhaps some combination of Oracle licensing surveillance agent style software and automated dependency management, i.e. 'solving' supply chain malware by whitelisting some other malware.
Node is the Visual Basic of our day, if Visual Basic had the ability to update itself from a thousand strangers, any minute of the day, without the user-developer having any clue what is going on behind the scenes unless they apply the very skills that would have precluded their use of Node/Visual Basic in the first place.
All that ease-of-development is being paid for by ease-of-rooting.
If you think about it, this is actually a new kind of security. Security by numbers. Overwhelm the attackers with so many compromised services and devices that they get a reverse denial of service. It's inspired by nature in herd animals.
1. the supply chain attacks I've seen are naive. They just leverage postinstall hooks. Malicious code also needs to be executed, not simply installed, so it's a lot less likely that an exploit would happen compared to postinstall since it can't just be buried in a transitive.
2. aube does the same. This is an extra level of protection if you've already whitelisted a package
Because there is a time and effort cost to swap over to another framework/runtime even if it brings benefits, and security is always considered "good for now" unfortunately.
That is what made Bun popular, and tools like uv/pip, oxlint/eslint, orbstack/docker desktop, and the list goes on. Drop-in replacements where we get 10x with little effort.
Because employers don't tend towards security. In fact, many actively punish somebody for "sandbagging" or simply taking too long if they even suggest a security concern.
Given general software quality of the js ecosystem, the proliferation of supply chain attacks was just matter of time. I’m curious how other ecosystems will hold (eg Rust)
NPM is an easy target because every package has the capability of interacting with your development environment by default, via pre/post-install scripts. Cargo has something similar with build.rs scripts, as does Cabal (Haskell).
In JVM-land, thanks to binary distribution being the default, the number of packages you can usurp to achieve the same compromise is fairly small; essentially Maven and Gradle plugins. Which is why you should be extremely wary when, say, Sentry tells you to add them as a dependency by setting up their Gradle plugin. Not sure about sbt. Clojure source dependencies can provide "prep" scripts, which are not automatically run as part of a build, but still execute code on your machine.
There's a pattern here: some build tools incorporate dependency-provided code as part of the consumer project's build, and that is a juicy attack surface. Packages which include such code, or are recently updated to include it, should be treated with extra scrutiny.
Yes... The only things I use from the JS ecosystem are {Claude,gemini}-cli which I fear will be compromised. Fortunately I run in them in their own user accounts with no e.g. ssh credentials, but I dead that's not enough especially for Gemini which probably has access to my entire Google account somehow.
JS is a fine frontend language I guess, if you're willing to hand-roll your own code instead of using frameworks and junk. Whoever decided that JS deserves to run as a backend is an idiot who should not be trusted to evaluate languages anymore, lest they decide cobol is a good idea
Please also collect responses from people, you'd find a pattern: a new attack is launched, people make noise, and later go back to installing packages the same way. Enterprises already use private registries to combat such attacks, vulnerable folks include individual devs and small teams.
I mean that's lovely, but you still store confidential data, I assume, like credentials on your development machine? This doesn't solve anything but the least important "reinstall the OS post-compromise" step.
On that machine there's Claude's authentication for my account so I can run Claude on it, a public key so I can SSH to the box, and a private key for GitHub access.
One solution I haven’t seen recommended much is to have a Claude instruction/skill that explicitly audits the diff of every upgrade, and force this manual audit as part of your upgrade workflow. This seems like it would work pretty reliably.
This is what many AI supply-chain security startups (like the one that posted the article) are already doing with all NPM packages, so save yourself the Claude tokens. All of these compromises were detected within minutes, but it takes some time (<1 hour) for NPM to unpublish all of the affected packages.
Sorry for my ignorance, but then couldn't we build this into NPM itself? So before a package is publicly available it would be quaranteened and checked.
Super dumb question as someone who has been using some form of AI for dev since 2023:
How does having an AI audit external code help? Can they not be prompt injected to ignore a malicious change?
I guess I am sort of concerned that they are a pretty thin layer and even if you put "DO NOT ALLOW PROMPT INJECTION", it's a bit like saying "make no mistakes". There _is_ a priority between `system` and `user` level messages as I had recalled, so a specifically made tool that has its own system prompt should prevent injection while asking Claude CLI could still allow for prompt injection.
There are prompt guard classifiers that can detect prompt injections, but they are not perfect (false positives, obfuscation) and should be only a part of the defense.
The concern is real and unsolved. I think security researchers have an advantage here because they still can fall back to manual audits if their automated analysis (or scores thereof) is off.
This had been going on for years. I'd expect there would be some kind of solution for now? Like banning scripts and only allowing them to be executed at the explicit user request, for example (not a guarantee but at least a barrier) or somehow restricting them from accessing stuff that does not belong to packaging system. I mean, we have millions of people in the ecosystem, and we keep having this problem for years. Maybe we should ask Claude Code how to fix it (sad laugh)?
The solution is to do exactly what you suggest - separate access. In CI this is a matter of having your "build/test" jobs happen separately from your "deploy/publish" jobs.
The trickier part is dev environments, but ideally you take a similar approach. The place that devs do `npm install` should be isolated from, say, your browser / ssh keys etc.
Package manager support would be an amazing win here since you'd have an easier time managing the isolation but you can do this today.
I'm very glad I never bought into fullstack JS/TS.
My JS is frontend only, served as a compiled bundle off a server that doesn't even have a JS runtime of its own. Whatever random vulnerabilities the frontend contains are limited in blast radius to the user's own browser, and since all frontends should be untrusted anyway, there is no real security risk to the server or backend. No reason to update more than a few times a year, if that.
Combine with obvious basic security practices like pnpm cooldowns + no build scripts. When you upgrade a few times a year, and frontend vulns don't matter, there's really no limit to the cooldown you can set. 60 days, why not.
JS programmers today are the PHP programmers of 25 years ago. Remember how many SQL injection bugs there was at the time? Little Bobby Tables remembers.
The standards haven't changed; for the vast majority of JS programmers, this is their first programming language and they have no solid foundation of architecture and security.
So what you get are these overly enthusiastic newbies that want to share their latest achievement with the world (say, a function to left pad a string), and why not include a fancy post-install script with emojis that makes adoption even simpler for other complete noobs? And this is the result.
I wrote a little PHP about 25 years ago and hated it, dropped it like a rock as soon as it wasn't my job. Before that basic, assembly, c, and c++. Then scheme, C#, Java, prolog, and others too minor to mention before I started using nodejs. My last company I chose full stack Typescript and it was great. There's a lot to love and a plenty to avoid.
You're not wrong that there's a lot of crap, a massive legacy, and some bad behavior. It is also approachable, flexible, and portable. Even if you sometimes need to say "Wat?!?"
And before PHP it was Visual Basic, and COBOL even further back. I'm convinced trying to teach people to program without them understanding the basics of how computers work will lead to this.
It is not about not teaching them, I am co-graduated with many people who do YOLO JS and AI slop right now. They had to pass the same architecture and microprocessor programming lectures.
Educated or not, there are many programmers who simply don't give a fuck. Unless you make all the IT jobs a regulated profession and enforce significant yearly recertification, the mediocres will dominate. You cannot be nice and inclusive to mediocre programmers and expect good baselines.
> Unless you make all the IT jobs a regulated profession and enforce significant yearly recertification, the mediocres will dominate.
This would merely optimise the industry for people who are good at wiggling through bureaucratic hurdles. It would entrench mediocrity while lopping off the extreme ends of the skill distribution - on both ends.
A lot of JS is nuts, but most of its basic behaviour that makes it not a good backend language make it an alright scripting language for the web specifically. I wish we would've used something tcl-y instead though, it's much cleaner and the "everything is a string"-ness makes sense for a fundamentally text based medium (the web)
If you drop-in replaced JS today with Lua, you'd have a conceptually similar language with a lot of warts chopped off. Backend devs can just cope with 1-based indexing though.
Nope, they've been targeting credentials so they can deploy whatever they like into prod. They prefer the build machine with it's broader rights than the individual dev boxes.
Only the frontend, which is already untrusted. Any risk would be restricted to individual users who used the compromised frontend to communicate something sensitive, and while that's not great, it is significantly better than the backend being compromised. A very significant reduction in blast radius.
Funnily enough, I don't actually think I do serve third party JS, though. Don't serve ads, don't use external telemetry, don't use JS CDNs. I don't think you have to go quite as far as I do, though - I imagine if your ads are Google AdSense or something, you're probably going to be fine.
Another day, another pre/postinstall script executed that could have easily have been prevented by any sane package manager. NPM really desperately needs an 'allowBuilds' style allowlist [1] and 'approve-builds' command [2].
After I upgraded pnpm to v11, I set all allowBuilds to false and have not observed any failures. Made me wonder why the packages even need build scripts. My guess is for obscure or old platforms, but for most users running on Linux or Darwin build scripts seem to be unnecessary.
Other times it was to avoid shipping binaries due to, erm, safety concerns. The package would include code in a different language, which in turn would compile into a binary library or executable.
If postinstall scripts are restricted the people behind these attacks will switch to something else. Package code is executed automatically by Node when imported, which could be a good replacement. It'll probably run when tests run instead but it's still going to run for most people.
Limiting post install as an attack vector is still a good thing.
Node is working on a similar permission model to Deno that allows explicitly granting certain system resource permissions https://nodejs.org/api/permissions.html. Using it should help reduce impact from malicious code, though if you allow wide permissions it's unlikely to help.
> Limiting post install as an attack vector is still a good thing.
If npm got rid of the post install scripts it would permanently break the install process of packages that use it. Affected systems will need to bypass it, stay on an old npm version, or upgrade the packages to versions that work without post install. Meanwhile, attackers switch to a different attack vector and continue.
I said limit post install, not remove them. Having an allow list in package.json of packages which can run post install would work fine. Pnpm already does this.
Having said that I'm not against full on removal of post install either. It would get more pushback, but would still be possible for people to manually run the post install for the few packages that require it, or to add them as a script in package.json.
Another supply chain attack found and blocked in a day. Everyone regularly using npm to install new packages should be using npm's min-release-age setting to avoid package versions that are newer than a few days old to avoid most attacks in practice like this. You can set it to two days with `npm config set min-release-age=2` for example. https://cooldowns.dev/ has info about equivalent settings in other dependency managers like PyPI and Cargo.
I wonder if npm could run a program where package uploads are automatically delayed for ~10min while they get distributed to an ecosystem of third-party code auditing companies for automatic checks. You could have a public leaderboard of which auditors detect problems fastest and most reliably, or even monetary compensation.
In the fictional universe of William Gibson's Sprawl trilogy, it is legal and normal for defenders to go kinetic on cyberattackers. How long until it is simply easier for governments and big business in the countries victimised by these criminal groups, to find the path of least resistance and go after them personally?
We can’t even prevent shoplifting, what expectation should there be that we’d be capable of mounting any sort of offence / defence against malware outfits.
I dunno, some people just have no sense of humour.
My understanding is that the problem is more that calling in the heavy artillery for what amounts to an annoyance, and maybe some financial harm, is generally considered impolite, even among nations that have conflicting worldviews.
It’s another thing to have the entire Iranian parliament broadcast chanting “death to America”.
I’m not even American.
I don’t even agree with half the stuff Trump says or does, but I’m onboard with at least 20% of it, and that’s infinitely better than the last bunch of clowns.
If that school was being used as cover for a military operation by Islamic extremist who recently shot up a music festival I was at, capturing some of the females and raping them then shooting out their eyes and vaginas.
Enough of the false equivocation.
My coffee was a little two hot this morning FUCKING JEWS!
Attackers are just having fun due to abysmal state of npm and some of the insecure design choices by GitHub and GitHub Actions. Every attack gives them credentials which in turn used to stage more attacks.
Vendor your dependencies, clone or port them where needed, and freeze them. Most good packages these days do not have a deep dependency tree, and we should stop using the ones that do.
I spent a week with claude and codex re-implementing several packages which had dependency trees deeper than I would like.
Most of these packages are trivial to clone.
"But now you're not getting the upstream fixes" they will say.
Wouldn't just having devs pin/not upgrade packages accomplish about as much, have the added benefit that if a package is discovered malicious it would be removed from npm but if already vendored you’d still have the vulnerability on your vendored copy, and pinning versions seems like it would be more likely for devs to do than vendoring?
I can't wait for npm/github to do literally anything at all to mitigate these attacks. Literally anything. Have we considered a basic WAF-style block on some postinstall script strings? LLM-assisted code scanning on publish? Is there anyone home? No I suspect not.
It still seems wild that npm hasn't gone allowlist-only for pre- and post-install scripts like every other JS package manager (yarn, pnpm, Deno, Bun). Obviously it would be a breaking change that might wreck some developers' day for a little while until that allowlist is built, but it would go a long way to eliminate some of the biggest vectors for these attacks.
Bun started out with npm compatibility as a key requirement, so has a very npm-like package manager from the beginning.
Deno at first tried to focus only on web-like/browser-like package management with a focus on full URLs ("https://mypackage.example.com/v2/mypackage.js") and importmaps (a JSON file mapping "mypackage" to a URL such as the previous example) and package/file caching over installing. Deno 2 made Node-compatibility a higher priority (partly because of Bun's competition, partly because of complaints that Deno was hard to migrate to piecemeal from existing Node codebases) and one of those initiatives was a more npm-compatible package manager out of the box (that can also speak Node package.json and manage a node_modules directory), even as Deno still encourages for greenfield projects the URL/importmap/caching approach (with the expansion that it also understands `npm:` pseudo-URLs, `jsr:` pseudo-URLs [an alternative package registry with a stronger focus on ESM and Typescript types], and `node:` pseudo-URLs [emulated node internals], beyond just browser-safe `http:` and `https:` URLs).
Why would a corporate company actually demonstrate responsibility and ethicality in its use of the open source commons from which it extracts every last penny and dollar?
Edit: a more suitable strategy is to do the minimal necessary actions for appearance purposes only, as its how to focus and optimize on its interest for revenue for its shareholders.
No look at the article of this post, it's by SafeDep they are in the same business as Microsoft with their Defender product line. They both publish near identical post mortems with subtle hints at how their product would've defended you against the attack. Why should Microsoft fix the cause instead of selling the cure to each business individually?
It's essential infrastructure there is only one node package manager. I'm not saying it's a good thing, I just describe the systemic reason why it's broken, because that's usually never expressed but its important.
What would be the required budget to host an alternative registry? I'm surprised any GAFAM still hasn't stepped-in and started building their alternative, at least for NPM to up its game in order not to become completely irrelevant.
at amazon, they maintain a private internal registry of packages with approved licenses and audits. this has been in place for several years. i assume other big corps enforce similar policies
This is Amazon, the company where the stuff they rolled their own now makes more money than the business it was rolled for: https://aws.amazon.com/codeartifact/
PSA - you can run something like `npm install -g npm@11.10.0; npm config set min-release-age=3` to update to a version of npm that supports the min-release-age configuration
I’m honestly at a point where I’m afraid to update any of my project’s dependencies, and I’m also afraid to run the locally without some locked down VM
I use Tanstack in my projects. Last week when Tanstack got compromised, it was only my laziness that saved me -- was thinking about doing pnpm upgrade but got lazy and played some dota... Finished game was just going to pnpm upgrade, opened hacker news and boom! news hit.
Since then, I had set up libvirt/qemu based VM with another Linux running in it specifically for development. Now I run all of docker, kubernetes, IDE, pnpm, uv, etc in that VM and removed them from host. The only write capable secret VM has access to, is my passphrase protected ssh key, which I can quickly revoke from my Github account in case of compromise. Feels much safer now.
Yeah, I agree, but then you are at the mercy of whatever vulnerability is found in the current version(s). It just feels like a lose-lose situation no matter what you do.
So long as we insist on everything from a light switch, the vacuum cleaner, security camera, clothes dryer, TV, car, and mobile phone being an always on, always online, Universal Turing Machine we’re not going to find a solution.
If you are using NPM for business, you should be paying for a repository which was checked by antivirus company and not rely on free repository supported by unpaid volunteers.
Every programming language package manager is affected. Any random person can sign up and push packages. They are all equivalent to the Arch Linux User Repository and have the exact same caveats.
How many other languages of any kind have a standard library that is so bizarrely lacking as JS?
Now how many other mainstream languages of any kind have a standard library that is so lacking?
Now how many other mainstream languages have a swathe of front-end developers that suddenly realised they can run code in a CLI or on a server, and spent exactly 0 minutes learning about how to make it not shit?
Now how many other languages get used in scenarios where it makes absolutely zero sense because the developer only knows javascript?
Yes, and it makes no sense. It's not a "swathe of front-end developers" problem.
Developers in general want to push packages. They don't want to experience friction while doing it. They especially don't want to have to do things like engage with Linux distribution maintainers in order to get their packages into official software repositories. They want to just run $pkgr publish on their repo and that's it. So they invariably end up creating their own distribution mechanisms with zero maintainers involved. Just untrusted randoms making accounts and pushing random stuff. It's easy, so naturally what happens is the repositories get filled with software.
It's only natural to use the stuff that is out there, so the packages get added to projects as dependencies despite the fact none of it is even slightly trusted. Developers hate friction when using libraries too. They very much want to just run $pkgr install x on their repositories and be done with it. They don't want to do things like read the source code or verify that it actually corresponds to what they've downloaded. That's somebody else's problem. On Linux distrubutions, that somebody else is the package maintainer, the exact person the programming language package managers aim to eliminate.
> It's only natural to use the stuff that is out there
Sure, if your entire "community" lives and dies by a nonsensical "don't reinvent the wheel, there is a package for that" chant that would rival the most fervent cult members.
That is quite literally the most persistent cargo cult in the entire computing industry. It's the rule, not the exception. Pretty much every community is guilty of it.
Using code from these repositories should be considered the same as licking toilet seats at highway rest stops. There’s something wrong with you if you do it.
As the article states, you can see in the package.json that the optionalDependencies references "@antv/setup": "github:antvis/G2#7cb42f57561c321ecb09b4552802ae0ac55b3a7a"
I'm pretty sure those commits have been removed from github:
Yeah funny, but npm is not the only development community where this regularly happens.
The Onion article this joke refers to [1] is funny because there is a very clear and obvious reason why the U.S. has far more gun deaths per capita. This doesn't apply for npm.
The analogy in npm is dependency proliferation, along with what appears to be weaker community norms around security. To the extent that either or both of these are true, I think the “no way to prevent this” quip is essentially accurate.
Edit: please explain. What other community has this rate of attacks? It's possible they are just detected or publicized less, too. Please help me understand what you're referring to.
some Pipy packages were also attacked recently. And is even more vulnerable due to many projects using requirements.txt which doesn’t lock sub dependencies
Sure, but the Python community isn't the paragon of software risk management excellence you may think it is.
Both Python and Node users (metaphorically) asked for a loaded revolver... They got a metaphorical high yield thermonuclear device with a large blast radius. (And then they skipped the safety tutorial for the B-83 they just bought.)
I initially read this to mean NPM has a pro-concealed-carry policy, which I don't think it does and I don't think is what you meant.
But... Node's culture does not reward "rational" policies with respect to dependency management in the same way that the US does not reward "rational" policies with respect to gun control *. But US gun control policy is a reflection of the "will of the electorate" -- i.e. there are a lot of Americans who want (or need) to own firearms. In the same way, NPM reflects the culture of high-speed, sili-valley web-devs.
I mention both not to criticize, but to comment it's not the tool that's at fault here, but the users who demanded it evolve the way it did. We moved fast. We broke things. And some of the things that remained broken were sociological: It's easy enough to add PGP/GPG signatures on packages, but whom do you trust? What is the meaning of a signature? Does it mean the signer warrants the package/version is free from defects?
NPM is working as designed. Users wanted the software construction equivalent of a loaded revolver. But we got something that was a bit more like a nuclear weapon with a large blast radius. At least the revolver user would more-likely only shoot their own feet six times (or twelve if they reload.)
[*] I'm trying very hard not to start a flame-war about gun control, I only mean to point out dependency management in node can be as contentious in it's domain as gun control policy is in the domain of US politics. Note that I am not making a pro or con argument about gun control, but only pointing out the issue exists. The word "rational" is intentionally chosen to reflect the fact that people's opinions on gun control and package management are often based on personal, often emotional beliefs (which should not be dismissed.)*
> npm is not the only development community where this regularly happens
That is like people defending IIS in the early days by pointing out that Apache occasionally had security problems too. Or, back to the gun control analogy people saying “gun control didn't stop Bondi Beach, did it?” or pointing out [incorrectly]⁴ that everywhere that has gun control has knife crime⁴ instead.
> because there is a very clear and obvious reason why … . This doesn't apply for npm.
I disagree. There are a number of reasons that stack together, the four that spring to the top of my head being:
1. Numbers. There are a lot of potential targets you can exploit if you manage to get something into a ecosystem that large. This “being a big fat target” combined with being easy to exploit makes NPM a very juicy target, and encouraging people to use such a target without trying to implement countermeasures for this sort of attack is IMO reprehensible. Numbers isn't a problem in itself, like in the bad old days when IIS was a mess but Apache got (successfully) attacked far less, but they do exacerbate the security issue by multiplying the attack surface area.
1b. A lot of those using it are relatively untrained or just following recipes so do not know how to protect themselves, and may not even update after an attack like this and remain vulnerable for some further time. While this is not NPM's fault, being due to the popularity/commonality thing, it is something those in control of NPM should care about, if, as I believe is claimed, they care about their users⁰.
2. It is an environment where a ridiculous amount of dependencies, nested impressively deep, is practically encouraged, making audit very difficult even for those who try.
3. A number of good suggestions have been made that would mitigate, or at least vastly reduce, the risks. But action on these has, as far as we know, not happened. Sometimes for good reasons, or at least for reasons¹ rather than “just because”/“cost to implement”/“we don't wanna”, and sometimes, well, not. And no alternatives from within those running NPM are being suggested/worked-on, as far as we know² at least.
4. Those who would make most noise about any change, especially a breaking change that affects them in the smallest way, simply do not care about the risk the situation poses to the wider population.
While the gun control analogy might be a little stretched, I think it is relevant enough particularly because of points 3 & 4.
--------
[0] I refuse to use the word community here. This isn't a cosy little village where everyone knows your name and everyone looks out for everyone else.
[1] Them bringing significant breaking changes, or being too complex to implement piecemeal to give time for those breaks are dealt with or otherwise prepared for, for instance.
[2] If something was being looked into, I'd expect it to be announced³ as that would quieten criticisms like these, at least a little.
[3] Maybe not immediately, but this has been a known problem for so long that we are well past immediately.
[4] When it isn't the case that the US doesn't have knife crime, it just doesn't get reported because the gun issues are worse. Like car travel killing more in total then flights, but you don't hear about every car crash. The UK is often given as an example in these comparisons, but if you look at the stats our knife crime rates are lower than the US's - it isn't that other countries have knife problems instead, the US has worse knife problems as well as the guns problem.
The Node ecosystem happens to be more vulnerable for social and software design reasons, it's true. But people need to be aware that PyPI and Cargo et. al. are not in any fundamental way less vulnerable. This will happen there too.
In fact, attacks like Shai-Hulud explicitly attempt to get into PyPI, and have succeeded to a lesser extent.
But aside from the package-size / -complexity issue pointed out in a sibling comment, PyPI also tries a fair bit to monitor for incoming malware (and there's a "report project as malware" button on each project page).
Also, there are no post-install scripts (of course, the code can detect when it's being run for the first time in the installed environment); and pre-install scripts are only included in sdists[0]. So you can easily[1] configure your installer such that you at least won't get pwned at install time, at the cost that some[2] packages can't be installed that way. And then you can go inspect, run a security scanner over, etc. whatever got installed; wheel installations just copy things to well-defined locations and generate simple wrapper scripts by strict rules.
[0]: I.e., when the project is "being built from source", which generally is only necessary when it includes non-Python code directly and the maintainer hasn't pre-built that code for your system.
[1]: Notwithstanding that, with pip, many actions that you'd expect not to get you pwned totally can. Such as, for example, explicitly telling it to download an sdist and not install it; as I discussed in https://zahlman.github.io/posts/python-packaging-3/ .
[2]: In practice, a pretty small fraction of what typical developers would actually care about, at least outside of specific niches. I'm told there are some niches where it's a big problem, but honestly they're lucky that this kind of build-install orchestration sort-of works at all.
To expand on this, PyPI is slightly less vulnerable because Python users tend to install mega packages (such as numpy or django) and do not frequently interact with their package manager. There is also not a culture of using sub-dependencies.
Cargo is essentially the same as NPM though, it's only "safer" because it's less popular.
Although the situation on NPM is extremely uncomfortable, you're probably less likely to get hit if you take reasonable precautions than on PyPI, simply because NPM is getting scanned more often. Most of these attacks on NPM have been detected and pulled days before my min age kicks in. A sleeper attack on PyPI could be devastating.
> Says Only Development Community Where This Regularly Happens
We've had such issues on other places as well... Shai-Hulud got into Maven [1] and PHP Composer [2], typosquatters got into Maven [3], and it's not new either [4].
No one is safe from skiddies, much less from nation state actors.
> Wake me up when the daily npm security breach headlines are typosquatting stories, not RCE-on-build or RCE-on-upgrade.
RCE-on-build/upgrade can be done in Maven if you manage to compromise one of the major Maven plugins, they run at build time. The thing keeping maven safe for now is that most people pin the plugin and dependency versions, with the obvious side effect that it's truly annoying to get all your dependencies updated.
> The thing keeping maven safe for now is that most people pin [...] versions
Yes, and also the signing of JARs that are uploaded to the repository, and the fact that most release processes are not fully automated, and the batteries-included standard library which reduces the total number of dependencies, and the fact that a run-of-the-mill third-party library can't execute code at build time, and the very small number of people with credentials to publish new versions of major Maven plugins, etc.
NPM by virtue of its popularity and the vagaries of the ecosystem is always going to be a prime target for attackers, and people (maintainers) will always be a weak point.
I agree generally, but I also think it's important to point out that in the NPM ecosystem it is culturally acceptable and even encouraged to install even trivial dependencies to avoid reinventing the wheel. Philosophically I completely agree with this, though practically we see the result and it is not good. The left pad debacle should have been a huge wake up call, but not really much seems to have changed after that.
As long as developers in the ecosystem are cavalier about installing huge chains of dependencies, NPM will be an attractive target for attackers.
Even though we wish it were not so, cultural problems seem to be the hardest technical problems to solve.
Most of these attacks have nothing to do with installing trivial dependencies. It’s usually because the authors npm tokens got hacked; often due to github actions.
The issue is that github actions has too many security gaps that are easy to miss.
No. It is partially due to trivial dependencies. With so many dependencies it is very difficult to evaluate the security posture of all the teams that are inserting themselves into your code.
When I publish commercial software for Unices that use shared object libraries, one of the things we do before publishing is review known vulnerabilities of our 10 dependencies. That is a tractable number. I get a senior engineer to spend time with an intern and step them through the evaluation criteria.
If the team managing a particular library grows lax over time with respect to responding to vulnerabilities, we move away from using that library.
And we can do these things because there are a tractable number of dependencies.
But yes, also GitHub is not pure as the driven slush. I agree with you on that.
Fair point, although when you have dependencies from dozens or hundreds of different publishers, the risk is much higher because it only takes one getting compromised. If instead you only had a handful of core things, there's less surface area
IPCMSes make it somewhat easy to MitM SMS. If your system poops a cookie in the wrong place it doesn't matter if the secret is in someone's head or if it's in a hardware dongle, like you say... the hacker is "in".
My recommendation for bad guys is to not attack the part of the system where it is strong. Just sniff around a bit until you find the weak part and attack that.
Also remember most devs couldn't use a static analysis tool to save their lives (which is why mythos is relevant.) I suspect that a 15 year old copy of Fortify or CoVerity could find bugs mythos missed.
And if that doesn't work, just start scanning github repos for entropy. That's where the credentials that were accidentally published live.
Yeah that’s fair. It’s only as secure as the weakest link.
That was one of the promises of wasm was sandboxing npm packages independently. Not sure what happened with that or not but I’d be curious to know now we’ve had a lot of recent supply-chain publicity.
For example, if every fetched module is sandboxed and even if they got compromised there would be more protection. It would be more “when” not “if” the package is compromised, nip it in the bud.
But then attackers will target the most exposed packages… :)
The "No way to prevent this" analogies seem to me to work better for Memory Safety because, as with Gun Safety, the simple fact is that everybody knows how to solve the problem, but one group insists it's impossible.
There is crowing from the "Actually copy-paste is better" people when this happens, but when it's their turn they just jam their fingers in their ears. The memory safety and gun safety problems are the actual problem. Shai-Hulud would ruin your day if it got into the Odin release you used to build your software, or it was copy-pasted into your "vendor everything" C++ project, the choice not to have automation doesn't mean you fixed the problem.
I'll disagree because the primary issues with gun control in the US are:
[1] Guns are a core part of culture for much of America, very deeply so outside coastal cities. Most of the left wing in the US lives in coastal cities and either grew up there or immigrated very recently and does not leave, so this is an alien concept to them, but even in very blue cities like D.C. you would be shocked how many liberal democrats have armories. It is literally amendment #2!
[2] They are already widely distributed and it would be a logistical impossibility to actually enforce gun control.
This is directly analogous to NPM where:
[1] The package registry working the way it does and people quickly installing packages without thinking much is deeply part of JS culture. It doesn't help that JS caters very heavily to as wide of a market as possible, of which the majority is going to be entry level/junior to associate engineers for whomst script kiddying or letting AI install whatever is essentially a way of life. As evidence, this type of thing is not really a problem with derivatives like Bun, especially in mature organizations where it's easy to enforce a minimum 72 hour wait time between publish and installation of a package.
[2] Packages are already widely distributed and part of dependency stacks (e.g. the infamous leftpad) where it is a logistical impossibility to change how things work.
I also view startups and companies like Vercel as essentially the NRA here, Next.js has taken over huge swathes of the ecosystem and highly encourages dependency-maxxing.
Another direct analogy: proponents of gun control say they are unnecessary for self defense (esp. because law enforcement is good now), too heavy duty to begin with, and fundamentally dangerous.
Similarly I would criticize dependency-maxxing as unnecessary for capability (esp. because AI is good now), too heavy duty to begin with, and fundamentally dangerous.
The whole reason this joke works is because of exactly your belief that somehow you're different and the solution which works for everybody else can't work for you. Charlie is always going to try to kick the ball and Lucy is always going to pull it away and Charlie will never learn from this experience no matter how often it is repeated.
>The "No way to prevent this" analogies seem to me to work better for Memory Safety because, as with Gun Safety, the simple fact is that everybody knows how to solve the problem, but one group insists it's impossible.
I'm not following. Whats the 2nd Amendment equivalent for memory safety? The amount of C/C++ in production or something?
> A well regulated Memory Operation, being necessary to the security of a free Computer, the right of the people to keep and bear Pointer Arithmetic, shall not be infringed.
"A well administered supply chain, being necessary to the freedom of an open internet, the right of the developers to keep and bear hundreds of uninspected transitive dependencies, shall not be infringed."
Language exclusive package managers like this are nightmares for security, but npm simply does things so poorly I feel like they wanted something insecure.
That isn't a language exclusive package manager. I mean things like npm and pip. It isn't necessarily that they're language exclusive, it's that they all tend to have features that aren't good security wise because they aid in developing in that language
At this point lifecycle script should be disabled by default in NPM.
It's a convenience feature that provides built-in Arbitrary Code Execution (even for transient dependencies), and every one of these widespread NPM worm style attacks has propagated through it, because of the default setting. Also enabling it for one command shouldn't automatically permit all transient dependencies to run lifecycle scripts, it should be required to explicitly mark each dependency to limit it to where it's absolutely necessary.
The vast majority of NPM packages do not depend on these scripts, and you should disable them globally if you haven't already.
There's an RFC for this: https://github.com/npm/rfcs/pull/868
Or simply use pnpm.
That's helpful to prevent individual secret harvesting, however it does not protect you from potential silent downstream effects in packages. i.e. inclusion of malicious code in libraries you depend on, this could be as simple as inserting a cryptominer, or as severe as data exfil in a front end package.
This default can affect all consumers of NPM packages, regardless of whether you use yarn, pnpm or npm itself, because most package maintainers use NPM. This is why it's NPM's responsibility to change this default in order to prevent spread of malware in packages.
Yeah. Or they should run in a sandbox. I would have no problem with a post install script which ran arbitrary commands in the context of the installed package itself. But arbitrary scripts + user level permissions is a recipe for disaster.
That said, packages could still just run whatever junk they want when they first get imported in a program.
“No way to prevent this” says only package manager where this regularly happens.
Is there anything about npm that makes it particularly susceptible to these attacks, other than the fact that it's the most popular package manager of all?
It's the general ecosystem.
There's not a word in the English language that really expresses how absolutely stupid the npm ecosystem is and the developers that perpetuate it by importing a package rather than writing 5 lines of code.
npm can execute code after install and most package managers don't do that
I'm far from an expert, but this feels like an oversimplification.
Python packages traditionally use setup.py to install code, and setup.py is all executable code under the installed package's control.
Native Ruby Gems execute arbitrary code via extconf.rb.
Pre .NET Core, NuGet packages could ship scripts like `install.ps1`. That's been removed, but they can still ship `.targets` and `.props` files that are incorporated into your build (and so can run code at build time).
PHP Composer packages can ship install scripts or configure themselves as Composer plugins.
The venerable .tar.gz approach to packaging, covering decades of C and C++ code, is all about executing code during installation.
There are measures that can help (e.g., PHP Composer doesn't run install scripts of _transitive_ dependencies) but the JS space is adopting measures that can help too (like pnpm's approve-builds).
> Python packages traditionally use setup.py
But nowadays prefer pyproject.toml, and most people use pre-built distributions (wheels) for their architecture from PyPI, so don't execute arbitrary code to install packages.
> PHP Composer packages can ship install scripts
Which requires the user to say yes to running them, but they can also say they only want a specific package to run scripts with something like "composer -n config allow-plugins.foo/bar true && composer -n require foo/bar"
> The venerable .tar.gz approach to packaging
Which most people don't install directly, but have already had built for them by their distro.
As more and more languages get "package managers", there's an expectation that installing what should just be inert package/library code should not run commands. Sometimes generated files are needed, and the direction seems to be that these package managers should be like distro package managers, where they take the risk of running the build instructions and generate those files for you, serving up os/architecture-specific builds.
This is the direction npm ought to take, and furthermore shouldn't allow things like electron being a small bundle of javascript code that fetches large lumps of binary code from somewhere else on the internet to install. It should all be uploaded to, and sourced from, NPM.
> But nowadays prefer pyproject.toml, and most people use pre-built distributions (wheels) for their architecture from PyPI, so don't execute arbitrary code to install packages.
Yes, and these are positive changes. But they aren't security boundaries, and they don't mean that pip won't execute arbitrary code: a malicious update could ship an update with sdist instead of wheels, a malicious pyproject.toml could provide an arbitrary-code `build-backend`, etc., and pip would still function as designed.
I appreciate the clarifications/corrections on PHP.
> Which most people don't install directly, but have already had built for them by their distro.
Yes, but the original claim was that npm is "particularly susceptible to these attacks" because "npm can execute code after install and most package managers don't do that." I don't think that's accurate: we've seen hundreds of NPM packages compromised in multiple high-profile attacks over the last several months, while .tar.gz was used for decades with nowhere near the same number of compromises.
Rather, I suspect it's a combination of factors: Early JS had a relatively anemic standard library in the early days, and NPM made code reuse dramatically simpler than before. This normalized the use of large and deep dependency trees among JS projects. And the extreme popularity of JS, the centralization of NPM + GitHub, and increased usage of automation makes attacks more practical and more lucrative.
Taking a step back from that particular debate, I'm very much in favor of changes like what you describe.
Taking still another step back, I'm not sure that even those will be enough. If I download a package, it's because I intend to run its code at some point: if it's malicious, I may be less automatically hosed than if its postinstall script runs, but I'm still hosed at execution time. I trust my distro packages, not because they don't execute arbitrary code on installation (RPMs and .debs both do), but because I _trust my distro_. NPM et al. simply cannot vouch for every package they host.
Thanks for the reply!
> Early JS had a relatively anemic standard library in the early days, and NPM made code reuse dramatically simpler than before. This normalized the use of large and deep dependency trees among JS projects.
This always seems like a very convenient excuse. C also have a very small standard library. And unless you're doing system programming, you often have to find utility library. It's just that those libraries tries to solve their domain instead of splitting themselves into molecules. Before npm, we had good js libraries too like jQuery as a fundamental one, backbone.js, dropzone.js,... where you import a few files (and vendor them in your project) and be done with it.
The issue with NPM is that it led to the creation of weird ecosystem like babel, webpack, eslint,... where instead of having a good enough solution, it was plugins ad infinitum. And other maintainers started doing the same thing, splitting their libraries, and writing libraries where a gist or a blog post would be enough[0]. Cargo is suffering the same[1]
[0]: https://github.com/Rob--W/proxy-from-env/blob/master/index.j...
[1]: https://docs.rs/is_executable/latest/src/is_executable/lib.r...
The whole JS ecosystem is insane. ESM, CJS, are still, and will be issues for the next 5-10 years.
> But nowadays prefer pyproject.toml
Couldn't you accomplish the same thing by adding a malicious [build-system] to a pyproject.toml file? You can pull in arbitrary code by providing exact URLs for requirements:
That's a very visible Ken Thompson style attack. The modern expectation is that PyPI would be evaluating this build-system section and would only accept build-systems that they trust to turn package distributions into wheels, and the end users only need the wheels. If you need a specific version of hatchling that they know of, that's fine. If you need something they haven't heard of, they should say no.
> most people use pre-built distributions (wheels) for their architecture from PyPI, so don't execute arbitrary code to install packages
Technically true, but wheels can include a `.pth` which will run arbitrary code as soon as Python is started, which is only marginally less dangerous. Recently exploited in the LiteLLM attack.
That appears to be an exploitable feature of the language, not the package manager per se.
We could then add the philosophical question of asking what's the difference between:
1. Adding malicious code to a package's .pth file that's evaluated automatically on every python invocation
2. Adding malicious code to the package itself that's evaluated automatically on every python invocation _that uses that package_
Packaging systems that don't run arbitrary code when you install a package are more trustworthy than ones that do, but there's still the essential trust you have to place in all code you're installing, directly and indirectly.
/me looks at the `build.rs` file in my Rust's project's `Cargo.toml` and laughs nervously...
(For non Rustaceans: "Placing a file named build.rs in the root of a package will cause Cargo to compile that script and execute it just before building the package.")
aaand many Cargo packages do use it, right now. It became an inseparable part of Rust ecosystem. You cannot build any embedded program or something slightly complex (omg Protobufs!) without build.rs. Not just your own program but all of your dependencies' build.rs is also executed.
I get the need for simple ways to make ecosystem inviting to the new developers. However, I think Cargo was completely mis-designed for simplicity only, where a system language like Rust should shine in its ability to control complexity. With the bad initial design, they invented hacky solutions like build.rs which speaks a string-based language to talk with Cargo!
On top of that crates.io is completely ripe for typosquatting and package overtakes. I think the ecosystem should be completely revamped to a Maven-style namespaced repos and it should require strong GPG signatures.
You're installing code from npm anyway, presumably to execute it at some point on some machine. If it's will be executed from an install script or later during app execution should not really make much of a difference for most devs.
Most of them definitely let you do that.
Yep.
1. Every day there's a new package.
2. Then five more packages appear so you don't have to write that one terrifying line of JavaScript yourself.
3. Then someone writes a wrapper around those five packages.
4. Then someone writes a "modern, lightweight, zero-config" wrapper around the wrapper.
5. Then a framework adopts it, a build tool requires it, and suddenly your todo app has a dependency graph that looks like international diplomacy.
6. Out of 100 devs building the same product, there are now 300 different dependency combinations, all somehow involving 'left-pad' spiritually if not literally.
7. Half the packages are maintained by one person, unpaid, at 2 a.m., after getting yelled at in GitHub issues.
8. The other half were abandoned three years ago but still have 40 million weekly downloads because removing them would break civilization.
9. Pinning dependencies sounds nice until the ecosystem tells you, "sorry, this package only works with Node 22, this plugin needs Node 18, and this transitive dependency has discovered ESM enlightenment."
10. So everyone lives on the bleeding edge, except nobody agrees where the edge is, and the bleeding part is very real.
So yeah, npm is not uniquely cursed because JavaScript devs are worse. It's cursed because it turned code reuse into a lifestyle, dependency trees into rainforests, and 'npm install' into an act of faith.
> 7. Half the packages are maintained by one person, unpaid, at 2 a.m., after getting yelled at in GitHub issues.
By a manager for for a >$1 billion market cap corporation who doesnt understand that the one person isnt an employee.
Pangram says this comment is %100 LLM generated.
It certainly reads as LLM generated!
It was.. but not in the way people generally think. Im not a native english speaker. Therefore, I use chatgpt to fix my comment sometimes. This was done the same way.
I get it and I'm not trying to get down on you, but I've seen people around say this and it bugs me.
I don't know Japanese at all. Sometimes I use LLMs to translate discord messages into Japanese so I can communicate with Japanese people. Then as verification I translate the messages back (with different LLMs) and they usually come out as a near-verbatim version of what I wanted to say. In other words, they don't come out in the chatgpt style of writing.
If I'm able to do that, then chatgpt should be able to fix your English without chatgpt-ifying the whole comment.
Point accepted. But,
If there are people who focus more on chatgpt "style" of writing, rather than what the message is conveying, frankly, I don't care.
These things are here, they are here to say, and expand into a lot more domains.
No. NPM's not particularly bad at all tbh.
OWNED BY MICROSOFT
At a certain point, is it better to just turn off Dependabot and freeze all NPM packages (minor/patch version and all), rather than continuously update? Particularly for frontend packages, meaningful security fixes seem less likely than supply chain attacks these days.
It's a sad state of affairs, for sure - but is there a reason we can't just switch our frontends to static BOMs, and trust that NPM at least gets their "you can't republish to an old version" bare-minimum constraint right?
> At a certain point, is it better to just turn off Dependabot and freeze all NPM packages (minor/patch version and all), rather than continuously update?
But then the compliance team gets annoyed because some CVE with a CVSS score of 3.1 that has a patch available sits unfixed.
I wonder if the only thing that will solve this is an insurer or regulator saying that: "A system that automatically pulls updates for dependencies without human review, where said updates are not protected by multi-factor authentication by their respective maintainers, shall not be considered secure."
That would wake NPM up at least to the notion that it's absolutely reasonable to require OSS maintainers to press a button on their phones when releases go out, and that's a good thing not a bad thing.
Pretty much what the EU Cyber Resiliency Act says re: OSS due diligence, actually.
The core problem is companies skimping on maintenance.
They don’t want to pay engineers to do the analysis manually, and they don’t want to pay for someone to figure out a better automated system.
Would anyone be surprised at their car having problems if they cheaped out on oil changes?
Enforce a “seasoning” period, for example don’t let any pull pull versions newer that 30 days. Perhaps with an exception for versions that address known CVEs.
Yes. This is partially why other ecosystems don’t see as many supply chain attacks.
> and trust that NPM at least gets their "you can't republish to an old version" bare-minimum constraint right?
... Does NPM not create full lockfiles, with hashes and pinned transitive dependencies and everything?
Yes, and the problem here is that most projects have automated systems that automatically update those lockfiles on every upstream release of a library, under an assumption that minor releases are either security patches or bugfixes that would immediately be useful to the consuming project.
IMO this is built on a pre-ShaiHulud, pre-AI set of assumptions, and should be evaluated from first principles with today's security situation.
My point was the "with hashes" part. You aren't in fact "trusting" NPM to ensure that old versions aren't replaced if the package installer is verifying the hash.
https://docs.npmjs.com/cli/v11/configuring-npm/package-lock-...
The situation is getting crazy ... personally I have already uninstalled node, python and all package managers from my machine and instead only use them in devcontainers / VMs.
But even if the dev community comes up with super hardened security, I fear in at least a year the models will be good enough in social engineering that we are still running a losing game.
> the models will be good enough in social engineering
Never mind questions of how good the models will/can get. I'm confused why people expect that, in principle, models getting really good at social engineering would have such huge impact. Seems to me like it has diminishing returns and is severely bottlenecked by the fact that the target operates at human speed.
The amount of effort involved in the XZ hack, for example, was immense, and it couldn't have been accelerated because it worked specifically by wearing the existing maintainer down over time. You could generate and send all the necessary nastygrams in seconds and it wouldn't speed up the human consuming them (and in fact, having all of them arrive at once would raise suspicion).
And there is a limit to how persuasive that input can be. Take any of the random nastygrams directed at the XZ maintainer; maybe they could have been made more nasty, more pointed, more aware of the maintainer's personal weaknesses and fears — but would that have actually been overall more effective? I think not, or at most only a tiny bit.
I'm imagining an influence campaign where someone weaponizes targeted advertising and recommendation engines against Mr. Trusted Maintainer, so that it's a The Truman Show effect, where the background of their entire digital bubble pushing the idea that they need retire and delegate.
This definitely happens. Perhaps more on a state actor level of sophistication but yeah. I’ve seen it. It’s wild.
how do containers solve the problem? if they are connected to the internet (and they are) you have got the same problem, if the credentials can be read by the container, at least to my understanding
Things on my workstation that the container does not have access to: browser cookies, 1Password cli, SSH keys (even if I allow the SSH agent socket), cargo publish tokens (unless it’s a rust project), npm tokens (unless it’s an npm project), and not to mention anything relating to my other clients (don’t compromise my employer when I vibe install some dep for a random side project).
On my personal machine, I run OpenSnitch. Much better defense against data exfil if you reject outbound connections to unexpected/unwanted hosts.
That wouldn't help in that case as exfiltrated data is committed to public GitHub repositories. Unless you have to accept every time an app posts or requests data from known hosts?
Personally I don't allow outbound connections from almost any app, except web browsers to port 80/443. So nodejs, pip, ruby, curl, wget, etc, opening unexpected outbound connections is a big red flag for me.
In some cases, maybe you need to allow permanently git to open outbound resquests to github.com (or gitlab, etc), but at least in my case, I'm okey allowing these connections manually.
> preinstall script: bun run index.js
> Dual exfiltration: > stolen data is committed as Git objects to public GitHub repositories (api.github.com) > and sent as RSA+AES encrypted HTTPS POSTs to hxxps://t.m-kosche[.]com/api/public/otel/v1/traces (disguised as OpenTelemetry traces)
> The Bun installer command (command -v bun >/dev/null 2>&1 || (curl -fsSL https://bun.sh/install | bash && export PATH=$HOME/.bun/bin:$PATH)) prepends every injected hook to guarantee Bun availability
> A separate gh-token-monitor daemon (decrypted from J7, deployed by class so) installs to ~/.local/bin/gh-token-monitor.sh with its own systemd service and LaunchAgent. It polls stolen GitHub tokens at 60-second intervals with a 24-hour TTL
This attack in particular would have caused OpenSnitch to go crazy, giving you the opportunity to review what's going on.
1) write a well crafted exfil payload to mozilla or chrome directory (there are sqlite databases and files that store eg. indexeddb content) 2) trigger a tab open to attacker's website, website takes the exfil data from indexeddb and posts it to the server (have something inocuous looking on that website - like a fake npm homepage or whatever, so you don't close it fast enough)
from one step process, this will become universally usable two step process
absolutely. These attacks will evolve for sure, like the malware evolved on Microslop for years.
But for the time being, the common entry vector is clear:
https://github.com/evilsocket/opensnitch/discussions/1119
> 2) trigger a tab open to attacker's website
be sure not to use extra cli parameters like "firefox --new-tab <url>", because if the rule is filtering by process path + cmdline it'll trigger a pop-up to allow the outbound request.
Damn, good call. Really reinforces the need for sandboxing.
Still doesn’t negate the value of OpenSwitch, since the majority of malware won’t do that. But really good to keep in mind.
This is genius. Let's hope the hackers aren't reading this...
> Personally I don't allow outbound connections from almost any app, except web browsers to port 80/443. So nodejs, pip, ruby, curl, wget, etc, opening unexpected outbound connections is a big red flag for me.
Yep, exactly. Reject by default, with reasonably judicious always-allow rules.
> That wouldn't help in that case as exfiltrated data is committed to public GitHub repositories
Correct in general that it doesn't protect against stuff like that. But this whitelisting is done per-command (in this case, the whitelisting is scoped to the node executable). I've had no need to allow node access to Git in the first place, so no problem there.
> Unless you have to accept every time an app posts or requests data from known hosts?
OpenSnitch doesn't have access to application-level information, so it has no concept of "post" or "request." It's got DNS names, layer 3 info, layer 4 info, and other such things that are visible to the kernel. Your rules get matched to network traffic based on these various properties.
btw, this analysis of a node linux malware with OpenSnitch and other tools was published on reddit a year ago (a malicious linkedin interview targeting web3/crypto devs that resulted in a system compromise):
https://markdownpastebin.com/?id=9c294c75f09349d2977a4ccd250...
Excellent example, thank you. This is the kind of stuff that skeeves me out and is entirely within the model of threats that I want to guard against. Sandboxing + OpenSnitch is good stuff. And, ofc, npm bad.
For credential stealing, that is true, but at least it would protect your local machine. But I just read these worms also try container escape ...
We need to prevent direct connections to internet for containers... once you have a proxy, predefined credentials (api keys) can maybe be added there (per container/target).
the model most people are talking about is in the cloud. for the harness to do useful work, it needs to talk to the cloud
the trouble is, we need protocols that are software determined that force AI interaxtions into limited scope but currently theyre all just bash adjacent and inherit your tools.
You need to use full isolated VM with its own kernel. But then again, I've read somewhere that this malware is also trying to escape the VM isolation as well...
Watertight subdivision in a ship doesn't promise: "there'll never ever be water in this ship". It says: "If there's water in this ship due to one hole, it'll stay in one compartment". Note that I said one hole: you have the titanic, many compartment gets holes, that one ship is still going to sink.
(btw that the Titanic sunk is not an excuse not to secure other ships. And it did save a great many other ships to have watertight subdivision.)
So... Although there are exploits escaping containers and VMs and then bad guys doing lateral moves across machines, you still want defense in depth.
One thing could be more clear, is that the Titanic's bulkheads didn't go all the way to the top deck. They did not seal completely and were not watertight.
Without node, how do you control your cloud resources? Cloudflare requires wrangler. AWS has a lot of node clis. Etc.
From CI pipelines with pinned package versions and OIDC instead of permanent secrets limited, least privileges.
AWS has CLIs and SDKs in many languages.
I want to go all in on zed now that it's 1.0 but it's security model is all or nothing AFAIK - either I allow it to download and install unknown to me NPM packages whenever it wants with no notice to me, or I turn off all LSP functionality. And then I keep seeing news like this.
> In regards to the whole ecosystem: TC39 should take a look into adding a better standard library to JS itself, which would reduce the amount of one-liner packages.
I concur, the best part of working with Deno way back was its standard library [0] and overall complete dev environment. It is just so damn obvious that a runtime comes with an integrated test runner and assertion library.
0 - https://docs.deno.com/runtime/reference/std/
To be fair, Node has `node:test` [0] and `node:assert/strict` [1] modules out of the box for several LTS versions now. `node --test` can easily replace Mocha and `node:assert/strict` is fine, but `chai` is still sometimes nicer (include ergonomics like `expect`). (Deno's @std includes an `expect` style assertion library.)
Of course, the trouble is that there are so many test runners in the Node ecosystem and many of them cannot be as easily replaced as Mocha, so the shift to the out-of-the-box test harness and assertion library will of course be painfully slow. People like the over-complicated nature of both Jest and Vitest for all sorts of reasons. Major companies thought Karma was a good idea. (I still don't understand why more developers didn't cringe at "Yo dog, I heard you like V8 for your unit tests, so we're gonna spin up a second, different copy of V8 from your existing V8 environment.")
[0] https://nodejs.org/api/test.html
[1] https://nodejs.org/api/assert.html#strict-assertion-mode
Much came after Deno added them. I don't know what will happen to Deno, but I am glad they showed the Node world than life can be a little more pleasant
I'm not sure any of the packages here would be part of a "better standard library".
Do any language standard libraries have a "3 hours ago" formatter? Thats what timeago.js does
Maybe slice.js, which just does python-style indexing with negative numbers. TC39 already made array.at() and array.slice() handle negative numbers.
> Do any language standard libraries have a "3 hours ago" formatter? Thats what timeago.js does
There's also a platform feature for that now: Intl.RelativeTimeFormat: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
It asks you to do the basic time math to determine your granularity so there's still a role for a library, but also that time math gets easier with Temporal: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
(Specifically `Temporal.Now.instant().since(somePastInstant)` returns a Temporal.Duration that you can relatively easily determine the highest granularity you want and pass that to an Intl.RelativeTimeFormat instance. Also Intl.DurationFormat which is what a Temporal.Duration's `toLocaleString()` uses may also be good enough in many "x hours ago" type situations, though it is over-precise for them.)
Thanks for that!
So, back to the GP post - TC39 should make a bigger stdlib.
Lets say timeago.js is warranted (as a polyfill and terser API) AND TC39 is taking action.
On slice.js, TC39 took action AND usage is unwarranted since the functionality is widely available. Maybe a stride.js would be needed.
There are 2 modules where npm's culture of "tiny modules because the stdlib is impoverished" holds - but the issue isn't TC39 really. There are 312 modules that aren't related to npm's culture of "tiny modules because the stdlib is impoverished".
> Lets say timeago.js is warranted (as a polyfill and terser API)
Though it may be useful to point out that timeago.js specifically is not a wrapper for Intl.RelativeTimeFormat. It implements its own unique internationalization beyond/instead of the platform capability. (Similarly, so do Moment, Luxon, and date-fns.)
I would argue that it is unwarranted to use such libraries, because we can do better in Vanilla with platform features now. Though yes, Safari still doesn't implement enough of Temporal today for it to be considered Baseline Widely Available. (There also are direct Temporal polyfills. Surprisingly I've found vanilla polyfilling with Date by hand isn't terrible enough to warrant a Temporal polyfill, but also my most complex Temporal math is usually server-side in Deno.)
But yeah, that's mostly quibbling outside of the point being made.
> There are 2 modules where npm's culture of "tiny modules because the stdlib is impoverished" holds - but the issue isn't TC39 really. There are 312 modules that aren't related to npm's culture of "tiny modules because the stdlib is impoverished".
Ah, yeah, I think that is a fair call and I mostly agree with it. "The stdlib is impoverished" argument has changed a lot since the "leftpad" days. So much so it has started to feel more like "the stdlib was impoverished for too long" (and/or "Node was too slow to adopt Browser platform stdlib features and accidentally forked the stdlib for too long"). Most of the damage happened in the past, but a lot of those libraries remain in the ecosystem as developers are slow to adopt new platform features or switch away from old "comfortable" libraries (timeago.js or slice.js versus more vanilla approaches; the quibbling above wasn't entirely tangential, I suppose).
But also the proliferation of modules in npm goes far beyond that. Of course there are a lot of blockchain libraries that will never be stdlib. Of course there are client modules to proprietary APIs that will never be stdlib. The giant size of the npm ecosystem includes a lot more than just "stdlib" style libraries.
I even think the "leftpad" debacle itself did a lot to accelerate npm out of the "tiny modules" approach in general. It also may have itself been a sign that that attitude was already dying. (It was caused by one of the earliest and most prolific "tiny modules" authors leaving the ecosystem in a huff because the ecosystem was changing.)
Worth noting that nowadays nodejs std lib keeps growing and includes the aforementioned assertion and testing support.
https://nodejs.org/api/
> Docker Container Escape
> The payload checks for the Docker socket and, if present, attempts container escape through three sequential methods:
So even if you're running devcontainers / VMs, these worms are already trying to escape.
Make sure you're running a rootless VM engine (e.g. podman instead of docker) !
Despite what some people will tell you (including many in the security indistry), Docker is not a strong security boundary, and it should not be treated as one. It shares a kernel with the running system.
It reminds me of the good old days when people would hand out low privilege Linux accounts and rely on the kernel to prevent privilige escalation. Docker is literally the same thing, just with extra steps. Especially today with new kernel LPE'S dropping every 5 minutes.
Yes, Podman is a bit better because you arent handing the attacker root, but... why hand them an account at all? Just use a grown up VM.
> Despite what some people will tell you (including many in the security indistry), Docker is not a strong security boundary, and it should not be treated as one. It shares a kernel with the running system.
Solaris Zones and FreeBSD Jails (their inspiration) also share a kernel with the running system and do not seem to have as many escape vulnerabilities.
(Though partly because there may not be as much scrutiny of course.)
Yeah, I'm reminded of 15 years ago being told Linux was super secure because people were popping Windows all the time. Turns out it was mostly just a function of effort pointed at the target, and I don't have any reason to believe that's not the case here too.
Linux has had 100% market penetration for 20 years or so now. Everyone using a computer uses Linux *somewhere* in it.
You'd think if it was that easy, there would be exploits all over the place.
Why isn't every single desk phone and router part of a botnet?
Kimwoolf and Aisuru kind of make my point better than yours, I think?
I mean? Many of them are…
I've been toying with using Jart's Blink, the "tiniest x86-64-linux emulator". If it were hardened against attacks by a smart community, it seems really attractive...
Then I've been toying with using WASM, with a strictly limited API surface... (API stands for what, "Attack, Please, Infiltrators"?)
I'd really love to make a service which can run "untrusted" code... I want this to exist. I want it to feel like the Sandstorm.io marketplace, and to use Tailscale or something like it... and then to make "Facebook but only for friends or friends-of-friends." No public figures. No random people or ads in my feeds. Just friends. And then other similar apps - mail, discord, etc. Don't let it make outbound HTTP connections. Don't let it access local filesystem, other than assets that were bundled with it. Give it a SQLite API... And let it handle inbound HTTP, REST, WebSocket requests... And use gRPC over Tailscale to only talk directly to approved contacts...
Thoughts? How to run untrusted code safely?
I tried using Firecracker and gVisor... But I'm dissatisfied with them. I was trying to make it easy for someone to download and run my service... But Firecracker and gVisor don't at all feel like "libraries that install cleanly." They feel more like, if you're enterprise scale, you can hire someone to make sure Firecracker and gVisor work the way you want... Maybe I'm just not trying hard enough... Maybe using WSL2 on Windows 11 is just asking for trouble. I tried several different ways to run Linux on Windows, and none of them seemed very happy, when I tried to add Firecracker / gVisor.
We've been leaning on Lima recently for a "dockeresque" experience with hard VM guarantees. Also keeping an eye on https://github.com/containers/crun/blob/main/krun.1.md
Docker is a lot more than just an unprivileged user. In particular, it comes with a seccomp filter. A lot of LPEs are blocked by that filter. Docker is actually a quite decent security boundary - in this case the attackers did not attempt to subvert docker by attacking the kernel attack surface, they attacked a weak configuration via the docker socket being mounted.
The reason you don't use a "grown up VM" is because it's significantly more difficult. Which VM? Firecracker requires KVM and a guest operating system - so how are you getting things in and out in a way that doesn't violate security? That's real work.
gVisor is great and my recommendation, certainly, but the difference between "nothing" and "docker" is actually pretty huge imo.
> the difference between "nothing" and "docker" is actually pretty huge
You've got me there, but it's not really saying much.
Seccomp, for example, is nice, but... It blocks ~44 system calls, and leaves 300 plus exposed. Any memory corruption issue in those remaining calls remain wide open. So better than nothing? Absolutely, but it leads to a false sense of security. I know actual security researchers that intentionally run malware inside Docker and think they're safe. They're not.
Then we can talk about docker itself. It had something like 6 public CVE's related to full escapes last year. If your patch cycle takes 30 days then you spend about half the year with a full, public, escape known. Even if you patched those all on day one you spent most of the year vulnerable to one of the many Kernel vulns that it doesn't stop. On any given calendar day it's statistically likely there is at LEAST one way to escape publicly known and unpatched.
So, yeah. It beats nothing by a huge margin, but it's WORSE than nothing if you think it's safe to run arbitrary untrusted code in. That was never what it was for, it's just what people treat it as. It's not a VM, wasn't designed to be, and people need to stop pretending it is.
Only works if the Docker socket is mounted which is pretty stupid thing to do and also not something you would do in a "frontend" container. Essentially is like having a password-less ssh
> the good old days when people would hand out low privilege Linux accounts and rely on the kernel to prevent privilige escalation.
... Did that actually stop?
... Is that not the purpose of being able to make multiple accounts (even on a random desktop or laptop system) and restrict their privileges?
Yeah because I'd rather trade one kernel surface for another.
[dead]
Or don't mount the Docker socket into containers
Or mount it read-only if you do, like for Traefik.
That doesn't actually do anything, connect(2) doesn't need write access to connect to a socket. If you think about it, if that did work then a socket with read-only permissions would be basically useless -- Docker uses HTTP for its API, how would the request for "read-only data" be sent without the ability to send messages?
I wrote a comment ~8 years about this[1], I'm kinda sad people still do this and seem to misunderstand just how big of a security hole they are opening...
Just don't do it. If you absolutely must then you can configure some very restrictive AuthZ plugin (but those are incredibly fickle and are almost certainly security theatre because they are basically just an application firewall).
[1]: https://news.ycombinator.com/item?id=17983623
I really wish we would’ve gotten something more like jails or zones. Or better yet put the containers in a jail or zone. Is there a comprehensive sandbox for Linux like the bsds have?
Worth noting that cgroups (kernel feature underlying containers) is "something more like jails or zones". Actually it is "something exactly like jails or zones".
Unprivileged LXCs get pretty close. Less unified design wise but on some aspects better - kernel escape doesn’t land you on a 0 UID
> "kernel escape doesn’t land you on a 0 UID"
I'm not sure I agree/understand. If you've somehow bypassed AppArmor and cgroup mechanisms then any UID/GID remapping is irrelevant. At this point you're in a position to directly manage memory.
What do you mean by "kernel escape"?
I'm enjoying how nobody in this thread seems to know what a container actually is, and folks may be surprised to learn kernel namespace underpins both docker and lxc.
> If you've somehow bypassed AppArmor and cgroup mechanisms then any UID/GID remapping is irrelevant. At this point you're in a position to directly manage memory.
Not really, user namespaces (despite all of the issues that unprivileged user namespaces have caused) provide an additional layer of protection as lots of privilege checks are either based on kuid/kgid or are userns-aware. These are some of the deepest security models that Linux has (in the sense that every codepath that involves operating on kernel objects involves a kuid/kgid check and possibly a capability check), so making full use of them is fairly prudent. The vast majority of container breakouts reported over the past decade were complete non-issues or very limited impact if you used user namespaces.
AppArmor is not a particularly strong security boundary (it's better than nothing, but there are all sorts of issues with having path-based policies and so they mostly act as final layer of defence against dumb attacks). cgroups are mostly just resource limits, but the devices cgroup (and devices eBPF filter) are are security barrier that prevent obviously bad stuff like direct write access to your host drive. However, those are not the only kinds of protections we need or use in containers, and breaking just those is not enough to "directly manage memory" (but /dev/kmem is inaccessible to user namespaced processes so if that is something you're worried about, user namespaces are another good layer of defence ;)).
It should also be noted that LXC is not the only runtime to support this, the OCI ecosystem supports this too and has for quite a long time now (and the latest release of Kubernetes officially supports isolated user namespaces). Most of my container runtime talks in the past decade have had a slide telling people to use user namespaces but sadly they are not widely used yet.
On the topic of whether containers are a security boundary, I consider them to be fairly secure these days if you use reasonable defaults (user namespaces, reasonable seccomp rules, ideally non-root inside the container). The main reason we struggle in ways that BSD Jails and Solaris Zones do not is because containers on Linux require putting together a lot of disparate components and while this does mean that you can harden non-container programs using them, it opens the door to more bugs. If we had a way to consolidate more operations and information in-kernel things would be much easier to secure (one perennial issue is that of the inatomicity of switching to the container security zone).
(Disclaimer: I am a maintainer of runc.)
Awesome, thanks for the explanation. I didn't know that kuid/kgid existed! That also explains why Proxmox manages the re-mapping in the `LXC.conf` rather than the AppArmor profile. The cascade of AppArmor configs seemed to focus quite a bit on access to `/proc` and `/sys` so I think I mixed that up cgroups with my comments about memory access.
I've been reading up on them (https://www.kernel.org/doc/html/latest/filesystems/idmapping...) seeing some of the notation for user IDs (e.g. `u20000`) reminded me that my Hetzner ZFS storage was accessed using a similar UID format for the username.
> The cascade of AppArmor configs seemed to focus quite a bit on access to `/proc` and `/sys` so I think I mixed that up cgroups with my comments about memory access.
Funnily enough that is a good example of how fickle AppArmor's protections are -- if you give containers mount privileges (needed to enable container nesting and most system container usecases) you can bypass most (if not all) AppArmor path rules because you can create alternative mounts that don't match the ones in the rules. With the fsopen(2) and open_tree(2) mount APIs, it's even easier -- AppArmor uses d_path to compute the path for policy purposes but detached mounts for procfs do not have a /proc prefix in their d_path form!
My general impression is SELinux is better as it applies to objects directly, but we've had security issues with it too.
Namespaces look dangerous to me because they break lot of assumptions software was built on before. For example, sudo relies on /etc/sudoers being accessible only to root. But with unprivileged containers one can easily create a filesystem namespace where /etc/sudoers would contain arbitrary data. I think, SUID bit won't work in container, but there might be other ways to confuse privileged software using containers. Or not?
Also, if the container has access to dbus, one can try to exploit multiple services listening on dbus.
Why not run a proper virtual machine?
That's the direction container runtimes are trending
Amazon has been doing it with Firecracker for a while and Kata containers is another popular one
https://github.com/firecracker-microvm/firecracker
https://github.com/kata-containers/kata-containers
I did the same. Claude coded my own solution.
https://github.com/matheusmoreira/virtdev
It's been working really well. Use it every day. Just make some machines and ssh in.
I think Google took a different approach with gVisor and limited userspace kernel interface, tradeoffs...
> Why not run a proper virtual machine?
Extra 'overhead' and heaviness (perceived or real).
I dunno, why own a car when you could ride your bike instead? They’re just different things and people may choose to use one over the other for a variety of reasons.
The joys of running podman/docker on macOS: you can't help but run it in a hypervisor.
> Make sure you're running a rootless VM engine (e.g. podman instead of docker) !
Aren't most people running docker rootless (at least on Linux)? Does podman do more?
The docker CLI tool is normally executed with user privileges, but there's dockerd, a daemon running as root that actually does the container execution.
im not sure people understand the security vectors. a user with docker permissions effectively has root permissions.
often, docker in docker is used to manage docker orchestration. putinng a user in a docker and peoviding docker access is security through obscurity.
on the flip side, i see people blindly installing tools and skills not understanding they are pushing context and capabilities without any significant security features.
Imagine mythos is actually exceptional hacker. if you give it a well crafted malicious prompt, its going to even more insecure.
the double edged sword is really fascinating to think about
Docker has not required root for a long time, at least on Linux. There's even a convenience script for it: https://get.docker.com/rootless
Almost everyone I know installs docker rootless.
Installing docker doesn't require root.
"Rootless mode lets you run the Docker daemon and containers as a non-root user."
https://docs.docker.com/engine/security/rootless/
This is how docker is best installed on Linux, and there's a convenience script for it as well (https://get.docker.com/rootless). I am surprised that's not how people are using docker.
If it’s not default, most people won’t do it.
> Aren't most people running docker rootless (at least on Linux)? Does podman do more?
In the HPC space Apptainer (previously "Singularity") was created precisely due to (multi-)user-level access, especially with the use of NFS.
I'd guess that most people who run Docker on linux install it through their distro's package manager, which has a dockerd running as root.
On Debian derivatives, you need some kind of extra privs to even talk to it (being a member of the "docker" group, iirc).
docker service that sets up containers runs as root. podman does away with that service.
I rely on podman for my "devcontainers": https://github.com/evertheylen/probox. If anyone can point me to the weak points in my setup I'd appreciate it!
i wish opencode would have a protocol that puts real guardrails around its agents. rather that gaving to try and transplant weve had ssh for decades, surely you can wire a xomms pathway that cant deciate.
You may run
> podman info --format '{{.Host.Security.Rootless}}'
to ensure podman is rootless in your config.
This list is incomplete - at least one other package (nx-console VS code extension; 2.2M downloads) was compromised yesterday with this worm: if suitably qualified/connected people are reading this, it could be worth following that dependency chain too in case there are more. See here:
https://github.com/nrwl/nx-console/security/advisories/GHSA-...
PS: I posted on HN to try and alert people right after it was compromised but sadly got almost no upvotes :-(
At this point I would very much like to get off Mr Bones' Wild Ride but I fear this is going to continue to happen because, from my own exploration at least, a large number of commercial detection strategies are directed at the repo/device/developer level when loading/using a package.
This seems analogous to how we tackle email spam and general malware. It means that there is almost always a target valuable enough for bad actors to continue trying. However, unlike email (mostly...), package managers are centralised authorities (and anything out-of-band is surely the developers problem?).
My ill-informed feeling is that we might need to change the culture of lazy versioning with rapid releases and focus on stable, deeply scanned versions at registries. There will be some effect of volume and scale so I could be off, but it still seems telling that this impacts high-churn languages more often.
I don't know, I would love a comprehensive article that explores the landscape right now.
Wondering about Mr Bones' Wild Ride and suspecting it might be a reference to the 1991 movie Nothing But Trouble I took a look, and found I had remembered it wrong.
The roller coaster in that movie was called Mr Bonestripper, https://www.youtube.com/watch?v=NEZEgd8GjJc .
Instead it comes from Roller Coaster Tycoon 2, https://knowyourmeme.com/memes/mr-bones-wild-ride .
As for the comparison with spam, there we kind of settled on making people accept spam by vacuuming up their email addresses in pretty much every commercial and social computer network setting, giving it a veneer of legitimacy. I think it is likely to happen in this area too, perhaps some combination of Oracle licensing surveillance agent style software and automated dependency management, i.e. 'solving' supply chain malware by whitelisting some other malware.
There's also Mr. Toad's Wild Ride at Disney parks, https://en.wikipedia.org/wiki/Mr._Toad%27s_Wild_Ride
I always remember that reference from mallrats ... I had never thought to look up what they were actually talking about
People who were born when this meme was current are 13 years old today. Time sucks. Time really sucks.
Node is the Visual Basic of our day, if Visual Basic had the ability to update itself from a thousand strangers, any minute of the day, without the user-developer having any clue what is going on behind the scenes unless they apply the very skills that would have precluded their use of Node/Visual Basic in the first place.
All that ease-of-development is being paid for by ease-of-rooting.
If you think about it, this is actually a new kind of security. Security by numbers. Overwhelm the attackers with so many compromised services and devices that they get a reverse denial of service. It's inspired by nature in herd animals.
Security by giving up
aube (npm/yarn/pnpm drop-in alternative) now has a "jailBuilds" flag that restricts access to network/filesystem access.
https://aube.en.dev/package-manager/jailed-builds.html
But this feels like a cat/mouse game.
> aube (npm/yarn/pnpm drop-in alternative) now has a "jailBuilds" flag that restricts access to network/filesystem access.
1. It seems like the restrictions are only for lifecycle scripts, so wouldn't help if/when the package's actual code had malicious code inserted?
2. Package managers like pnpm seem to entirely block lifecycle scripts by default, so I guess this is an in-between solution.
Still, I guess it's a step in the right direction for those want or need to run lifecycle scripts specifically.
1. the supply chain attacks I've seen are naive. They just leverage postinstall hooks. Malicious code also needs to be executed, not simply installed, so it's a lot less likely that an exploit would happen compared to postinstall since it can't just be buried in a transitive.
2. aube does the same. This is an extra level of protection if you've already whitelisted a package
dino was wired with security in mind, but you can see developers dont tend towards aecurity
Because there is a time and effort cost to swap over to another framework/runtime even if it brings benefits, and security is always considered "good for now" unfortunately.
That is what made Bun popular, and tools like uv/pip, oxlint/eslint, orbstack/docker desktop, and the list goes on. Drop-in replacements where we get 10x with little effort.
Because employers don't tend towards security. In fact, many actively punish somebody for "sandbagging" or simply taking too long if they even suggest a security concern.
What is dino?
Pretty wary of the entire JS/nodejs ecosystem at this stage.
Given general software quality of the js ecosystem, the proliferation of supply chain attacks was just matter of time. I’m curious how other ecosystems will hold (eg Rust)
NPM is an easy target because every package has the capability of interacting with your development environment by default, via pre/post-install scripts. Cargo has something similar with build.rs scripts, as does Cabal (Haskell).
In JVM-land, thanks to binary distribution being the default, the number of packages you can usurp to achieve the same compromise is fairly small; essentially Maven and Gradle plugins. Which is why you should be extremely wary when, say, Sentry tells you to add them as a dependency by setting up their Gradle plugin. Not sure about sbt. Clojure source dependencies can provide "prep" scripts, which are not automatically run as part of a build, but still execute code on your machine.
There's a pattern here: some build tools incorporate dependency-provided code as part of the consumer project's build, and that is a juicy attack surface. Packages which include such code, or are recently updated to include it, should be treated with extra scrutiny.
And now, vibe-coding is just gonna take this to a whole another level.
Yes... The only things I use from the JS ecosystem are {Claude,gemini}-cli which I fear will be compromised. Fortunately I run in them in their own user accounts with no e.g. ssh credentials, but I dead that's not enough especially for Gemini which probably has access to my entire Google account somehow.
Same. At this point I just started using virtual machines for any project that pulls in packages from outside Linux distribution repositories.
JS is a fine frontend language I guess, if you're willing to hand-roll your own code instead of using frameworks and junk. Whoever decided that JS deserves to run as a backend is an idiot who should not be trusted to evaluate languages anymore, lest they decide cobol is a good idea
Collected all the attacks that happened in the npm ecosystem in past 1 year here:
https://npm-supply-chain-attacks-25-26.pagey.site/
Please also collect responses from people, you'd find a pattern: a new attack is launched, people make noise, and later go back to installing packages the same way. Enterprises already use private registries to combat such attacks, vulnerable folks include individual devs and small teams.
Want to.. but I'll need to first integrate social-media and forum scanning in my tool. Haven't done that yet.
But, it's a good feature to have.
Thanks.. gonna integrate now :)
Did Claude generate this site?
The scroll seems to jump all over the place on my device.
yep... fixed it in new version. Thanks for mentioning.
No worries, works fine on my end now!
And this is partly why my development machine is a Raspberry Pi that I can image any time by removing the SD card: https://blog.jgc.org/2026/04/raspberry-pi-as-isolated-ai-cod...
I mean that's lovely, but you still store confidential data, I assume, like credentials on your development machine? This doesn't solve anything but the least important "reinstall the OS post-compromise" step.
On that machine there's Claude's authentication for my account so I can run Claude on it, a public key so I can SSH to the box, and a private key for GitHub access.
One solution I haven’t seen recommended much is to have a Claude instruction/skill that explicitly audits the diff of every upgrade, and force this manual audit as part of your upgrade workflow. This seems like it would work pretty reliably.
This is what many AI supply-chain security startups (like the one that posted the article) are already doing with all NPM packages, so save yourself the Claude tokens. All of these compromises were detected within minutes, but it takes some time (<1 hour) for NPM to unpublish all of the affected packages.
Sorry for my ignorance, but then couldn't we build this into NPM itself? So before a package is publicly available it would be quaranteened and checked.
Super dumb question as someone who has been using some form of AI for dev since 2023:
How does having an AI audit external code help? Can they not be prompt injected to ignore a malicious change?
I guess I am sort of concerned that they are a pretty thin layer and even if you put "DO NOT ALLOW PROMPT INJECTION", it's a bit like saying "make no mistakes". There _is_ a priority between `system` and `user` level messages as I had recalled, so a specifically made tool that has its own system prompt should prevent injection while asking Claude CLI could still allow for prompt injection.
What are your thoughts and experience?
There are prompt guard classifiers that can detect prompt injections, but they are not perfect (false positives, obfuscation) and should be only a part of the defense.
The concern is real and unsolved. I think security researchers have an advantage here because they still can fall back to manual audits if their automated analysis (or scores thereof) is off.
This had been going on for years. I'd expect there would be some kind of solution for now? Like banning scripts and only allowing them to be executed at the explicit user request, for example (not a guarantee but at least a barrier) or somehow restricting them from accessing stuff that does not belong to packaging system. I mean, we have millions of people in the ecosystem, and we keep having this problem for years. Maybe we should ask Claude Code how to fix it (sad laugh)?
The solution is to do exactly what you suggest - separate access. In CI this is a matter of having your "build/test" jobs happen separately from your "deploy/publish" jobs.
The trickier part is dev environments, but ideally you take a similar approach. The place that devs do `npm install` should be isolated from, say, your browser / ssh keys etc.
Package manager support would be an amazing win here since you'd have an easier time managing the isolation but you can do this today.
I'm very glad I never bought into fullstack JS/TS.
My JS is frontend only, served as a compiled bundle off a server that doesn't even have a JS runtime of its own. Whatever random vulnerabilities the frontend contains are limited in blast radius to the user's own browser, and since all frontends should be untrusted anyway, there is no real security risk to the server or backend. No reason to update more than a few times a year, if that.
Combine with obvious basic security practices like pnpm cooldowns + no build scripts. When you upgrade a few times a year, and frontend vulns don't matter, there's really no limit to the cooldown you can set. 60 days, why not.
Also, I feel like a lot of fullstack JS is written by people who started with frontend JS and don't have nearly enough backend experience
JS programmers today are the PHP programmers of 25 years ago. Remember how many SQL injection bugs there was at the time? Little Bobby Tables remembers.
The standards haven't changed; for the vast majority of JS programmers, this is their first programming language and they have no solid foundation of architecture and security.
So what you get are these overly enthusiastic newbies that want to share their latest achievement with the world (say, a function to left pad a string), and why not include a fancy post-install script with emojis that makes adoption even simpler for other complete noobs? And this is the result.
I wrote a little PHP about 25 years ago and hated it, dropped it like a rock as soon as it wasn't my job. Before that basic, assembly, c, and c++. Then scheme, C#, Java, prolog, and others too minor to mention before I started using nodejs. My last company I chose full stack Typescript and it was great. There's a lot to love and a plenty to avoid.
You're not wrong that there's a lot of crap, a massive legacy, and some bad behavior. It is also approachable, flexible, and portable. Even if you sometimes need to say "Wat?!?"
And before PHP it was Visual Basic, and COBOL even further back. I'm convinced trying to teach people to program without them understanding the basics of how computers work will lead to this.
It is not about not teaching them, I am co-graduated with many people who do YOLO JS and AI slop right now. They had to pass the same architecture and microprocessor programming lectures.
Educated or not, there are many programmers who simply don't give a fuck. Unless you make all the IT jobs a regulated profession and enforce significant yearly recertification, the mediocres will dominate. You cannot be nice and inclusive to mediocre programmers and expect good baselines.
> Unless you make all the IT jobs a regulated profession and enforce significant yearly recertification, the mediocres will dominate.
This would merely optimise the industry for people who are good at wiggling through bureaucratic hurdles. It would entrench mediocrity while lopping off the extreme ends of the skill distribution - on both ends.
I came from systems programming, so I think half of what happens in JS is nuts.
A lot of JS is nuts, but most of its basic behaviour that makes it not a good backend language make it an alright scripting language for the web specifically. I wish we would've used something tcl-y instead though, it's much cleaner and the "everything is a string"-ness makes sense for a fundamentally text based medium (the web)
If you drop-in replaced JS today with Lua, you'd have a conceptually similar language with a lot of warts chopped off. Backend devs can just cope with 1-based indexing though.
> compiled bundle
Where does that compilation happen again? Not on the front end and it happens exactly where the exploits have been targeting (dev and build boxes).
Hardly a factor when you're (deliberately) months behind.
It is your development machine's AWS keys they want. The server's keys should be too finly scooped anyway.
Not likely to be a major risk if you update every few months, to some major version that's already over a month old.
I would imagine it's the opposite. Most dev's machines can't query the prod database, for example, whereas a prod server can.
Nope, they've been targeting credentials so they can deploy whatever they like into prod. They prefer the build machine with it's broader rights than the individual dev boxes.
If you serve any third party logic, such as advertisements, your code is just as potentially compromised.
Only the frontend, which is already untrusted. Any risk would be restricted to individual users who used the compromised frontend to communicate something sensitive, and while that's not great, it is significantly better than the backend being compromised. A very significant reduction in blast radius.
Funnily enough, I don't actually think I do serve third party JS, though. Don't serve ads, don't use external telemetry, don't use JS CDNs. I don't think you have to go quite as far as I do, though - I imagine if your ads are Google AdSense or something, you're probably going to be fine.
Another day, another pre/postinstall script executed that could have easily have been prevented by any sane package manager. NPM really desperately needs an 'allowBuilds' style allowlist [1] and 'approve-builds' command [2].
1. https://pnpm.io/settings#allowbuilds
2. https://pnpm.io/cli/approve-builds
After I upgraded pnpm to v11, I set all allowBuilds to false and have not observed any failures. Made me wonder why the packages even need build scripts. My guess is for obscure or old platforms, but for most users running on Linux or Darwin build scripts seem to be unnecessary.
> Made me wonder why the packages even need build scripts
As the name implies it's for building stuff. Most (all?) packages that use C++ FFI with node-gyp need it. A popular package that needs it is re2.
Many newer packages bundle prebuilt native code as transitive dependencies, so build scripts are less needed than before.
> Made me wonder why the packages even need build scripts.
Historically it was to accommodate packages like the original SASS compiler:
https://sass-lang.com/ruby-sass/
Other times it was to avoid shipping binaries due to, erm, safety concerns. The package would include code in a different language, which in turn would compile into a binary library or executable.
Agree. Postinstall scripts should be explicit opt-in, not ambient capability.
Most packages should not need arbitrary code execution during install. And when they do, that should be obvious during review.
The default should probably be: install files, don’t run code.
If postinstall scripts are restricted the people behind these attacks will switch to something else. Package code is executed automatically by Node when imported, which could be a good replacement. It'll probably run when tests run instead but it's still going to run for most people.
Limiting post install as an attack vector is still a good thing.
Node is working on a similar permission model to Deno that allows explicitly granting certain system resource permissions https://nodejs.org/api/permissions.html. Using it should help reduce impact from malicious code, though if you allow wide permissions it's unlikely to help.
> Limiting post install as an attack vector is still a good thing.
If npm got rid of the post install scripts it would permanently break the install process of packages that use it. Affected systems will need to bypass it, stay on an old npm version, or upgrade the packages to versions that work without post install. Meanwhile, attackers switch to a different attack vector and continue.
Who does that help?
I said limit post install, not remove them. Having an allow list in package.json of packages which can run post install would work fine. Pnpm already does this.
Having said that I'm not against full on removal of post install either. It would get more pushback, but would still be possible for people to manually run the post install for the few packages that require it, or to add them as a script in package.json.
pre- and post- install scripts should just be abolished. It should require a special dispensation from npm to even publish one.
Yes, I choose to use pnpm but opt-in safety isn't going to get the developer community to herd immunity.
Another supply chain attack found and blocked in a day. Everyone regularly using npm to install new packages should be using npm's min-release-age setting to avoid package versions that are newer than a few days old to avoid most attacks in practice like this. You can set it to two days with `npm config set min-release-age=2` for example. https://cooldowns.dev/ has info about equivalent settings in other dependency managers like PyPI and Cargo.
Or just use dependencies from 6 years ago and never update them like most companies i ever worked for.
Accidental true leadership
[dead]
I wonder if npm could run a program where package uploads are automatically delayed for ~10min while they get distributed to an ecosystem of third-party code auditing companies for automatic checks. You could have a public leaderboard of which auditors detect problems fastest and most reliably, or even monetary compensation.
In the fictional universe of William Gibson's Sprawl trilogy, it is legal and normal for defenders to go kinetic on cyberattackers. How long until it is simply easier for governments and big business in the countries victimised by these criminal groups, to find the path of least resistance and go after them personally?
We can’t even prevent shop lifting.
Shoplifting isn't a national security matter.
Most malware isn't either, especially with Node.
It’s an issue affecting the nation.
Compared to what?
Are you ok?
Compared to calling in air support on cyber criminals.
How’s your reading comprehension coming along?
wtf does shop lifting have to do with the conversation and why would it take priority vs anything else ? how is that an argument
don't attack people like this on this forum or you will get banned
We can’t even prevent shoplifting, what expectation should there be that we’d be capable of mounting any sort of offence / defence against malware outfits.
I dunno, some people just have no sense of humour.
> or you will get banned
Oh yeah, run and tell the teacher. Dibberdobber.
Tell me, did you grow up in a fatherless home?
[dead]
What they do is already illegal. The problem is finding out who they are and where they live to go and arrest them.
I’m not convinced.
My understanding is that the problem is more that calling in the heavy artillery for what amounts to an annoyance, and maybe some financial harm, is generally considered impolite, even among nations that have conflicting worldviews.
You are allowed to have a conflicting world-view than USA as long as you have nukes, or, turns out, control a strait.
It’s one thing to have a conflicting world view.
It’s another thing to have the entire Iranian parliament broadcast chanting “death to America”.
I’m not even American.
I don’t even agree with half the stuff Trump says or does, but I’m onboard with at least 20% of it, and that’s infinitely better than the last bunch of clowns.
If I were to bomb a school near you, you'd hold no grudge?
Come now.
If that school was being used as cover for a military operation by Islamic extremist who recently shot up a music festival I was at, capturing some of the females and raping them then shooting out their eyes and vaginas.
Enough of the false equivocation.
My coffee was a little two hot this morning FUCKING JEWS!
<eye roll>
Israel is the country where most journalists get killed. But sure, mass killing is completely equivalent to bad coffee.
You people disgust me.
It’s also the most culturally diverse country in the region.
Israel has a Muslim judge in their Supreme Court.
Are there any Muslim majority countries in the region with a Jewish Supreme Court justice?
https://en.wikipedia.org/wiki/Khaled_Kabub
Attackers are just having fun due to abysmal state of npm and some of the insecure design choices by GitHub and GitHub Actions. Every attack gives them credentials which in turn used to stage more attacks.
perfect capitalism writ technology
You making a good point.
Socialism or our right communism would probably fix this.
But only as a second order effect of fucking everything.
3.5 to 6 million Ukrainians died in the Holodomor.
None of them really had any issues with technology. Not so much because they didn’t have any, but more so because they were dead.
When you have AI models finding vulnerabilities by themselves in a for-loop, it isn't long before there is no human to go kinetic on.
Vendor your dependencies, clone or port them where needed, and freeze them. Most good packages these days do not have a deep dependency tree, and we should stop using the ones that do.
I spent a week with claude and codex re-implementing several packages which had dependency trees deeper than I would like.
Most of these packages are trivial to clone.
"But now you're not getting the upstream fixes" they will say.
"So what?" I reply
Wouldn't just having devs pin/not upgrade packages accomplish about as much, have the added benefit that if a package is discovered malicious it would be removed from npm but if already vendored you’d still have the vulnerability on your vendored copy, and pinning versions seems like it would be more likely for devs to do than vendoring?
Pinning the version also avoids the risk of accidentally introducing new vulnerabilities while reimplementing the package yourself.
Not to support the GPs choices (they are not mine) but pinned dependencies can get grumpy over time with endless LTS migrations.
writing your own code is only harbor in this storm. vanilla js cries out into the night sky. will you answer?
That and vendored HTMX. Nobuild. JS on the backend looks more and more like a very bad idea.
I can't wait for npm/github to do literally anything at all to mitigate these attacks. Literally anything. Have we considered a basic WAF-style block on some postinstall script strings? LLM-assisted code scanning on publish? Is there anyone home? No I suspect not.
It still seems wild that npm hasn't gone allowlist-only for pre- and post-install scripts like every other JS package manager (yarn, pnpm, Deno, Bun). Obviously it would be a breaking change that might wreck some developers' day for a little while until that allowlist is built, but it would go a long way to eliminate some of the biggest vectors for these attacks.
... Deno and Bun manage packages? I thought they were just runtimes.
Bun started out with npm compatibility as a key requirement, so has a very npm-like package manager from the beginning.
Deno at first tried to focus only on web-like/browser-like package management with a focus on full URLs ("https://mypackage.example.com/v2/mypackage.js") and importmaps (a JSON file mapping "mypackage" to a URL such as the previous example) and package/file caching over installing. Deno 2 made Node-compatibility a higher priority (partly because of Bun's competition, partly because of complaints that Deno was hard to migrate to piecemeal from existing Node codebases) and one of those initiatives was a more npm-compatible package manager out of the box (that can also speak Node package.json and manage a node_modules directory), even as Deno still encourages for greenfield projects the URL/importmap/caching approach (with the expansion that it also understands `npm:` pseudo-URLs, `jsr:` pseudo-URLs [an alternative package registry with a stronger focus on ESM and Typescript types], and `node:` pseudo-URLs [emulated node internals], beyond just browser-safe `http:` and `https:` URLs).
Third parties can detect compromised packages. It’s ridiculous Microsoft doesn’t.
They will once they figure out how to charge for it (which GitHub is trying according to the security scanning banner they keep showing me).
Why would a corporate company actually demonstrate responsibility and ethicality in its use of the open source commons from which it extracts every last penny and dollar?
Edit: a more suitable strategy is to do the minimal necessary actions for appearance purposes only, as its how to focus and optimize on its interest for revenue for its shareholders.
They can and do indeed detect those attacks, it's just from Microsoft's POV a feature of Microsoft Defender (on Windows and Cloud) they sell:
https://www.microsoft.com/en-us/security/blog/2025/12/09/sha...
https://azure.microsoft.com/en-us/pricing/details/defender-f...
So this is presumably why they will never address this in npm itself.
Maybe they should prove their shit works first.
What a wonderful marketing opportunity! Leave it to Microsoft to blindly ignore it.
No look at the article of this post, it's by SafeDep they are in the same business as Microsoft with their Defender product line. They both publish near identical post mortems with subtle hints at how their product would've defended you against the attack. Why should Microsoft fix the cause instead of selling the cure to each business individually?
Is the complete loss of trust in the platform they want to profit off a better alternative?
It's essential infrastructure there is only one node package manager. I'm not saying it's a good thing, I just describe the systemic reason why it's broken, because that's usually never expressed but its important.
What would be the required budget to host an alternative registry? I'm surprised any GAFAM still hasn't stepped-in and started building their alternative, at least for NPM to up its game in order not to become completely irrelevant.
at amazon, they maintain a private internal registry of packages with approved licenses and audits. this has been in place for several years. i assume other big corps enforce similar policies
Do you know if they are using any product like JFrog for this or rolling their own?
This is Amazon, the company where the stuff they rolled their own now makes more money than the business it was rolled for: https://aws.amazon.com/codeartifact/
If your company not running an internal proxy at minimum you're stupid - you have no audit function for what libraries are being pulled.
Not every company has tons of available funds to run 300 different internal services to "protect" itself.
PSA - you can run something like `npm install -g npm@11.10.0; npm config set min-release-age=3` to update to a version of npm that supports the min-release-age configuration
As similar to 1st wave of Shai Hulud, this also got it through opentionalDependency. intresting
Libc still unaffected. Write native code. Your users will thank you
npm, the ecosystem: https://npm-the-ecosystem.pagey.site/
i run all my stuff in vm's built with nix
not as easy as docker, but i have a few bash scripts that simplify things for me a lot
i hope that this protects me from the sweep attacks at least
I’m honestly at a point where I’m afraid to update any of my project’s dependencies, and I’m also afraid to run the locally without some locked down VM
I use Tanstack in my projects. Last week when Tanstack got compromised, it was only my laziness that saved me -- was thinking about doing pnpm upgrade but got lazy and played some dota... Finished game was just going to pnpm upgrade, opened hacker news and boom! news hit.
Since then, I had set up libvirt/qemu based VM with another Linux running in it specifically for development. Now I run all of docker, kubernetes, IDE, pnpm, uv, etc in that VM and removed them from host. The only write capable secret VM has access to, is my passphrase protected ssh key, which I can quickly revoke from my Github account in case of compromise. Feels much safer now.
I also was at this point, and I decided to add cooldowns to every project.
Yeah, I agree, but then you are at the mercy of whatever vulnerability is found in the current version(s). It just feels like a lose-lose situation no matter what you do.
You can still update to new versions even if the new release is still in cooldown phase. You just have to be explicit that you want the new release.
pnpm audit —fix for example will whitelist releases in cooldown phase when theres a known security issue for a version you currently use.
So long as we insist on everything from a light switch, the vacuum cleaner, security camera, clothes dryer, TV, car, and mobile phone being an always on, always online, Universal Turing Machine we’re not going to find a solution.
If you are using NPM for business, you should be paying for a repository which was checked by antivirus company and not rely on free repository supported by unpaid volunteers.
Because of course it’s npm
Does npm stand for "newly packaged malware"?
Akshully, "Node Packaged Malware."
Needs more recursion.
NPM - NPM Packaged Malware
should be a meme
Every programming language package manager is affected. Any random person can sign up and push packages. They are all equivalent to the Arch Linux User Repository and have the exact same caveats.
How many other languages of any kind have a standard library that is so bizarrely lacking as JS?
Now how many other mainstream languages of any kind have a standard library that is so lacking?
Now how many other mainstream languages have a swathe of front-end developers that suddenly realised they can run code in a CLI or on a server, and spent exactly 0 minutes learning about how to make it not shit?
Now how many other languages get used in scenarios where it makes absolutely zero sense because the developer only knows javascript?
This has nothing to do with standard libraries or popularity. It's about trust.
I signed up for an npm account and pushed packages. Same for PyPI. Same for Ruby gems.
There's no actual reason why anyone should believe I pushed anything but malware in there.
Did you not read point 3?
Developer experience level is a part of the problem.
Yes, and it makes no sense. It's not a "swathe of front-end developers" problem.
Developers in general want to push packages. They don't want to experience friction while doing it. They especially don't want to have to do things like engage with Linux distribution maintainers in order to get their packages into official software repositories. They want to just run $pkgr publish on their repo and that's it. So they invariably end up creating their own distribution mechanisms with zero maintainers involved. Just untrusted randoms making accounts and pushing random stuff. It's easy, so naturally what happens is the repositories get filled with software.
It's only natural to use the stuff that is out there, so the packages get added to projects as dependencies despite the fact none of it is even slightly trusted. Developers hate friction when using libraries too. They very much want to just run $pkgr install x on their repositories and be done with it. They don't want to do things like read the source code or verify that it actually corresponds to what they've downloaded. That's somebody else's problem. On Linux distrubutions, that somebody else is the package maintainer, the exact person the programming language package managers aim to eliminate.
> It's only natural to use the stuff that is out there
Sure, if your entire "community" lives and dies by a nonsensical "don't reinvent the wheel, there is a package for that" chant that would rival the most fervent cult members.
That is quite literally the most persistent cargo cult in the entire computing industry. It's the rule, not the exception. Pretty much every community is guilty of it.
Which other community imports a third party package rather than writing `% 2 === 0` (or it's equivalent) code to test if a number is even or not?
I last added a new dependency to a large project I work on a couple of years ago. So no, not all ecosystems are the same.
The culture makes a difference.
it seems obvious to me the ability to push code to public repositories shluld be tied to real human identity.
Using code from these repositories should be considered the same as licking toilet seats at highway rest stops. There’s something wrong with you if you do it.
are these fixed removed now?
Seems like it is fixed/removed: https://github.com/antvis/.github/commit/cb641113703e531ee43...
Some are still on npm but marked "deprecated":
https://www.npmjs.com/package/size-sensor/v/1.0.4?activeTab=...
As the article states, you can see in the package.json that the optionalDependencies references "@antv/setup": "github:antvis/G2#7cb42f57561c321ecb09b4552802ae0ac55b3a7a"
I'm pretty sure those commits have been removed from github:
https://github.com/antvis/G2/issues/7401#issuecomment-448480...
https://github.com/antvis/G2/issues/7394
another day, another npm hack.
‘No way to prevent this’, Says Only Development Community Where This Regularly Happens
— <https://itnext.io/no-way-to-prevent-this-says-only-developme...>
Yeah funny, but npm is not the only development community where this regularly happens.
The Onion article this joke refers to [1] is funny because there is a very clear and obvious reason why the U.S. has far more gun deaths per capita. This doesn't apply for npm.
[1] https://theonion.com/no-way-to-prevent-this-says-only-nation...
The analogy in npm is dependency proliferation, along with what appears to be weaker community norms around security. To the extent that either or both of these are true, I think the “no way to prevent this” quip is essentially accurate.
Oh?
Edit: please explain. What other community has this rate of attacks? It's possible they are just detected or publicized less, too. Please help me understand what you're referring to.
I'd also like to know who monitors the casualties.
some Pipy packages were also attacked recently. And is even more vulnerable due to many projects using requirements.txt which doesn’t lock sub dependencies
Sure, but the Python community isn't the paragon of software risk management excellence you may think it is.
Both Python and Node users (metaphorically) asked for a loaded revolver... They got a metaphorical high yield thermonuclear device with a large blast radius. (And then they skipped the safety tutorial for the B-83 they just bought.)
I initially read this to mean NPM has a pro-concealed-carry policy, which I don't think it does and I don't think is what you meant.
But... Node's culture does not reward "rational" policies with respect to dependency management in the same way that the US does not reward "rational" policies with respect to gun control *. But US gun control policy is a reflection of the "will of the electorate" -- i.e. there are a lot of Americans who want (or need) to own firearms. In the same way, NPM reflects the culture of high-speed, sili-valley web-devs.
I mention both not to criticize, but to comment it's not the tool that's at fault here, but the users who demanded it evolve the way it did. We moved fast. We broke things. And some of the things that remained broken were sociological: It's easy enough to add PGP/GPG signatures on packages, but whom do you trust? What is the meaning of a signature? Does it mean the signer warrants the package/version is free from defects?
NPM is working as designed. Users wanted the software construction equivalent of a loaded revolver. But we got something that was a bit more like a nuclear weapon with a large blast radius. At least the revolver user would more-likely only shoot their own feet six times (or twelve if they reload.)
[*] I'm trying very hard not to start a flame-war about gun control, I only mean to point out dependency management in node can be as contentious in it's domain as gun control policy is in the domain of US politics. Note that I am not making a pro or con argument about gun control, but only pointing out the issue exists. The word "rational" is intentionally chosen to reflect the fact that people's opinions on gun control and package management are often based on personal, often emotional beliefs (which should not be dismissed.)*
> Yeah funny, but npm is not the only development community where this regularly happens.
What are the other ones? Does this happen with the same sort of frequency?
> npm is not the only development community where this regularly happens
That is like people defending IIS in the early days by pointing out that Apache occasionally had security problems too. Or, back to the gun control analogy people saying “gun control didn't stop Bondi Beach, did it?” or pointing out [incorrectly]⁴ that everywhere that has gun control has knife crime⁴ instead.
> because there is a very clear and obvious reason why … . This doesn't apply for npm.
I disagree. There are a number of reasons that stack together, the four that spring to the top of my head being:
1. Numbers. There are a lot of potential targets you can exploit if you manage to get something into a ecosystem that large. This “being a big fat target” combined with being easy to exploit makes NPM a very juicy target, and encouraging people to use such a target without trying to implement countermeasures for this sort of attack is IMO reprehensible. Numbers isn't a problem in itself, like in the bad old days when IIS was a mess but Apache got (successfully) attacked far less, but they do exacerbate the security issue by multiplying the attack surface area.
1b. A lot of those using it are relatively untrained or just following recipes so do not know how to protect themselves, and may not even update after an attack like this and remain vulnerable for some further time. While this is not NPM's fault, being due to the popularity/commonality thing, it is something those in control of NPM should care about, if, as I believe is claimed, they care about their users⁰.
2. It is an environment where a ridiculous amount of dependencies, nested impressively deep, is practically encouraged, making audit very difficult even for those who try.
3. A number of good suggestions have been made that would mitigate, or at least vastly reduce, the risks. But action on these has, as far as we know, not happened. Sometimes for good reasons, or at least for reasons¹ rather than “just because”/“cost to implement”/“we don't wanna”, and sometimes, well, not. And no alternatives from within those running NPM are being suggested/worked-on, as far as we know² at least.
4. Those who would make most noise about any change, especially a breaking change that affects them in the smallest way, simply do not care about the risk the situation poses to the wider population.
While the gun control analogy might be a little stretched, I think it is relevant enough particularly because of points 3 & 4.
--------
[0] I refuse to use the word community here. This isn't a cosy little village where everyone knows your name and everyone looks out for everyone else.
[1] Them bringing significant breaking changes, or being too complex to implement piecemeal to give time for those breaks are dealt with or otherwise prepared for, for instance.
[2] If something was being looked into, I'd expect it to be announced³ as that would quieten criticisms like these, at least a little.
[3] Maybe not immediately, but this has been a known problem for so long that we are well past immediately.
[4] When it isn't the case that the US doesn't have knife crime, it just doesn't get reported because the gun issues are worse. Like car travel killing more in total then flights, but you don't hear about every car crash. The UK is often given as an example in these comparisons, but if you look at the stats our knife crime rates are lower than the US's - it isn't that other countries have knife problems instead, the US has worse knife problems as well as the guns problem.
The Node ecosystem happens to be more vulnerable for social and software design reasons, it's true. But people need to be aware that PyPI and Cargo et. al. are not in any fundamental way less vulnerable. This will happen there too.
In fact, attacks like Shai-Hulud explicitly attempt to get into PyPI, and have succeeded to a lesser extent.
But aside from the package-size / -complexity issue pointed out in a sibling comment, PyPI also tries a fair bit to monitor for incoming malware (and there's a "report project as malware" button on each project page).
Also, there are no post-install scripts (of course, the code can detect when it's being run for the first time in the installed environment); and pre-install scripts are only included in sdists[0]. So you can easily[1] configure your installer such that you at least won't get pwned at install time, at the cost that some[2] packages can't be installed that way. And then you can go inspect, run a security scanner over, etc. whatever got installed; wheel installations just copy things to well-defined locations and generate simple wrapper scripts by strict rules.
[0]: I.e., when the project is "being built from source", which generally is only necessary when it includes non-Python code directly and the maintainer hasn't pre-built that code for your system.
[1]: Notwithstanding that, with pip, many actions that you'd expect not to get you pwned totally can. Such as, for example, explicitly telling it to download an sdist and not install it; as I discussed in https://zahlman.github.io/posts/python-packaging-3/ .
[2]: In practice, a pretty small fraction of what typical developers would actually care about, at least outside of specific niches. I'm told there are some niches where it's a big problem, but honestly they're lucky that this kind of build-install orchestration sort-of works at all.
To expand on this, PyPI is slightly less vulnerable because Python users tend to install mega packages (such as numpy or django) and do not frequently interact with their package manager. There is also not a culture of using sub-dependencies.
Cargo is essentially the same as NPM though, it's only "safer" because it's less popular.
Although the situation on NPM is extremely uncomfortable, you're probably less likely to get hit if you take reasonable precautions than on PyPI, simply because NPM is getting scanned more often. Most of these attacks on NPM have been detected and pulled days before my min age kicks in. A sleeper attack on PyPI could be devastating.
Isn't this more like a "Macs don't have viruses" type scenario though?
I thought it was «windows has all the viruses» since there are a lot of npm packages and it’s used by a lot of people?
> Says Only Development Community Where This Regularly Happens
We've had such issues on other places as well... Shai-Hulud got into Maven [1] and PHP Composer [2], typosquatters got into Maven [3], and it's not new either [4].
No one is safe from skiddies, much less from nation state actors.
[1] https://thehackernews.com/2025/11/shai-hulud-v2-campaign-spr...
[2] https://semgrep.dev/blog/2026/malicious-intercom-php-package...
[3] https://www.esecurityplanet.com/threats/malicious-jackson-lo...
[4] https://socket.dev/blog/malicious-maven-package-exfiltrates-...
Your example of security issues in Maven is... npm guys setting up processes to auto-publish infected npm packages into the Maven Central repository?
Wake me up when the daily npm security breach headlines are typosquatting stories, not RCE-on-build or RCE-on-upgrade.
> Wake me up when the daily npm security breach headlines are typosquatting stories, not RCE-on-build or RCE-on-upgrade.
RCE-on-build/upgrade can be done in Maven if you manage to compromise one of the major Maven plugins, they run at build time. The thing keeping maven safe for now is that most people pin the plugin and dependency versions, with the obvious side effect that it's truly annoying to get all your dependencies updated.
> The thing keeping maven safe for now is that most people pin [...] versions
Yes, and also the signing of JARs that are uploaded to the repository, and the fact that most release processes are not fully automated, and the batteries-included standard library which reduces the total number of dependencies, and the fact that a run-of-the-mill third-party library can't execute code at build time, and the very small number of people with credentials to publish new versions of major Maven plugins, etc.
NPM by virtue of its popularity and the vagaries of the ecosystem is always going to be a prime target for attackers, and people (maintainers) will always be a weak point.
I agree generally, but I also think it's important to point out that in the NPM ecosystem it is culturally acceptable and even encouraged to install even trivial dependencies to avoid reinventing the wheel. Philosophically I completely agree with this, though practically we see the result and it is not good. The left pad debacle should have been a huge wake up call, but not really much seems to have changed after that.
As long as developers in the ecosystem are cavalier about installing huge chains of dependencies, NPM will be an attractive target for attackers.
Even though we wish it were not so, cultural problems seem to be the hardest technical problems to solve.
Most of these attacks have nothing to do with installing trivial dependencies. It’s usually because the authors npm tokens got hacked; often due to github actions.
The issue is that github actions has too many security gaps that are easy to miss.
No. It is partially due to trivial dependencies. With so many dependencies it is very difficult to evaluate the security posture of all the teams that are inserting themselves into your code.
When I publish commercial software for Unices that use shared object libraries, one of the things we do before publishing is review known vulnerabilities of our 10 dependencies. That is a tractable number. I get a senior engineer to spend time with an intern and step them through the evaluation criteria.
If the team managing a particular library grows lax over time with respect to responding to vulnerabilities, we move away from using that library.
And we can do these things because there are a tractable number of dependencies.
But yes, also GitHub is not pure as the driven slush. I agree with you on that.
Fair point, although when you have dependencies from dozens or hundreds of different publishers, the risk is much higher because it only takes one getting compromised. If instead you only had a handful of core things, there's less surface area
Yeah I was thinking the difference between open source and enterprise to mitigate this is a simple hardware security module right?
If the attackers spearfish folks who hold “keys to the castle” and everything is digital it’s game over no matter what ecosystem.
Those things should be locked down by “something you have” because that’s much more difficult.
What will cryptography buy you in this instance? Do digital signatures imply virtuous behaviour? Does the lack of a signature imply vice?
It’s not just cryptography when a developers password manager gets compromised and all, the hacker is “in”.
If it’s secured by a hardware key, they need to have the key physically.
Two step could work as well with the proper Authenticator.
IPCMSes make it somewhat easy to MitM SMS. If your system poops a cookie in the wrong place it doesn't matter if the secret is in someone's head or if it's in a hardware dongle, like you say... the hacker is "in".
My recommendation for bad guys is to not attack the part of the system where it is strong. Just sniff around a bit until you find the weak part and attack that.
Also remember most devs couldn't use a static analysis tool to save their lives (which is why mythos is relevant.) I suspect that a 15 year old copy of Fortify or CoVerity could find bugs mythos missed.
And if that doesn't work, just start scanning github repos for entropy. That's where the credentials that were accidentally published live.
Yeah that’s fair. It’s only as secure as the weakest link.
That was one of the promises of wasm was sandboxing npm packages independently. Not sure what happened with that or not but I’d be curious to know now we’ve had a lot of recent supply-chain publicity.
For example, if every fetched module is sandboxed and even if they got compromised there would be more protection. It would be more “when” not “if” the package is compromised, nip it in the bud.
But then attackers will target the most exposed packages… :)
Security is hard.
The "No way to prevent this" analogies seem to me to work better for Memory Safety because, as with Gun Safety, the simple fact is that everybody knows how to solve the problem, but one group insists it's impossible.
There is crowing from the "Actually copy-paste is better" people when this happens, but when it's their turn they just jam their fingers in their ears. The memory safety and gun safety problems are the actual problem. Shai-Hulud would ruin your day if it got into the Odin release you used to build your software, or it was copy-pasted into your "vendor everything" C++ project, the choice not to have automation doesn't mean you fixed the problem.
I'll disagree because the primary issues with gun control in the US are:
[1] Guns are a core part of culture for much of America, very deeply so outside coastal cities. Most of the left wing in the US lives in coastal cities and either grew up there or immigrated very recently and does not leave, so this is an alien concept to them, but even in very blue cities like D.C. you would be shocked how many liberal democrats have armories. It is literally amendment #2!
[2] They are already widely distributed and it would be a logistical impossibility to actually enforce gun control.
This is directly analogous to NPM where:
[1] The package registry working the way it does and people quickly installing packages without thinking much is deeply part of JS culture. It doesn't help that JS caters very heavily to as wide of a market as possible, of which the majority is going to be entry level/junior to associate engineers for whomst script kiddying or letting AI install whatever is essentially a way of life. As evidence, this type of thing is not really a problem with derivatives like Bun, especially in mature organizations where it's easy to enforce a minimum 72 hour wait time between publish and installation of a package.
[2] Packages are already widely distributed and part of dependency stacks (e.g. the infamous leftpad) where it is a logistical impossibility to change how things work.
I also view startups and companies like Vercel as essentially the NRA here, Next.js has taken over huge swathes of the ecosystem and highly encourages dependency-maxxing.
Another direct analogy: proponents of gun control say they are unnecessary for self defense (esp. because law enforcement is good now), too heavy duty to begin with, and fundamentally dangerous.
Similarly I would criticize dependency-maxxing as unnecessary for capability (esp. because AI is good now), too heavy duty to begin with, and fundamentally dangerous.
The whole reason this joke works is because of exactly your belief that somehow you're different and the solution which works for everybody else can't work for you. Charlie is always going to try to kick the ball and Lucy is always going to pull it away and Charlie will never learn from this experience no matter how often it is repeated.
Gun deaths in the US are just the cost of doing business, and business is booming (for some).
Sad.
>The "No way to prevent this" analogies seem to me to work better for Memory Safety because, as with Gun Safety, the simple fact is that everybody knows how to solve the problem, but one group insists it's impossible.
I'm not following. Whats the 2nd Amendment equivalent for memory safety? The amount of C/C++ in production or something?
> A well regulated Memory Operation, being necessary to the security of a free Computer, the right of the people to keep and bear Pointer Arithmetic, shall not be infringed.
(maybe workshop this)
> the simple fact is that everybody knows how to solve the problem
Don't make braindead My First C Program mistakes?
"A well administered supply chain, being necessary to the freedom of an open internet, the right of the developers to keep and bear hundreds of uninspected transitive dependencies, shall not be infringed."
Language exclusive package managers like this are nightmares for security, but npm simply does things so poorly I feel like they wanted something insecure.
What about language exclusivity makes a package manager a security nightmare?
Now I have to wonder if someone is targeting Homebrew too, thanks
That isn't a language exclusive package manager. I mean things like npm and pip. It isn't necessarily that they're language exclusive, it's that they all tend to have features that aren't good security wise because they aid in developing in that language
after running the original version of this into the ground we can run this one into the ground too
[dead]
[flagged]
[flagged]
[dead]
[dead]
[flagged]