Somebody is spamming kernel mailing lists under the name Marian Corcodel with a 26 MByte message multiple times per day containing a collection of nonsensical patches. Looks AI-generated, perhaps with the intention to poison LLMs. This has been going on for a few days now.
I'd warn HN users not to click on that link simply because it will load a 26Mb message that will likely cause quite a strain on kernel.org's servers if everyone here does it.
I was curious how much of an impact HN could have. Napkin math:
HN gets 24M views a day. Assume those views are evenly distributed across the front page (they aren’t), and that’s about 1M views for each front page post, assuming each user clicks on one post.
By the rule of 10s (also not exact), there are 10x less views on comment threads. So assume around 100k views on a comment thread as a theoretical average.
If everyone in this thread clicked on the link, that would be 2.6 TB of transfer across the day. But by the rule of 10’s we have to assume 10x fewer people will interact (upvote, click, anything) than view. So we’re down to 260GB transfer over the course of a day.
I wonder how close that is. It seems plausible that a link in the top comment of a thread could garner 10,000 clicks.
That’s still about one click every 8 seconds, which at 10Mbit/s would indeed overwhelm the server by a factor of about 2.5x. But I clicked through and it loaded in just a few seconds, so presumably the pipe is faster than 10Mbit/s.
Another caveat is that many websites are already several megabytes, so it seems strange that 26Mb would be the breaking point for a reasonable web host.
> There's no stats page but last I checked it was around 5M monthly unique users (depending on how you count them), perhaps 10M page views a day (including a guess at API traffic), and something like 1300 submissions (stories) and 13k comments a day.
> The most interesting number is the 1300 submissions because that hasn't grown since 2011 - it just fluctuates. Everything else has been growing more or less linearly for a long time, which is how we like it.
Does a 26MB message actually cause noticeable strain on the server much beyond loading the page? I would think serving a contiguous 26MB chunk would be relatively similar to say 20 normal sized messages.
Way off. I went to an arbitrary message on lore.kernel.org. Firefox's network inspector says 7.37kB was transferred, including stylesheets. 26MB is roughly 3500x 7.37kB.
IA's infra is slightly better for big loads though, they tend to just have higher latency rather than aborted/timed out requests, for better or worse. It can be bit slow, but as long as you're ready to wait, you'll eventually get the response. Usually hosts just cut you off with a hardcoded timeout instead, which for people on high latency/low bandwidth connections can be super fun.
Actual context: Linux 7.1-rc4 release, Linus remarked on a specific documentation change.
The Register somehow turned this into an "article" that says a lot less with roughly the same number of words, and provides "context" by linking to a number of unrelated articles.
I think people are used to bias in mainstream media, but theregister seems to show it to computer/tech folks. I don't mind, I like opinionated stuff and can make up my own mind, but I wonder what folks with less context/experience think when they read some of their articles.
(and at least weird isn't like uncanny-valley-ai-written-weird)
"Torvalds' remarks contrast with recent comments from fellow kernel maintainer Greg Kroah-Hartman, who recently told The Register that AI has become an increasingly useful tool for the FOSS community."
Does it? Both points can be true at the same time.
“AI tools are great, but only if they actually help, rather than cause unnecessary pain and pointless make-believe work,” he wrote. “Feel free to use them, but use them in a way that is productive and makes for a better experience.”
So I think the closing remark from the register isn’t really appropriate given the context from the quotes they pulled.
the problem here is that many of the submissions are not "make-believe work" but actual existing security issues
it's just that in the past people most times didn't find security vulnerabilities independently of each other without knowing about the others en mass
worse it's non trivial to dedup on the submitter side, nor on the receiver site (as long as we stay with a classical mailing list format)
and while this might be fixable with an AI auto grouping duplicates etc. getting that right is _hard_ especially if we consider that there can be a lot to gain for an adversary to use prompt injection and similar to cause an effective "hiding" of "useful" security issues (e.g. by wrongly causing them being labeling as duplicate).
In addition to all the technical problems this causes some other problems: 1.) additional cost you can intentional (maliciously) increase 2.) dependence on some LLM provider 3.) trust problem wrt. the used LLM provider. Some of this can be avoided by running open models on sponsored owned hardware, but at the cost of often outdated LLM tech, higher cost, now needing to maintain additional hardware etc.
> the problem here is that many of the submissions are not "make-believe work" but actual existing security issues
Not exactly, the submissions are reports about actual existing security issues. They are make-believe work because everybody has access to AI, and anybody could have done it. Deduping is not productive work, it's a search for productive work.
Instead of spamming bug reports generated by AI, people should spam cash or token credit of some sort so the project can generate these themselves. The real unnecessary part of the entire process is the submitter. There's no need for an AI middleman.
If somebody comes up with some witty trick that gets an AI to find a bug that it wouldn't have found on its own, submit the prompt.
I will argue that ON AVERAGE, humans are lazy, and will use LLMs to generate walls of text and code. We like the easy way out - just pop a pill. Here we have a technology that can finally help us manage the crippling firehose of data, and instead, we are going to make it much worse. As expected.
A few of us will actually use these tools to reduce toil and achieve something useful.
I mean, they are two (of many) contrasting results of AI. The writer didn't say "contradict". But I agree they probably could have chosen better wording.
The problem is people trying to get individual credit for merely running a script that spams a mailing list. Many of those people are likely not even C programmers or programmers at all.
Without the immense personal reward and recognition and job offers as a motivation, the problem will disappear.
The problem will also disappear with time as the people lauding and celebrating and hiring security researchers of the past will quickly abandon LLM generated spam as a positive signal; running a prompt that sends spam is, if anything, a strong negative indicator of infosec ability and skill.
LLMs are a tool. Like all tools, most people can't or won't use them responsibly or profitably although they are useful in the correct hands.
People will spam continuously anyways, for whatever motivation they have (trolling, ai poisoning, hate, etc) and it will become harder to filter through the noise.
If you have emails attached, I can confirm that Linus in fact posted something and its probably worth reading. You can gain reputation and I can filter my attention based upon those who have reputation.
It's not clear to me that the spammer discussed in this thread is motivated by recognition and job offerings. I don't think this solves it.
I really like this idea. Removes the fame, blog & resume/job hunting incentive from it.
The kernel isn't the only OSS project with this issue either. Requiring submissions & issues to be anonymous could help a ton of other open source projects currently drowning in AI slop issues.
Will never understand why some people prefer mailing lists to do development, it always feels like the most convoluted way to hold a discussion, especially if there are multiple topics at the same time.
It probably doesn't really change that much in this scenario but with a forum or any other topics-based platform you can at least just close and ignore these things without it affecting everyone else.
A good mailclient allows a skilled user a much more efficient communication than most forums.
> It probably doesn't really change that much in this scenario but with a forum or any other topics-based platform you can at least just close and ignore these things without it affecting everyone else.
True, external moderation is a benefit of centralized platforms, but a mailclient allows personalized moderation, which allows with a well organized list to only filter out anything you are not interested in. Usenet had the benefit of both, a centralized platform with moderation, and powerful clients for further personalization. Too bad it died for most usages.
Usenet is probably better, but to a rough approximation, nobody has access to a usenet feed anymore.
Mailing lists allow people to use threads if they want (assuming nobody thrashes the threads headers by using terrible email software from Microsoft/Google), and also allows people to read from the firehose if the want. And there's plenty of threaded web views of mailing lists available for lurkers.
I remember when it was popular, computers had limited storage and bandwidth, and you had to "sip" usenet articles because it was a giant firehose of data, mostly hand-typed text. (excepting alt.binaries.*)
You can still use it. Eternal September provides free access to everything but binaries. There's a number of relatively active groups and some that'll get a post or two a month
Show me a forum or topics based platform that handle threads as good as proper mail clients? Don’t mistake the poor HTML view for how managing threads with thousands of replies look like.
Local filtering is the key to ignoring threads you are not interested in. Depending on the client with 2 or 3 keystrokes you are ignoring the whole thread or this particular sub branch of it and automatically jumping to the next interesting, unread message.
Because ther don't have to keep switching from discussion client to discussion client or whatever tools the use for each project they are involved with, at the same time being distracted by adverts, geegaws, random emojis and other kinds of nonsense.
They are simply more efficient and more importantly censoring is done by the user themselves, not by politically motivated admins who ban discussions based on their ideologies and whims.
It’s an open format and with basic tools, you can create a very good pipeline to consume the information. There’s no ceiling on the convenience and automation. Gmail and other webmail client are not a representation of good email workflow.
This is the reason behind essentially every reply I've ever seen to this question.
"I like it this way because it's always been this way and once you change your entire email workflow and customize your email client, it's almost as good as PHPbb"
Forums are built for threads and are immediately visible and accessible for everyone, not just people who want to spend their limited time dicking with email clients.
Mailing lists are the proto-discord: knowledge locked away from the public behind a special frontend and elitist attitudes. It's only better because the list is technically visible, but only in the worst, most low-effort way possible. You dump a raw txt copy of the entire thread unstructured onto the user and make it their problem to figure out. After all, your email client makes it easy to read, so why should you care about what anyone else needs?
I understand it is out of fashion, but technologically more advanced are systems that use well defined interfaces and allow pieces to be exchanged easily. After all, we’re communicating over a number of open protocols here. A forum merges all elements into one silo. I prefer web over AOL/Compuserve. If you want a forum-like interface, there’s no technical reason why this couldn’t be done on top of a mailing list. In fact, Discourse and others attempted it.
This discussion has been happening since forever. And also the idea that it serves anyone to complain how others are obviously doing it wrong, without even attempting to understand why they’re doing it a certain way. And then be irritated when the response is negative, and labeling others as elitist for using and providing open platforms over decades and not silos.
If you don’t know, feel free to ask. And then suggest (or provide!) improvements that factor in current requirements and goals instead of dismissing them as stupid.
Life advice: if you want anybody to change what they do, you need to first understand why they’re doing it, and then offer suggestions based on that understanding that improve it with them. Otherwise you’re going to continue to recreate your own victim position, and an “elite” position that you will never belong to.
I think it's unsurprising that the people maintaining the tangled plumbing of the clunky, customizable, fiercely independent operating system most popular with highly-opinionated power-users prefer the clunky, customizable, open format that's a little inconvenient for non power-users but allows them to set up their own personal bespoke client.
You like forums for the same reason - because you grew up with them.
Today's preteens will be pining for Discord in the 2050s era of AI Neuralink vibe-telepathy. They'll say once you change your entire workflow and customize your client, phpBB is almost as good as Roblox chat or TikTok comments.
This could be read as reductive, presumptuous, ignorant, and insulting.
At the same time, it's often technically true, but for a good reason that you neglected to mention:
Those old tools tend to be very capable email clients, not web apps with their awkward attempts to simplify complex conversation structure. A good email client can handle large, high-traffic, frequently branching, long-lived threads with ease. All the web forums I've ever used fail miserably here.
The people who are tasked with participating in large scale discussion groups (like the LKML) know this through experience. They prefer email because it works better. It makes their lives easier. It helps them to be more efficient, which is absolutely necessary given the sheer volume of messaging that they handle.
Yes, a specialized tool is required to get these benefits, just as a specialized tool is required to make web server output easily readable. Thankfully, these tools have existed for decades.
maybe the old tools are prevailing for a good reason.
I prefer people to email me because half of the time they figure out their problems while writing them.
it's not an absolute rule but people who don't do their homework gravitate towards calls and messaging because they just don't prepare their questions.
asynchronous communication puts the burden on the sender, where it belongs.
Many "new" tools are basically the old tools with a few additions like emojis and pictures.
I do think upvoting was one "technology" that did make discussion forums different than mailing lists, but that put them under control of someone and had different drawbacks. hn is a good example, reddit is one devolving example
I think it's time the report-only intake should stop. If a reporter can't reproduce at least one use case or can't summarise it in two sentences, it should be classified as spam. LLMs write beautiful reports, it's just that sometimes it doesn't bear anything resembling the truth.
couldn't an llm be used for verification like we're seeing some OSS projects do? Some projects are moving so fast, its almost certain there's little human involvement.
> Torvalds' remarks contrast with recent comments from fellow kernel maintainer Greg Kroah-Hartman, who recently told The Register that AI has become an increasingly useful tool for the FOSS community
Thats kinda a misrepresentation. They are talking about two different things. Linus is trying to point out incorrect use of a tool while GKH is praising a correct use. This sentence felt weird at the end of the article, kind like rage bait. And I took it :P.
It seems like LLMs are actually pretty good at the sorts of things needed to manage a high-volume mailing list (summarizing, looking for dupes, sentiment, flagging things, etc), even if only as augmentation for human eyes.
That said, I get why this would rankle a lot of the folks involved.
That's just a security/protection racket with extra steps: "Someone is paying us to hurt your business/site; pay us money to defend your site against our attacks".
I like to imagine that LLM's ability to optimize code is like an extension of the training-loop in deep learning. The loss function is some kind of metric representing security and/or performance (or the lack of it) of the code and we use the LLM as the gradient/diff generator to iterate in batches over the code and fine tune it.
Imagine the current state being for the most part a collection of local maxima in security. To push the system in a more optimal state, you either need skilled people and time to overcome the barrier to a new local maximum or you throw AI at it and evaluate whether you land in a more optimal state.
I think after some time of turbulent exploit/patch cycles we will reach a stable state again, where the code converges against a new local minimum that even with AI requires significant effort (time and tokens) to overcome. Or ideally a global maximum.
With time, the LLMs improve, so the diffs/gradients get better and we will be able to reach optimal points for any software faster.
My problem with the idea is that apparently it is assumed that OSS contributors and especially maintainers will generously donate their time to get this machinery into a state that makes the optimization loop work well - just for the AI labs to turn around and sell access to the optimized models for increasingly larger amounts of money.
AI generated code can be great. Hand rolled code can be bad. The rules are the same in both cases. Make sure your code changes are focused (no random changes just because you happen to be in the file/dir or notice something) and make sure you don't break anything else along the way.
Mailing lists are a bad fit for this problem once spam starts taking over. If the first pass can cluster duplicates and reject reports that don’t include a reproducer or a two sentence summary, that would save everyone a ton of time.
The bug reports are helpful! Many Linux developers including Linus, Greg Kroah-Hartman, Andrew Morton, Chris Mason, and Willy Tarreau have all commented positively on all the legitimate problems that are being found with LLM. Here is just one example article[1].
This is just a workflow issue. In the past it was very rare for multiple people to find and report a security vulnerability at once, so it made sense to keep the discussion private until they were ready to release a fix. With AI that is happening all the time, so it makes more sense for the discussion to be in public to avoid duplication. So they changed the policy accordingly. That is it.
I think this will sort itself out over time, as people realise that it’s no longer impressive whatsoever to land an AI-assisted PR to the Linux kernel.
An open visibility tracker would be a goldmine for finding new exploits before a fix is even available.
From what I’ve seen many of the AI bug search operators are newer to security research. They’re burning their tokens trying to find kernel bugs as their claim to fame before other people with AI tools find them first. They don’t spend time de-duplicating their own bugs.
Some of them may not be coming from real people. There are honeypot repos that are entirely fake and only have folders of simple files with clear security problems. They collect automated reports they get from all of the AI bots that people are running.
The issue highlighted in Linus's message isn't that the LLM is hallucinating fake bugs; it's that 100 people running the same LLM on the same codebase find the same real bug 100 times, and if they all send it to the private security mailing list, it's (1) unmanageably high volume and (2) stupid security theater [because by definition any bad actor with the same LLM would find that bug — it's effectively public at that point].
You don't need an LLM to deduplicate bugs, just categorize by files affected. The real security problem is LLMs have a ~499/500 false positive rate and the new 'security research' post this slop and DDoS the mailing list.
Analysis of changes in the source code is already heavily analyzed by both threat actors and researchers in a desperate attempt to find code which likely was changed as a result of a bug or exploit patch.
If the AI is awesome at identifying security bugs in the linux kernel, it likely can also identify if the thing it's found is similar to something that is already found in the security mailing list?
Or, put another way -- what flags the duplicate? The filer or the system? If my cheese factory is measured by the volume of cheese instead of the quality, I'll churn out the cheese even if it's sloppy duplicated cheese. And that is the case if a person has to flag a new ticket as "same as this" or not.
What's that law that says that any sufficiently large problem turns into a moderation problem?
The problem is that the tech companies are paying their research/marketing departments for headlines that go "Researcher uses powerful new Saga 6.2 release to find 597 kernel vulnerabilities! (Can your company afford NOT getting their $1000/month subscription?)", not for headlines that go "Researcher spends $50.000 to find 597 bugs, then spends $25.000 figuring out 540 of them are duplicates".
Unless the kernel community starts banning & publicly shaming repeat offenders, there's zero incentive for them to put any effort in filtering out duplicates. They are mostly doing it for marketing after all, not out of a genuine interest in making the kernel better.
> “AI detected bugs are pretty much by definition not secret, and treating them on some private list is a waste of time for everybody involved – and only makes that duplication worse because the reporters can't even see each other's reports.”
Ah; so it _is_ a tool problem. It is _also_ a moderation problem.
One could ban orgs that flood the zone with AI generated trash, but is there some potential middle ground where there are sets of filters to identify duplicated bugs, and possibly just internally dump "AI spam" to a lower queue?
This seems like the sort of problem I'd addressed in the 90s with killfiles and spamassassin. In other words, can't the ingestion just go through some filters to shield the humans at the end of the pipe?
And with a mailing list you don't even have to do that! The problem doesn't really change, because you have to figure out whether it is a duplicate before you can mark it as duplicate, and that's the 'managing' part of 'unmanageable'.
So ... first, AI slop is killing mankind slowly. Skynet is winning here.
On the other hand ... IF the bug report is real, and let's assume that AI slop reports at the least a few bugs that are indeed real, then I really think it should not make a difference WHO or WHAT reports these bugs. I would not disagree on fake bugs or bogus bug reports wasting time of humans, but this is a quality difference then. Surely people can tweak AI models to be better at finding bugs too. Besides, they should auto-fix that. Is AI still too stupid to fully replace humans? Other than killing them with spam, as it does right now.
I hate to be "that guy" but there's a reason why most of the industry stopped using mailing lists for things like this. Extremely impressive that Linux lasted this long.
Nonsense advice, he's just asking for duplicate slop patches too this way.
It's a catch 22. Why not make a separate list for AI generated reports that can be subscribed to instead? If the claim is that these are not private anyhow, no reason not to, and then a reasonable expectation could be held against submitters to check against existing reports.
That is unless it is still absolutely sensitive, in which case the only way forward that I see is to start using AI for triaging and duplicate detection as well.
I'd really like maintainers to get their hands dirty with AI agents as well to help speed up the reviews.
Over the last year there have been way too many stories and Twitter posts like these.
Yes, maintainers are overloaded, but that's only because we haven't yet built the tools to support them.
Other than such statements, I would, as a builder like to hear the sorts of tools and requirements maintainers are looking for which would make their work easier!
I'm confused by your answer, the previous post doesn't seem to be about vibe-coding at all.
It seems to be more about:
1. auto grouping duplicate security reports
2. auto validating if they are likely viable or likely nonsense
3. auto checking if they have recently been patched
4. auto assessing if they likely "invalide" for other reasons (e.g. they are for a very old long time no longer maintained Linux version, out of tree drivers, etc.)
I mean practically all of that isn't trivial to get working in a way appropriate for the Linux security mailing list and comes with many not so obvious complications. But also non of that is vibe coding and in most cases this is is more about AI doing a per-assemsment of send security issues to speed up the review of them, then it is about the AI doing the final decision.
At the end of the day, we would rather have a more stable and bug-free kernel than not.
It's not that much work for me anymore to report and even fix that obscure monitor driver bug that sometimes causes my machine to bootloop, unless I boot without graphics and start the XOrg server manually.
I often find myself surprised at how easily frontier models are able to find bugs across abstraction layers, that only original authors can comprehend. We need more positivity around these contributions as well.
The world doesn't need to support the projects and research areas that interest you. How about we do something better: No one is allowed to say or write anything about AI or AI generated slop until AI is 100% perfect and produces zero errors and does everything with perfect efficiency.
AI trash like this is like showing up to a baseball game with a pitching machine and demanding that they let you join in and be the pitcher using your machine. Just because your slop cannon is fun and exciting to you doesn't make anyone else obligated to join your club just because you fired your slop on them.
Fun fact (or not so fun if you're a subscriber):
Somebody is spamming kernel mailing lists under the name Marian Corcodel with a 26 MByte message multiple times per day containing a collection of nonsensical patches. Looks AI-generated, perhaps with the intention to poison LLMs. This has been going on for a few days now.
https://lore.kernel.org/all/CAGg4U=GNtCObd_Nbm_1Rr5FEvPb69Yz...
I'd warn HN users not to click on that link simply because it will load a 26Mb message that will likely cause quite a strain on kernel.org's servers if everyone here does it.
I was curious how much of an impact HN could have. Napkin math:
HN gets 24M views a day. Assume those views are evenly distributed across the front page (they aren’t), and that’s about 1M views for each front page post, assuming each user clicks on one post.
By the rule of 10s (also not exact), there are 10x less views on comment threads. So assume around 100k views on a comment thread as a theoretical average.
If everyone in this thread clicked on the link, that would be 2.6 TB of transfer across the day. But by the rule of 10’s we have to assume 10x fewer people will interact (upvote, click, anything) than view. So we’re down to 260GB transfer over the course of a day.
I wonder how close that is. It seems plausible that a link in the top comment of a thread could garner 10,000 clicks.
That’s still about one click every 8 seconds, which at 10Mbit/s would indeed overwhelm the server by a factor of about 2.5x. But I clicked through and it loaded in just a few seconds, so presumably the pipe is faster than 10Mbit/s.
Another caveat is that many websites are already several megabytes, so it seems strange that 26Mb would be the breaking point for a reasonable web host.
Don't forget scrapers. Scrapers can be biased towards top posts and comments.
Arn't AI agents worse than scrapers now since they're basically a DDoS that runs over and over where scrapers will actually cache data.
> HN gets 24M views a day
This is available info?
https://news.ycombinator.com/item?id=33450094
2022 from dang:
> There's no stats page but last I checked it was around 5M monthly unique users (depending on how you count them), perhaps 10M page views a day (including a guess at API traffic), and something like 1300 submissions (stories) and 13k comments a day.
> The most interesting number is the 1300 submissions because that hasn't grown since 2011 - it just fluctuates. Everything else has been growing more or less linearly for a long time, which is how we like it.
Plenty of people deliberately posting to HN have their servers overwhelmed.
It's mirrored by Akamai, which is designed to repeatedly serve the same thing over and over. It won't really hurt anyone.
Does a 26MB message actually cause noticeable strain on the server much beyond loading the page? I would think serving a contiguous 26MB chunk would be relatively similar to say 20 normal sized messages.
Way off. I went to an arbitrary message on lore.kernel.org. Firefox's network inspector says 7.37kB was transferred, including stylesheets. 26MB is roughly 3500x 7.37kB.
Data transferred is not what generates load. sendfile() is about the lowest-overhead thing a web server does.
https://web.archive.org/web/20260518134447/https://lore.kern...
I don't think needlessly straining the Internet Archive's servers is any better.
IA's infra is slightly better for big loads though, they tend to just have higher latency rather than aborted/timed out requests, for better or worse. It can be bit slow, but as long as you're ready to wait, you'll eventually get the response. Usually hosts just cut you off with a hardcoded timeout instead, which for people on high latency/low bandwidth connections can be super fun.
IA's resources are very limited as is. There is so many people (emulation/roms) YouTubers linking to Archive.org downloads for full ROM Sets.
It's a big problem. Donate to Archive.org if you can!
Will clicking on this link download a 26MB message putting extra load on archive.org's servers?
Thank you for the warning. I rarely click on links these days though; only exception I make for HN links for main articles.
How do you navigate the web, everything is CTRL+L then manually type the address, or you have some fancier solution?
the web is useless outside of hn
90% of it yeah, but the 10% is still worth it, like HN.
The page is gzipped in transit - only 5 MB of traffic are generated.
> perhaps with the intention to poison LLMs
How does that work?
This is just nonsensical changes and slurs, but particularly degenerate input data can cause big issues in training:
https://x.com/gabriberton/status/2051873677998956851
Here's the actual mailing list post: https://lore.kernel.org/lkml/CAHk-=wi+JvcuKF2NaD_rGiYrwkR6rx...
Actual context: Linux 7.1-rc4 release, Linus remarked on a specific documentation change.
The Register somehow turned this into an "article" that says a lot less with roughly the same number of words, and provides "context" by linking to a number of unrelated articles.
here is what seems to be the relevant documentation: https://docs.kernel.org/process/security-bugs.html
see "If you resorted to AI assistance to identify a bug, you must treat it as public." and https://docs.kernel.org/process/security-bugs.html#responsib...
The Register has always been a... weird 'news' source, but they've gotten significantly worse over the last year or two.
I think people are used to bias in mainstream media, but theregister seems to show it to computer/tech folks. I don't mind, I like opinionated stuff and can make up my own mind, but I wonder what folks with less context/experience think when they read some of their articles.
(and at least weird isn't like uncanny-valley-ai-written-weird)
"Torvalds' remarks contrast with recent comments from fellow kernel maintainer Greg Kroah-Hartman, who recently told The Register that AI has become an increasingly useful tool for the FOSS community."
Does it? Both points can be true at the same time.
Linus also said
“AI tools are great, but only if they actually help, rather than cause unnecessary pain and pointless make-believe work,” he wrote. “Feel free to use them, but use them in a way that is productive and makes for a better experience.”
So I think the closing remark from the register isn’t really appropriate given the context from the quotes they pulled.
the problem here is that many of the submissions are not "make-believe work" but actual existing security issues
it's just that in the past people most times didn't find security vulnerabilities independently of each other without knowing about the others en mass
worse it's non trivial to dedup on the submitter side, nor on the receiver site (as long as we stay with a classical mailing list format)
and while this might be fixable with an AI auto grouping duplicates etc. getting that right is _hard_ especially if we consider that there can be a lot to gain for an adversary to use prompt injection and similar to cause an effective "hiding" of "useful" security issues (e.g. by wrongly causing them being labeling as duplicate).
In addition to all the technical problems this causes some other problems: 1.) additional cost you can intentional (maliciously) increase 2.) dependence on some LLM provider 3.) trust problem wrt. the used LLM provider. Some of this can be avoided by running open models on sponsored owned hardware, but at the cost of often outdated LLM tech, higher cost, now needing to maintain additional hardware etc.
> the problem here is that many of the submissions are not "make-believe work" but actual existing security issues
Not exactly, the submissions are reports about actual existing security issues. They are make-believe work because everybody has access to AI, and anybody could have done it. Deduping is not productive work, it's a search for productive work.
Instead of spamming bug reports generated by AI, people should spam cash or token credit of some sort so the project can generate these themselves. The real unnecessary part of the entire process is the submitter. There's no need for an AI middleman.
If somebody comes up with some witty trick that gets an AI to find a bug that it wouldn't have found on its own, submit the prompt.
So if a thing is good then it is good, but if a thing is bad then it is bad? Got it!
I will argue that ON AVERAGE, humans are lazy, and will use LLMs to generate walls of text and code. We like the easy way out - just pop a pill. Here we have a technology that can finally help us manage the crippling firehose of data, and instead, we are going to make it much worse. As expected.
A few of us will actually use these tools to reduce toil and achieve something useful.
Torvalds didn't say AI isn't useful. He is saying everybody use AI to file same duplicate bug report causing extra churn.
AI can amplify your intelligence just as easily as it can amplify your stupidity. All while telling you how smart and brilliant you are.
[dead]
I mean, they are two (of many) contrasting results of AI. The writer didn't say "contradict". But I agree they probably could have chosen better wording.
yeah reading the original post linus seems to be pretty constructive in his words and open to AI in a common-sense way.
Make it anonymous and the problem will go away.
The problem is people trying to get individual credit for merely running a script that spams a mailing list. Many of those people are likely not even C programmers or programmers at all.
Without the immense personal reward and recognition and job offers as a motivation, the problem will disappear.
The problem will also disappear with time as the people lauding and celebrating and hiring security researchers of the past will quickly abandon LLM generated spam as a positive signal; running a prompt that sends spam is, if anything, a strong negative indicator of infosec ability and skill.
LLMs are a tool. Like all tools, most people can't or won't use them responsibly or profitably although they are useful in the correct hands.
Won't you have the adverse affect?
People will spam continuously anyways, for whatever motivation they have (trolling, ai poisoning, hate, etc) and it will become harder to filter through the noise.
If you have emails attached, I can confirm that Linus in fact posted something and its probably worth reading. You can gain reputation and I can filter my attention based upon those who have reputation.
It's not clear to me that the spammer discussed in this thread is motivated by recognition and job offerings. I don't think this solves it.
I really like this idea. Removes the fame, blog & resume/job hunting incentive from it.
The kernel isn't the only OSS project with this issue either. Requiring submissions & issues to be anonymous could help a ton of other open source projects currently drowning in AI slop issues.
Will never understand why some people prefer mailing lists to do development, it always feels like the most convoluted way to hold a discussion, especially if there are multiple topics at the same time.
It probably doesn't really change that much in this scenario but with a forum or any other topics-based platform you can at least just close and ignore these things without it affecting everyone else.
A good mailclient allows a skilled user a much more efficient communication than most forums.
> It probably doesn't really change that much in this scenario but with a forum or any other topics-based platform you can at least just close and ignore these things without it affecting everyone else.
True, external moderation is a benefit of centralized platforms, but a mailclient allows personalized moderation, which allows with a well organized list to only filter out anything you are not interested in. Usenet had the benefit of both, a centralized platform with moderation, and powerful clients for further personalization. Too bad it died for most usages.
Is there a demo of such communication on YouTube, or at least some article with screenshots?
http://www.kroah.com/log/blog/2019/08/14/patch-workflow-with...
Because it is an open and widely distributed system that is difficult to take down or otherwise have an extended outage.
"Will never understand why some people prefer mailing lists to do development ..."
The people who have this preference are processing the mailing lists with a highly specialized mailtool, not a web browser.
If you have only ever accessed email with a web browser it is not surprising that you find the mailing list format weird.
Usenet is probably better, but to a rough approximation, nobody has access to a usenet feed anymore.
Mailing lists allow people to use threads if they want (assuming nobody thrashes the threads headers by using terrible email software from Microsoft/Google), and also allows people to read from the firehose if the want. And there's plenty of threaded web views of mailing lists available for lurkers.
I miss usenet.
I remember when it was popular, computers had limited storage and bandwidth, and you had to "sip" usenet articles because it was a giant firehose of data, mostly hand-typed text. (excepting alt.binaries.*)
You can still use it. Eternal September provides free access to everything but binaries. There's a number of relatively active groups and some that'll get a post or two a month
Show me a forum or topics based platform that handle threads as good as proper mail clients? Don’t mistake the poor HTML view for how managing threads with thousands of replies look like.
Local filtering is the key to ignoring threads you are not interested in. Depending on the client with 2 or 3 keystrokes you are ignoring the whole thread or this particular sub branch of it and automatically jumping to the next interesting, unread message.
Because ther don't have to keep switching from discussion client to discussion client or whatever tools the use for each project they are involved with, at the same time being distracted by adverts, geegaws, random emojis and other kinds of nonsense.
They are simply more efficient and more importantly censoring is done by the user themselves, not by politically motivated admins who ban discussions based on their ideologies and whims.
It’s an open format and with basic tools, you can create a very good pipeline to consume the information. There’s no ceiling on the convenience and automation. Gmail and other webmail client are not a representation of good email workflow.
old people like the old tools that they grew up using
This is the reason behind essentially every reply I've ever seen to this question.
"I like it this way because it's always been this way and once you change your entire email workflow and customize your email client, it's almost as good as PHPbb"
Forums are built for threads and are immediately visible and accessible for everyone, not just people who want to spend their limited time dicking with email clients.
Mailing lists are the proto-discord: knowledge locked away from the public behind a special frontend and elitist attitudes. It's only better because the list is technically visible, but only in the worst, most low-effort way possible. You dump a raw txt copy of the entire thread unstructured onto the user and make it their problem to figure out. After all, your email client makes it easy to read, so why should you care about what anyone else needs?
I understand it is out of fashion, but technologically more advanced are systems that use well defined interfaces and allow pieces to be exchanged easily. After all, we’re communicating over a number of open protocols here. A forum merges all elements into one silo. I prefer web over AOL/Compuserve. If you want a forum-like interface, there’s no technical reason why this couldn’t be done on top of a mailing list. In fact, Discourse and others attempted it.
This discussion has been happening since forever. And also the idea that it serves anyone to complain how others are obviously doing it wrong, without even attempting to understand why they’re doing it a certain way. And then be irritated when the response is negative, and labeling others as elitist for using and providing open platforms over decades and not silos.
If you don’t know, feel free to ask. And then suggest (or provide!) improvements that factor in current requirements and goals instead of dismissing them as stupid.
Life advice: if you want anybody to change what they do, you need to first understand why they’re doing it, and then offer suggestions based on that understanding that improve it with them. Otherwise you’re going to continue to recreate your own victim position, and an “elite” position that you will never belong to.
yes, exactly this.
I think it's unsurprising that the people maintaining the tangled plumbing of the clunky, customizable, fiercely independent operating system most popular with highly-opinionated power-users prefer the clunky, customizable, open format that's a little inconvenient for non power-users but allows them to set up their own personal bespoke client.
You like forums for the same reason - because you grew up with them.
Today's preteens will be pining for Discord in the 2050s era of AI Neuralink vibe-telepathy. They'll say once you change your entire workflow and customize your client, phpBB is almost as good as Roblox chat or TikTok comments.
This could be read as reductive, presumptuous, ignorant, and insulting.
At the same time, it's often technically true, but for a good reason that you neglected to mention:
Those old tools tend to be very capable email clients, not web apps with their awkward attempts to simplify complex conversation structure. A good email client can handle large, high-traffic, frequently branching, long-lived threads with ease. All the web forums I've ever used fail miserably here.
The people who are tasked with participating in large scale discussion groups (like the LKML) know this through experience. They prefer email because it works better. It makes their lives easier. It helps them to be more efficient, which is absolutely necessary given the sheer volume of messaging that they handle.
Yes, a specialized tool is required to get these benefits, just as a specialized tool is required to make web server output easily readable. Thankfully, these tools have existed for decades.
maybe the old tools are prevailing for a good reason.
I prefer people to email me because half of the time they figure out their problems while writing them.
it's not an absolute rule but people who don't do their homework gravitate towards calls and messaging because they just don't prepare their questions.
asynchronous communication puts the burden on the sender, where it belongs.
Don't throw the baby out with the bathwater.
would you give up text messages or email?
Many "new" tools are basically the old tools with a few additions like emojis and pictures.
I do think upvoting was one "technology" that did make discussion forums different than mailing lists, but that put them under control of someone and had different drawbacks. hn is a good example, reddit is one devolving example
its battle tested and you can use it even with outdated computer by 20 years. no JavaScript needed hehe
easy to archive and open format too
I think it's time the report-only intake should stop. If a reporter can't reproduce at least one use case or can't summarise it in two sentences, it should be classified as spam. LLMs write beautiful reports, it's just that sometimes it doesn't bear anything resembling the truth.
couldn't an llm be used for verification like we're seeing some OSS projects do? Some projects are moving so fast, its almost certain there's little human involvement.
At my job, multiple people have vibe-coded bug-triage utilities. They're great for grouping duplicates.
But now we need an AI tool to consolidate the triage utilities.
[flagged]
> Torvalds' remarks contrast with recent comments from fellow kernel maintainer Greg Kroah-Hartman, who recently told The Register that AI has become an increasingly useful tool for the FOSS community
Thats kinda a misrepresentation. They are talking about two different things. Linus is trying to point out incorrect use of a tool while GKH is praising a correct use. This sentence felt weird at the end of the article, kind like rage bait. And I took it :P.
It seems like LLMs are actually pretty good at the sorts of things needed to manage a high-volume mailing list (summarizing, looking for dupes, sentiment, flagging things, etc), even if only as augmentation for human eyes.
That said, I get why this would rankle a lot of the folks involved.
That's just a security/protection racket with extra steps: "Someone is paying us to hurt your business/site; pay us money to defend your site against our attacks".
I like to imagine that LLM's ability to optimize code is like an extension of the training-loop in deep learning. The loss function is some kind of metric representing security and/or performance (or the lack of it) of the code and we use the LLM as the gradient/diff generator to iterate in batches over the code and fine tune it.
Imagine the current state being for the most part a collection of local maxima in security. To push the system in a more optimal state, you either need skilled people and time to overcome the barrier to a new local maximum or you throw AI at it and evaluate whether you land in a more optimal state.
I think after some time of turbulent exploit/patch cycles we will reach a stable state again, where the code converges against a new local minimum that even with AI requires significant effort (time and tokens) to overcome. Or ideally a global maximum.
With time, the LLMs improve, so the diffs/gradients get better and we will be able to reach optimal points for any software faster.
My problem with the idea is that apparently it is assumed that OSS contributors and especially maintainers will generously donate their time to get this machinery into a state that makes the optimization loop work well - just for the AI labs to turn around and sell access to the optimized models for increasingly larger amounts of money.
AI generated code can be great. Hand rolled code can be bad. The rules are the same in both cases. Make sure your code changes are focused (no random changes just because you happen to be in the file/dir or notice something) and make sure you don't break anything else along the way.
Mailing lists are a bad fit for this problem once spam starts taking over. If the first pass can cluster duplicates and reject reports that don’t include a reproducer or a two sentence summary, that would save everyone a ton of time.
So ... who, exactly, is AI supposed to be "helping"???
The bug reports are helpful! Many Linux developers including Linus, Greg Kroah-Hartman, Andrew Morton, Chris Mason, and Willy Tarreau have all commented positively on all the legitimate problems that are being found with LLM. Here is just one example article[1].
This is just a workflow issue. In the past it was very rare for multiple people to find and report a security vulnerability at once, so it made sense to keep the discussion private until they were ready to release a fix. With AI that is happening all the time, so it makes more sense for the discussion to be in public to avoid duplication. So they changed the policy accordingly. That is it.
[1]https://lwn.net/Articles/1066581/
You're literally commenting on an article that says the influx of low-effort reports caused by AI rendered the security mailing list unusable.
The "security researches" who post those bugs. Their goal being self promotion.
I think this will sort itself out over time, as people realise that it’s no longer impressive whatsoever to land an AI-assisted PR to the Linux kernel.
Maybe it's time to require public zero-knowledge proofs of a working exploits before privately-delivered exploit details can be considered.
Isn't it mostly the medium that's problematic? With an issue tracker it's easier to close as duplicate
An open visibility tracker would be a goldmine for finding new exploits before a fix is even available.
From what I’ve seen many of the AI bug search operators are newer to security research. They’re burning their tokens trying to find kernel bugs as their claim to fame before other people with AI tools find them first. They don’t spend time de-duplicating their own bugs.
Some of them may not be coming from real people. There are honeypot repos that are entirely fake and only have folders of simple files with clear security problems. They collect automated reports they get from all of the AI bots that people are running.
So make it a closed issue tracker with a public email gateway. Get Anthropic to donate LLM time to classify and combine incoming reports.
If the LLM hallucinates bugs what makes you think any classification won't be hallucinated?
The issue highlighted in Linus's message isn't that the LLM is hallucinating fake bugs; it's that 100 people running the same LLM on the same codebase find the same real bug 100 times, and if they all send it to the private security mailing list, it's (1) unmanageably high volume and (2) stupid security theater [because by definition any bad actor with the same LLM would find that bug — it's effectively public at that point].
You don't need an LLM to deduplicate bugs, just categorize by files affected. The real security problem is LLMs have a ~499/500 false positive rate and the new 'security research' post this slop and DDoS the mailing list.
Analysis of changes in the source code is already heavily analyzed by both threat actors and researchers in a desperate attempt to find code which likely was changed as a result of a bug or exploit patch.
You still spend time identifying duplicates and doing triage. That can be very significant for a project like Linux.
Interestingly enough doing that type of triage is something LLMs are actually great at
If the AI is awesome at identifying security bugs in the linux kernel, it likely can also identify if the thing it's found is similar to something that is already found in the security mailing list?
Or, put another way -- what flags the duplicate? The filer or the system? If my cheese factory is measured by the volume of cheese instead of the quality, I'll churn out the cheese even if it's sloppy duplicated cheese. And that is the case if a person has to flag a new ticket as "same as this" or not.
What's that law that says that any sufficiently large problem turns into a moderation problem?
The problem is that the tech companies are paying their research/marketing departments for headlines that go "Researcher uses powerful new Saga 6.2 release to find 597 kernel vulnerabilities! (Can your company afford NOT getting their $1000/month subscription?)", not for headlines that go "Researcher spends $50.000 to find 597 bugs, then spends $25.000 figuring out 540 of them are duplicates".
Unless the kernel community starts banning & publicly shaming repeat offenders, there's zero incentive for them to put any effort in filtering out duplicates. They are mostly doing it for marketing after all, not out of a genuine interest in making the kernel better.
> it likely can also identify if the thing it's found is similar to something that is already found in the security mailing list?
It can not because this mailing list is not public.
> “AI detected bugs are pretty much by definition not secret, and treating them on some private list is a waste of time for everybody involved – and only makes that duplication worse because the reporters can't even see each other's reports.”
Ah; so it _is_ a tool problem. It is _also_ a moderation problem.
One could ban orgs that flood the zone with AI generated trash, but is there some potential middle ground where there are sets of filters to identify duplicated bugs, and possibly just internally dump "AI spam" to a lower queue?
This seems like the sort of problem I'd addressed in the 90s with killfiles and spamassassin. In other words, can't the ingestion just go through some filters to shield the humans at the end of the pipe?
And with a mailing list you don't even have to do that! The problem doesn't really change, because you have to figure out whether it is a duplicate before you can mark it as duplicate, and that's the 'managing' part of 'unmanageable'.
While true, security reports should be treated as confidential until a patch is widely available.
So ... first, AI slop is killing mankind slowly. Skynet is winning here.
On the other hand ... IF the bug report is real, and let's assume that AI slop reports at the least a few bugs that are indeed real, then I really think it should not make a difference WHO or WHAT reports these bugs. I would not disagree on fake bugs or bogus bug reports wasting time of humans, but this is a quality difference then. Surely people can tweak AI models to be better at finding bugs too. Besides, they should auto-fix that. Is AI still too stupid to fully replace humans? Other than killing them with spam, as it does right now.
I feel like the ability to speed up finding bugs will exceed your ability to fix them and/or review ai PRs
It will almost likely find issues that require fundamental design changes to even fix some of these
Dangerous waters ahead for data security and vital infrastructure
I hate to be "that guy" but there's a reason why most of the industry stopped using mailing lists for things like this. Extremely impressive that Linux lasted this long.
Nonsense advice, he's just asking for duplicate slop patches too this way.
It's a catch 22. Why not make a separate list for AI generated reports that can be subscribed to instead? If the claim is that these are not private anyhow, no reason not to, and then a reasonable expectation could be held against submitters to check against existing reports.
That is unless it is still absolutely sensitive, in which case the only way forward that I see is to start using AI for triaging and duplicate detection as well.
AI (read: LLM technology) is the most powerful spam weapon ever invented.
[flagged]
[flagged]
[flagged]
[flagged]
I'd really like maintainers to get their hands dirty with AI agents as well to help speed up the reviews.
Over the last year there have been way too many stories and Twitter posts like these.
Yes, maintainers are overloaded, but that's only because we haven't yet built the tools to support them.
Other than such statements, I would, as a builder like to hear the sorts of tools and requirements maintainers are looking for which would make their work easier!
We need to move fast without breaking things.
I'm a huge AI advocate but even I can't get on board with this.
Feel free to fork the kernel and maintain your own vibe-coded disaster.
I'm confused by your answer, the previous post doesn't seem to be about vibe-coding at all.
It seems to be more about:
1. auto grouping duplicate security reports
2. auto validating if they are likely viable or likely nonsense
3. auto checking if they have recently been patched
4. auto assessing if they likely "invalide" for other reasons (e.g. they are for a very old long time no longer maintained Linux version, out of tree drivers, etc.)
I mean practically all of that isn't trivial to get working in a way appropriate for the Linux security mailing list and comes with many not so obvious complications. But also non of that is vibe coding and in most cases this is is more about AI doing a per-assemsment of send security issues to speed up the review of them, then it is about the AI doing the final decision.
Exactly.
At the end of the day, we would rather have a more stable and bug-free kernel than not.
It's not that much work for me anymore to report and even fix that obscure monitor driver bug that sometimes causes my machine to bootloop, unless I boot without graphics and start the XOrg server manually.
I often find myself surprised at how easily frontier models are able to find bugs across abstraction layers, that only original authors can comprehend. We need more positivity around these contributions as well.
AI slop causes additional noise on the mailing list. Your suggestion is to use more AI to filter the noise?
How about we just reduce the noise?
The world doesn't need to support the projects and research areas that interest you. How about we do something better: No one is allowed to say or write anything about AI or AI generated slop until AI is 100% perfect and produces zero errors and does everything with perfect efficiency.
AI trash like this is like showing up to a baseball game with a pitching machine and demanding that they let you join in and be the pitcher using your machine. Just because your slop cannon is fun and exciting to you doesn't make anyone else obligated to join your club just because you fired your slop on them.