Even though the video is somewhat sensationalized at some points, it is well worth a watch for people who are interested in computers but don't have a background in it. There is a nice mixture of everything from history (e.g. the founding of the FSF) to a clear explanation of a compression algorithm (clear enough that one should be able to implement it). It also makes claims that should make some people stop and think about the industry as a whole (such as Linux being the most important contemporary operating system).
I'm not sure if it is HN-crowd type material since it is easy enough information for most of us to dig up, assuming we didn't already know it. Yet it does not simplify things to the point of, "technology is magic."
The huffman tree, LZ77 and LZMA explanation is truly excellent for how concise the explanation is.
The earlier Veritasium video on Markov Chains in itself is linked if you don't know what a markov chain is.
I expected Veritasium to tank when it got sold to private equity & Derek went to Australia, but been surprised to see the quality of the long form stuff churned out by Casper, Petr, Henry & Greg.
I liked the presentation about paint mixing, however I think this is not impossible to find the missing key paint having the public paint and the message paint, but still, this is really close to what RSA is
This is IMO one of the coolest tech stories to ever happen, seriously amazing spycraft & hacking skills, but I haven't been keeping up with new developments from this story since it broke. Last I heard, the best guess at what happened was some state-sponsored actor worked very hard to get this merged, and it was caught luckily at the last minute. But no one had any smoking gun as to who did it or why or who they were targeting. Any new developments since then? Are we still just totally in the dark about what was going on here?
This isn't correct at all. The changes were merged into xz and made it into testing branches of major Linux distros.
It was caught at T plus a few minutes only because a neurotic Microsoft employee performing debugging noticed an obscure performance issue.
You can literally say Microsoft saved Linux that day. Imagine thinking this 25 years ago.
It's the difference between something really bad which happened, and something really, really, really, really bad: a malicious actor having RCE credentials to every new Debian and Red Hat box on planet Earth.
Redhat actually stumbled on the bug separately with valgrind errors triggering, so it's days were likely numbered regardless. Probably saved them a lot of debugging but the writing was on the wall.
Red Hat noticed that something was off, but there was a new version published by "Jia Tan" that fixed the warnings and the performance issue, so it's not really clear that the original version would have still gotten as deep of an investigation as would have been needed to find the issue.
It's possible though. The noise around it did at least put Freund on alert and we should be very glad both that "Jia Tan" made the mistakes they made originally and that Freund followed up on their gut feeling
The irony being that 'Jia Tan' went out of their way to ensure the backdoor was very well obfuscated, to the point it inadvertently caused bugs and slight, but noticeable, performance issues.
One wonders whether the xz backdoor would have been discovered if slightly less obfuscation was used.
The whole xz incident is a pretty strong argument to:
a) change practice from including binary (opaque) test files themselves to human-readable scripts and tooling that build test files on-demand,
b) raise suspicion of any binaries included in open source projects, and
c) create much more scrutiny around dependencies of 'highly scrutinised' packages like OpenSSH.
It's a shame that there isn't a foundation (that I'm aware of) that can donate time and effort of vetted developers to foundational open source projects like xz.
Imo it just proves there's a 99% chance a standard distro has a current zero day in it.
If a state actor (it almost has to be a state actor at the time frame they were operating under) could put in this much effort once, they clearly could afford to do it X times. And when you look through the history of communications from the author, it just reads like 'another day at the office'.
The problem is that there are many many people that are falling over themselves to believe bogus claims about false positives.
Outside of Valgrind bugzilla bug reports these claims almost never stand up to close scrutiny. Not that the people making the claims ever perform any scrutiny. It's usually "my application doesn't crash so it must be a false positive" or "I'm sure that I initialised that variable" or "it's not really a leak, the OS will reclaim the memory".
> A lot of the aliases, like Jia Tan, they sound like Asian names, and the published changes are all timestamped in UTC+8, Beijing time. So the signs point to China. And that's why it's probably not China. I mean, why would they make it that obvious? Every other part of the operation has been so meticulous, so cautious.
> And they also worked on Chinese New Year, but not on Christmas. And over the years, there were nine changes that fall outside of the Beijing time into UTC+2, which is a time zone that includes Israel and parts of Western Russia. That's why some experts have speculated that this could be the work of APT29, a Russian-state-backed hacker group also known as Cozy Bear. But again, do we know? No, of course we don't know who it is, and we likely will never know.
UTC+2 isn't very convincing as an argument for Russia. Only the Kaliningrad exclave uses that timezone, and if I were in a state-backed group, I'd live in one of the big cities.
Also quick search suggested UTC+3 was seen during the summer, and Russia doesn't do DST either.
Edit: some of the UTC+2/3 times are attributable to being differences in git committer and author dates (e.g. email patches)
I couldn't let this be, so I went through the commits and as far as I can tell, that's the case. The committer/author names and timestamps are consistent with using --author on a commit (... or in a few cases, --amend --author).
Except one: commit 3d1fdddf9 has Jia Tan as both author and committer but the author timestamp is in +0300 while the commit timestamp is +0800.
That was also what I took away when watching the video. Russians don't celebrate Christmas on the 25th (they Celebrate on January 7th), but even more than that: Russians don't celebrate Christmas the same way we do in the west.
Their "Christmas" family celebrations are on New Years Eve.
So if you're drawing conclusions from them not working on the 25th (which is a literal normal day in eastern europe) then signs point elsewhere unfortunately.
ah, Eastern Europe is a homogeneous slop, good to know. The 25th is only celebrated as a national holiday in like 90% of Eastern European countries. Russia and Serbia celebrate it on January 7th instead, and even that is only due to their use of the old Julian calendar. But sure, normal day in a very narrow and obscure definition of Eastern Europe.
Those anecdotes don’t mean anything. If I were China and wanted plausible deniability I would work on CNY and take off on foreign holidays. Of course that leaves Beijing time as a weird oversight though it’s always Beijing time anywhere in China.
I don't think it's a money thing really. IIRC the regular XZ creator/maintainer had a regular job and enough money already, and it was more of a burnout thing from dealing with the usual hassles of OSS. Which means what it really needs is to be taken over by an actual business organization, with a team of developers and professional project managers and customer support people etc so no one person gets too burnt out and if anyone does, they have plenty of backup.
I had a question. People are claiming that the spike in time for ssh or the performance degradation is not much.
But in the video itself, they show that the actual ssh time was about 100 ms and the new time it took was about 600 ms. It is almost 6 times the actual time. I am expecting the performance of the benchmark to significantly drop with these times. And it should be obvious to see that something was wrong.
( I am taking nothing from Andres here. I think he's a brilliant engineer to actually find the root cause of this himself. He is a hero. I am just pointing that 500 ms is not something obscure time interval).
...and yet, zero mention of systemd's recommendation for programs to link in the libsystemd kitchen sink just to call sd_notify() (which should really be its own library)
...and no mention of why systemd felt the need to preemptively load compression libraries, which it only needs to read/write compressed log files, even if you don't read/write log files at all? Again, it's a whole independent subsystem that could be its own library.
The video showed that xz was a dependency of OpenSSH. It showed on screen, but never said aloud, that this was only because of systemd. Debian/Redhat's sshd [0] was started with systemd and they added in a call to the sd_notify() helper function (which simply sends a message to the $NOTIFY_SOCKET socket), just to inform systemd of the exact moment sshd is ready. This loads the whole of libsystemd. That loads the whole of liblzma. Since the xz backdoor, OpenSSH no longer uses the sd_notify() function directly, it writes its own code to connect to $NOTIFY_SOCKET. And the sd_notify manpage begrudgingly gives a listing of code you can use to avoid calling it, so if you're an independent program with no connection to systemd, you just want to notify it you've started... you don't need to pull in the libsystemd kitchen sink. As it should've been in the first place.
Is the real master hacker Lennart Poettering, for making sure his architectural choices didn't appear in this video?
[0]: as an aside, the systemd notification code is only in Debian, Redhat et al because OpenSSH is OpenBSD's fork of Tatu Ylönen's SSH, which went on to become proprietary software. systemd is Linux-only and will never support OpenBSD, so likewise OpenBSD don't include any lines of code in OpenSSH to support systemd. Come to think of it, "BSD" is another thing they don't mention in the script, despite mentioning the AT&T lawsuit (https://en.wikipedia.org/wiki/USL_v._BSDi)
When I was being interviewed, we did talk about exactly this, including that libsystemd is a kitchen sink, and that eventually OpenSSH went with open-coding the equivalent to sd_notify instead of depending on libsystemd. (Also that ahem Red Hat added the dependency on libsystemd in a downstream patch oops).
However the editors (correctly IMHO) took the decision to simplify the whole story of dependencies. In an early draft they simplified it too much, sort of implying that sshd depended directly on liblzma, but they corrected that (adding the illustration of dependencies) after I pointed out it was inaccurate.
I agree with everything you say, but you have to pick your battles when explaining very complicated topics like shared libraries to a lay audience.
In general I was impressed by their careful fact checking and attention to detail.
Sadly they missed the misspelling (UNRESOVLED) even though I pointed it out last week :-( But that's literally the only thing they didn't fix after my feedback.
It did get mentioned - in the context of the upstream change to dynamically load those libraries being a threat to the hack's viability which may have caused "Jia Tan" to rush and accidentally make mistakes in the process.
They say "an open-source developer requests to remove the dependency that links xz to OpenSSH" while showing https://github.com/systemd/systemd/pull/31550 on screen, zoomed and focused so the word "systemd" does not appear.
They never once utter the word "systemd", anywhere in the script... isn't that strange for such a key dependency?
It probably is because of video length, mentioning systemd would mean explaining init system which could add another 5 min runtime. At least they showed it in diagram of dependencies.
From my vague memory of xz backdoor, I don't even recall systemd being involved. Now, I get what people are talking about when they said systemd is taking over everything and why there was so much pushback to systemd when it was being added to distros. For me as a end user/dev, it mattered little whether services were started by systemd, openrc etc.
OpenSSH is maintained by the OpenBSD developers. OpenSSH does not use liblzma (xz) at all.
Linux distros which chose to switch to systemd also chose to patch OpenSSH to call systemd's sd_notify() function, to inform systemd when sshd is fully started.
This sd_notify() function is in the huge, sprawling kitchen sink of a library called libsystemd. sd_notify() is only a few lines of code, but it's convenient (to Linux distro packagers) to make systemd a dependency of OpenSSH, link in the whole library and call that one function. It makes their patches of the upstream software smaller and easier to review for correctness.
In the sprawling libsystemd is an entire subsystem for reading/writing systemd's famous binary log files, and the user can choose compression (xz, zstd or lz4). It depended on and loaded all three of these compression libraries, whether you read/write compressed logs or not. In the video you hear about the imminent request to load these libraries dynamically on demand -- https://github.com/systemd/systemd/pull/31550 -- but this arrives many years adding these functions to the libsystem kitchen sink, and generally speaking most programs shouldn't use the libsystemd functions for reading/writing log files, they only need to send log messages to journald via syslog() or sd_journal_print()
So you can see this unwarranted dependency chain was introduced by Linux distros adding systemd to everything, and nation-state level hackers saw and tried to exploit it, seeking out the xz maintainer for social engineering.
I actually watched this last night, and while I totally understand that criticism is easy, and making things is hard (and the production quality here is great); I got a weird vibe from the video when it comes to who it is for.
The technical explanations are way too complex (even though they're "dumbed down" somewhat with the colour mixing scenario), that anyone who understands those will also know about how dependencies work and how Linux came to be.
It feels almost like it's made for people like my mum, but it will lose them almost immediately at the first mention of complex polynomials.
The actual weight of the situation kinda lands though, and that's important. It's really difficult to overstate how incredibly lucky we were to catch it, and how sophisticated the attack actually was.
I'm really sad that we will genuinely never know who was behind it, and anxious that such things are already in our systems.
My partner who is an accountant, so intelligent but not technical, watched some Veritasium documentaries the other day.
Her comment was that she was really impressed that it didnt dumb anything down like normal documentaries do. She was able to follow along more technical stuff than she anticipated, and that made her enjoy it even more.
I think we need to give people more credit when it comes to complex or techincal explanations. If people are enjoying the context but dont understand the techincal, they can just gloss over that if they prefer. But I felt this was quite telling at how and why Veritasium is such a popular channel.
Veritasium started out as a physics channel, and they've covered a wide variety of physics, math and science topics. They are never afraid of showing you the math, but one of the things I think they are really good at is not losing the human part of the story even if you can't follow the numbers exactly. At the end of the day it's humans who came up with this stuff in the first place, so it must be possible to understand it.
They aren't really a technology channel though, at least as it relates to software/computers, so that's probably why the video starts out with a brief history of Linux.
With the enormous budgets we allocate in the name of "national security", this is exactly the kind of work I expect TLAs to do.
Instead we have come to expect them to cowardly sit on exploits, or actively introduce them, rather than working to secure the general public from adversaries.
Something that's puzzling in that XZ backdoor attempt is that the attacker had to hide the evil payload. And he hid it in test files and AIUI it was injected at build time through a modified build script and that went unnoticed (it's a compiled, deployed, version that got caught by someone and raised alarm bell).
Why are build scripts not operating in a clean directory, stripping away all test related files?
Isn't this something we should begin to consider doing, seen that it's all too easy to put arbitrary things in test files (you can just pretend stuff is "fuzzed" or "random" or "test vectors" and whatnots: there's always going to be room to hide mischief in test files)?
Like literally building, but only after having erased all test directories/files/data.
Or put it this way: how many backdoors are actually live but wouldn't be if every single build was only done after carefully deleting all the irrelevant files related to tests?
Even though the video is somewhat sensationalized at some points, it is well worth a watch for people who are interested in computers but don't have a background in it. There is a nice mixture of everything from history (e.g. the founding of the FSF) to a clear explanation of a compression algorithm (clear enough that one should be able to implement it). It also makes claims that should make some people stop and think about the industry as a whole (such as Linux being the most important contemporary operating system).
I'm not sure if it is HN-crowd type material since it is easy enough information for most of us to dig up, assuming we didn't already know it. Yet it does not simplify things to the point of, "technology is magic."
> a clear explanation of a compression algorithm
The huffman tree, LZ77 and LZMA explanation is truly excellent for how concise the explanation is.
The earlier Veritasium video on Markov Chains in itself is linked if you don't know what a markov chain is.
I expected Veritasium to tank when it got sold to private equity & Derek went to Australia, but been surprised to see the quality of the long form stuff churned out by Casper, Petr, Henry & Greg.
I liked the presentation about paint mixing, however I think this is not impossible to find the missing key paint having the public paint and the message paint, but still, this is really close to what RSA is
The paint mixing is actually way closer to the idea of the Diffie-Hellman key exchange, more than RSA.
I couldn't help but notice it was quite similar to a Numberphile video 8 years earlier...
https://youtu.be/NmM9HA2MQGI
And they interviewed some people involved in this event, which is quite nice.
This is IMO one of the coolest tech stories to ever happen, seriously amazing spycraft & hacking skills, but I haven't been keeping up with new developments from this story since it broke. Last I heard, the best guess at what happened was some state-sponsored actor worked very hard to get this merged, and it was caught luckily at the last minute. But no one had any smoking gun as to who did it or why or who they were targeting. Any new developments since then? Are we still just totally in the dark about what was going on here?
> and it was caught luckily at the last minute
This isn't correct at all. The changes were merged into xz and made it into testing branches of major Linux distros.
It was caught at T plus a few minutes only because a neurotic Microsoft employee performing debugging noticed an obscure performance issue.
You can literally say Microsoft saved Linux that day. Imagine thinking this 25 years ago.
It's the difference between something really bad which happened, and something really, really, really, really bad: a malicious actor having RCE credentials to every new Debian and Red Hat box on planet Earth.
Redhat actually stumbled on the bug separately with valgrind errors triggering, so it's days were likely numbered regardless. Probably saved them a lot of debugging but the writing was on the wall.
Red Hat noticed that something was off, but there was a new version published by "Jia Tan" that fixed the warnings and the performance issue, so it's not really clear that the original version would have still gotten as deep of an investigation as would have been needed to find the issue.
It's possible though. The noise around it did at least put Freund on alert and we should be very glad both that "Jia Tan" made the mistakes they made originally and that Freund followed up on their gut feeling
The irony being that 'Jia Tan' went out of their way to ensure the backdoor was very well obfuscated, to the point it inadvertently caused bugs and slight, but noticeable, performance issues.
One wonders whether the xz backdoor would have been discovered if slightly less obfuscation was used.
The whole xz incident is a pretty strong argument to:
a) change practice from including binary (opaque) test files themselves to human-readable scripts and tooling that build test files on-demand,
b) raise suspicion of any binaries included in open source projects, and
c) create much more scrutiny around dependencies of 'highly scrutinised' packages like OpenSSH.
It's a shame that there isn't a foundation (that I'm aware of) that can donate time and effort of vetted developers to foundational open source projects like xz.
> create much more scrutiny around dependencies of 'highly scrutinised' packages like OpenSSH.
But xz is not a dependency of upstream OpenSSH you see. It was a dependency of a patch created by Linux distros for systemd integration.
> Red Hat noticed that something was off, but there was a new version published by "Jia Tan" that fixed the warnings and the performance issue
Video of Jia Tan fixing the valgrind bugs: https://www.youtube.com/watch?v=A16YuzuKN58&t=138s
A lot of people fail to fully grasp how bad this could have been on the off chance the authors were slightly less sloppy.
Imo it just proves there's a 99% chance a standard distro has a current zero day in it.
If a state actor (it almost has to be a state actor at the time frame they were operating under) could put in this much effort once, they clearly could afford to do it X times. And when you look through the history of communications from the author, it just reads like 'another day at the office'.
The problem is that there are many many people that are falling over themselves to believe bogus claims about false positives.
Outside of Valgrind bugzilla bug reports these claims almost never stand up to close scrutiny. Not that the people making the claims ever perform any scrutiny. It's usually "my application doesn't crash so it must be a false positive" or "I'm sure that I initialised that variable" or "it's not really a leak, the OS will reclaim the memory".
Still no smoking gun, but possibly Russia. From the video https://youtu.be/aoag03mSuXQ?t=2883:
> A lot of the aliases, like Jia Tan, they sound like Asian names, and the published changes are all timestamped in UTC+8, Beijing time. So the signs point to China. And that's why it's probably not China. I mean, why would they make it that obvious? Every other part of the operation has been so meticulous, so cautious.
> And they also worked on Chinese New Year, but not on Christmas. And over the years, there were nine changes that fall outside of the Beijing time into UTC+2, which is a time zone that includes Israel and parts of Western Russia. That's why some experts have speculated that this could be the work of APT29, a Russian-state-backed hacker group also known as Cozy Bear. But again, do we know? No, of course we don't know who it is, and we likely will never know.
UTC+2 isn't very convincing as an argument for Russia. Only the Kaliningrad exclave uses that timezone, and if I were in a state-backed group, I'd live in one of the big cities.
Also quick search suggested UTC+3 was seen during the summer, and Russia doesn't do DST either.
Edit: some of the UTC+2/3 times are attributable to being differences in git committer and author dates (e.g. email patches)
I couldn't let this be, so I went through the commits and as far as I can tell, that's the case. The committer/author names and timestamps are consistent with using --author on a commit (... or in a few cases, --amend --author).
Except one: commit 3d1fdddf9 has Jia Tan as both author and committer but the author timestamp is in +0300 while the commit timestamp is +0800.
I’ve always found this an amusing method of attribution considering top tier hackers are unlikely to be writing code only during office hours.
Russians don't celebrate Christmas on the 25th.
That was also what I took away when watching the video. Russians don't celebrate Christmas on the 25th (they Celebrate on January 7th), but even more than that: Russians don't celebrate Christmas the same way we do in the west.
Their "Christmas" family celebrations are on New Years Eve.
So if you're drawing conclusions from them not working on the 25th (which is a literal normal day in eastern europe) then signs point elsewhere unfortunately.
ah, Eastern Europe is a homogeneous slop, good to know. The 25th is only celebrated as a national holiday in like 90% of Eastern European countries. Russia and Serbia celebrate it on January 7th instead, and even that is only due to their use of the old Julian calendar. But sure, normal day in a very narrow and obscure definition of Eastern Europe.
Calm your tits, it’s pretty clear I meant slavic nations, and yes: even the January 7th stuff is not celebrated like we do in the west.
I’m trying to convey to people why its materially a red-herring that “russians” wouldn’t work on the 25th: since its a normal day.
>And that's why it's probably not China. I mean, why would they make it that obvious?
That's just what they want you to think!
Those anecdotes don’t mean anything. If I were China and wanted plausible deniability I would work on CNY and take off on foreign holidays. Of course that leaves Beijing time as a weird oversight though it’s always Beijing time anywhere in China.
One can also make commits with any arbitrary date/time(zone) values they want that don't reflect reality.
Stuxnet is also another mindblowing case. Wired write-up on it is a recommended reading: https://web.archive.org/web/20141028182107/http://www.wired....
https://boehs.org/node/everything-i-know-about-the-xz-backdo...
This is the scariest part to me:
> A pull request (https://github.com/jamespfennell/xz/pull/2) to a go library by a 1Password employee is opened asking to upgrade the library to the vulnerable version
People are always trying to bump versions because it's (usually) an easy contribution.
Oxide and Friends also had a great podcast with Andres about the discovery:
https://oxide-and-friends.transistor.fm/episodes/discovering...
Ireland recently created a Basic Income scheme for artists.
Europe should have an equivalent scheme for programmers of important Open Source projects such as this one.
Just German, not European, but still a start: https://en.wikipedia.org/wiki/Sovereign_Tech_Agency
The problem was more than remuneration. It was burnout and mental health issues. They may have been moderated by income but we don’t know.
Also today as I understand it much of OSS is done in-house by major companies (red hat, Ubuntu, ibm, Google, etc)
I don't think it's a money thing really. IIRC the regular XZ creator/maintainer had a regular job and enough money already, and it was more of a burnout thing from dealing with the usual hassles of OSS. Which means what it really needs is to be taken over by an actual business organization, with a team of developers and professional project managers and customer support people etc so no one person gets too burnt out and if anyone does, they have plenty of backup.
I had a question. People are claiming that the spike in time for ssh or the performance degradation is not much.
But in the video itself, they show that the actual ssh time was about 100 ms and the new time it took was about 600 ms. It is almost 6 times the actual time. I am expecting the performance of the benchmark to significantly drop with these times. And it should be obvious to see that something was wrong.
( I am taking nothing from Andres here. I think he's a brilliant engineer to actually find the root cause of this himself. He is a hero. I am just pointing that 500 ms is not something obscure time interval).
Lovely video, going into almost everything...
...and yet, zero mention of systemd's recommendation for programs to link in the libsystemd kitchen sink just to call sd_notify() (which should really be its own library)
...and no mention of why systemd felt the need to preemptively load compression libraries, which it only needs to read/write compressed log files, even if you don't read/write log files at all? Again, it's a whole independent subsystem that could be its own library.
The video showed that xz was a dependency of OpenSSH. It showed on screen, but never said aloud, that this was only because of systemd. Debian/Redhat's sshd [0] was started with systemd and they added in a call to the sd_notify() helper function (which simply sends a message to the $NOTIFY_SOCKET socket), just to inform systemd of the exact moment sshd is ready. This loads the whole of libsystemd. That loads the whole of liblzma. Since the xz backdoor, OpenSSH no longer uses the sd_notify() function directly, it writes its own code to connect to $NOTIFY_SOCKET. And the sd_notify manpage begrudgingly gives a listing of code you can use to avoid calling it, so if you're an independent program with no connection to systemd, you just want to notify it you've started... you don't need to pull in the libsystemd kitchen sink. As it should've been in the first place.
Is the real master hacker Lennart Poettering, for making sure his architectural choices didn't appear in this video?
[0]: as an aside, the systemd notification code is only in Debian, Redhat et al because OpenSSH is OpenBSD's fork of Tatu Ylönen's SSH, which went on to become proprietary software. systemd is Linux-only and will never support OpenBSD, so likewise OpenBSD don't include any lines of code in OpenSSH to support systemd. Come to think of it, "BSD" is another thing they don't mention in the script, despite mentioning the AT&T lawsuit (https://en.wikipedia.org/wiki/USL_v._BSDi)
When I was being interviewed, we did talk about exactly this, including that libsystemd is a kitchen sink, and that eventually OpenSSH went with open-coding the equivalent to sd_notify instead of depending on libsystemd. (Also that ahem Red Hat added the dependency on libsystemd in a downstream patch oops).
However the editors (correctly IMHO) took the decision to simplify the whole story of dependencies. In an early draft they simplified it too much, sort of implying that sshd depended directly on liblzma, but they corrected that (adding the illustration of dependencies) after I pointed out it was inaccurate.
I agree with everything you say, but you have to pick your battles when explaining very complicated topics like shared libraries to a lay audience.
In general I was impressed by their careful fact checking and attention to detail.
Sadly they missed the misspelling (UNRESOVLED) even though I pointed it out last week :-( But that's literally the only thing they didn't fix after my feedback.
It did get mentioned - in the context of the upstream change to dynamically load those libraries being a threat to the hack's viability which may have caused "Jia Tan" to rush and accidentally make mistakes in the process.
They say "an open-source developer requests to remove the dependency that links xz to OpenSSH" while showing https://github.com/systemd/systemd/pull/31550 on screen, zoomed and focused so the word "systemd" does not appear.
They never once utter the word "systemd", anywhere in the script... isn't that strange for such a key dependency?
It probably is because of video length, mentioning systemd would mean explaining init system which could add another 5 min runtime. At least they showed it in diagram of dependencies.
From my vague memory of xz backdoor, I don't even recall systemd being involved. Now, I get what people are talking about when they said systemd is taking over everything and why there was so much pushback to systemd when it was being added to distros. For me as a end user/dev, it mattered little whether services were started by systemd, openrc etc.
systemd was the key to the whole backdoor.
OpenSSH is maintained by the OpenBSD developers. OpenSSH does not use liblzma (xz) at all.
Linux distros which chose to switch to systemd also chose to patch OpenSSH to call systemd's sd_notify() function, to inform systemd when sshd is fully started.
This sd_notify() function is in the huge, sprawling kitchen sink of a library called libsystemd. sd_notify() is only a few lines of code, but it's convenient (to Linux distro packagers) to make systemd a dependency of OpenSSH, link in the whole library and call that one function. It makes their patches of the upstream software smaller and easier to review for correctness.
In the sprawling libsystemd is an entire subsystem for reading/writing systemd's famous binary log files, and the user can choose compression (xz, zstd or lz4). It depended on and loaded all three of these compression libraries, whether you read/write compressed logs or not. In the video you hear about the imminent request to load these libraries dynamically on demand -- https://github.com/systemd/systemd/pull/31550 -- but this arrives many years adding these functions to the libsystem kitchen sink, and generally speaking most programs shouldn't use the libsystemd functions for reading/writing log files, they only need to send log messages to journald via syslog() or sd_journal_print()
So you can see this unwarranted dependency chain was introduced by Linux distros adding systemd to everything, and nation-state level hackers saw and tried to exploit it, seeking out the xz maintainer for social engineering.
libsystemd was the indirect dependency that caused liblzma to be pulled into sshd.
> Is the real master hacker Lennart Poettering, for making sure his architectural choices didn't appear in this video?
systemd is doing what it was designed to do... Cute videos are doing what they were designed to do too - hiding that!
> OpenBSD don't include any lines of code in OpenSSH to support systemd. Come to think of it, "BSD" is another thing they don't mention in the script
And this!
I actually watched this last night, and while I totally understand that criticism is easy, and making things is hard (and the production quality here is great); I got a weird vibe from the video when it comes to who it is for.
The technical explanations are way too complex (even though they're "dumbed down" somewhat with the colour mixing scenario), that anyone who understands those will also know about how dependencies work and how Linux came to be.
It feels almost like it's made for people like my mum, but it will lose them almost immediately at the first mention of complex polynomials.
The actual weight of the situation kinda lands though, and that's important. It's really difficult to overstate how incredibly lucky we were to catch it, and how sophisticated the attack actually was.
I'm really sad that we will genuinely never know who was behind it, and anxious that such things are already in our systems.
My partner who is an accountant, so intelligent but not technical, watched some Veritasium documentaries the other day.
Her comment was that she was really impressed that it didnt dumb anything down like normal documentaries do. She was able to follow along more technical stuff than she anticipated, and that made her enjoy it even more.
I think we need to give people more credit when it comes to complex or techincal explanations. If people are enjoying the context but dont understand the techincal, they can just gloss over that if they prefer. But I felt this was quite telling at how and why Veritasium is such a popular channel.
Veritasium started out as a physics channel, and they've covered a wide variety of physics, math and science topics. They are never afraid of showing you the math, but one of the things I think they are really good at is not losing the human part of the story even if you can't follow the numbers exactly. At the end of the day it's humans who came up with this stuff in the first place, so it must be possible to understand it.
They aren't really a technology channel though, at least as it relates to software/computers, so that's probably why the video starts out with a brief history of Linux.
I'm still floored that Andres both found this and didn't ignore it. It's such a testament to an incredible engineer.
(But also, my conspiratorially-inclined mind is quite entertained by the thought of some sort of parallel construction or tip from a TLA.)
With the enormous budgets we allocate in the name of "national security", this is exactly the kind of work I expect TLAs to do.
Instead we have come to expect them to cowardly sit on exploits, or actively introduce them, rather than working to secure the general public from adversaries.
What a mess.
Wow when you put it like that it really hits hard.
> (But also, my conspiratorially-inclined mind is quite entertained by the thought of some sort of parallel construction or tip from a TLA.)
For sure you were/are not alone in this thinking. How fast the whole thing was exposed in decent enough details was... surprising.
Something that's puzzling in that XZ backdoor attempt is that the attacker had to hide the evil payload. And he hid it in test files and AIUI it was injected at build time through a modified build script and that went unnoticed (it's a compiled, deployed, version that got caught by someone and raised alarm bell).
Why are build scripts not operating in a clean directory, stripping away all test related files?
Isn't this something we should begin to consider doing, seen that it's all too easy to put arbitrary things in test files (you can just pretend stuff is "fuzzed" or "random" or "test vectors" and whatnots: there's always going to be room to hide mischief in test files)?
Like literally building, but only after having erased all test directories/files/data.
Or put it this way: how many backdoors are actually live but wouldn't be if every single build was only done after carefully deleting all the irrelevant files related to tests?