bootc and OSTree are both very neat, but the leading edge of immutable Linux distros (GNOME OS, KDE Linux) is currently converging on a different proposal by systemd developers that's standardized by the UAPI Group (https://uapi-group.org/specifications/). It fixes quite a few of the complexities with OSTree (updates are handled by `systemd-sysupdate`/`updatectl` and are just files served via HTTP) and is quite a bit easier to extend with things like an immutable version of the Nvidia drivers or codecs thanks to system extensions handled by `systemd-sysext` (which in turn are just simple squashfs files overlayed over `/usr`) and configuration via `systemd-confext`. `mkosi`, also by systemd, is quickly becoming _the_ way to build custom images too, and is somewhat tied to these new standards.
GNOME OS uses BuildStream and as a result has no concept of packages at all and no relationship to any distro, KDE Linux is based on Arch. There is no relationship between the two. GNOME OS used to be OSTree based but switched to systemd-sysupdate a while ago.
Sorry, not completely separate, yes, but some of the parts of the standard (e.g. systemd-sysext for layering "packages") and closely related things like systemd-sysupdate do actually replace parts of this (esp. ostree).
It is very odd to me to watch OStree-based distros starting to take off and win recruits.
The only reason Red Hat needed to invent this very complex mechanism was because RH does not officially have a COW-snapshot capable filesystem in its enterprise distro.
A filesystem with snapshots makes software installation transactional. You take a snapshot, install some software, and if it doesn't work right, you can revert to the snapshot. (With very slightly more flexible snapshots, you can limit the snapshot to just some part of the directory tree, but this is not essential; it merely permits more flexibility.)
In other words, you are a long way toward what in database language is called ACID:
Atomicity, consistency, isolation, durability. It makes your software inastallation transactional: an update either happens completely (A), you can check it is valid (C) and works (I), or it can be totally reverted, and the system restored to the earlier state (D).
That's a good thing. It means you can safely automate software deployment knowing that if it goes wrong you have an Undo mechanism. Databases got this 50+ years ago; in the 21st century it's making its way to FOSS OSes.
Do this in the filesystem and it's easy. SUSE's implementation is so simple, it's basically a bunch of shell scripts, and it can be turned on and off. You can run an immutable OS, reboot for updates, and if you need, disable it, go in and fix the system, and then turn it back on again.
This is because SUSE leans very heavily on Btrfs and that is the critical weakness -- Btrfs is only half finished and is not robust.
But RH removed Btrfs from RHEL and Btrfs was the only GPL COW filesystem, so core infrastructure in the distro means no COW on RH. Oracle Linux has Btrfs -- the FS was developed at Oracle, after all -- and so does Alma.
(Yes I know, Fedora put it back, but the key thing is, it only uses Btrfs only for compression so that Flatpak looks less horrendously inefficient. Fedora doesn't use snapshots.)
With no COW FS, RH had to invent a way to do transactional updates without filesystem support. Result, OStree. Git, but for binaries.
And yes, everyone developing FOSS uses Git, but almost nobody understands Git:
You know that if there's an Xkcd about it, it must be true.
Embedding something you don't understand in your OS design is a VERY BAD PLAN.
With OStree your FS is a virtual one, it's not real, it's synthesized on the fly from a local repository. The real FS is hidden and can't be hand-edited or anything. It generates the OS filesystem tree on the fly, you see. OS-tree.
Use it just for GUI apps, that's Flatpak.
Use it for the whole OS, that's OStree. It is so mind-shreddingly complicated that you can't do package management any more, you can't touch the underlying FS. So you need a whole new set of layers on top: virtual directories on top of the main virtual directory, and some bits with extra pseudo-filesystems layered on top of that to make some bits read-write.
It's like the scene in the Wasp Factory where under the skull plate it's just writhing maggots. I recall in horror and revulsion when I see it.
So it's deeply bizarre to read blog posts praising all the cool stuff you can do with it.
> A filesystem with snapshots makes software installation transactional. You take a snapshot, install some software, and if it doesn't work right, you can revert to the snapshot. (With very slightly more flexible snapshots, you can limit the snapshot to just some part of the directory tree, but this is not essential; it merely permits more flexibility.)
Eh, you don't typically have a lock mechanism for the filesystem equivalent to that of a database.
Who's to say something like this doesn't happen:
- snapshot fs
- op/system adjust firewall rules
- "you" install updates
- you rollback
- firewall rules is now missing patches
Don't get me wrong zfs is great - but it doesn't come with magical transactions.
bootc and OSTree are both very neat, but the leading edge of immutable Linux distros (GNOME OS, KDE Linux) is currently converging on a different proposal by systemd developers that's standardized by the UAPI Group (https://uapi-group.org/specifications/). It fixes quite a few of the complexities with OSTree (updates are handled by `systemd-sysupdate`/`updatectl` and are just files served via HTTP) and is quite a bit easier to extend with things like an immutable version of the Nvidia drivers or codecs thanks to system extensions handled by `systemd-sysext` (which in turn are just simple squashfs files overlayed over `/usr`) and configuration via `systemd-confext`. `mkosi`, also by systemd, is quickly becoming _the_ way to build custom images too, and is somewhat tied to these new standards.
Typo: (CoreOS and Fedora Silverblue) are the bleeding edge of immutable distros. Those mentioned are just users.
GNOME OS uses BuildStream and as a result has no concept of packages at all and no relationship to any distro, KDE Linux is based on Arch. There is no relationship between the two. GNOME OS used to be OSTree based but switched to systemd-sysupdate a while ago.
> bootc and OSTree are both very neat
May I rephrase that?
bootc and OStree are both Cthulhoid nightmare horrors that only exist because of corporate politics, but the leading edge...
> the bleeding edge of immutable Linux distros (GNOME OS, KDE Linux)
These are words but they don't make sense.
Kdelinux uses pacman for now, but the eventual goal is systemd-sysext based mkosi images.
They are also considering moving to buildstream and join gnome.
Corrected - I meant leading edge.
Context re:distros mentioned:
GNOME OS: https://os.gnome.org/ KDE Linux: https://kde.org/linux/
Doubling progression-free survival (17.6 vs 7.4 months) is a large effect size for recurrent prostate cancer.
I’m pretty sure Linux doesn’t have a prostate, even with all the changes in the leading edge distros, and you’re commenting in the wrong post.
"some of the newer ideas happening in this space are in the GNOME OS project and the KDE Linux project"
My Gentoo box is immutable. Right up until I run emerge.
From https://uapi-group.org/ :
> Contributing members include people from Ubuntu Core, Debian, GNOME OS, Fedora CoreOS, Endless OS, Arch Linux, SUSE, Flatcar, systemd, image-builder/osbuild, mkosi, tpm2-software, System Transparency, buildstream, BTRFS, bootc, composefs, (rpm-)ostree, Microsoft, Amazon, and Meta.
Note systemd, (rpm-)ostree and bootc.
My understanding is that uapi is another initiative but not completely separated from bootc and ostree. Maybe complementary.
Sorry, not completely separate, yes, but some of the parts of the standard (e.g. systemd-sysext for layering "packages") and closely related things like systemd-sysupdate do actually replace parts of this (esp. ostree).
For those looking for a more extensive article about bootc, I recently wrote about using it in https://yorickpeterse.com/articles/self-hosting-my-websites-..., including a comparison to some other existing tools.
We use TorizonOS, which is also based on OSTree: https://www.torizon.io/blog/ota-best-linux-os-image-update-m....
It works quite well for our edge devices. It’s tightly integrated with Toradex hardware, but not limited to it.
It may seems litte a niche, but it has strong potential for long‑term supported edge products. Any additional experiences to share?
bootc is kind of perfect for edge. delivering OS update as a whole. ease of update/rollback.
developers will do anything but to use a cow fs
I like the idea of using the same format for kernel-included VMs as I use for containers.
Next up, backups stored as layers in the same OCI registries.
I am not, however, sure ostree is going to be the final image format. Last time I looked work was in progress to replace that.
It is not, the future is currently pointing to composefs:
https://github.com/bootc-dev/bootc/issues/1190
There's a GitHub org that builds bootc-ready images for non-Red Hat family distributions using this backend.
https://github.com/bootcrew
It is very odd to me to watch OStree-based distros starting to take off and win recruits.
The only reason Red Hat needed to invent this very complex mechanism was because RH does not officially have a COW-snapshot capable filesystem in its enterprise distro.
A filesystem with snapshots makes software installation transactional. You take a snapshot, install some software, and if it doesn't work right, you can revert to the snapshot. (With very slightly more flexible snapshots, you can limit the snapshot to just some part of the directory tree, but this is not essential; it merely permits more flexibility.)
In other words, you are a long way toward what in database language is called ACID:
https://en.wikipedia.org/wiki/ACID
Atomicity, consistency, isolation, durability. It makes your software inastallation transactional: an update either happens completely (A), you can check it is valid (C) and works (I), or it can be totally reverted, and the system restored to the earlier state (D).
That's a good thing. It means you can safely automate software deployment knowing that if it goes wrong you have an Undo mechanism. Databases got this 50+ years ago; in the 21st century it's making its way to FOSS OSes.
Do this in the filesystem and it's easy. SUSE's implementation is so simple, it's basically a bunch of shell scripts, and it can be turned on and off. You can run an immutable OS, reboot for updates, and if you need, disable it, go in and fix the system, and then turn it back on again.
This is because SUSE leans very heavily on Btrfs and that is the critical weakness -- Btrfs is only half finished and is not robust.
But RH removed Btrfs from RHEL and Btrfs was the only GPL COW filesystem, so core infrastructure in the distro means no COW on RH. Oracle Linux has Btrfs -- the FS was developed at Oracle, after all -- and so does Alma.
(Yes I know, Fedora put it back, but the key thing is, it only uses Btrfs only for compression so that Flatpak looks less horrendously inefficient. Fedora doesn't use snapshots.)
With no COW FS, RH had to invent a way to do transactional updates without filesystem support. Result, OStree. Git, but for binaries.
And yes, everyone developing FOSS uses Git, but almost nobody understands Git:
https://xkcd.com/1597/
You know that if there's an Xkcd about it, it must be true.
Embedding something you don't understand in your OS design is a VERY BAD PLAN.
With OStree your FS is a virtual one, it's not real, it's synthesized on the fly from a local repository. The real FS is hidden and can't be hand-edited or anything. It generates the OS filesystem tree on the fly, you see. OS-tree.
Use it just for GUI apps, that's Flatpak.
Use it for the whole OS, that's OStree. It is so mind-shreddingly complicated that you can't do package management any more, you can't touch the underlying FS. So you need a whole new set of layers on top: virtual directories on top of the main virtual directory, and some bits with extra pseudo-filesystems layered on top of that to make some bits read-write.
It's like the scene in the Wasp Factory where under the skull plate it's just writhing maggots. I recall in horror and revulsion when I see it.
So it's deeply bizarre to read blog posts praising all the cool stuff you can do with it.
> A filesystem with snapshots makes software installation transactional. You take a snapshot, install some software, and if it doesn't work right, you can revert to the snapshot. (With very slightly more flexible snapshots, you can limit the snapshot to just some part of the directory tree, but this is not essential; it merely permits more flexibility.)
Eh, you don't typically have a lock mechanism for the filesystem equivalent to that of a database.
Who's to say something like this doesn't happen:
Don't get me wrong zfs is great - but it doesn't come with magical transactions.