Nested virt on x86 is curiously painful; you'd kind of think each layer would be isolated, so that the L0 (hardware) would only have to worry about it's VM (L1), and L1 would have to worry about it's VM (L2); but nope - the L0 top level hypervisor sees faults from the L2 and has to figure out that they are actually L2 not L0.
IMHO the extra complexity (and historical flakiness of it) - makes me say that enabling nesting is a bad idea for public VM hosts.
> LPE: On distributions such as RHEL, /dev/kvm is world-writable (0666), so an unprivileged user can also use this vulnerability as a reliable LPE to gain root.
Why on Linux device files are accessible by untrusted applications?
Not all device files, only /dev/kvm. I assume the logic was "with /dev/kvm access the user can ...allocate memory and execute code, which they already can, so why not allow it?". Could also make rootless isolation easier
Because if /dev/kvm isn't accessible to unprivileged users, then people will start using `sudo` to run anything involving virtualization, which would be much worse for security overall.
This is a very nasty vulnerability and risks any service that uses and allows nested x86 virtualization features at risk. Including those running VMs as a service.
> Running the PoC inside a guest VM can trigger a host kernel panic. A full escape exploit that works in a controlled environment also exists, but it is not released at this time and is planned to be released in the very distant future.
The first commit that introduced this vulnerability was in 2010. [1] So it was undiscovered for 16 years until now [2].
It was only a matter of time that a vulnerability in KVM would appear. This one is really not good as it is the first KVM guest-to-host exploit working on both AMD and Intel.
"If you operate an x86 KVM host that accepts multi-tenant guests and supports nested virtualization, or use an instance on top of one"
does this mean that you must have nested virtualization enabled to br vulnerable.
does disabling this feature in the host os or bios, make you immune to this bug?
Nested virtualization prompts the use of shadow paging (where the bug is), is my understanding, where non-nested cases use hardware accelerated translation instead.
If you share resources, that reduces costs, but increases security risks.
choose whether to share a filesystem, an OS, a kernel, hardware, or just use a dedicated server.
The economics of sharing resources are all in a tiny sliver of the budget spectrum, the shoestring budget range :
0-1$/mo: serverless
1$-5$/mo containers
5$-200$/mo Virtual Machine(s)
200$-1Billion$/month , at least one dedicated server
So if your hourly is worth anywhere upwards of 5$/hr, and your project has any semblance of seriousness, just use a dedicated server, and avoid a whole class of LPE vulnerabilities just to save some $.
Businesses have expenses, let's stop pretending that all of these non dedicated server infrastructures are serious. Shell out 200$/month or stick to hobby status.
I run 3 servers for 200 EUR, thanks to Hetzner, exactly for this reason (and I’m cheap and I never understood cloud/services like Vercel and Railway as serious alternatives ;-)).
If you can run everything you need on two or three servers, what you describe can work. But it’s still hobby status, basically. The equation changes when the scale gets significantly bigger. Managing a non-trivial hardware fleet requires people, and people cost money.
The reason “managed services” of all kinds, including cloud services, are so widespread in business is because someone else is managing things so that you don’t have to. This is as serious as it gets in business. Managing your own hardware makes very little sense for many, if not most companies.
One server is enough for many small businesses. And for large business (like X or Instagram) it is economically more profitable to own their servers. For example, in my country top companies like VK or Yandex own their datacenters and sell cloud services instead of paying for someone's else cloud.
Also if you have several servers you do not need to hire a full-time sysadmin.
> Managing a non-trivial hardware fleet requires people, and people cost money.
People in AWS also cost money and guess who is going to cover this cost?
Suppose that you are a midsize company or a b2b service, so you want to make sure that your service has minimum downtime.
This means that you need sysadmins in close proximity to your hardware to do hardware swapping/troubleshooting. Or you need to engineer your system to not have a SPOF (which is not easy). So you're looking at employing at least 2 engineers near your datacenter.
It does not make sense for a lot of time and scaling. You need 3+ people maintaining it, you have upfront costs in the hundreds of thousands of euros on the very lower end. If you don't utilize that money spent, sucks to be you. You have planning times in the area of months, not hours, unless you keep capacity you don't use around (rackspace, cabling, power/cooling capacity).
On the other hand, if you have that hardware management running, it's very amazing. Before the AI nuke, We were looking at moving various systems fully bare metal, because it would simplify management on both sides a lot, and a common statement I heard is "We don't deal with systems that small. If we do bare metal container hosting, we don't measure in dozens of gigabytes of memory. Your business case validates that investment. Here is btw three test systems about double your requirement, just old".
Before the AI nonsense (HBM Memory Demand -> RAM & SSD prices), this would result in very competitive hosting costs after some scale, when amortized across 5 years and then tossed into the testing environment until it stops functioning. And these testing environments allow for a lot of experimentation and failover testing.
Nested virt on x86 is curiously painful; you'd kind of think each layer would be isolated, so that the L0 (hardware) would only have to worry about it's VM (L1), and L1 would have to worry about it's VM (L2); but nope - the L0 top level hypervisor sees faults from the L2 and has to figure out that they are actually L2 not L0. IMHO the extra complexity (and historical flakiness of it) - makes me say that enabling nesting is a bad idea for public VM hosts.
> LPE: On distributions such as RHEL, /dev/kvm is world-writable (0666), so an unprivileged user can also use this vulnerability as a reliable LPE to gain root.
Why on Linux device files are accessible by untrusted applications?
Not all device files, only /dev/kvm. I assume the logic was "with /dev/kvm access the user can ...allocate memory and execute code, which they already can, so why not allow it?". Could also make rootless isolation easier
Different kernel modules might have different vulnerabilities.
Because if /dev/kvm isn't accessible to unprivileged users, then people will start using `sudo` to run anything involving virtualization, which would be much worse for security overall.
1: As siblings note, some device files are wide open, some are limited to a given user group, and some are root-only.
2: Because it's desirable for users to be able to run VMs.
???
That's been the case forever: /dev/null, /dev/zero, /dev/stdin, ...
Very many "devices" aren't at all device-like.
The full write up is here: [0].
This is a very nasty vulnerability and risks any service that uses and allows nested x86 virtualization features at risk. Including those running VMs as a service.
> Running the PoC inside a guest VM can trigger a host kernel panic. A full escape exploit that works in a controlled environment also exists, but it is not released at this time and is planned to be released in the very distant future.
The first commit that introduced this vulnerability was in 2010. [1] So it was undiscovered for 16 years until now [2].
It was only a matter of time that a vulnerability in KVM would appear. This one is really not good as it is the first KVM guest-to-host exploit working on both AMD and Intel.
[0] https://github.com/V4bel/Januscape/blob/main/assets/write-up...
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
> So it was undiscovered for 16 years until now
Publicly undiscovered
"If you operate an x86 KVM host that accepts multi-tenant guests and supports nested virtualization, or use an instance on top of one"
does this mean that you must have nested virtualization enabled to br vulnerable. does disabling this feature in the host os or bios, make you immune to this bug?
Nested virtualization prompts the use of shadow paging (where the bug is), is my understanding, where non-nested cases use hardware accelerated translation instead.
KVM, or x86 identified mail validation.
hey, here's a good rule of thumb.
If you share resources, that reduces costs, but increases security risks.
choose whether to share a filesystem, an OS, a kernel, hardware, or just use a dedicated server.
The economics of sharing resources are all in a tiny sliver of the budget spectrum, the shoestring budget range :
0-1$/mo: serverless
1$-5$/mo containers
5$-200$/mo Virtual Machine(s)
200$-1Billion$/month , at least one dedicated server
So if your hourly is worth anywhere upwards of 5$/hr, and your project has any semblance of seriousness, just use a dedicated server, and avoid a whole class of LPE vulnerabilities just to save some $.
Businesses have expenses, let's stop pretending that all of these non dedicated server infrastructures are serious. Shell out 200$/month or stick to hobby status.
No, I don't sell dedicated servers, but I should
I run 3 servers for 200 EUR, thanks to Hetzner, exactly for this reason (and I’m cheap and I never understood cloud/services like Vercel and Railway as serious alternatives ;-)).
If you can run everything you need on two or three servers, what you describe can work. But it’s still hobby status, basically. The equation changes when the scale gets significantly bigger. Managing a non-trivial hardware fleet requires people, and people cost money.
The reason “managed services” of all kinds, including cloud services, are so widespread in business is because someone else is managing things so that you don’t have to. This is as serious as it gets in business. Managing your own hardware makes very little sense for many, if not most companies.
One server is enough for many small businesses. And for large business (like X or Instagram) it is economically more profitable to own their servers. For example, in my country top companies like VK or Yandex own their datacenters and sell cloud services instead of paying for someone's else cloud.
Also if you have several servers you do not need to hire a full-time sysadmin.
> Managing a non-trivial hardware fleet requires people, and people cost money.
People in AWS also cost money and guess who is going to cover this cost?
Suppose that you are a midsize company or a b2b service, so you want to make sure that your service has minimum downtime.
This means that you need sysadmins in close proximity to your hardware to do hardware swapping/troubleshooting. Or you need to engineer your system to not have a SPOF (which is not easy). So you're looking at employing at least 2 engineers near your datacenter.
Really? We basically never go onsight. Ticket in with the colo and they can help.
Own hardware has a weird scaling curve.
It does not make sense for a lot of time and scaling. You need 3+ people maintaining it, you have upfront costs in the hundreds of thousands of euros on the very lower end. If you don't utilize that money spent, sucks to be you. You have planning times in the area of months, not hours, unless you keep capacity you don't use around (rackspace, cabling, power/cooling capacity).
On the other hand, if you have that hardware management running, it's very amazing. Before the AI nuke, We were looking at moving various systems fully bare metal, because it would simplify management on both sides a lot, and a common statement I heard is "We don't deal with systems that small. If we do bare metal container hosting, we don't measure in dozens of gigabytes of memory. Your business case validates that investment. Here is btw three test systems about double your requirement, just old".
Before the AI nonsense (HBM Memory Demand -> RAM & SSD prices), this would result in very competitive hosting costs after some scale, when amortized across 5 years and then tossed into the testing environment until it stops functioning. And these testing environments allow for a lot of experimentation and failover testing.
Though now it's all very different and not clear.
i dont get this dedicated servers = hobby mentality
you can do a ton with just a couple of dedicated servers with the redundancy you need
There seems to be some confusion, although one other user interpreted the same thing, I still believe it's a misinterpretation.
I said dedicated servers, I never said anything about owning or managing the hardware. You can rent a dedicated server.
The decision to virtualize and the decision to own the hardware are separate decisions.