Enable nested virtualization

[felipecrs]

I don't know which is the hypervisor used to spin the VMs, but enabling nested virtualization would be a great enhancement. For example, this way we could build Vagrant boxes in GitHub Actions.

[felipecrs]

This feature is enabled by default in Travis CI. So I wonder why not here?

[dalehamel]

It looks like this is likely because azure only supports nested virtualization on dv3 or above https://azure.microsoft.com/en-ca/blog/nested-virtualization-in-azure/ but azure runs on ds2 instances: https://help.github.com/en/actions/automating-your-workflow-with-github-actions/virtual-environments-for-github-hosted-runners#cloud-hosts-for-github-hosted-runners

This seems to be older than dv3 from the catalogue: https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes-general#dsv2-series vs https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes-general#dv3-series-1

I haven't checked the exact CPU features, but i wonder if Hyper-V powering azure needs to work with a specific CPU family / featureset?

Has anyone tried just running KVM on azure? What happens? Is there even support for it in the stock azure / ubuntu kernel for bionic?

[dalehamel]

My use-case is I'd like to run an android emulator on Github Actions btw, like with https://github.com/google/android-emulator-container-scripts

[dalehamel]

When I run:

$egrep -c '(vmx|svm)' /proc/cpuinfo
0

This indicates there isn't CPU support, at least not passed through to the guest.

I installed kvm packages and was able to modprobe kvm, but there is no /dev/kvm, which is what I would expect.

Unless Azure adds support for this, or github actions switches / offers other machine classes, I don't think nested virtualization can be done.

[felipecrs]

Travis supports not only nested virtualization but also xvfb, that I guess you also need to run your Android tests.

https://github.com/felipecassiors/ubuntu1804-4dev/blob/master/.travis.yml

The support for nested virtualization is out-of-the-box there, where I use to build a Vagrant box using VirtualBox.

[AkihiroSuda]

https://github.com/actions/virtual-environments/blob/master/images/macos/macos-10.15-Readme.md

Intel x86 Emulator Accelerator (HAXM installer) | 7.5.1

macOS instances seem to support nested virtualization?
(I couldn't install VirtualBox though: #429)

[hannes]

+1, I would also really like to see this. I wonder what went wrong, if I manually spin up an equivalent Azure VM, it has vmx set and VirtualBox works fine with hardware acceleration.

[felipecrs]

Seems like it is possible indeed: https://docs.microsoft.com/en-us/azure/virtual-machines/windows/nested-virtualization

[dalehamel]

Yes azure supports it but not the instance type GitHub actions runs on

[hannes]

Works fine on Travis ^^

[AjkayAlan]

+1 - Building vagrant boxes using packer is not currently possible with GitHub Actions, unless we self-host a runner. Same with Azure DevOps (which appears to use the same hosted images as GitHub Actions).

So, if this got prioritized, you would maybe solve this on both platforms.

[dalehamel]

I suspect that with self-hosted runners on azure you could circumvent this issue, as jobs could be dispatched to a host that supports nested virtualization.

The current worker that actions runs on in their free offering just doesn’t have a processor that supports this.

I suspect that if this feature is ever offered, it will be when a compatible processor is bumped into this low-tier instance class. Until then, self-hosted runners could be feasible but only really for corporate users.

[njsmith]

I wouldn't mind a self-hosted runner, but I need nested virtualization for a public project, and those aren't really usable with self-hosted runners: https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories

[pkit]

That's bogus. If you run self-hosted on a VM you're already protected.

[hannes]

I wouldn't mind a self-hosted runner, but I need nested virtualization for a public project, and those aren't really usable with self-hosted runners: https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories

Yes, exactly, that's also the reason we can't use them.

[mvgijssel]

Would also love this to be able to use packer to build images.

[maxkratz]

@maxkratz not right now :( this is a test for us to see how people use it and to ensure our ability to detect bad actors abusing Actions when we have nested virt as another tool in the mix.

Thank you for the clarification.

[Steve-Glass]

👋 @Gustl22 With regard to your prior comment:

@nebuk89 You mean a separate issue for GPU support on MacOS (maybe with github/roadmap#528) or for the specific Flutter 3.7 issue?

I would be great if a separate issue is filed for GPU support on macOS. Having customer submitted issues for this feature helps with establishing the priority of upcoming investigations. You can tag me in it as well 😄

[gsauthof]

@nebuk89 wrote:

sudo adduser $USER kvm
sudo chown -R $USER /dev/kvm

Hm, this is suboptimal:

• On Ubuntu, /dev/kvm is owned by group kvm - thus, adding your user to that group is a valid strategy for using it as a normal user - iff you are able to re-login with that user - which is usually cumbersome in a CI environment.
• The second part is a hack to work-around the re-login requirement. First, it includes the -R recursive option for no reason. Second, this change might be reversed by udev. I guess one can live with such a unstable hack to some extend, but then you don't have to add your user to the kvm group, in the first place - as well as calling chown recursively.

The better alternative is to widen the permission via udev, since it doesn't require a re-login nor such a hack.

See my Stackoverflow answer for details.

[pkit]

re-login with that user

man newgrp

[alexellis]

Hey folks, hope you don't mind us chiming in here.

actuated supports launching VMs within CI, so long as Linux + KVM are all you need.

Launching Firecracker in a GitHub Action: https://twitter.com/welteki/status/1625487015650410496?s=20

Running a NixOS build in a GitHub Action https://twitter.com/welteki/status/1625419864029466624?s=20

Nested virtualisation extensions are required on the host, so this only works on x86_64 for now.

[nebuk89]

https://github.blog/changelog/2023-02-23-hardware-accelerated-android-virtualization-on-actions-windows-and-linux-larger-hosted-runners/ << announced as live

and thank you @gsauthof for the recommendation on how to better approach the KVM group :) it works and is what I have in the change log (I hope it was ok to credit you there!)

[jacob-carlborg]

@benja-M-1 macOS Intel runners do support hardware accelerated nested virtualization, I depend on that in my GitHub action. There are no regular macOS runners running on Apple Silicon, only large runners [1]. So I don't think it's possible to randomly get Apple Silicon runners. That would be pretty bad API from GitHub's side since Intel and Apple Silicon are two completely different architectures. I mean, you don't randomly get a macOS runner if you have specified ubuntu-latest.

[1] https://github.blog/2023-10-02-introducing-the-new-apple-silicon-powered-m1-macos-larger-runner-for-github-actions/

[benja-M-1]

I mean, you don't randomly get a macOS runner if you have specified ubuntu-latest.

Did you meant macos-latest?

This was more a supposition because this is what I observed with ubuntu-latest.
Here is a build where the virtualisation was not enabled. It is an Intel processor.
And in this one, which is another run of the previous job, the virtualisation is enabled. It is an AMD processor.

On the same repo, we tried to make it work with macos-runner and never managed to. It used to work and failed mid September. We thought it was related to Docker Destkop but it happened to work once or twice again...
I am very confused at the moment and fail to understand what is possible and what is not :(

[jacob-carlborg]

Did you meant macos-latest?

No. I mean that I don't think you will randomly get an Apple Silicon or Mac Intel runner. Forget about it. I tried to make a point, didn't work that well 😄.

Here is a build where the virtualisation was not enabled. It is an Intel processor.
And in this one, which is another run of the previous job, the virtualisation is enabled. It is an AMD processor.

Wow, I did not know that some of the regular Linux runners supported virtualization. Then there's no excuse for GitHub to not support virtualization for regular Linux runners.

On the same repo, we tried to make it work with macos-runner and never managed to

Do you have an example of a job that fails on macOS?

I'm successfully using QEMU with the Hypervisor framework (accel=hvf) and Xhyve on macos-latest [1][2][3].

[1] https://github.com/cross-platform-actions/action/blob/bacc3477d8996e2a495657ba82b050de3ce8b371/.github/workflows/ci.yml#L44
[2] https://github.com/cross-platform-actions/action/actions/runs/6749647583/job/18350449757#step:3:267
[3] https://github.com/cross-platform-actions/action/actions/runs/6749647583/job/18350450893#step:3:269

[gtardif]

Hi @nebuk89! good to see nested virt on GHA nodes!
I work with @benja-M-1 and as he mentioned we tried to run Docker Desktop on linux nodes, it works well on large linux runners, but we noticed that it works "sometimes" on "ubuntu-latest", depending on the node we get.

Sometimes we have a node "Intel Xeon" that does not support it, like here.
Sometimes we get an "AMD" with nested virt support and Docker Desktop starts nicely, like here. (apparently a single proc, cpu_core: 2 so we don't need a huge machine)

But since we get random nodes, we can't rely on it for our Desktop action.
This is sad because 1) large runners cost more (quite a lot), but also 2) it requires access to non-standard nodes, users might not have access to these nodes.

It would be really cool if we had a way to specify that we need an ubuntu-latest, but the AMD one with nested virt support (even if slightly more expensive, but part of the standard nodes, that would be great)

[gtardif]

Thx @jacob-carlborg for the detailed info, we need to investigate more on our side, as indeed it seems nested virt is still enabled on mac nodes as before...

[alexellis]

That's so cool @nebuk89, congrats on shipping a much requested feature!

[jacob-carlborg]

This is great news. Now we just want this for the regular runners as well so the rest of us mortals can use it as well 😄.

[amrsa1]

Hello All,

Not sure what Im missing but using the example provided through this error

      - name: Enable KVM group perms
        run: |
            echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
            sudo udevadm control --reload-rules
            sudo udevadm trigger --name-match=kvm

KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"
Failed to open the device 'kvm': No such file or directory
Error: Process completed with exit code 1.

[antoineco]

Are you using a large runner?

Starting on February 23, 2023, Actions users of GitHub-hosted larger Linux runners will be able to make use of hardware acceleration for Android testing.

-- https://github.blog/changelog/2023-02-23-hardware-accelerated-android-virtualization-on-actions-windows-and-linux-larger-hosted-runners/

[amrsa1]

Aha just noticed, its only featured for large runner unfortunately

[alexellis]

Feel free to check out what we're doing - https://actuated.dev/blog/kvm-in-github-actions

[rfc2822]

Can I somehow use it with an own image? We have our own Android emulator image: https://github.com/bitfireAT/docker-android-ci/

Currently it runs on a self-hosted runner, but I'd love to run it on Large Runners instead. Unfortunately I don't know how to adapt the image so that it can use the kvm functionality. I have tried to install the kvm host in the Dockerimage and so on, but failed… would be great if there were instructions to use this feature with other images!

[amrsa1]

I have been using this image https://github.com/amrsa1/Android-Emulator

Tried alot of options and different configs but no use, seems nested virtualization is not working at all.

[vincent-chung-paypay]

Is there any update about using KVM in self-hosted runner?
Would be grateful if somebody gives me some details.

[JojOatXGME]

@vincent-chung-paypay With self-hosted runners, you are in control of the system you are using. So I would expect that you are also in control whether your system supports virtualization. However, maybe I am wrong. I haven't used self-hosted runners.

EDIT: Also note that somebody from https://actuated.com/ has pointed out earlier that they are supporting KVM in their (self-hosted) runners. https://actuated.com/blog/kvm-in-github-actions

[maxkratz]

@JojOatXGME Using KVM with self-hosted runners is possible. We use this setup to provision VM images.

[storenth]

CPU still doesn't support hardware virtualization under Github hosted ubuntu runner. Check it out my test with 0 output. See related article on Android:

An output of 1 or greater means that virtualization is supported. An output of 0 means that your CPU doesn't support hardware virtualization.

[storenth]

No, I am talking about free plan.

[AkechiShiro]

I would also like to know what in free plan, has support for nested virtualization beside MacOS free runners.

[ssbarnea]

@AkechiShiro AFAIK, only macos has nested virtualization support and likely only because they failed to find a way to disable it! Still, be sure that once they will add arm macs, that will stop working too as I doubt they will jump directly to M2 series.

[felipecrs]

I.e. it would be really nice if the free runners could support virtualization for a multitude of good reasons in which the community could benefit from.

However, if there is any extra cost added into that (let's say Azure charges more for VMs with nested virtualization support), I believe it's understandable.

[pkit]

@ssbarnea for MacOS they use non-Azure instances from Akka.
AFAIK MacOS on non-Intel does not support nested virt at all yet. Although CPU do support it, OS - doesn't.

[mpartel]

Currently with free runners, /dev/kvm sometimes exists (and works!), and sometimes it doesn't.
Not sure if this is by accident, or if this means free KVM support is rolling out 🤞

My use case: testing bindfs on multiple Linux / BSD distros, since GH Actions only supports Ubuntu. Can't do this properly with Docker, because FUSE has a kernel component.

[jacob-carlborg]

@mpartel FYI you can use https://github.com/marketplace/actions/cross-platform-action to test on FreeBSD, OpenBSD and NetBSD. Supports x86-64 and ARM64.

[alexvanyo]

Since the runner update yesterday (https://github.blog/2024-01-17-github-hosted-runners-double-the-power-for-open-source/), I am seeing that /dev/kvm exists for all free/non-large ubuntu-latest runners.

This is amazing, since with https://github.blog/changelog/2023-02-23-hardware-accelerated-android-virtualization-on-actions-windows-and-linux-larger-hosted-runners/ now working on these runners it allowed me to replace all my usages of macos-latest with ubuntu-latest to run instrumented Android tests on emulators, which also had the bonus of running those tests about twice as fast.

Can this new support be confirmed?

[ssbarnea]

Hold your horses..., the new macos-14 runners which are using arm architecture, have any kind of virtualization disabled. That means that is currently impossible to use any container engines on them (docker or podman), as both needs virtualization.

At least on intel ones you can run container engines.

Basically there is no way to build a native arm64 container using any github runners.

[maxkratz]

Basically there is no way to build a native arm64 container using any github runners.

I think it depends on what you mean by "native arm64 container". Currently, you can't build an arm64 container without any kind of virtualization/emulation on GitHub-hosted runners. But this doesn't mean you can't build any arm-based container images. You can use, for example, QEMU on Intel-based runners.

This is an example configuration to build a Docker image for amd64, arm64, and arm/v7:
https://github.com/maxkratz/docker_texlive/blob/main/.github/workflows/build-only.yml#L11-L26

[sebrandon1]

Another user here trying things on the free tier since the free-tier runner update this week.

When I try to add the user to the libvirt group on ubuntu-latest runners today with:

sudo usermod -a -G libvirt $USER

I run groups immediately after and I see:

docker adm systemd-journal

Is there a way to get the user to show up as part of the libvirt group without reloading the terminal?

[pkit]

try sudo bash -c 'exec su -l $USER'
I think these images have passwordless sudo only

[pkit]

It will probably kill all env variables though. Eh...

[pkit]

sudo -E bash -c 'exec su -l '$USER

[sebrandon1]

That seemed to "work" but groups still didn't reflect the change.

[pkit]

Debug it. It works.

$ sudo useradd test1
$ sudo usermod -a -G libvirt test1
$ sudo -E bash -c 'exec su -l 'test1
su: warning: cannot change directory to /home/test1: No such file or directory
$ id
uid=1002(test1) gid=1002(test1) groups=1002(test1),135(libvirt)
$ 

Just in case: you need to run in in EVERY action. As it falls back to previous login session each time.

[Stromweld]

Is this confirmation then that nested virtualization works? Would love to get the bento project working on public runners. Need hyper-v vmware fusion, parallels, virtualbox, qemu(libvirt, kvm) to work so we can build vagrant boxes for all major providers.

[hassanazharkhan]

I attempted to use it in a private repository today, but it's not functioning. Do you have any suggestions on how to activate this feature for private repos? or is it only for public repos?

[antoineco]

@hassanazharkhan only public repositories get access to these runners. Private repositories default to smaller runners. See https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#cloud-hosts-used-by-github-hosted-runners

[pkit]

@hassanazharkhan just host your own runners for private repos. These runner-images are created for hosting your own runners after all.

[alexellis]

@hassanazharkhan we support nested virt with actuated - feel free to take a look, reach out - https://actuated.dev/blog/kvm-in-github-actions

[nebuk89]

We are currently migrating the private runners to a new underlying SKU which should also support nested virt as well :) watch this space.

[nebuk89]

This has now shipped :) All Linux runners are now on a SKU that supports nested virtualization

[kasperk81]

@maxkratz and if we try to switch it with dockercli it deosn't work and that is because nested virtualization in hyper-v is not enabled. hence my original question "can we now run linux containers on windows?"

[JojOatXGME]

@kasperk81 Mhh... I haven't tried docker, but (nested) virtualization was definitely working for WSL2 on Windows when I tested it at the end of January. WSL2 is also using Hyper-V under the hood. Maybe Nested Virtualization is somehow disabled for docker in GitHub's runner configuration? I mean, you also still have to re-configure your Linux runner when you want to use KVM. Anyway, I am not very familiar with Docker, so no idea if that is actually possible. Could you share or link the specific error message you are getting? That might improve the chance that someone familiar with Docker could help you.

EDIT: While reading the stackoverflow page linked above, I have seen that Docker on Windows may use WSL2 under the hood. WSL2 was not installed on Windows Runners when I tested it. While I have no idea whether it makes any difference, you may try to install WSL2 first? I used the following action to install it back in January. Anyway, as I have said, I don't really know Docker and I have no idea if that actually makes sense to try.^^

      - name: Update WSL to enable support of WSL2
        run: |
          wsl --update
          wsl --version

[kasperk81]

@JojOatXGME docker is "configured" by a script in this repo so it is the right place to discuss this stuff

[JojOatXGME]

@kasperk81 I haven't questioned that? I just wanted to help you find the reason why it may not work, so that we may be able to find a solution. Anyway, if you just wanted to make GitHub aware that Docker is not working for Linux containers out of the box, that is also fine for me. I don't have any other suggestions you could try anyway. Have a nice day.

[nebuk89]

Sadly this isn't feasible on Windows :( the SKU we are using doesn't have this support yet. As soon as it does (I chased the MSFT team again) we will happily do the work to get this enabled <3