Interactively Configuring Alpine as a KVM Host

Last week, I did a shallow dive into why I want to use virtual machines instead of dual-booting, and why I picked Alpine Linux as my host operating system.

The associated scripts are on Github (https://github.com/sureshjoshi/alpine-kvm) and I want to spend a bit of time explaining parts of the configuration script. It's not a line-by-line replica (a lot of these lines won't run in isolation) - so if you want to run the script, make sure you grab it from: https://github.com/sureshjoshi/alpine-kvm/blob/main/configure-alpine.sh.

This configuration assumes no GUI or desktop environment - it's all from the terminal.

Prerequisites

  • A fresh install of Alpine Linux 3.19 (Extended) using 'sys' or 'cryptsys'
  • Intel CPU
  • Two GPUs (one for passthrough, one for host)
  • A working internet connection

Why Extended?

In an ideal world, I would have built this from the lighter Standard installer - but I needed some packages and firmware installed via the Extended installer to work with my wireless network card. If that's not the case, you can probably install off of Standard (untested).

I did experiment with making a custom Alpine ISO with exactly what I required, but I decided that this wasn't something I wanted to bother with for each new version of Alpine.

Alpine's scripts to make new ISOs or chroots are awesome though. It only took about 10 minutes before I had setup my dev environment and built a custom ISO with new packages and post-install scripts. Compare that with Debian's custom installation for BeagleBone ...shudder... never again...

The other reason to use Extended is because it defaults to GRUB as a bootloader, and I'm just more familiar configuring GRUB.

Why sys/cryptsys?

Those just happen to be what I'm using, and what I've tested against. I had the idea to run the OS straight out of RAM - but just decided it wasn't worth the time or hassle to mess around with.

Running the script

After a fresh install of Alpine, you can curl/wget the configure-alpine.sh and run it on your system.

wget https://raw.githubusercontent.com/sureshjoshi/alpine-kvm/main/configure-alpine.sh
# Read through and closely examine the code in this file before running it
sh ./configure-alpine.sh

It is an interactive script which will best-effort setup your system as a QEMU/KVM host, allowing GPU passthrough to a Windows (or other) virtual machine.

The Alpine/Intel requirements are not necessarily dealbreakers, but this version of the script was only tested using those conditions. For example, this script could probably be modified for AMD by changing intel_iommu to amd_iommu in the configuration script, but that's untested.

Additionally, this script could theoretically work on a single-GPU system, but it would likely require some modifications and debugging/troubleshooting would be more difficult as this script binds the GPU at boot time, rather than on-demand (which a single-GPU system should prefer).

Preamble

These steps must be done in your BIOS and without them, there is no reason to go any further. Check out your motherboard's manual or just Google DuckDuckGo the manufacturer, since each BIOS is slightly different, or may have slightly different names for each of these steps.

  • Have you enabled virtualization in your BIOS? [y/n]
  • Have you enabled VT-d in your BIOS? [y/n]
  • Have you setup your initial display output to your secondary GPU (or iGPU) in your BIOS? [y/n]

APKs

Several of the following APKs are only available in the community repo, so this step adds main and community in case they weren't setup automatically.

# In /etc/apk/repositories
http://dl-cdn.alpinelinux.org/alpine/latest-stable/main
http://dl-cdn.alpinelinux.org/alpine/latest-stable/community

Just in case a new repo was added, update the apk cache - but just good practice anyways, as if you have internet problems, this should fall over.

apk update

Add the following sets of APKs.

These are just general utility libraries that are good to have installed in Alpine, ones I needed to support my hardware, or ones I have installed to learn more about my hardware. For a slightly smaller footprint, I could have also used dropbear instead of openssh - but I'm using this machine with added proxying and other, more advanced, ssh features that I don't yet know how to do in dropbear.

chrony
doas
less
lm-sensors
lm-sensors-detect
logrotate
openssh
wpa_supplicant

pciutils installs some tools on Alpine which are used later in the script. For example lspci which allows inspection of connected PCI devices (e.g. GPU).

pciutils

These are the libraries directly related to virtualization, QEMU, libvirt, and virt-install. If you're not on headless Alpine, you can also install virt-manager or similar for graphical control of your virtual machines.

Special shout-out to qemu-hw-usb-host, as without this I wasn't able to passthrough USB devices to my virtual machines and instead they errored out with a cryptic message ('usb-host' is not a valid device model name). I didn't realize I specifically needed to install a QEMU USB utility (instead assuming it would have been baked in or a dep).

libvirt-daemon
qemu-hw-usb-host
qemu-img
qemu-system-x86_64
virt-install

OVMF it the Open Virtual Machine Firmware, which allows enabling UEFI support in virtual machines (which makes launching Windows much easier).

ovmf

eudev and udev-init-scripts allow udev functionality without systemd. What this means practically is that the /dev/input/by-id directory will exist, and give your mouse/keyboard stable names as you plug and unplug them into different ports.

The alternative to eudev is to run cat /proc/bus/input/devices and then iterate cating through the associated /dev/input/eventX nodes and typing/moving your mouse until you find the event that represents your device. That will give you the associated virt-install input (e.g. --input evdev,source.dev=/dev/input/event123) to setup your device on the specific USB port it's currently inserted into. I did it for a few days, and it kinda sucks if you're someone who moves peripherals around a lot because you need to accommodate other USB devices.

eudev
udev-init-scripts

By default, Alpine uses lbusb from busybox - which would usually be fine, but it doesn't use available hardware data to provide human-readable device names. The alternative here is to use cat /proc/bus/input/devices - but this was miserable when I wanted to figure out my Bluetooth adapter - so I highly encourage this package.

usbutils

# busybox lsusb
# Bus 001 Device 006: ID 05e3:0608
# Bus 001 Device 005: ID 045e:0750

# usbutils lsusb
# Bus 001 Device 004: ID 1532:006e Razer USA, Ltd DeathAdder Essential
# Bus 001 Device 005: ID 045e:0750 Microsoft Corp. Wired Keyboard 600

Starting Services

Nothing too special here - just need to make sure the appropriate services are installed and started on boot. I read through the eudev files, and I thought that installing the udev service and/or installing udev-init-scripts would also trigger setting up the services, but after multiple reboots - it appears not. So, those are also done manually.

rc-update add libvirtd
rc-service libvirtd restart

rc-update add udev sysinit
rc-update add udev-trigger sysinit
rc-update add udev-settle sysinit
rc-update add udev-postmount default

rc-service udev restart
rc-service udev-trigger restart
rc-service udev-settle restart
rc-service udev-postmount restart

IOMMU

IOMMU is an acronym for Input/Output Memory Management Unit. The linked Wiki page is worth a read, but what it functionally means for VMs is that host hardware (e.g a GPU) can be directly used by guest VMs without the host needing to perform address, data, or memory translations, which would incur performance and/or latency hits on each call.

When setting these up via the kernel, IOMMU devices are mapped into "IOMMU groups" which are the physical devices which must be passed together to a VM. For example, in the following, IOMMU group 13 must be passed together (the GPU audio and video device), but passing group 13 won't affect any other groups.

IOMMU Group 0:
	00:02.0 Display controller [0380]: Intel Corporation Raptor Lake-S GT1 [UHD Graphics 770] [8086:a780] (rev 04)
...
IOMMU Group 13:
	01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] [10de:2504] (rev a1)
	01:00.1 Audio device [0403]: NVIDIA Corporation GA106 High Definition Audio Controller [10de:228e] (rev a1)

Enabling IOMMU is as simple as adding one (errr... two) flags to the GRUB kernel command line:

# In /etc/default/grub - update the following line
GRUB_CMDLINE_LINUX_DEFAULT="... quiet rootfstype=ext4 intel_iommu=on iommu=pt"

intel_iommu=on is the one that does the magic for Intel to enable IOMMU (and on AMD, supposedly it's amd_iommu=on - but I haven't tested).

I've completely cargo-culted iommu=pt, as depending on who you talk to - they can't pass-through their GPU at all (or there is reduced performance) without it. My system worked perfectly without that flag, but adding it didn't harm anything - and if it adds better support to other people, so be it. Intel also recommended it, so who am I to not include it?

Anyways, after adding that line to the default GRUB, run grub-mkconfig and then reboot. After rebooting, look for the IOMMU message in dmesg/

grub-mkconfig -o /boot/grub/grub.cfg

# Reboot

dmesg | grep "IOMMU enabled"
# [    0.084049] DMAR: IOMMU enabled

VFIO

VFIO is an acronym for Virtual Function I/O. It's a Linux kernel module that allows userspace access to IOMMU devices. So, basically, safe-ish hardware access for PCI passthrough.

Enabling VFIO requires enabling kernel modules and features, and then specifically calling out the GPU we want to passthrough so that it's accessible to VFIO and not the host.

# In /etc/mkinitfs/features.d/vfio.modules - add the following lines:
kernel/drivers/vfio/vfio.ko.*
kernel/drivers/vfio/vfio_iommu_type1.ko.*
kernel/drivers/vfio/pci/vfio-pci.ko.*

# In /etc/mkinitfs/mkinitfs.conf - add the vfio feature
features="ata base ide scsi usb virtio ext4 nvme cryptsetup keymap nvme vfio"

# list_iommu_groups
# IOMMU Group 13:
#   01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] [10de:2504] (rev a1)
#   01:00.1 Audio device [0403]: NVIDIA Corporation GA106 High Definition Audio Controller [10de:228e] (rev a1)

# In /etc/modprobe.d/vfio.conf - add the following lines, where YOUR:PCIID is determined via `list_iommu_groups` and the 8-digit ID in square brackets
options vfio-pci ids=YOUR:PCIID
options vfio_iommu_type1 allow_unsafe_interrupts=1
softdep igb pre: vfio-pci

After updating, you'll need to re-build your kernel ramdisk.

mkinitfs

Finally, go back to your GRUB config and add vfio modules to ensure they're loaded at boot, so the GPU is accessible as early as possible (deferring this to after the OS loaded never worked reliably for me).

The two main items to look for are that the VFIO driver is loaded, and that your passed through GPU is "in use" by vfio-pci.

# In /etc/default/grub - update the following line to add the vfio, vfio-pci, and vfio_iommu_type1 modules
GRUB_CMDLINE_LINUX_DEFAULT="modules=sd-mod,usb-storage,ext4,nvme,vfio,vfio-pci,vfio_iommu_type1 ..."
grub-mkconfig -o /boot/grub/grub.cfg

# Reboot

dmesg | grep VFIO
# [    0.739825] VFIO - User Level meta-driver version: 0.3

lspci -nnk | grep -A2 VGA
# 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] [10de:2504] (rev a1)
#   Kernel driver in use: vfio-pci

But don't do all that

The stuff I mentioned above was strictly informative. As I mentioned from the start, use the interactive configuration script in my alpine-kvm repo instead.

For those who prefer ansible (like me), I'm looking into turning this into a playbook and leverage ansible's ensure mechanism to prevent unnecessary backups and reboots - rather than my budget version in the shell script.

System info

I mentioned in the previous post that it was annoying to see KVM wikis and docs that didn't explicitly mention versions, because this content is absolutely not evergreen - the lifetime is probably closer to 1-2 years, unless it's constantly updated.

CPU: 13th Gen Intel Core i5-13600K
Motherboard: Gigabyte Z790I Aorus Ultra
GPU: Nvidia GeForce RTX 3060
cat /etc/alpine-release
# 3.19.1

qemu-system-x86_64 --version
# QEMU emulator version 8.1.5

libvirtd --version
# libvirtd (libvirt) 9.10.0

virt-install --version
# 4.1.0

What's next?

References

Here is a list of the resources I used to build this script. The ArchLinux wiki was generally the most useful, followed by the Alpine Linux wiki for specific substitutions, and the Fedora documentation is generally great. After those, I used a few other blogs and Reddit to cherry pick snippets and compare/contrast implementations over time.

I'd say the previous 5 links are required reading for QEMU/KVM with GPU passthrough. The following are moreso nice references that I used for specific parts of the KVM, GPU Passthrough, and VM creation process.