vfio notes

## VFIO from scratch

by https://www.reddit.com/user/Chapo_Rouge

*The goal of this document is to provide an up-to-date (July 2016) way to setup a VGA
passthrough on QEMU/KVM with the help of VFIO on any Linux host.*

*This guide focus on Intel CPU and Nvidia GPU as I don't own another type of hardware, nevertheless, a good part of it is hardware agnostic.*

*This guide use the most recent KVM VGA passthrough method which is OVMF + vfio-pci (compared to Seabios + pci-stub) and requires a GPU with EFI ROM*

#### Hardware prerequisites :

+ VT-x & VT-d enabled hardware (CPU + motherboard)
+ 2 GPU
+ (optional) coffee or cold beers

#### Software used :

    Gentoo amd64, Linux kernel 4.6.x
    QEMU 2.5.1 + KVM
    virt-manager & libvirt
    OVMF (if using Windows >= 8.0)

### Part 1 : Host system setup

#### Kernel configuration
todo

    CONFIG_VFIO_IOMMU_TYPE1=m
    CONFIG_VFIO=m
    CONFIG_VFIO_PCI=m
    CONFIG_INTEL_IOMMU=y
    CONFIG_IOMMU_HELPER=y
    CONFIG_VFIO=m
    CONFIG_IOMMU_API=y
    CONFIG_IOMMU_SUPPORT=y
    CONFIG_KVM_VFIO=y

#### Kernel boot options

edit `/etc/default/grub` and add to GRUB_CMDLINE_LINUX

    intel_iommu=on

> not always needed ?

Then regenerate grub2 menu with

    grub2-mkconfig -o /boot/grub/grub.cfg

or add the kernel option at boot time.

#### Check IOMMU groups

> This command should report the various IOMMU groups from your machine (An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine.)

> if there's nothing it means IOMMU is not properly enabled.

    #!/bin/bash

    for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);
        	do echo "IOMMU group $(basename "$iommu_group")";
    	for device in $(ls -1 "$iommu_group"/devices/);
    		do echo -n $'\t'; lspci -nns "$device";
    	done;
    done

#### Isolating the GPU with vfio-pci

Get your vendor-id :

    lspci | grep -i vga

> note down the first number, it is the slot number i.e 01:00.0

    lspci -nns 01:00.0

> Note down the last number between "[]", this is the vendor-id, i.e 10de:1380

    lspci -nnk -d vendor-id

Edit `/etc/modprobe.d/vfio.conf` with the vendor-id from your GPU you want to isolate gathered from the previous command, in this __example__, the vendor-id is __10de:13c2__ for the GPU and __10de:0fbb__ for the audio

    options vfio-pci ids=10de:13c2,10de:0fbb

Add these modules to `/etc/conf.d/modules` (Gentoo/OpenRC specific)

    modules="vfio vfio-pci vfio_iommu_type1 vfio_virqfd"

module loading at boot is enabled by

        rc-update add modules boot

Reboot

#### Check that your GPU is correctly isolated :

    $ dmesg | grep -i vfio
    [    0.329224] VFIO - User Level meta-driver version: 0.3
    [    0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000 <-- Good

    $ lspci -nnk -d <VENDOR_ID>
    Kernel driver in use: vfio-pci <-- Good

Same check for your sound card

If lspci -nnk -d reports a "Kernel driver in use" which is not vfio-pci, there's something wrong.

You can blacklist module loading by settings up the blacklist in /etc/modprobe.d/blacklist.conf

    blacklist snd_hda_intel

Also check IOMMU & DMAR, output should contains :

    dmesg | grep "IOMMU\|DMAR"
    [    0.000000] ACPI: DMAR 0x00000000BD9373C0 000080 (v01 INTEL  HSW      00000001 INTL 00000001)
    [    0.019360] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap d2008c20660462 ecap f010da
    [    0.019362] IOAPIC id 8 under DRHD base  0xfed90000 IOMMU 0
    [    0.292166] DMAR: No ATSR found
    [    0.292235] IOMMU: dmar0 using Queued invalidation
    [    0.292237] IOMMU: Setting RMRR:
    [    0.292246] IOMMU: Setting identity map for device 0000:00:14.0 [0xbd8a6000 - 0xbd8b2fff]
    [    0.292301] IOMMU: Prepare 0-16MiB unity mapping for LPC
    [    0.292307] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

#### Kernel modules (.ko)

Make sure these modules are present when you run lsmod

    $ lsmod
    [...]
    vfio (?)
    vfio_iommu_type1 (?)
    kvm
    kvm_intel
    vfio-pci
    intel_iommu (?)
    [...]

#### QEMU / KVM / libvirt

Below example is for Gentoo & OpenRC, adjust for your distro.

    # emerge --ask qemu virt-manager libvirt git
    # rc-update add libvirt-guests
    # service libvirt-guests start
    # service libvirtd start

#### OVMF setup

Get OVMF via https://www.kraxel.org/repos/jenkins/edk2/ and choose the

`edk2.git-ovmf-x64-[...] RPM.`

Unpack it and copy `OVMF-pure-efi.fd` and `OVMF_VARS-pure-efi.fd` to /usr/share/ovmf/x64/, creating the directory if needed.

Add this line to /etc/libvirt/qemu.conf :

    nvram = [
            "/usr/share/ovmf/x64/ovmf_x64.bin:/usr/share/ovmf/x64/ovmf_vars_x64.bin"
    ]

Create a new machine with virt-manager and check "edit settings" before finishing the setup, change firmware from BIOS to UEFI (or to "Custom: /usr/share/ovmf/x64/ovmf_x64.bin") then boot the VM.

> If you're dropped to an EFI shell, make sure the ISO you wish to boot is correct, try with a recent Ubuntu release for instance.

#### Networking (Poor man's)

    emerge bridge-utils

Create bridge :

    brctl addbr br0
    brctl addif br0 eth0
    dhcpcd br0

Delete bridge :

    ifconfig br0 down
    brctl delif br0 eth0
    brctl delbr br0

Add a network card on guest, attached to br0

#### Unhinge QEMU (needed ?) :

Ref : http://www.firewing1.com/howtos/fedora-20/create-gaming-virtual-machine-using-vfio-pci-passthrough-kvm

Because QEMU normally runs sandboxed, we need to 'unhinge' it and give it root privileges so it can control hardware.

In /etc/libvirt/qemu.conf, add:

	user = "root"
	group = "root"
	clear_emulator_capabilities = 0

#### Common patches explained

__vgaarb__ aka __i915__ :

`x-vga=on`
If OVMF : vgaarb is not needed
If SeaBIOS : Requires vgaarb

__ACS Override__

If you find your PCI devices grouped among others that you do not wish to pass through, you may be able to seperate them using Alex Williamson ACS override patch.

### Part 2 : Guest VM setup

The XML configuration of the VM is located at `/etc/libvirt/qemu`

#### Possible bugs :

+ Nvidia Code 43 :

Recent nvidia drivers block the driver if it runs on top of an hypervisor, the workaround is to hide KVM :

via cli :

    -cpu [type],kvm=off

via libvirt :

> edit your VM XML definition with virsh edit <VM_NAME>

    <domain type='kvm'>
	...
      <features>
        <kvm>
          <hidden state='on'/>
        </kvm>
	...
      </features>
    </domain>


### Part 3 : Optimizations

Chipset : 440FX is ok

### Appendices : Biblography

#### Online resources :

+ https://www.reddit.com/r/VFIO
+ https://www.redhat.com/archives/vfio-users/
+ https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF