Virtual Machine with vGPU Unlock for single GPU desktop

This document catalogs my working setup for configuring an Arch Linux host OS to run a Windows 10 VM while passing through a virtualized GPU.

Sources

The resources I used during my own install. This document is a compilation of the instructions and files provided by these people.

Guides

• vGPU Wiki
• PCI passthrough via OVMF
• Proxmox 7 vGPU by wvthoog
• vGPU YouTube guide by Craft Computing
• Looking Glass
• Looking Glass YouTube guide by Pavol Elsig

Sources

• vGPU unlock scripts by DualCoder
• vGPU_LicenseBypass by KrutavShah
• vGPU unlock patch for kernel 5.12 by rupansh

Configuration

The what and how. Mostly described in suggested installation order unless otherwise stated.

Host

Operating system

Configuration of the host OS for virtualization using KVM and device passthrough using OVMF, IOMMU, and VFIO.

Arch Linux

Kernel 5.12.15-arch1-1

Kernel modules

• vfio
• vfio_iommu_type1
• vfio_pci
• vfio_virqfd

How to add

1. Edit /etc/mkinitcpio.conf
2. Add MODULES=(... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ...) to the MODULES line.
3. Add HOOKS=(... modconf ...) to the HOOKS line.
4. Update update-initramfs -P (use mkinitpico -P for manjaro)

Boot parameters

• amd_iommu=on
• iommu=pt
• vfio_iommu_type1.allow_unsafe_interrupts=1
• kvm.ignore_msrs=1
• blacklist=nouveau

How to add

1. edit /etc/default/grub
2. Add to GRUB_CMDLINE_LINUX_DEFAULT:

GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 blacklist=nouveau ..."

1. Update GRUB update-grub

Hardware

The physical devices present on my system.

• CPU: AMD Ryzen 7 5800X (16) @ 3.800GHz
• GPU: NVIDIA GeForce GTX 1060 6GB
• Memory: 16MiB
• 1440p Monitor

Software

What needs to be installed.

Arch Packages

linux

The OS kernel.

• Version: 5.12.15

1. Install downgrade from AUR (using yay or other AUR helper)
2. downgrade linux
3. Select 5.12.15

1. Download package curl -O https://archive.archlinux.org/packages/l/linux/linux-5.12.15.arch1-1-x86_64.pkg.tar.zst
2. Install with pacman -U linux-5.12.15.arch1-1-x86_64.pkg.tar.zst

linux-headers

Needed to build kernel modules, particularly the Nvidia driver.

• Version: 5.12.15

1. Install downgrade from AUR (using yay or other AUR helper)
2. downgrade linux-headers
3. Select 5.12.15

1. Download package curl -O https://archive.archlinux.org/packages/l/linux-headers/linux-headers-5.12.15.arch1-1-x86_64.pkg.tar.zst
2. Install with pacman -U linux-headers-5.12.15.arch1-1-x86_64.pkg.tar.zst

virt-manager

Used to create and run virtual machines.

libvirt

kvm

(Kernel-based Virtual Machine). A hypervisor. Used for virtualization.

qemu

looking-glass

Used to view VM in graphical display.

iptables-nft

For networking with the VM.

edk2-ovmf

dnsmasq

Used for VM networking.

mdevctl

Used to create vGPU devices

scream

Used to read audio stream from network.

1. Install using package manager.
2. Create systemd-file

   [Unit]
   Description=Scream network audio client
   
   [Unit]
   Description=Scream Receiver
   After=pulseaudio.service network-online.target
   Wants=pulseaudio.service
   
   [Service]
   Type=simple
   ExecStartPre=/bin/sleep 3
   ExecStart=scream -i virbr0
   Restart=always
   RestartSec=1
   
   [Install]
   WantedBy=default.target
         

Python packages

frida

1. Install python-pip using pacman.
2. Run /bin/pip install frida

1. pip install frida

Graphics setup

Nvidia Driver

1. merged driver
2. vgpu_unlock
3. vgpu_unlock_5.12 patch

cd /home/user/Downloads/NVIDIA-Linux-x86_64-460.73.01-grid-vgpu-kvm-v5
  

./nvidia-installer --dkms
  

nano /lib/systemd/system/nvidia-vgpud.service
  

Replace ExecStart with:

ExecStart=/root/vgpu_unlock/vgpu_unlock /usr/bin/nvidia-vgpud
  

nano /lib/systemd/system/nvidia-vgpu-mgr.service
  

Replace ExecStart with:

ExecStart=/root/vgpu_unlock/vgpu_unlock /usr/bin/nvidia-vgpu-mgr
  

systemctl daemon-reload
  

nano /usr/src/nvidia-460.73.01/nvidia/os-interface.c
  

Under #include “nv-time.h” insert this line

#include "/root/vgpu_unlock/vgpu_unlock_hooks.c"
  

nano /usr/src/nvidia-460.73.01/nvidia/nvidia.Kbuild
  

Add this to the bottom of the file.

ldflags-y += -T /root/vgpu_unlock/kern.ld
  

dkms remove -m nvidia -v 460.73.01 --all
dkms install -m nvidia -v 460.73.01
  

chmod -R +x vgpu_unlock

vGPU Profiles

• nvidia-48 GRID P40-3Q

mdevctl types
  

I chose nvidia-47 (GRID P40-3Q) because it has 3Q at the end of its name. This means I can have two 3GB vGPU’s running.

lspci | grep VGA
PCI_ADDRESS=$(lspci | grep VGA | grep -Po '\d\w:\d\d.\d')
echo $PCI_ADDRESS
  

UUID1=$(uuidgen)
UUID2=$(uuidgen)
echo $UUID1 $UUID2
  

mdevctl start -u $UUID1 -p 0000:$PCI_ADDRESS -t nvidia-48
mdevctl start -u $UUID2 -p 0000:$PCI_ADDRESS -t nvidia-48

mdevctl define -a -u $UUID1
mdevctl define -a -u $UUID2
  

VM Image

The virtual machine configuration.

• Chipset: Q35
• Firmware: UEFI x86_64: /usr/share/edk2-ovmf/x64/OVMF_CODE.fd
• Hypervisor KVM
• Spice Server
• Video QXL
• MDEV
• shmem looking-glass

How to install

In virt-manager, create virtual machine.

Apply above settings minus “MDEV” and “shmem looking-glass”.

Start VM and install OS.

Shutdown guest.

Add looking-glass shmem device.

The size is determined like this.

<shmem name='looking-glass'>
  <model type='ivshmem-plain'/>
  <size unit='M'>64</size>
</shmem>

Make a file /etc/tmpfiles.d/10-looking-glass.conf

#Type Path               Mode UID  GID Age Argument

f /dev/shm/looking-glass 0660 user kvm -

Remove tablet device

Add MDEV device.

Where $UUID1 is the UUID of one of the mdev vGPU’s created earlier.

<hostdev mode="subsystem" type="mdev" managed="no" model="vfio-pci" display="on">
  <source>
    <address uuid="$UUID1"/>
  </source>
</hostdev>

Install all guest software.

Shutdown guest.

Either set Video model to “none” or install guest Nvidia driver​.

How to run

1. Start VM. (May need to click run several times if error related to “vfio-<bus>” occurs.
2. Run looking-glass client

   export SDL_VIDEO_X11_VISUALID=
   looking-glass-client
       

Guest

Inside the VM.

Operating system

• Windows 10

To download (within guest)

• virtio guest tools
• spice guest tools
• looking glass host
• virtio PCI RAM controller driver
• Nvidia GRID driver
• vGPU_LicenseBypass
• Scream audio driver

Software

Looking Glass Host

How to install

1. Download components.
2. Update driver in Device Manager->System Devicess->PCI standard RAM Controller to virtio PCI RAM controller driver.
3. Install virtio guest tools and spice guest tools.
4. Install looking glass host by running looking-glass-host-setup.exe as administrator.
5. VM.

Nvidia GRID driver

How to install

1. Run Nvidia GRID driver installer.
2. Reboot.
3. In Nvidia Console set resolution. My monitor used 2560x1440.
4. Run vGPU_LicenseBypass.

Scream audio driver

How to install

1. Install driver.
2. Change sound source to Scream (WDM).

My files

• Final QEMU Windows 10 VM image
• mdev device creation script

Suggestions

Since there is a lot of messing with the kernel and graphics drivers, it’s important to have a plan for recovering your host OS.

• Use timeshift or other auto-snapshotting of the host.
• Have a recovery drive available.

Maintenance

Preventing accidental upgrades

In order to avoid accidentally upgrading these carefully chosen package versions, make sure to ignore updates to them:

IgnorePkg = linux linux-headers nvidia-dkms nvidia nvidia-utils nvidia-settings opencl-nvidia libxnvctrl