Troubles with GPU Passthrough

Software & Operating Systems Linux
, ,
1

Hello folks, long time no post.

Hope you are all keeping well.

I recently picked up some kit to start looking at GPU passthrough with KVM, but I’m now running into a little trouble.

I have an Intel Z75 board with a CPU supporting integrated graphics, and an Nvidia graphics card that I want to pass through. The board supports Intel VT-d.

My Linux flavour of choice is Ubuntu 16.04 Gnome. I’m keen to get it working on this version.

After installing and setting up a network bridge, KVM and IOMMU I can see the IOMMU groups and the graphics card shows up as two devices, a GPU and an audio device.

This is where I’m encountering the issue.

I want to use the integrated graphics for the host system and pass through the GPU for the virtual machine, however both the nvidia GPU and Intel integrated are grabbed by Nouveau at startup.

I need Nouveau for the host, but somehow make it leave the GPU alone.

So far I have added the PCI IDs into etc/initramfs-tools/modules so that it looks like the below.
softdep nouveau pre: vfio vfio_pci
softdep nvidiafd pre: vfio vfio_pci
vfio
vfio_iommu_type1
vfio_pci ids=10de:1244,10de:0bee
options vfio_pci ids=10de:1244,10de:0bee
options vfio_pci ids=10de:1244
options vfio_pci ids=10de:0bee
nouveau

This seems to have worked for the audio device, which shows the kernel driver as vfio_pci, but the GPU is still controlled by Nouveau.

Could anybody give some insight as to how I may be able to get this working?

Thanks in advance, Fantom.

2

Have you tried changing the Nvidia GPU driver to the proprietary one? Also, make sure you pass through both the video and audio device to the VM, you cannot split them or just do one or the other

Also, I’m assuming your GPU is plugged into a PCIe slot that is using lanes directly connected to the CPU, and not through a PLX chip

3

Yes, the GPU is connected directly to the CPU.

I had a look around and found this thread:

Even though the replies suggested it may not work I tried it out anyway.

I added vfio_pci.ids=10de:1244,10de:0bee to my boot line in /etc/default/grub and commented all previously added lines in /etc/initramfs-tools/modules, before updating both GRUB and initramfs.

Interestingly this worked for me, then showing both devices using the vfio_pci driver.

I even tested it passing the PCI devices through to a Ubuntu VM. The system didn’t crash and both devices appeared as expected.

Hopefully I will now be able to install Windows and install the drivers to test it fully.

Thank you for your assistance so far, I’ll probably be back for more soon :slight_smile:

1 Like
4

It’s also possible you may need to boot the WIndows VM without any devices passed through and install the VM tools, but I’d say just try it without doing that first

5

Hi again folks!

Almost can’t believe it’s been 19 days! I’ve been working on this on and off for a while.

Apologies for bumping this thread, please let me know if I should have started a new one instead as I know this borders on a Necrothread.

Anyways, I’ve done some tinkering and once again am here to seek some more insight into GPU passthrough with KVM.

So far I’ve been able to variously passthrough PCI devices with some success, however I’ve tripped up on the inevitable Code 43 Nvidia driver error on my Windows guest.

I initially set up the guest with the below settings in the hope that I could sidestep the whole Code 43 thing, but to no avail.

<os>
 <smbios mode='host'/>
</os>

<cpu mode='host-passthrough'/>

<features>
 <kvm>
  <hidden state='on'/>
 </kvm>
</features>

This shows as expected in Windows, appearing mostly to be the host system, including the model of the motherboard.

No luck, still getting Code 43.

I updated the CPU config to the below.

<cpu mode='host-passthrough'>
 <feature policy='disable' name='hypervisor'/>
</cpu>

Still Code 43.

I then tried adding the below to my config, but ran into a different issue.

<features>
 <hyperv>
  <vendor_id state='on' value='1234567890ab'/>
 </hyperv>
</features>

When trying to save this config it throws an “unsupported HyperV Enlightenment feature: vendor_id”.

Not sure where to go from here, any help would be much appreciated.

In case it’s necessary, the guest is Windows 7, and the Nvidia driver did let me install, appeared normal and restarted, but then gave Code 43. Not sure if that’s normal.

Many thanks in advance, Fantom.

6

Hi,

I think you need to go back and fix your post because some of what you were trying to show did not appear. That said, please try this. The value for vendor_id has to be 12 characters, too bad adding the a makes it 13. :wink:

  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='fuckyounvidi'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <feature policy='disable' name='monitor'/>
  </cpu>
1 Like
7

Thanks for that, I found the Preformatted text button eventually :slight_smile:

I tried to make the changes you suggested, but it still throws “unsupported HyperV Enlightenment feature: vendor_id”.

I did add the other suggested changes, but it won’t allow check=‘none’ in the cpu section.

8

What version of libvirt and QEMU are you running? That option has been supported since QEMU 2.5. Does the top of your domain XML contain this? Might be helpful to provide the whole thing.

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
9

Hi,

My config file doesn’t have xmlns:qemu at all, just type=‘kvm’.

I will insert the whole config as it is currently below. Wall of XML incoming.

<domain type='kvm'>
  <name>Windows7</name>
  <uuid>b148234b-1401-4040-9d44-7993f2da664a</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static' current='1'>16</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-xenial'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='host'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='4'/>
    <feature policy='disable' name='monitor'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/Windows7.qcow2'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:86:53:85'/>
      <source bridge='br0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Thanks very much for your help so far :slight_smile:

Edit: I have now run “/usr/bin/qemu-system-x86_64 --version”, which returns “QEMU emulator version 2.5.0”.

10

Right on the edge of supporting the feature there with support added since libvirt 1.3.3 and qemu 2.5, not sure what libvirt version you’re on but you can check with libvirtd --version, may be a good idea to try to upgrade libvirt and qemu but I’m not sure of 16.04 will offer an updates for those packages. Okay try replacing the first line with the one mentioned and see if that does it, that’s usually required. I also see you’re not running UEFI which is okay but may not be happy for the GPU you’re trying to pass through even if you manage to start it with vendor_id in the domain XML but one step at a time.