VM Passthrough (Ryzen + RX 580) (Working but issues with Driver updating)

joykiller · February 21, 2019, 4:44am

So as the title suggests everything is working fine. I use a modified version of @powerhouse qemu-kvm build. https://heiko-sieger.info/running-windows-10-on-linux-using-kvm-with-vga-passthrough/

However, I did rebuild qemu from source as current LM/ubuntu repo is old and added networking/usb/lan etc to configuration build.

Problem is when I go to update RX 580 AMD Drivers, It gets to around 40% then screen goes to reset the drivers (Flicker stage) it just goes black screen, never to return. Also on boot it just doesn’t boot up, freezes on the spinning dots.

No errors in the console, but im not running verbose (Not sure how, let me know if you know how to add that to the config file to spitout) If you have had this issue and learned a way to fix it let me know.

Would love to have most recent drivers that work.

Was doing some research into it, supposedly older drivers work somewhat better. So I will try a few revisions down. See how my luck goes.

also, I duno how it happened maybe some of my hardware was “used” but sold as new on amazon. But I ended up with a fully licensed VM windows 10. First time installing windows on any of this hardware since i used to be linux for the past 6 years.

-Edit- As per the Drivers… Keep getting Error 43.

joykiller · February 21, 2019, 10:20am

However, with windows loaded 580 drivers, I still can game so cant complain…

Getting great FPS on Apex Legends that game is super fluid and polished. Max of around 90fps lows around 64fps with maxed out graphics settings. on my rx580.

itoffshore · February 21, 2019, 2:43pm

I had the same problem with AMD driver installation stopping at 40ish% on a Rx 570 (but this probably applies to all AMD GPUs).

To solve this problem you need 2 x monitors:

Windows Monitor (2) start up the Windows VM & login - enable Remote Desktop Connections.
Linux Monitor (1) - connect to the VM using RDP (e.g with Remmina) - this will logout the Windows Desktop on monitor (2) - leave it logged out & monitor (2) connected.
On the RDP session from Linux start the AMD driver installation.

The Windows Monitor (2) will flicker at the 40ish% mark & you may lose the RDP session from Linux but don’t worry if it does - simply reconnect the RDP session from Linux & finish the driver installation.

Tested up to the current stable drivers 19.1.1

joykiller · February 21, 2019, 8:35pm

Interesting I got 5 other monitors in my build on linux host side so I’ll give this a try thx for info!

joykiller · February 22, 2019, 5:44am

reconnected via RDP when main monitor whch was logged out went blank. Doesnt want to connect. since the screens black even on RDP. Just sits there spinning… So basically no change, rebooted vm and crash / corruption as it does.

Was able to restore then went into safemode, installed drivers that way. It worked fine, Then rebooted and boot screen stalled again. lol… doesnt seem like there is anyway to do this lol.

Guess this is the best i can hope for, Not entirely base install, but went into device manager, did online update and it loaded a slightly newer month. But still a few years old lol…
drivers

I’m at a loss for Idea’s if you got any sling it my way, Maybe if i try a diffrent approach to passthrough maybe through libvirt xml might have a better outcome? But dont they all use basically qemu cli?

itoffshore · February 22, 2019, 6:58pm

I do passthrough with libvirt & have a 2nd monitor inside the vm with the Red Hat QXL driver that Virt-Manager automatically creates - but I have the Windows Display settings set to only show on monitor (1) with the AMD GPU.

You need to start the driver install from the RDP session in Linux not from the windows monitor,

joykiller · February 22, 2019, 8:45pm

Yep, rdp logs you out of main vm monitor that started, installed driver rdp monitor froze at same time main monitor went black. Tried to rdp back in but got consistant spinning no reconnection.

While was in rdp, had to remove driver from device manager, because I was getting library 87 path error, once I did that it allowed install which caused black screen and boot corruption.

I have main monitor inside Passthrough GPU, and RDP from a linux host.

itoffshore · February 24, 2019, 1:05pm

The only difference then is the fact I’m using libvirt - it must be doing something extra behind the scenes.

lessaj · February 24, 2019, 4:05pm

SInce you’re using libvirt what CPU type is configured? Host-passthrough? Curious if it works with type set to core2duo.

joykiller · February 24, 2019, 8:33pm

Didn’t think about that, mines still set to the original for the tutorial I used. I guess I probably should look at redoing the kvm via libvirt. It’s possible to just run the os already installed with libvirt ya? Just wondering if it would require a reinstall I don’t see why it would but just curious.

lessaj · February 24, 2019, 10:18pm

Yes you should be able to re-use the existing install.

itoffshore · February 25, 2019, 9:53am

@lessaj - my host CPU is an FX 8370 & I set host passthrough in libvirt so the Windows VM recognises the CPU as an FX 8370.

I passthrough all 8 cores to the VM with the first 6 cores pinned & set to use the FIFO real time scheduler. I also set the first 6 cores as tickless via grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1G hugepages=12 nohz_full=0-5 rcu_nocbs=0-5 rcu_nocb_poll"

if you have plenty of RAM allocating hugepages via grub saves a lot of hassle.

For the last 2 cores I also pin one to the emulator thread & one to iothread. Linux doesn’t care if it has free use of core zero (Windows does).

This setup gives me good enough performance to run the Division / Hitman / Dishonored 2 / Watch Dogs / GTA5 all on high. I no longer reboot into Windows 10.

loadrunner · March 10, 2019, 1:44am

I also have this problem.
After reading this I installed 18.9.3, which works without any issues.

I don’t need the latest driver, but I would be curious to understand the issue, as this doesn’t seem to affect everyone.

ArnC · July 30, 2019, 9:06pm

HI people,

I know that this thread is a bit old, but I am running into the same problem than @loadrunner . @itoffshore, would you mind sharing your xml configuration ?

Thanks a lot,

ArnC

itoffshore · July 31, 2019, 7:16pm

@ArnC - setting up a Windows rdp session with Remmina when updating the drivers is still working for me.

If it helps here is my libvirt xml - this should also help anyone trying to do vfio with AMD FX 83xx:

<domain type='kvm'>
  <name>win10</name>
  <uuid>7be611bf-4036-45d8-9ef6-5a376ccb56a5</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>11718750</memory>
  <currentMemory unit='KiB'>9765625</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <iothreads>1</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='5'/>
    <vcpupin vcpu='6' cpuset='6'/>
    <vcpupin vcpu='7' cpuset='7'/>
    <emulatorpin cpuset='6'/>
    <iothreadpin iothread='1' cpuset='7'/>
    <vcpusched vcpus='0' scheduler='fifo' priority='1'/>
    <vcpusched vcpus='1' scheduler='fifo' priority='1'/>
    <vcpusched vcpus='2' scheduler='fifo' priority='1'/>
    <vcpusched vcpus='3' scheduler='fifo' priority='1'/>
    <vcpusched vcpus='4' scheduler='fifo' priority='1'/>
    <vcpusched vcpus='5' scheduler='fifo' priority='1'/>
    <iothreadsched iothreads='1' scheduler='fifo' priority='99'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-3.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/ovmf/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <bootmenu enable='no'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vpindex state='on'/>
      <synic state='on'/>
      <stimer state='on'/>
      <reset state='on'/>
      <vendor_id state='on' value='KVM Hv'/>
      <frequencies state='on'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough' check='full'>
    <topology sockets='1' cores='8' threads='1'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='invtsc'/>
    <feature policy='require' name='svm'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='apic'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' present='no' tickpolicy='catchup'/>
    <timer name='pit' present='no' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='kvmclock' present='no'/>
    <timer name='hypervclock' present='yes'/>
    <timer name='tsc' present='yes' mode='native'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='block' device='lun'>
      <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
      <source dev='/dev/disk/by-id/ata-KINGSTON_SA400S37120G_50026B768268BFFE'/>
      <target dev='sda' bus='scsi'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/disk/by-id/ata-TOSHIBA_MG03ACA400_16B6KI91F'/>
      <target dev='sdb' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='3'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/data/vm-images/iso/Win10_1903_V1_English_x64.iso'/>
      <target dev='sdc' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/data/vm-images/iso/virtio-win-0.1.160.iso'/>
      <target dev='sdd' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <driver queues='1' iothread='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xc'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:86:b4:dc'/>
      <source network='default'/>
      <model type='virtio'/>
      <link state='up'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x14' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x02' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </memballoon>
    <shmem name='looking-glass'>
      <model type='ivshmem-plain'/>
      <size unit='M'>32</size>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x01' function='0x0'/>
    </shmem>
  </devices>
</domain>

With linux-hardened-5.2.18.a-2 kernel in Manjaro the above libvirt settings & these grub settings currently give me 60fps in Hitman on an FX 8370 / Rx 570 8gb.

ArnC · August 13, 2019, 9:36pm

Thanks a lot @itoffshore, and sorry for the very late reply.

I finally solved my problem by updating to an intermediate driver version (04/18 if I remember well), then updating to the latest version using the RDP technique which worked. I also noticed that disabling Windows defender helped for the first update step, so I’m currently fully patched, even if I am not sure what went wrong in the first place.

This is a real pain in 2019 that Windows is more and more difficult to setup with KVM.

Anyway, thanks again Sir !