Newbie need some help for GPU passthrough

Hi there

I am currently struggeling with the GPU passthrough on Fedora 28 to a Windows VM and need some help.
I have two nVidia Grafik cards (780/1080), so i did install the nVidia Driver in Fedora (https://www.if-not-true-then-false.com/2015/fedora-nvidia-guide/). After that i did the tutorial from Levelone (https://level1techs.com/article/ryzen-gpu-passthrough-setup-guide-fedora-26-windows-gaming-linux). With some help from the skript of another post (GPU passthrough for lazy ppl on fedora 28 (using ansible)) i did get the VM running, except for the GPU passthrough.
For that i did:

  1. dnf update (4.18.10-200)
  2. /etc/default/grub -> iommu=1 amd_iommu=on rd.driver.pre=vfio-pci
  3. Updated grub (grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg)
  4. /etc/modprobe.d/vfio.conf -> options vfio-pci ids=…
  5. /etc/dracut.conf.d/local.conf -> add_drivers+=“vfio vfio_iommu_type1 vfio_pci vfio_virqfd”
  6. Installed Windows on a NVME SSD (directly from an USB iso on the SSD without Fedora running)
  7. Installed Windows nVidia drivers
  8. Passthrough NVME SSD in VM Manager

Without GPU passthrough the VM is running fine. When i add the GPU as PCI device an start the VM, then the VM Manager is crashing. After that i have to reboot Fedora, otherwise the Windows VM does not show in the manager again.

What did i miss?
How can i control my gpu settings and to be shure, that the second GPU is not running?

thank you for helping me and sorry for my bad english :blush:

have you tried blacklisting the nvidia and nouveau drivers in modprobe.d? they might still be binding to the card. Also what is the output of dmesg | grep -i vfio and lspci -nnk -d 10de:13c0 (with 10de:13c0 being the id of the card you want to pass through) they should tell you if the vfio driver has bound to the card or if its still grabbing the nvidia drivers.

You were right. The GPU runs still with nvidia driver.
With dmesg | grep -i vfio the output is empty
With lspci -nnk -d 10de:1b06 the ouput is: Kernel driver in use: nvidia, Kernel modules: nouveau, nvidia_drm, nvidia
I did try to blacklist the drivers (nouveau,nvidia,nvidia_drm) but did not work. I tryed to pass through the other GPU as well but it did not work either. i tried different options from google but nothing works. It does look like, that no settings in the /etc/modprobe.d/vfio.conf will change somthing.

Is the problem, that i have two nVidia GPUs?

I did the guide first as well and it didn’t work for me either. After some experimenting I realized my board is uefi, but for some reason the actual grub configuration which is used on startup is /boot/grub2/grub.cfg

I’m by no means an expert on grub and its configuration. I got confused on that topic and its on my list for a closer examination. As I understand from what I read so far, it should have worked with the guide.

So instead of the the guides suggestion I used:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Thank you for the advise. With your Command it worked :smile:
Now i have a problem with Windows, but hopefully i will be able to get it running.

Thank you two for helping me.

ok let me know cause I’m also hanging on the windows part :smiley:

Yesterday i had some issue with the CPU passthrough of my TR 1950x (Windows 10 1803 as guest with QEMU/KVM, BSOD under install). But i did get it running after i did what was suggested in the last post.
Now i have the same problem again. After switch back to coreduo, the Windows VM startet successfully. Now i have the Error Code 43 to deal with of the nVidia GPU. It is like a never ending storry :face_with_raised_eyebrow: but i feel i am getting closer to the finish line :yum:

problem 43, with a lot of other issues, is covered here

Thank you for the additional information.
Sadly i can not get it going without the error 43. Thank you nVidia for that … :pensive:

To get past the error 43 open your VMs XML file and change the domain line to:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

and somewhere within the domain tag usually at the end of the file just before the domain closes:

<qemu:commandline>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='host,hv_time,kvm=off,hv_vendor_id=null'/>
</qemu:commandline>

which should hide to the Nvidia drivers that your using a virtual machine and get past the error.

Not working for me. Still the Error 43. I have reinstalled the Driver but nothing is working. I tried so many things. It is realy frustrating.
What was the Order you did to get it work?

Did you run virsh define afterwards and made sure it hasn’t overwritten with the old config? sometimes this can happen if you have the vm, virt-manager running or if you made a mistake with the changes. All I did was make the changes to the file and redefine the configuration before reloading the vm again, here is a copy of my configuration with the changes for reference.

https://hastebin.com/posakanozi.xml

Yes i did the virsh define. Too be shure i did a reboot and controlled id again. I did compare your and my configuration and found that i have a hyperv and kvm section under features. I did delete that for a test but it did not work.

Here is my configuration maybe you see what is wrong: https://hastebin.com/owihabopid.xml

I don’t see anything overly wrong with your configuration. Are you using OVMF for this machine? I’m not sure if it’d make a difference but I can’t see the configuration for it and its the only major change between the two.

Is this the option under Overview Hypervisor Details Firmware? I saw some tutorial where they change there to UEFI, but on my machine i can not change the setting. There is only BIOS.

How can i change that?

You have to install the OVMF firmware with

sudo dnf install edk2-ovmf

Try the following in your XML file

<features>
    <kvm>
      <hidden state='on'/>
    </kvm>
</features>

if you already have a “features” block, make sure to add it inside that.

It does work, i can not beleve it.
I think it was the OVMF instead of the BIOS after all, thank you @MichaelLindman.
I think that went wrong:
I installed Windows from USB directly to the SSD. With the UEFI instead of the BIOS it did not boot. So i made a new installation of Windows over the VM setup to the nvme SSD. After that i did take my old configuration xml and it is working now. May be its clear for others but it was not for me that i have to install Windows that way.

Thank you all for the support :smile:

1 Like

If you start from scratch, run my ansible play. On fedora it should work like a charm. If not let me know so I can update it. I already put the fix for error 43 in the vm template that is part of the ansible play. If you need help running it let me know.

sorry i did saw you post too late. I had already installed it from scratch by hand. But i did use your script as guideline, so its nearly the same. What is different from your template is the problem i had to solve with the CPU passthrough.