Unable to setup IOMMU/VT-d and VM on compatible hardware - Ubuntu 19.04

Hi,

I’d like to setup a VM via qemu/kvm/virt-manager and passing trough my GTX1070.
Long story short: I want to have a system running a windows 10 vm to run all games (and some CAD programs), instead of messing with proton or wine.
And go a step further by using some scripts to give the gpu back to the host so I could use it there. But that is optional. (something like this: https://www.youtube.com/watch?v=ssfvpLXK8po&t=208s)

My system specs are as follows:

  • Palit GTX1070
  • Intel i5 6500
  • Asrock Z170 extreme4
  • The rest is not important I assume.

According to the docs of arc.intel.com the CPU has VT-d capability.
In the docs of the motherboard it states also this should support VT-d.

The tutorials I’m following are these:

They are very much alike so I don’t think it matters which I use…

In the bios I have both “Intel virtualization technology” and VT-d enabled.
When looking at the chipset config it also states: “Virtualization technologies: Supported”
So it should work right?..

The steps I took so far:

  • Updated the bios to the newest version to ensure the problem does not ly somewhere there…
  • Did a clean install of Ubuntu 19.04
  • Installed the necessary packages for the vm to work properly: sudo apt install qemu-kvm libvirt-clients libvirt-daemon-system bridge-utils virt-manager ovmf
  • Blacklisted the nouveau drivers (is this even needed in my case?)
    /etc/modprobe.d/blacklist-nouveau.conf contains the following:
    blacklist nouveau
    options nouveau modeset=0
  • /etc/default/grub contains:
    –GRUB_CMDLINE_LINUX_DEFAULT=“quiet splash intel_iommu=on”
    – In a later stage i also tried to add iommu=1 but that does not help.
  • Ran “lspci -nn | grep -i nvidia” to look for the pci id’s that should be put in the vfio.conf file
    – That gave:
    01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
    01:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
  • put the correct id’s in /etc/modprobe.d/vfio.conf , which are:
    options vfio-pci ids=10de:1b81,10de:10f0
  • updated initramfs: sudo update-initramfs -u
  • And then rebooted my device.

So all is well at this stage.
After the reboot I checked with: dmesg | grep -E “DMAR|IOMMU” if is configured correctly. This returns the following:
[ 0.008716] ACPI: DMAR 0x0000000067F58DB8 0000A8 (v01 INTEL SKL 00000001 INTL 00000001)
[ 0.232850] DMAR: Host address width 39
[ 0.232851] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.232855] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 7e3ff0505e
[ 0.232856] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.232858] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.232859] DMAR: RMRR base: 0x00000067553000 end: 0x00000067572fff
[ 0.232860] DMAR: RMRR base: 0x00000068800000 end: 0x00000078ffffff
[ 0.232861] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.232862] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.232862] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[ 0.232862] DMAR-IR: Use ‘intremap=no_x2apic_optout’ to override the BIOS setting.
[ 0.234292] DMAR-IR: Enabled IRQ remapping in xapic mode

There should be a “IOMMU” line somewhere in this result i read somewhere. But there isn’t. The tutorial states that if there are results, all is well…
So I began to be a bit sceptical if this is going to work or not…
Next i executed “dmesg | grep -i vfio”
[ 3.027184] VFIO - User Level meta-driver version: 0.3
[ 3.042504] vfio-pci: probe of 0000:01:00.0 failed with error -22
[ 3.042507] vfio_pci: add [10de:1b81[ffff:ffff]] class 0x000000/00000000
[ 3.042511] vfio-pci: probe of 0000:01:00.0 failed with error -22
[ 3.042553] vfio-pci: probe of 0000:01:00.1 failed with error -22
[ 3.042554] vfio_pci: add [10de:10f0[ffff:ffff]] class 0x000000/00000000

And also in this case: “If there are results, all is well…”
It returned results so all should be well. But since I saw 3 lines with error -22 I don’t think any of this is going to work…

But I kept going since the two lines of my GPU id’s didnt gave an error.

Lastly I tried creating a new VM in virt-manager. That went ok, until I had to add both the audio and gpu pci-express id’s to the vm. I can select them, but when trying to run the vm I get something like: “Passtrough is not supported” or “virtualisation is not supported/enabled for this hardware”

I double checked the settings in the BIOS: both settings are enabled.

Does anyone know what to try?
Did I overlook something?

If there is some more info needed, please let me know!

Any help would be appreciated! :slight_smile:

What about IOMMU groups?
https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Ensuring_that_the_groups_are_valid

I created a test.sh with those lines and executed it but it yielded no results…

It’s weird. :frowning:

Did you try other parameters?

“intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 iommu=pt”

1 Like

Hi PetebLazar,
The current line in the grub config is: GRUB_CMDLINE_LINUX_DEFAULT=“quiet splash intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 iommu=pt”

I rebooted and executed the dmesg commands. Both resulting in no results at all now.

Maybe try “intremap=no_x2apic_optout”

https://lists.xenproject.org/archives/html/xen-devel/2015-12/msg00225.html

I started fiddling some more. And also took a look at this topic: Play games in Windows on Linux! PCI passthrough quick guide
find /sys/kernel/iommu_groups/ -type l yielded iommu group results. With one that is my gpu. So thats a first step.
Then i changed the /etc/modules (and updated initramfs) file and did a reboot. Then tried to create a new VM in virt-manager.
When doing the steps exactly as described in the linked topic I got the vm to boot up but unable to boot from the cd.
Then I created a vm as described in the first two tutorials I linked in my initial post and it seems I can install it the normal way.
One thing i noticed is that it keeps “typing” zeros? Is that a bug or something? I luckily could create a new partition in the instaler and go on with the installation.

So given this, is it really mandatory to have those dmesg commands giving results?
And is the intremap still needed?

I am now able to install the latest nvidia drivers in the VM. But it’s not exactly fast. CPU has a constant 100% load. Would that intremap solve that? It seems it has something to do with cpu anyway…

EDIT: Nevermind the cpu problem. I rebooted the VM and that problem was gone.
Now it seems it does not work properly with mouse and keyboard since i use it trough the virt-manager display and then switching monitors if you get what i mean…
I’ll try connecting a mouse and keyboard and dedicate that to the vm to see if that would fix the problem!

EDIT2: i tried the intermap option. but still the dmesg commands do not show any results.

I’m using AMD-based solution Threadripper 1950X & Asrock Taichi X399 for GPU passthrough (on Xubuntu 18.04). We had own issues. :wink:

I think, best “HowTo” in this case is Archi Wiki. https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

I finally (sort of) got it working. Now i have the problem that the VM only sees 1 cpu core.
I did install windows 10 home, is that a drawback from that version? Should i try installing pro to try to fix this?
In the VM config i gave it 3 cores/threads to work with.
When opening the task manager its almost constantly at 100% load, which results in a very “laggy” experience.

Yeah you have to setup to XML file to 4 vcpu, and then set 1 socket, 2 real, 2 threads in the CPU xml line You probably want to set host-passthrough as the architecture as well. That should solve your CPU issues.

As far as using SPICE for USB devices you’re not going to enjoy gaming like that. Setup an xml file for attaching and detaching devices. Reverse SSH back into computer and detach when done using VM. So you’re never locked out. Bootleg KVM switch on the $$$free$$$

Also yeah Arch iz t3h b3st.

The XML file you mean is the XML you edit with virsh edit I guess?
But isn’t it actually the same as editing the config trough virt-manager?

I wasn’t planning to use SPICE for usb devices in the end. I will try to setup Synergy/Barrier once everything is working correctly and then use LookingGlass (and a dummy hdmi stick) for video output. For now a dedicated keyboard and mouse would work.
But do you have some topic/tutorial to setup XML files for peripherals anyway?

Ha, Arch… yeah… Before committing to install Ubuntu I wondered if I should try Arch again. I have some experience with it, but since I use Ubuntu servers all the time for my work I thought it would be better to use Ubuntu while I’m more familiar with it.
Maybe I change my mind in the next couple of days…

You cannot access all of the options via the GUI. There are many that are hidden such as CPU pinning and SMT(amd) vt-d (intel).

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Swap_peripherals_to_and_from_the_Host

Thanks for pointing that out. I think I discard the idea of Synergy/Barrier all together since passing trough the devices directly as described in the Arch wiki should be the fastest way.
The main reason im creating the VM is gaming. LookingGlass gives some latency already but having the input to be even later would be very annoying.

If i understand it correctly, would it be possible to have the keyboard+mouse bind to the guest after login? And before logging out unbind them so they can be used on the host again? (or even do so when going to the logon screen instead of actually logging out?)
Then it won’t be needed to turn of the VM in order to give the peripherals back to the Host.

No. You’re creating a script to bind and unbind the MKB. You run the bind script on the host to bind the MKB to the guest. When you’re done using guest you reverse SSH into host and unbind MKB. Then you’re free to either suspend, shutdown, or simply close the LG client until you need it again.

You’re creating the ability to freely bind and unbind MKB from guest and host without needing to restart or log out.

Ok! For the time being I’ll try to do that with the “reverse” SSH.
I’ll just try to create some shortcuts on desktop to do this “automatically”.
It’s very annoying to have to create the SSH connection and manually locate the script and execute it to bind/unbind the MKB.

I’ll give this a try tonight!

I mean I set it up in like 30 minutes. it really wasn’t that bad. The hardest part was converting my RSA 4096 pubkey to whatever fucked up format PuTTy uses. Once you figure that out it makes life super convenient.

It seems the changes in the XML config solved the cpu problems. At least its not always at 100%. Still seems a tad higher than i’m used to when windows is installed natively.

Now I setup the audio (pulseaudio) according to the tutorial of GrayBoltWolf
But i have a very noticable crackling and glitching in the audio.
Any way to fix this?
I could follow through the steps of the zerosector link i posted in the first post. But to have a 10ms delay is not that nice to have.

EDIT: I use an ASUS Xonar D2X soundcard by the way. I could just pass it through… but that would result in the sound not working on the host. Not really acceptable.

EDIT2: Just tried playing HL2 DM. But the fps drops are horrible. Also when i open the dev console it shows me many many red warnings. Like at least 20 a second. I think that has something to do with it.
Any idea?

CPU ISSUES:
Yeah I don’t know what to tell you I play Rainbow Six Siege, which is graphically far more demanding, and I get around 120FPS with no frame drops. Maybe try setting up CPU Isolation/pinning? I have 3 cores with SMT enabled pinned to my VM and its pretty rock solid. Since I’m missing 3 cores I’m missing about 20-30 frames but passing cores not on a single CCX incurs massive latency issues for Ryzen.

AUDIO ISSUES:
If you’re using QEMU version 4.0 or greater all of those steps are pointless and out of date now. If you’re using old debian poop than yeah its still relevant sort of, but you should learn the new correct way

Could you please clarify what to do exactly with cpu pinning?
Its not really that clear in the libvirt manual to set that up.
But if i understand it correctly (by searching google) it means that the vm has some dedicated cores of the cpu that the host cannot use? So theoretically that should also solve the audio problems.

Where do i check the QEMU version?
I tried apt show qemu-system-x86, that gives me1:3.1+dfsg-2ubuntu3.1
So that is version 3.1 i guess? Is there already any way to install the 4.x version?