Am I totally screwed? (Virt-io / passthrough / KVM)

Power_Max · April 24, 2021, 6:14pm

I have a AMD rig desktop that I would like to get a Windows VM going. I have a 3700XT, 32GB RAM, and 6900XT. There are certain software packages I need to use that are windows only and so the solution has been to dual-boot. Rather than deal with a bunch of Grub bootloader nonsense, I have Windows installed on a SATA M.2 SSD and Pop!OS installed on a 1.2TB U.2 enterprise drive connected to the primary M.2 slot via an adapter. I simply press F12 prior to POST to select which drive to boot. Here’s why:

Wouldn’t it be amazing if I could have my 2 disks serve to allow dual boot, but also be able to pass-through the whole Windows SSD into the VM, so that I can also optionally boot the same instance of Windows inside of Linux? And with the 6900XT, supposedly there is some sort of SR-IOV support that allows me to also pass through a portion of the computing capability and vram to the guest! Or failing that maybe get a PCIe bifucation card and install my spare 2070 Super into it since nVidia bestowed upon us some new drivers that allow GPU passthrough.

This is the dream. But I am still fairly inexperienced in this topic. I got the QEMU/KVM stuff installed, got the Virt-manager GUI going, and enabled the relevant SVM stuff in BIOS. I followed through with the wizard in virt-manager to get a windows ISO installer booting, great! Now time to modify this VM to boot my existing disk.

I run lspci | grep SATA to determine what PCIe devices exist for me to dabble with, specifically for SATA:

08:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
09:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
0f:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
10:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

Awesome! 4 seemingly separate SATA controllers, I could just pass through the one that happens to correspond to my Windows SSD… right? With a bit of digging it’s found that my Windows SSD is the first one, 08:00.0.

And of course, murphy catches up. If I try to pass this through, I get the following error when attempting to boot the VM:

Error starting domain: internal error: qemu unexpectedly closed the monitor: 2021-04-24T17:59:24.817382Z qemu-system-x86_64: -device vfio-pci,host=0000:08:00.0,id=hostdev0,bus=pci.6,addr=0x0: vfio 0000:08:00.0: group 0 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 66, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/domain.py", line 1279, in startup
    self._backend.create()
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1234, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2021-04-24T17:59:24.817382Z qemu-system-x86_64: -device vfio-pci,host=0000:08:00.0,id=hostdev0,bus=pci.6,addr=0x0: vfio 0000:08:00.0: group 0 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

This seems to be the dreaded issue of having to ensure I pass through the whole IOMMU group. So what else do I have to give up from the Host in order to get this to work?

I found some random script that reports this stuff in a reasonably human-readable format:

Please be patient. This may take a couple seconds.
    Group:  0   0000:00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
    Group:  0   0000:00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]   Driver: pcieport
    Group:  0   0000:00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]   Driver: pcieport
    Group:  0   0000:01:00.0 Non-Volatile memory controller [0108]: Intel Corporation DC P3520 SSD [8086:0a53] (rev 02)   Driver: nvme
    Group:  0   0000:02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad]   Driver: pcieport
    Group:  0   0000:03:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]   Driver: pcieport
    Group:  0   0000:03:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]   Driver: pcieport
    Group:  0   0000:03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]   Driver: pcieport
    Group:  0   0000:03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]   Driver: pcieport
    Group:  0   0000:03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]   Driver: pcieport
    Group:  0   0000:03:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]   Driver: pcieport
    Group:  0   0000:04:00.0 Ethernet controller [0200]: QLogic Corp. cLOM8214 1/10GbE Controller [1077:8020] (rev 54)   Driver: qlcnic
    Group:  0   0000:04:00.1 Ethernet controller [0200]: QLogic Corp. cLOM8214 1/10GbE Controller [1077:8020] (rev 54)   Driver: qlcnic
    Group:  0   0000:05:00.0 Network controller [0280]: Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak] [8086:24fb] (rev 10)   Driver: iwlwifi
    Group:  0   0000:06:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)   Driver: igb
    Group:  0   0000:07:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
    Group:  0   0000:07:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]   Driver: xhci_hcd
    Group:  0   0000:07:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]   Driver: xhci_hcd
    Group:  0   0000:08:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)   Driver: ahci
    Group:  0   0000:09:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)   Driver: ahci
    Group:  1   0000:00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
    Group:  2   0000:00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
    Group:  2   0000:00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]   Driver: pcieport
    Group:  2   0000:0a:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c0)   Driver: pcieport
    Group:  2   0000:0b:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]   Driver: pcieport
    Group:  2   0000:0c:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73bf] (rev c0)   Driver: amdgpu
    Group:  2   0000:0c:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab28]   Driver: snd_hda_intel
    Group:  2   0000:0c:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6]   Driver: xhci_hcd
    Group:  2   0000:0c:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a4]
    Group:  3   0000:00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
    Group:  4   0000:00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
    Group:  5   0000:00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
    Group:  5   0000:00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]   Driver: pcieport
    Group:  5   0000:0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
    Group:  6   0000:00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
    Group:  6   0000:00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]   Driver: pcieport
    Group:  6   0000:00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]   Driver: pcieport
    Group:  6   0000:00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]   Driver: pcieport
    Group:  6   0000:0e:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
    Group:  6   0000:0e:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]   Driver: ccp
    Group:  6   0000:0e:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]   Driver: xhci_hcd
    Group:  6   0000:0e:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]   Driver: snd_hda_intel
    Group:  6   0000:0f:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)   Driver: ahci
    Group:  6   0000:10:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)   Driver: ahci
    Group:  7   0000:00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
    Group:  7   0000:00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
    Group:  8   0000:00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
    Group:  8   0000:00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
    Group:  8   0000:00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
    Group:  8   0000:00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]   Driver: k10temp
    Group:  8   0000:00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
    Group:  8   0000:00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
    Group:  8   0000:00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
    Group:  8   0000:00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]

[Insert that overused “oh no” meme song here]

It looks like I will lose:

boot SSD
1/10G Qlogic ethernet NIC
wifi card
on-board ethernet NIC
USB controllers
some other SATA controller

in other words,

it cost… EVERYTHING..

OK so what are my options? Can I split these up into seperate IOMMU groups? Or am I totally screwed?

Is there an alternative way to pass through the existing /dev/sdc2 partition or the entire block device that I’m unaware of?

oegat · April 24, 2021, 6:49pm

It should be pretty straightforward to pass the block device, just as you would pass a any disk image file. Do you use libvirt? Have a look at my example in this thread, under the heading “VM setup notes”.

I pass the entire sda to boot a windows VM from it, and TRIM is enabled and works.

TheCakeIsNaOH · April 24, 2021, 7:30pm

There are lots of people (including me) that REALLY would want this, but I am not aware of anything at the moment to get this to work.

Yes, or at least some of them, see the ACS patch.

It is a kernel patch, which after it is applied can be activated by a kernel parameter in your bootloader.

If that makes sense, then you know what to do, otherwise if is all gobbledygook then there are instructions for a number of different distros available I could point you to.

Power_Max · April 24, 2021, 7:37pm

Yeah, I’ve heard rumors that it’s supported in the hardware somewhere but of course I can’t find the source after having given in to the scalper prices for the GPU. I might scalp it myself and wait until I can get a couple more modest 6800 or 6800xt’s and maybe a motherboard that can support them along with my 10G network card.

I’ve heard of the ACH patch, although I’m not particularly a fan, it looks to have some security implications. What hardware changes would be necessary to get everything into its own IOMMU group? server grade hardware? Or perhaps the right choice of motherboard for consumer platforms?

TheCakeIsNaOH · April 24, 2021, 7:49pm

Yes and no.

Its kind of like the more estoric side channel attacks as it probably would require root/admin on the VM side. And I don’t think there are any public exploits or proofes of concept that would use this.

So if you are planning to rent out this VM to someone, and let them run their own OS, then yes, it might be a concern. But on a home network, you almost certainly have bigger issues to worry about if someone malicious gets root/admin on the VM.

Sometimes, switching around what cards are in what slots is good enough. Other times, making sure that IOMMU is set to on/enabled instead of auto in the UEFI helps. Sometimes a UEFI upgrade or downgrade (be careful downgrading, it can brick motherboard) can fix or help.

I’m not up to date on what is best for Ryzen.