I have a AMD rig desktop that I would like to get a Windows VM going. I have a 3700XT, 32GB RAM, and 6900XT. There are certain software packages I need to use that are windows only and so the solution has been to dual-boot. Rather than deal with a bunch of Grub bootloader nonsense, I have Windows installed on a SATA M.2 SSD and Pop!OS installed on a 1.2TB U.2 enterprise drive connected to the primary M.2 slot via an adapter. I simply press F12 prior to POST to select which drive to boot. Here’s why:
Wouldn’t it be amazing if I could have my 2 disks serve to allow dual boot, but also be able to pass-through the whole Windows SSD into the VM, so that I can also optionally boot the same instance of Windows inside of Linux? And with the 6900XT, supposedly there is some sort of SR-IOV support that allows me to also pass through a portion of the computing capability and vram to the guest! Or failing that maybe get a PCIe bifucation card and install my spare 2070 Super into it since nVidia bestowed upon us some new drivers that allow GPU passthrough.
This is the dream. But I am still fairly inexperienced in this topic. I got the QEMU/KVM stuff installed, got the Virt-manager GUI going, and enabled the relevant SVM stuff in BIOS. I followed through with the wizard in virt-manager to get a windows ISO installer booting, great! Now time to modify this VM to boot my existing disk.
I run lspci | grep SATA
to determine what PCIe devices exist for me to dabble with, specifically for SATA:
08:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
09:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
0f:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
10:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
Awesome! 4 seemingly separate SATA controllers, I could just pass through the one that happens to correspond to my Windows SSD… right? With a bit of digging it’s found that my Windows SSD is the first one, 08:00.0.
And of course, murphy catches up. If I try to pass this through, I get the following error when attempting to boot the VM:
Error starting domain: internal error: qemu unexpectedly closed the monitor: 2021-04-24T17:59:24.817382Z qemu-system-x86_64: -device vfio-pci,host=0000:08:00.0,id=hostdev0,bus=pci.6,addr=0x0: vfio 0000:08:00.0: group 0 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.
Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
callback(asyncjob, *args, **kwargs)
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb
callback(*args, **kwargs)
File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 66, in newfn
ret = fn(self, *args, **kwargs)
File "/usr/share/virt-manager/virtManager/object/domain.py", line 1279, in startup
self._backend.create()
File "/usr/lib/python3/dist-packages/libvirt.py", line 1234, in create
if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2021-04-24T17:59:24.817382Z qemu-system-x86_64: -device vfio-pci,host=0000:08:00.0,id=hostdev0,bus=pci.6,addr=0x0: vfio 0000:08:00.0: group 0 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.
This seems to be the dreaded issue of having to ensure I pass through the whole IOMMU group. So what else do I have to give up from the Host in order to get this to work?
I found some random script that reports this stuff in a reasonably human-readable format:
Please be patient. This may take a couple seconds.
Group: 0 0000:00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Group: 0 0000:00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] Driver: pcieport
Group: 0 0000:00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] Driver: pcieport
Group: 0 0000:01:00.0 Non-Volatile memory controller [0108]: Intel Corporation DC P3520 SSD [8086:0a53] (rev 02) Driver: nvme
Group: 0 0000:02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad] Driver: pcieport
Group: 0 0000:03:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3] Driver: pcieport
Group: 0 0000:03:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3] Driver: pcieport
Group: 0 0000:03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3] Driver: pcieport
Group: 0 0000:03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4] Driver: pcieport
Group: 0 0000:03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4] Driver: pcieport
Group: 0 0000:03:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4] Driver: pcieport
Group: 0 0000:04:00.0 Ethernet controller [0200]: QLogic Corp. cLOM8214 1/10GbE Controller [1077:8020] (rev 54) Driver: qlcnic
Group: 0 0000:04:00.1 Ethernet controller [0200]: QLogic Corp. cLOM8214 1/10GbE Controller [1077:8020] (rev 54) Driver: qlcnic
Group: 0 0000:05:00.0 Network controller [0280]: Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak] [8086:24fb] (rev 10) Driver: iwlwifi
Group: 0 0000:06:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03) Driver: igb
Group: 0 0000:07:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
Group: 0 0000:07:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] Driver: xhci_hcd
Group: 0 0000:07:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] Driver: xhci_hcd
Group: 0 0000:08:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51) Driver: ahci
Group: 0 0000:09:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51) Driver: ahci
Group: 1 0000:00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Group: 2 0000:00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Group: 2 0000:00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] Driver: pcieport
Group: 2 0000:0a:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c0) Driver: pcieport
Group: 2 0000:0b:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479] Driver: pcieport
Group: 2 0000:0c:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73bf] (rev c0) Driver: amdgpu
Group: 2 0000:0c:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab28] Driver: snd_hda_intel
Group: 2 0000:0c:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6] Driver: xhci_hcd
Group: 2 0000:0c:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a4]
Group: 3 0000:00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Group: 4 0000:00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Group: 5 0000:00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Group: 5 0000:00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] Driver: pcieport
Group: 5 0000:0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
Group: 6 0000:00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Group: 6 0000:00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] Driver: pcieport
Group: 6 0000:00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] Driver: pcieport
Group: 6 0000:00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] Driver: pcieport
Group: 6 0000:0e:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
Group: 6 0000:0e:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486] Driver: ccp
Group: 6 0000:0e:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] Driver: xhci_hcd
Group: 6 0000:0e:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487] Driver: snd_hda_intel
Group: 6 0000:0f:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51) Driver: ahci
Group: 6 0000:10:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51) Driver: ahci
Group: 7 0000:00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
Group: 7 0000:00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
Group: 8 0000:00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
Group: 8 0000:00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
Group: 8 0000:00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
Group: 8 0000:00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443] Driver: k10temp
Group: 8 0000:00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
Group: 8 0000:00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
Group: 8 0000:00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
Group: 8 0000:00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
[Insert that overused “oh no” meme song here]
It looks like I will lose:
- boot SSD
- 1/10G Qlogic ethernet NIC
- wifi card
- on-board ethernet NIC
- USB controllers
- some other SATA controller
in other words,
it cost… EVERYTHING..
OK so what are my options? Can I split these up into seperate IOMMU groups? Or am I totally screwed?
Is there an alternative way to pass through the existing /dev/sdc2 partition or the entire block device that I’m unaware of?