I’ve been reading through other threads and attempting to get this working. My objectives are:
- Fedora Silverblue
- Host: W6600 connected to four displays via displayport
- Audio via USB DAC (no need for W6600 audio capabilities)
- Guest: 6800XT connected to single display via HDMI
- USB-C passed to guest, with USB-C based hub for keyboard+mouse+headset
- Mostly Linux guest, and mostly headless for vulkan accelerated containers
- Windows guest for some games
This might be the first silverblue focused post on here? I’m attempting this on silverblue now but I have similar hardware I was planning to run headless-ly with Fedora IoT for doing moonlight. The approach with silverblue/ostree is a bit different than other distros.
Hardware is:
- Gigabyte X570 Aorus Master
- AMD Reference W6600
- AMD Reference 6800 XT
-[0000:00]-+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
+-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
+-01.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-01.1-[01]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
+-01.2-[02-0a]----00.0-[03-0a]--+-00.0-[04]----00.0 Toshiba Corporation XG6 NVMe SSD Controller
| +-01.0-[05]----00.0 Toshiba Corporation XG6 NVMe SSD Controller
| +-03.0-[06]----00.0 Intel Corporation Wi-Fi 6 AX200
| +-05.0-[07]----00.0 Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller
| +-08.0-[08]--+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
| | +-00.1 Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
| | \-00.3 Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
| +-09.0-[09]----00.0 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
| \-0a.0-[0a]----00.0 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
+-02.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-03.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-03.1-[0b-0d]----00.0-[0c-0d]----00.0-[0d]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Device 73e3
| \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
+-03.2-[0e-10]----00.0-[0f-10]----00.0-[10]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT]
| +-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
| +-00.2 Advanced Micro Devices, Inc. [AMD/ATI] Device 73a6
| \-00.3 Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 USB
+-04.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-05.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-07.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-07.1-[11]----00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
+-08.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-08.1-[12]--+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
| +-00.1 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
| +-00.3 Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
| \-00.4 Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
+-14.0 Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
+-14.3 Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
+-18.0 Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0
+-18.1 Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1
+-18.2 Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2
+-18.3 Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3
+-18.4 Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4
+-18.5 Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5
+-18.6 Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6
\-18.7 Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7
I have done the following and had minimal success:
vim /etc/dracut.conf.d/vfio.conf
add_drivers+=" vfio vfio_iommu_type1 vfio_pci vfio_virqfd "
Need to enable initramfs, where “In Silverblue, the initramfs is prebuilt and included in the system image”, enabling will apparently create the local initramfs and read the dracut.conf.d
we configured above:
rpm-ostree initramfs --enable
Install the necessary hypervisor packages, choosing to do this as a layered package instead of something like flatpak:
rpm-ostree install virt-manager qemu-kvm
- Update BIOS (f35b at time of this posting)
- Set some BIOS settings:
- Settings → IO Ports → Initial Display Output → PCIe 1 Slot (W6600)
- Tweaker → Advanced CPU Settings → SVM Mode → Enabled
- Settings → Miscellaneous → IOMMU → Enabled
- Settings → NBIO Common Options → IOMMU → Enabled
Locate the GPU that we want to pass through (6800XT and it’s USB-C controller):
#!/bin/bash
shopt -s nullglob
for g in `find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V`; do
echo "IOMMU Group ${g##*/}:"
for d in $g/devices/*; do
echo -e "\t$(lspci -nns ${d##*/})"
done;
done;
IOMMU Group 34:
10:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1)
IOMMU Group 35:
10:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] [1002:ab28]
IOMMU Group 36:
10:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6]
IOMMU Group 37:
10:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 USB [1002:73a4]
Adjust our kernel boot parameters:
rpm-ostree kargs --append=“amd_iommu=pt rd.driver.pre=vfio-pci vfio-pci.ids=1002:73bf,1002:ab28,1002:73a6,1002:73a4 video=efifb:off”
Reboot and observe:
sudo lspci -vn
However I’ve truncated to only pull in the two GPU(s):
- W6600 is 0d:00:0,1
- 6800XT is 10:00.0,1,2,3
0d:00.0 0300: 1002:73e3 (prog-if 00 [VGA controller])
Subsystem: 1002:0e0c
Flags: bus master, fast devsel, latency 0, IRQ 213, IOMMU group 30
Memory at d0000000 (64-bit, prefetchable) [size=256M]
Memory at e0000000 (64-bit, prefetchable) [size=2M]
I/O ports at e000 [size=256]
Memory at fcc00000 (32-bit, non-prefetchable) [size=1M]
Expansion ROM at fcd00000 [disabled] [size=128K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [64] Express Legacy Endpoint, MSI 00
Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150] Advanced Error Reporting
Capabilities: [200] Physical Resizable BAR
Capabilities: [240] Power Budgeting <?>
Capabilities: [270] Secondary PCI Express
Capabilities: [2a0] Access Control Services
Capabilities: [2d0] Process Address Space ID (PASID)
Capabilities: [320] Latency Tolerance Reporting
Capabilities: [410] Physical Layer 16.0 GT/s <?>
Capabilities: [440] Lane Margining at the Receiver <?>
Kernel driver in use: amdgpu
Kernel modules: amdgpu
0d:00.1 0403: 1002:ab28
Subsystem: 1002:ab28
Flags: fast devsel, IRQ 255, IOMMU group 31
Memory at fcd20000 (32-bit, non-prefetchable) [disabled] [size=16K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [64] Express Legacy Endpoint, MSI 00
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150] Advanced Error Reporting
Capabilities: [2a0] Access Control Services
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
10:00.0 0300: 1002:73bf (rev c1) (prog-if 00 [VGA controller])
Subsystem: 1002:0e3a
Flags: bus master, fast devsel, latency 0, IRQ 255, IOMMU group 34
Memory at b0000000 (64-bit, prefetchable) [size=256M]
Memory at c0000000 (64-bit, prefetchable) [size=2M]
I/O ports at d000 [disabled] [size=256]
Memory at fc600000 (32-bit, non-prefetchable) [size=1M]
Expansion ROM at fc700000 [disabled] [size=128K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [64] Express Legacy Endpoint, MSI 00
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150] Advanced Error Reporting
Capabilities: [200] Physical Resizable BAR
Capabilities: [240] Power Budgeting <?>
Capabilities: [270] Secondary PCI Express
Capabilities: [2a0] Access Control Services
Capabilities: [2d0] Process Address Space ID (PASID)
Capabilities: [320] Latency Tolerance Reporting
Capabilities: [410] Physical Layer 16.0 GT/s <?>
Capabilities: [440] Lane Margining at the Receiver <?>
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
10:00.1 0403: 1002:ab28
Subsystem: 1002:ab28
Flags: fast devsel, IRQ 255, IOMMU group 35
Memory at fc724000 (32-bit, non-prefetchable) [disabled] [size=16K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [64] Express Legacy Endpoint, MSI 00
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150] Advanced Error Reporting
Capabilities: [2a0] Access Control Services
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
10:00.2 0c03: 1002:73a6 (prog-if 30 [XHCI])
Subsystem: 1002:73a6
Flags: bus master, fast devsel, latency 0, IRQ 90, IOMMU group 36
Memory at fc500000 (64-bit, non-prefetchable) [size=1M]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [64] Express Endpoint, MSI 00
Capabilities: [a0] MSI: Enable- Count=1/8 Maskable- 64bit+
Capabilities: [c0] MSI-X: Enable+ Count=8 Masked-
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150] Advanced Error Reporting
Capabilities: [2a0] Access Control Services
Kernel driver in use: xhci_hcd
10:00.3 0c80: 1002:73a4
Subsystem: 1002:0408
Flags: bus master, fast devsel, latency 0, IRQ 109, IOMMU group 37
Memory at fc720000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [64] Express Endpoint, MSI 00
Capabilities: [a0] MSI: Enable+ Count=1/2 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150] Advanced Error Reporting
Capabilities: [2a0] Access Control Services
Kernel driver in use: i2c-designware-pci
From here I create a VM with virt-manager, will attempt Windows first:
- Q35 + OVMF
- CPU host-passthrough with manually set topology of socket 1 and cores 16
- virtio qcow2 disk (works fine with proper drivers)
- leaving all the default devices (for a windows 10 detected system)
- kvm hidden state on
- vendor_id state on with random value
- pass in the 6800XT, and the three other devices
<domain type="kvm">
<name>windblows</name>
<uuid>d17d1f9f-361b-40f3-87fe-da5f735bd5e9</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/10"/>
</libosinfo:libosinfo>
</metadata>
<memory unit="KiB">33554432</memory>
<currentMemory unit="KiB">33554432</currentMemory>
<vcpu placement="static">16</vcpu>
<os>
<type arch="x86_64" machine="pc-q35-5.2">hvm</type>
<loader readonly="yes" type="pflash">/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
<bootmenu enable="no"/>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state="on"/>
<vapic state="on"/>
<spinlocks state="on" retries="8191"/>
<vendor_id state="on" value="randomid"/>
</hyperv>
<kvm>
<hidden state="on"/>
</kvm>
<vmport state="off"/>
</features>
<cpu mode="host-passthrough" check="none" migratable="on">
<topology sockets="1" dies="1" cores="16" threads="1"/>
</cpu>
<clock offset="localtime">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
<timer name="hypervclock" present="yes"/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2"/>
<source file="/var/home/agd/.kvm/win10.qcow2"/>
<target dev="vda" bus="virtio"/>
<boot order="1"/>
<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
</disk>
<disk type="file" device="cdrom">
<driver name="qemu" type="raw"/>
<source file="/var/home/agd/.kvm/Win10_21H1_English_x64.iso"/>
<target dev="sdb" bus="sata"/>
<readonly/>
<address type="drive" controller="0" bus="0" target="0" unit="1"/>
</disk>
<disk type="file" device="cdrom">
<driver name="qemu" type="raw"/>
<source file="/var/home/agd/.kvm/virtio-win-0.1.208.iso"/>
<target dev="sdc" bus="sata"/>
<readonly/>
<address type="drive" controller="0" bus="0" target="0" unit="2"/>
</disk>
<controller type="usb" index="0" model="qemu-xhci" ports="15">
<address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
</controller>
<controller type="sata" index="0">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
</controller>
<controller type="pci" index="0" model="pcie-root"/>
<controller type="pci" index="1" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="1" port="0x10"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="2" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="2" port="0x11"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
</controller>
<controller type="pci" index="3" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="3" port="0x12"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
</controller>
<controller type="pci" index="4" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="4" port="0x13"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
</controller>
<controller type="pci" index="5" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="5" port="0x14"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
</controller>
<controller type="pci" index="6" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="6" port="0x15"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
</controller>
<controller type="pci" index="7" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="7" port="0x16"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
</controller>
<controller type="pci" index="8" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="8" port="0x17"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
</controller>
<controller type="pci" index="9" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="9" port="0x18"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
</controller>
<controller type="virtio-serial" index="0">
<address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
</controller>
<interface type="network">
<mac address="52:54:00:3c:0e:32"/>
<source network="network"/>
<model type="e1000e"/>
<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
<serial type="pty">
<target type="isa-serial" port="0">
<model name="isa-serial"/>
</target>
</serial>
<console type="pty">
<target type="serial" port="0"/>
</console>
<channel type="spicevmc">
<target type="virtio" name="com.redhat.spice.0"/>
<address type="virtio-serial" controller="0" bus="0" port="1"/>
</channel>
<input type="tablet" bus="usb">
<address type="usb" bus="0" port="1"/>
</input>
<input type="mouse" bus="ps2"/>
<input type="keyboard" bus="ps2"/>
<graphics type="spice" autoport="yes">
<listen type="address"/>
<image compression="off"/>
</graphics>
<video>
<model type="qxl" ram="65536" vram="65536" vgamem="16384" heads="1" primary="yes"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
</video>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x10" slot="0x00" function="0x0"/>
</source>
<address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x10" slot="0x00" function="0x1"/>
</source>
<address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x10" slot="0x00" function="0x2"/>
</source>
<address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x10" slot="0x00" function="0x3"/>
</source>
<address type="pci" domain="0x0000" bus="0x09" slot="0x00" function="0x0"/>
</hostdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="2"/>
</redirdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="3"/>
</redirdev>
<memballoon model="virtio">
<address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
</memballoon>
</devices>
</domain>
Observations and Questions:
- Is it going to cause problems to pass
1002:ab28
since both the W6600 and 6800XT seem to have the same device? - What power state does the 6800XT idle in?
-
sudo lspci -vv -s 10:00.0
showsStatus: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
which seems like it’s going to be a spicy boy while it’s sitting there doing nothing most of the time.
-
- VM doesn’t consistently work:
- Seems to only get video on HDMI when I cold boot the entire system. Rebooting the VM gives just a blank screen.
- USB-C on GPU doesn’t passthrough
- Hardware acceleration is good (seems to drive the display, amd drivers install and recognize the 6800XT)
Looking for assistance!