VFIO 2023 / Radeon 7000 Edition [WIP]

> Enter The PC Giveaway Here! <

The Build

  • Motherboard: ASRock Taichi X670E
  • AMD 7950X 16 core CPU
  • Corsair Vengeance DDR5 2x32gb
  • Corsair RM1200x Shift Series modular PSU
  • Corsair 7000D Airflow
  • Corsair 140mm iCUE H170ibElite LCD XT
  • Corsair 3x140mm AIO LCD Cooler
  • Corsair MP600 2TB ( x2 )
  • Corsair MP600 500gb
  • Tested GPUs:
    • ASRock Taichi White 7900XTX
    • ASRock Taichi 7900XTX
    • ASRock Aqua 7900XTX
    • ASRock OC Formula 6950 XT
    • ASRock OC Formula 6900 XT
    • ASRock Taichi 6800XT
    • Powercolor Hellhound 7900XTX
    • AMD Reference 7900XTX
    • AMD Reference 7900XT
    • Merc 319 6950XT
    • AMD Reference 6800XT

Huge Thanks to AMD & Corsair for most of this gear we’re using in our giveaway. Big thanks TO YOU for being part of the Level1Techs Community. Link below

TODO LINK

TODO RULES

Ubuntu 23.04 notes

sudo apt install wget
wget https://raw.githubusercontent.com/pimlie/ubuntu-mainline-kernel.sh/master/ubuntu-mainline-kernel.sh
chmod +x ubuntu-mainline-kernel.sh
sudo mv ubuntu-mainline-kernel.sh /usr/local/bin/

sudo ubuntu-mainline-kernel.sh -i 6.4.3

Don’t worry about this script making errors like Failed to execute child process “dbus-launch” (No such file or directory) or GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.Notifications was not provided by any .service files – it doesn’t matter for our purposes.

It will look like this when it succeeds:

Downloading index from kernel.ubuntu.com
Will download 6 files from kernel.ubuntu.com:
Downloading amd64/linux-headers-6.4.3-060403-generic_6.4.3-060403.202307110536_amd64.deb: 100%   
Downloading amd64/linux-headers-6.4.3-060403_6.4.3-060403.202307110536_all.deb: 100%   
Downloading amd64/linux-image-unsigned-6.4.3-060403-generic_6.4.3-060403.202307110536_amd64.deb: 100%   
Downloading amd64/linux-modules-6.4.3-060403-generic_6.4.3-060403.202307110536_amd64.deb: 100%   
Downloading amd64/CHECKSUMS: 100%   
Downloading amd64/CHECKSUMS.gpg: 100%   
Importing kernel-ppa gpg key ok
Signature of checksum file has been successfully verified
Checksums of deb files have been successfully verified with sha256sum
Installing 4 packages
Cleaning up work folder

QEMU Changes

Important Syntax Changes

 <qemu:commandline>
... stuff to turn x-vga on 
 </qemu:commandline>

became

  <qemu:override>
    <qemu:device alias="ua-stupid">
      <qemu:frontend>
        <qemu:property name="x-vga" type="bool" value="true"/>
      </qemu:frontend>
    </qemu:device>
  </qemu:override>

for a device like:


<hostdev mode="subsystem" type="pci" managed="yes">
  <driver name="vfio"/>
  <source>
    <address domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
  </source>
  <alias name="ua-stupid"/>
  <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
</hostdev>

What’s with ua-stupid ? Well if you have a reasonable alias like name=hostdev0 it vanishes from the config mysteriously. Pretty WTF at the moment.

also have to update the <domain> line to allow more stuff to something like

<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">

Re-size Bar Complications

Normally the Base Address Register is 256 megabytes. Moving data 256mb at a time is too slow. ReBar driver enablement usually means that you can get 8, 16 or 32 gigabytes of memory-mapped I/O space (meaning that your GPU can move data in much larger chunks).

This is another layer of complication as explained in the video. You should start by getting your setup working for VFIO with rebar disabled and then move on to enabling resizable bar.

Be prepared to need to write a script that unbinds vfio from your GPU, sets the BAR to something like 8gb, then rebinds your GPU (making it ready to start the VM).

Otherwise you can face a blue-screen of death from AMD’s driver when you install them – a common cause of BSODs from amdkmdag.sys for vfio/gpu passthrough is when Resizable Bar is enabled, but the actual BAR values are a little wacky from where you are in a GPU passthrough situation. Disabling BAR in bios as a temporary workaround/to test is fine.

Another strategy for this situation is to boot linux and examine the BAR/pcie mmio situation in order to take troubleshooting steps.

Yet another layer on top of this is when you have three GPUs in the system (integrated, plus two PCIe). I’m not sure AMD has done much testing with two 7000 series PCIe GPUs in the same system (given that multi-gpu gaming is dead… and VFIO is even more esoteric than when I started doing this a decade agao… this probably makes sense).

Hellhound 7900XTX Notes

It is possible to trigger D0 to D3 hot bugs on this card with kernel 6.4.3; however this card seems to work okay with ancient kernels – pre 5.15. I’m not sure if this is truly an AMD issue or just a problem with PCIe passthrough in general.

From a clean setup it is possible to install the AMD GPU Adrenaline drivers, and reinstall them, and to shutdown / restart the VM without triggering this issue.

For me the invalid header type 127 was triggered from an unexpected BSOD; hard restarting the system restored the ability to bind the GPU with no other configuration changes.

Re-size Bar

[    6.218099] amdgpu 0000:1e:00.0: amdgpu: ACPI VFCT table present but broken (too short #2),skipping
[    6.221042] [drm] BIOS signature incorrect 0 0
[    6.221044] amdgpu 0000:1e:00.0: BAR 6: can't assign [??? 0x00000000 flags 0x20000000] (bogus alignment)
[    6.224015] [drm] BIOS signature incorrect 0 0
[    6.224016] amdgpu 0000:1e:00.0: amdgpu: Unable to locate a BIOS ROM
[    6.224018] amdgpu 0000:1e:00.0: amdgpu: Fatal error during GPU init

The bogus alignment is a direct result, seems to be, of the kernel parameter
pci=realloc,assign-busses,nocrs (I tried also without realloc and nocrs).

I suspect, but not sure, an additional wrinkle here is this board’s thunderbolt capability. Thunderbolt can make some bad assumptions about memory I/O regions.

It could also be related to AMD integrated GPU assumptions about GOP:

… CSM is disabled on our test platform, but sometimes it is useful to patch the GOP even for the SOC (integrated) GPU.

Fine, I’ll resize it myself

Since assign-busses craps out on us, we’ll do it this way:

lspci |grep VGA

03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX] (rev c8)
59:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Raphael (rev c1)

lspci -vvvs 03:00.0

...
	Capabilities: [200 v1] Physical Resizable BAR
		BAR 0: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
		BAR 2: current size: 2MB, supported: 2MB 4MB 8MB 16MB 32MB 64MB 128MB 256MB

That’s the part of the lspci we care about. 256mb aka Re-Bar is off in bios.

Enabled Re-Bar in bios:

      Capabilities: [200 v1] Physical Resizable BAR
                BAR 0: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
                BAR 2: current size: 256MB, supported: 2MB 4MB 8MB 16MB 32MB 64MB 128MB 256MB

Let’s set it to 8gb Re-bar support to make it a little easier for VM memory mapping…

 ls *resize -l
-rw-r--r-- 1 root root 4096 Jul 17 21:39 resource0_resize
-rw-r--r-- 1 root root 4096 Jul 17 21:39 resource2_resize

I’m not sure why, but this is bar 0, 2 from lspci but here it’s called resource 0,2. (Where’s resource 1??)

to resize we can do:

# echo 13 > resource0_resize 
bash: echo: write error: Device or resource busy
# echo 3 > resource2_resize
bash: echo: write error: Device or resource busy
root@level1techs-X670E-Taichi:/sys/bus/pci/devices/0000:03:00.0# rmmod vfio_pci
root@level1techs-X670E-Taichi:/sys/bus/pci/devices/0000:03:00.0# echo 13 > resource0_resize 
root@level1techs-X670E-Taichi:/sys/bus/pci/devices/0000:03:00.0# echo 3 > resource2_resize
root@level1techs-X670E-Taichi:/sys/bus/pci/devices/0000:03:00.0# modprobe vfio_pci

Note that we got device or resource busy becauase the vfio module was loaded. The gpu cannot CANNOT be in use while you do this!

What’s #13? That’s 2 ^ to the power of. 2^13 is 8 gigabytes. Except that’s not math, 1 is 1 megabyte in this case. Then it works. Here’s a Handy table:

15 = 32gb
14 = 16gb
13 = 8gb
12 = 4gb 
11 = 2gb
10 = 1gb
09 = 512mb
08 = 256mb 
07 = 128mb
06 = 64mb
05 = 32mb
04 = 16mb
03 = 8mb

… and so on.

for 8gb BAR you want resource2 to be 8mb,
16gb 16mb and so on.

But lspci confirms:

Capabilities: [200 v1] Physical Resizable BAR
                BAR 0: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
                BAR 2: current size: 8MB, supported: 2MB 4MB 8MB 16MB 32MB 64MB 128MB 256MB

8gb now! Nice!

When using a 32gb BAR / 32mb resource2, the computer would black screen and not show anything. But 8gb works fine. This may or may not, not really, impact game performance.

Bar Sizes, and weird stuff IDK about

So, and this is really interesting to me, but it seems like almost all of the frustration I was experiencing previously where ReBar on in bios = doesn’t work is that there is some implicit relationship between BAR0 and BAR2 space that matters a lot on Linux (but seems not to matter on windows…)

The system firmware automatically resizes BAR0 (the big one) to be as big as it can, but Linux Does Not Like if the size of BAR2 does not correspond to BAR0 as with the above table.

In other words please set 8 gigabytes/8mb. Curiously, it doesn’t seem to always follow this rule but I haven’t found an instance where ReBar initializaiton failes if Bar0/Bar2 are BOTH sized along this rule on the Linux side of things.

What about Anti-Cheat?

This is becoming a huge problem. I am not sure what the solution is. Cheaters will always cheat, but perhaps game companies don’t realize the valid use case here is also being trampled on. We should all make some noise about this… fortunately even folks that use Proton (a la Valve’s Steamdeck) often also run afoul of anticheat.

Here are some things you can do to hide from anitcheat in a vm.

Enable Hyper-V in the host OS

This helps you be not flagged as a cheater. Historically this would flag you, but fortunately, even unreasonable game companies thought this was too unreasonable and relented.

KVM XML Config Tweaks

    <kvm>
      <hidden state="on"/>
    </kvm>

Processor ID & Raw Timer Patch RDTSCP

13 Likes

I don’t think you need the x-vga option anymore. I don’t use it on any of my VFIO systems.

I use this re-bar hack for my 3090, would it work for you instead of your resizing trick?

<domain type='kvm' id='1' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <..snip..>
  <qemu:commandline>
    <qemu:arg value='-fw_cfg'/>
    <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/>
  </qemu:commandline>
</domain>

EDIT: Also, check this out as an alternative to the linked mainline script. You can install it from a PPA and everthing.

1 Like

The x-vga option in combination with hiding the Hypervisor is still usefull if you want to keep the GPU scaling options in the nvidia (Windows) driver.

For gaming purposes, does Re-bar really make a performance difference?

Ah, good to know. Thank you. I’ll have to check it out.

For gaming, on NVIDIA, not usually. But there are other use-cases where it can be beneficial, like machine learning, and my wife does some 3D rendering in Windows. This way I can just leave it enabled in the BIOS and not worry about it.

I have no reason to build this but Jesus I want to.

2 Likes

So with this, the Task Manager will show Virtualzation as “enabled”, but anti-cheat will be lenient cuz of Hyper-V?

I thought there would’ve been improvements, since then, but Hyper-V is still a complete pain in the butt.


It’s pretty fun experience, IMO. Got Halo: Infinite running on it since its more stable there. I did play Halo: MCC for a while, but now I’m with Linux for that one.
Also got some Linux VMs if I wanna play with their GUI without feeling the slowness of software-rendered GUI. though I wish I can get macOS up and running with an NVIDIA GPU

4 Likes

Nice tweak, will add that above. Thanks.

I also have half a writeup about huge pages… which is not super required these days but still helps

1 Like

Oh neat, I used the last guide and ran that for a few years until it wasn’t really performing very well in new games on my old hardware. I’ll have to see if I can get this working again! Looking glass has come so far!

Threadripper vs AM5 for PCI passthrough.

While it seems pretty easy to passthrough multiple devices on a server based motherboard like Threadripper and Epyc. (dedicated pcie lanes that are not shared)

What about consumer based motherboards? I am currently using AM4 5700G but when updating my my mouse firmware in windows VM it would disconnect the mouse and try to reconnect it. Which fails cause as soon as the mouse is disconnected and reconnected the device is no longer connected to the VM but to the host.

It seems most pcie slots and devices are shared on consumer boards. So I don’t know how I can passthrough a USB port or hub. so that i can easily plug and unplug devices from my VM while its active.

I’ve been running a pretty solid setup with solid performance that I passthrough the cores from one CCD from my 7950x and keep on host the other CCD. The only thing I haven´t have luck so far is with ReBAR, I’ll test with the above approach to see if it works this weekend.

My system:
AMD Ryzen 7950X (igpu for host)
Asus ProArt X670E
64gb of RAM
XFX 6800XT 16GB (passthrough)

1 Like

Hey everyone, I’ve been a longtime reader of the level1 forums and the youtube channel, this is my first post. Wendell’s VFIO call to action was the push I needed.

I’ve been using GPU passthrough on my daily driver Ubuntu 22.04 system for about a year now with no issues. I won’t say getting it working was all smooth sailing, but once I got everything all dialed in and working everything has been pretty great.

I think your updated 2023 VFIO guide is SORELY needed, a lot of the information out there is quite outdated, conflicting or just flat out wrong and sorting through it can be a daunting task for someone trying VFIO (especially GPU passthrough) for the first time.

My Specs:

Motherboard: MSI MAG Mortar B760M Wifi
CPU: Intel Core i5-12600K
GPU: MSI Radeon RX 6750XT 12GB (iGPU for Host)
RAM: 32GB

I use the iGPU on the 12600K for the Linux host and Looking Glass with the KVMFR kernel driver instead of using a SHMEM pipe.

I had a little bit of trouble getting the audio in Looking Glass to work with the stock Ubuntu audio configuration and at first I was using Scream over SHMEM but I was getting audio stuttering issues so I investigated further and found out that switching the audio backend to Pipewire ( I think I used the guide on UbuntuHandbook, but basically just google “Switching Ubuntu to PipeWire” ) pretty much immediately got sound working natively in Looking Glass.

Before my current system, I was using GPU passthrough also with Looking Glass on my Sager (Clevo) Laptop with an RTX 2070 Super GPU. I was super proud of myself when I actually got this working, trying to pass through a mobile GPU will definitely try your patience. To make matters worse, the available documentation is even worse for Laptops. What ended up making that work was a combination of adding a fake battery, and also spoofing the Sub Device ID and Sub Vendor ID of the GPU, without that last step even if I did everything else it was Code 43.

Fake Battery and Device spoofing XML:

<qemu:commandline>
    <qemu:arg value='-acpitable'/>
    <qemu:arg value='file=/usr/share/qemu-kvm/acpitable.bin'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.hostdev0.x-pci-sub-vendor-id=0x1558'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.hostdev0.x-pci-sub-device-id=0x67E3'/>
</qemu:commandline>

(P.S. change your sub-vendor-id and sub-device-id to match your system)

Laptop Specs:

OS: Ubuntu 22.04
CPU: Intel Core i7 10870H (8 Core / 16 Thread)
GPU: RTX 2070 Super Max-Q
RAM: 32GB DDR4

Sorry for the long post, I love the ability to use VFIO / GPU passthrough in Linux and I really hope keeps getting better and easier to use in the future.

P.S. shout out to @gnif for Looking Glass, thank you so much for all your hard work, Looking Glass is awesome.

3 Likes

I have a PCI-E based USB controller with 4x USB 3.0 ports that I pass through to my windows VM. If I have a USB device that needs some windows app to do a firmware update on, I plug it into one of those ports. If the update process power cycles the device, it will reconnect to the guest as the entire controller is connected to the VM.

1 Like

From my limited experience with AMD platforms, B series motherboards seem to have devices that are connected to chipset(e.g. SATA controller, M.2 slot, on-board Ethernet, USB controller and some PCIe slots) in a single IOMMU group. I can either passthrough the whole chipset or use ACS override patch, which is probably not the best security practice. Devices or PCIe slots that are connected to CPU directly are in their own IOMMU group.

On the other hand, X series motherboards that I konw of, break down every possible devices into their own IOMMU group, including those conneted to chipset.

For passing through USB controllers, I used an awesome utility usbview, plugged my keyboard into every USB port so I would know what USB ports goes to which USB controller. Even my cheapest B450 motherboard have mutiple USB controllers.

1 Like

TLl;DR: There needs to be quality of life improvements and documentation for hardcore esports gamers and audio engineer guys.

Just putting this out there. Is there any information/guides for emulating more than just emulating an old q35/i440fx system? If I were an anti cheat developer I would just ban all users with those chipsets, especially if they have an AMD CPU on an Intel only chipset.

Also, why can’t you modify Qemu smbios strings without recompiling in Qemu/LibVirt? Why can’t we then passthrough the acpi table in its entirety? You can’t edit type 17 memory data in the smbios without using qemu commandline.

Finally audio and it’s related latency. Documenting how to get that sweet, sweet realtime mic/input audio without usb passthrough in a VM would be awesome. Scream/audiodev pulseaudio, work fine, just not for low latency audio, unless I’m doing something wrong.

Has anybody been able to enable Hyper-V in guest Windows 11, with AMD Linux host without disabling hypervisor extension?
Some games, like PUBG, kick you with VMs, and this trick works, but cripples performance.
So I was able to play PUBG, with

  • Hyper-V installed
  • Hypervisor Enabled
  • SVM Disabled
    but this, slows down FPS up to -75% !

While this works fine

  • Hyper-V installed
  • Hypervisor Disbled
  • SVM Enabled
    but PUBG detects VM and kicks me out of the game.
1 Like

Is this the A770 thread you mention in the video this one?https://www.chiphell.com/thread-2512734-1-1.html

I need more info on the resize bar issue, been running with it off for months now.

Experiment with what’s explained in the “to resize bar” section. Youll need to resize "both* bar regions and one corresponds to the other