Motherboard: Biosstar X570GT8
CPU: Ryzen 3600
Memory: 32GB
Boot Dev: M.2 SSD
OS: Ubuntu 19.10
Kernel: 5.3.45, 5.4.21, 5.6
Display Manager: lightdm
In the case of the RX580 single GPU configuration, the first pass-through of the VM after boot was successful, but the second pass-through after reloading the admgpu driver after shutting down the VM failed.
Failure message
vfio-pci 0000: 0c: 00.1: vfio_bar_restore: reset recovery-restoring BARs
AMD-Vi: Completion-Wait loop timed out
iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device = 0c: 00.0 address = 0x80b59f680]
If the link speed of PCI-e is fixed to Gen2 and set to 5G/sT in UEFI BIOS, it is possible to pass through without fail even after reloading the admgpu driver, but the link speed of all PCI-e is limited to 5GT/s There was a problem.
By specifying the following in the amdgpu module option, it is possible to pass through even after reloading the admgpu driver after shutting down the VM.
The link speed of the GPU after pass-through becomes the maximum of the GPU.
Other PCI-e link speeds are also maximized.
options amdgpu pcie_gen_cap=0x2
0x1:2.5GT/s(Gen1) x
0x2:5GT/s(Gen2) o
0x3:8GT/s(Gen3) x
0x4:16GT/s(Gen4) x
If it is other than 5GT/s, an error occurs and restart is not possible.
It is unknown why setting to 5GT/s will be successful.
Link speed change without option
OS boot: 8GT/s
Display Manager end: 2.5GT/s
amdgpu unload: 2.5GT/s
VM startup: 8GT/s
VM termination: 2.5GT/s
amdgpu load: 2.5GT/s
Display Manager startup: 2.5GT/s
After that it is fixed to 2.5GT/s
When the option is set, 2.5GT/s becomes 5GT/s.
When the options are set, if the amdgpu is unloaded or loaded without starting or passing through the VM, the next VM startup will fail (OS crashes).
When logging on to the OS and starting the VM, unloading of the GPU driver may fail because the Desktop process remains.
In Lightdm, LXQT environment, it may take 90 seconds to finish lxqt-panel, so it can be avoided by forcibly terminating with killall.
Even in the Intel environment, if the link speed of PCI-e was fixed to Gen2, pass-through was successful, so it seems that the same setting will succeed.
(Cannot be confirmed because there is no environment)