Is PCIPassthrough possible on ESXi installed on AM5 motherboards?

Operating Systems & Open Source Virtualization
, ,
1

Here are the current hardware specs:

AMD Ryzen 7900x
ASUS ROG STRIX 650e-e gaming Wifi
Samsung DDR5 16GB PC5-44800 * 2EA
SK hynix P31 1TB
WD Ultrastar 4TB*2EA
MSI GeForce GTX 1660 SUPER Ventus S OC D6 6GB * 2EA

I tried ESXi 8.0 for this spec, ESXi loaded just fine.

I installed Windows 10 pro here, and tried to PCIPassthrough only one graphics card, but it failed with code 43.

Parameters should not be set to hypervisor.cpuid.v0 FALSE as a workaround for Code 43, nor should SMBIOS.reflectHost = TRUE, pciPassthru0.msiEnabled = FALSE. The GTX 1660 SUPER has 4 types of PCI, so I put all 4 variables in msiEnabled.

I added pciPassthru.use64bitMMIO=TRUE as a parameter, but it failed, and I set pciPassthru.64bitMMIOSizeGB = 16. (Since the graphics card memory is 6 gigabytes, 6 x 2 = 12, the closest number to the square of 2 is 16, so I set it to 16 .)

I also have a 5900x system running (x570), this system has a GTX 1060 3GB, and on this system the ESXi works fine, so the PCIPassthrough is fine. After installing this graphics card on a 7900x system, I tried it, but the same code 43 comes out.

After that, I gave up on the GeForce graphics card and tried with the Radeon 5700XT, but the same code 43 error occurs.

I thought it was a bug in ESXi 8.0, uninstalled 8.0, tried again with 7.0U3 and it failed the same.

I confirmed that the mainboard SVM is enabled, I also enabled the address decoding option of 4GB or more in PCI, I tried enabling and disabling it anyway, although it is not a SR-IOV supported graphics card, I also enabled PCI ARI Support, enabled ACS, and enabled IOMMU. I did it. TPM doesn’t seem to matter much, but I disabled it, but all failed. BIOS has been updated to the latest version.

The only unusual symptom is that if you change the Resize BAR to OFF, and PCIPassthrough the GeForce graphics card to the virtual computer, the same symptom, but Radeon boots normally once.
However, when I try to restart the Radeon virtual machine after shutting it down, I get a DevicePowerOn error and the virtual machine cannot be turned on. In order to run the virtual machine again, the entire ESXi system must be rebooted to turn it back on, but even after turning it back on, the symptoms are the same.

When I installed Proxmox and tried it, I was able to PCIPassthrough normally in Proxmox. I expected it to be a hardware problem, but it seems to be a software problem. Does anyone else have the same symptoms?

I am a user from a non-English speaking country. Please forgive me if the sentence is not smooth.

2

“However, when I try to restart the Radeon virtual machine after shutting it down, I get a DevicePowerOn error and the virtual machine cannot be turned on. In order to run the virtual machine again, the entire ESXi system must be rebooted to turn it back on, but even after turning it back on, the symptoms are the same.”

If I understand correctly then this is normal behavior for many AMD GPU cards. I do not know of any workarounds.

3

According to testimonies from people who have experienced AMD GPUs, they were able to do PCIPassthrough on ESXi or Proxmox more easily than on NVIDIA GPUs. However, I was able to do PCIPassthrough more easily on NVIDIA GPUs.

I also don’t know the cause.

That was when trying PCIPassthrough on the 5900x. None of my attempts have been successful with the 7900x.

4

I moved away from VMs because of the reset problem and also USB problems. I replaced the VM with an ITX system and things work again.

5

Did you attempt USB Passthrough? And was your motherboard a b650 or x670?

6

Yes, I had USB passthrough mostly working. The MB was an AM4 board, ASROCK AMD X470 Taichi. There was problems with KVM USB switch that only worked with keyboards without volume controls on the keyboard. I originally thought it was the keyboard, not so. Then I thought it was the switch, not so. In my opinion the USB passthrough is not fully hashed out. FYI, the same gear all works fine now that I am not using USB passthrough. The ITX system is half the size of a mid tower. I plan to convert the midtower to an ITX system to reclaim some of the desktop back. I suppose maybe the MB was sensitive to USB passthrough?

7

The USB controllers in Ryzen systems tend not to cooperate with passthrough, having reset issues. I’ve found that adding ‘pcie_acs_override=downstream,multifunction’ to the kernel options on Linux systems seems to resolve it though. At least on the 3 boards I’ve used.

1 Like
8

Did you ever get this resolved? I have the same issue. I tried rtx3060, rtx4070, rtx4070TI, all with the same issue. Funny enough, my gtx980 works no problem. Maybe something to do with the cards being over 4gb?