Supermicro M12SWA-TF and GPU Passthrough

Operating Systems & Open Source Virtualization
1

Recently I setup a Supermicro M12SWA-TF motherboard with a 5975WX and have spent quite some time getting GPU passthrough to work in Proxmox. BIOS is flashed with v2.0b (latest).

For anyone else considering this motherboard beware that Slot 7 (closest to the CPU) does not allow GPU pass through without GFX card related errors around FLR reset (see below) which cause the VM to fail to start up.

Jan 11 14:51:38 pve kernel: [83097.716315] vfio-pci 0000:41:00.0: not ready 1023ms after FLR; waiting
Jan 11 14:51:39 pve kernel: [83098.771575] vfio-pci 0000:41:00.0: not ready 2047ms after FLR; waiting
Jan 11 14:51:41 pve kernel: [83101.011621] vfio-pci 0000:41:00.0: not ready 4095ms after FLR; waiting

Move the GPU to another slot and it works without issue.

I have the following cards successfully running;

  • Gigabyte GeForce RTX 3060 WINDFORCE OC 12GB, Model number GV-N3060WF2OC-12GD
  • EVGA GeForce RTX 3060 XC LHR Graphics Card 12GB, Model number 12G-P5-3657-KR
  • Nvidia RTX A4000

Cards I have not been able to run.

  • XFX Radeon RX6700 SWFT 309, Model number RX-67XLKW, seems to be effected by Reset error and others have reported the same trouble.

My only other complaint about the motherboard is the PCH fan is to noisy.

2

Thank you for posting this! I am on 2.0c (currently the latest) and struggled with this for two hours. Moving the card to another slot solved the problem, and GPU passthrough in Proxmox using GTX1080, GTX 980ti, and RTX A2000 cards worked immediately without any new issues.

My symptoms were slightly different. I saw the following messages in the log:

Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff

pci_irq_handler: Assertion `0 <= irq_num && irq_num < PCI_NUM_PINS' failed
3

I am experiencing the same issue, using 2.0c. This is a major bummer, I bought this motherboard because I needed 4x PCIE 4.0 16x slots for 4x RTX4070s through for a cloud gaming rig I am building. You have any luck on a more permanent resolution to this?

My other issues with the board is that it seems to get stuck in a boot loop, it shows the first splash screen before you get the options to press F2 to enter the BIOS, then it will sometimes reboot on that page or sometimes it makes it to the F2 screen. If I power off the machine then start it back up it will boot correctly.