Mellanox CX3 40G card causes issues with GPU passthrough on Asus X99-E-10G WS

Operating Systems & Open Source Virtualization
1

I have an Asus X99-E-10G WS board with an Intel Xeon E5 2679 V4. I know the CPU has 40 PCIe lanes and has support for IOMMU so passing through GPUs in proxmox is trivial.

This board has PLX chips to split 32x PCIe 3.0 lanes into its 7x PCIe slots that operate at all 8x or 4 of them at 16x. I have passed through multiple GPUs to VMs on this board without issues before, but I just got a Mellanox ConnectX3 FCBT to connect to my NAS and it seems like this is causing issues with passing through a GPU that’s also on the same PLX chip as the Mellanox card.

I have a Tesla P100 that I am trying to pass through that’s plugged into a PCIe slot coming from the second PLX chip that also has the Mellanox card plugged into another port from the same PLX chip. This causes a code 10 error in windows device manager that said there is not enough resources to start the API and the GPU won’t start and can’t be used by the driver.

I have 4G Decoding, Virtualization, VT-D and ACS enabled in bios as well as CSM disabled and it still does not work. It will only work if I plug my Tesla P100 into another slot that is connected to the first PLX chip while the mellanox card is in the second PLX chip. This is an issue because then I would effectively reduce the number of PCIe slots available for use for GPUs on the board.

Is this fixable or just an inherent behaviour of Mellanox 40G cards? Thanks for any help.

2

How is that the final question after mentioning it working if they’re connected through separate PLX chips?

How many slot combination did you actually check though? I was recently disappointed by a board having static slot enumeration. Many devices show up as multiple, I have a couple slots that don’t allow that.

3

Yea I am asking if Mellanox cards somehow doesn’t like sharing a PLX chip with other devices or not. I was thinking maybe it requests a larger PCI address than other devices and causes issues when it shares a PLX chip with other devices. Or something.

I tried all the slot combinations and it only works if it’s the only device connected to the PLX chip. Considering this board puts 3 slots through the first PLX and 4 slots through the second PLX, this just wastes so much PCIe slots.

4

HWInfo64 might tell you some things. See what it says when working, and compare that to when it isn’t.
eDIT: FOR EXAMPLE

image
image687×888 88.6 KB