Proxmox with two pass-thru GPUs?

computer_cerb · January 13, 2021, 5:57pm

I want to convert my current system (ubuntu 20.10) that already has an AMD GPU for the host and a KVM Windows with passed-thru NVIDIA.

System is based on Gigabyte Aorus Master X570.

I would like to add a third GPU and set the system up with Proxmox (using AMD GPU). A second (Ubuntu 20.10) VM should then have the added GPU passed-thru and be able to use it.

Since I would need to by a new, beefier powersupply for that, I’d like to get a feeling first if this was possible at all. (I am aware that the GPUs in slot one and two would be required to share the x16 lanes as 2 x8). But, would there be any further pitfalls?

Will the second slot (sharing the x16 lanes as 2x8) have it’s own IOMMU group?

Or will it be the same as the first GPU?

And if so, is it possible to isolate it for the ubuntu VM?

Oh, and the two guests should be able to run simultaneously.

GoldenAngel1997 · January 13, 2021, 7:30pm

Entertaining a 3rd GPU, would likely need to be a LOW-Powered chip [x4 electrical]

computer_cerb · January 13, 2021, 7:44pm

Right now the “3rd” card (meaning in the third (x4) slot) is an AMD FirePro W7100, and it hast it’s own 6-pin connector for power. Works just fine for powering an HD and 21:9 UW monitor.

The first card (read: first (x16) slot) is a 3090 (for the Windows VM).

The spare card (after the upgrade to the 3090) is my previous 2080ti.

All of them bring their own power-connectors with them - obviously. The two NVIDIA’s have - again, obviously - two 8-pin.

But since my current (750w) PSU only has 2 PCIE 12v outlets, it needs to be upgraded for this to work. Now, the replacement would be a rather expensive PSU (something 1200+ watts (with more 12v’s), to support both of the power-hungry GPUs and the rest of the system). So this would be a non-trivial expense which I’d like to avoid if something on the “logic side” was destined to fail.

Since I want to get into ML it would be nice to be able to play some games while waiting for calculations to finish. So, that’s the motivation - and also, just because it can (possibly).

robbbot · January 13, 2021, 8:57pm

Depending on how many NVMe drives you have in there as well, your going to start to limit your PCIe lane availability.

This will disable some of the sata ports on the controller, something to consider.

computer_cerb · January 13, 2021, 9:08pm

You’re right. I have three 1TB NVMe’s in there, so this disables two of the six SATA ports. The other four carry a 4TB harddrive each. Planning to also setup a Freenas VM for storage in there. 16(12)TB of RAID5 seems enough, though. Especially since this is only to play with Freenas, those 4Tb drives come from a recent upgrade of my Synology NAS.

The third GPU should not change this, correct? The first NVMe is connected to the CPU and the other two to the PCH - which is also supporting the 3rd card (config now). The second PCIe slot should only split the x16 into two x8? That wouldn’t do much to the 3090, since it’s PCIe 4.0 anyway. And the 2080ti for Linux is just for display and a little ML. Maybe I’ll swap the two … depends on how the memory situiation turns out.

2080ti is still enough for gaming, if it needs to be.

robbbot · January 13, 2021, 9:12pm

I’ve only used it with 2 cards, so you’ll have to let us know if it works with 3. My guess would be that it will since the IOMMU groups for the board for me have always been really good, but that’s just a guess.

computer_cerb · January 13, 2021, 9:15pm

Thanks. Guess I’ll just get that PSU then, when they are in stock.
I’ll keep you guys posted.

computer_cerb · January 23, 2021, 10:37am

Guess what showed up today?

computer_cerb · January 23, 2021, 10:37am

Now waiting for level one KVM to arrive

computer_cerb · January 23, 2021, 4:36pm

Ok, stupid me.

It doesn’t fit.

There is actually too little space on the Aorus Master to fit the Gigabyte RTX 2080 ti Gaming OC (2.something slots).

The 3090 is not the problem, but the (on AMD) necessary third GPU in the lower (4x) slot. A 2-slot card (like the 2060 super) will fit, but mine has no USB-C port …

The 2080 ti does.

I am open to suggestions.

New case with Threadripper and a board with way more PCIe slots? (would also allow me to run both VFIO GPUs on x16 …)
Water-cooled GPUs?
2-slot 2080 ti FE? RTX 3000 doesn’t seem to sport a USB controller any more …

I’m a bit bummed right now, would have really loved to give this a try …

computer_cerb · January 23, 2021, 8:40pm

Ok. Got myself together and bought a used founder’s edition card. Can always sell the Gigabyte card anyways.