Idea: Mix IOMMU GPU Passthrough with coin mining

Communities Science, Engineering, & Security
1

So I had a thought. The limitations on GPUs per system is at 8 for both Windows and Linux from what I’ve read. And IOMMU works by passing through a grouping of devices to a Virtual Machine.

What if you combined the two to increase the maximum number of GPUs on a single system (i.e. motherboard, CPU, and RAM)?

There are PCI-e splitters that essentially turn one slot into three. IOMMU grouping, when it works correctly, usually splits devices into groups based on the PCI-e slot on the motherboard.

So if I had a motherboard with three PCI-e slots, I could attach splitters to each and have up to 9 GPUs on a single board like that, but then the OS limitation comes in.

Scaling up to a board with more PCI-e slots, you could then have more GPUs as well. So, what if instead of just running them all on one OS, you passed them through to a VM to be handled there?

I figure if I had a motherboard with 8 PCI-e slots, and a CPU with 32 PCI-e lanes (or with the chipset, 32 total), you could theoretically do up to 24 GPUs on a single board. 8x3 = 24, and assuming they used a max of 4 lanes, that’s 32 lanes.

Then just pass through 2 PCI-e slots on the motherboard (so 6 total with the splitters and so 6 GPUs), and you could have four VMs, each with 6 GPUs.

Driver instability with mixed GPUs (not AMD vs Nvidia, but different AMD generations) is one annoyance with mining. And configuring the Linux host to monitor the mining on each VM and restarting the VM automatically when an issue occurs just seems like it’d save a lot of time. Plus, snapshots for when updates break things.

Overall, if this could work, that’d be very interesting. Reasons I think it might not work:

  1. Linux still has to be able to pass the GPU device to the VM, but if it can’t see more than 8 GPUs, how would the other 16 be handled?
  2. RAM constraints (though I figured at 32GB of RAM, with 5 OSes, you could give each Windows VM 6GB and the Linux host 8GB or so).
1 Like
2

I'd like to see it done just for laughs. Would make for a cool video.

2 Likes
3

in theory if you had everything setup in the EFI or bios plus boot loader then yes it could work. all of the split lanes must use the same type of card. and each set of split lanes must use a different model/gen of card than the other set of split lanes ( wokeyness insures i am told). and then we get to the bus limitations and how it could negatively effect performance .

4

PCIe throughput is irrelevant for mining. Many miners have their GPUs connected via x1 riser cables anyway.

1 Like
5

These are the risers I'm talking about.

And these are the splitters I'm talking about:

And they work apparently (from what I've read online, and I've got one in the mail) without affecting performance.

6

As I understand the GPU limitation is more an issue with the driver, not the OS. So if the devices aren't initialised (which is what you want with IOMMU/passthrough as I understand), the OS would know there is a device, but wouldn't know what it is because it's not initialised, so the driver is not an issue at that point.

In theory it should work, the question is if you can find a board that has each PCIe slot in a different IOMMU group.

Aside from that, what about those 12 PCIe-slot mining boards (Biostar TB250-BTC Pro), there must be a way to use those to begin with, no? So if there's a limitation how would they use it?

1 Like
7

If the limitation really comes from the driver, as you say, those boards could be used by mixing Nvidia/AMD cards. Just guessing here.

8

Yeah that's my guess too since they 6x AMD and 6x nvidia in one of the pictures too.

I don't remember where I read about that driver thing, but there was something about AMD working on the issue so... yeah.