Virtual Machines Dual Chip GPUS

There are video cards out there that have 2 GPU chips on them I was wonder if anyone out there has one and if they are whiling to see if they can do a GPU pass-through to 1 of the chips on the GPU to. This is more so see if this is possible. Below is list of GPUs. If I forgot to mention some out there please tell me I will add them on the list.

Nvidia GeForce GTX 690
AMD Radeon 7990
AMD Radeon R9 295X2
AMD Radeon Pro Duo
AMD Radeon Pro Duo (Polaris)

2 Likes

Here are some more dual chip GPUs-
Radeon V340(Has SR-IOV so definitely works with multiple VMs)
FirePro S7150 x2(Has SR-IOV so definitely works with multiple VMs)
GTX 590
GTX Titan Z
R9 390x2(No reference model, only one available is the powercolor devil 13 390)
Asus MARS GTX 760
Radeon HD 6990
Radeon HD 5970
GeForce GTX 295
Radeon HD 4870 X2
GeForce 9800 GX2
GeForce 7950 GX2
Radeon HD 3870 X2

I think these are all the PCIe ones, there are more AGP or PCI cards that I did not include, and I might have missed a PCIe card or two.

3 Likes

I would expect they are both in the same IOMMU group. It will be interesting.

I registered just to give you a reply:

I have tested GTX690 with UnRaid.
You are able to get each GPU in a separate IOMMU group with ACS patch.
You can boot inside the VM with only one GPU, but I am unable to get the drivers working. In windows VM I get code 43 and thus the nVidia driver cannot be installed.

Also the GTX690 behaves weird in native windows 10 on my machine Ryzen 1700, Asus X370F. I get the same code 43 almost every time.

I have tested this with 2 GTX690 so I assume the todays motherboards just do not understand the PLX chip on the card. Buildzoid mentioned in his video, that the MARS 760 does not work correctly in todays mobos and it uses basically the same components as GTX690.

If anyone feels like helping with the issue, I would appreciate it. If anyone has a UEFI compatible bios(PLX included) for the GTX690, that would be most helpful. Flashing the EVGA files from gdrive(linked from nVidia forums) does not seem to work for me.

4 Likes

Tested the card on Maximus V gene with i7-2700k and I was getting code 43 in windows.

I have a new test machine i5-6600 asus b150m-c.

I just plugged the card in and in windows i was able to get the card working, no code 43, even gamestream works. Happy to see the card finally working under normal usage.

I will update here once I test the passthrough with unraid.

The code 43 is when Nvidia card detects it is running in a VM. You generally need to add a line in the hypervisor to say tell the VM ‘No really, you’re really a computer, promise!’

Unsure on exact syntax on how to do this in unRAID though.

1 Like

In this case it is not driver detecting VM. GTX 1080 is working flawlessly. (unRAID is very foolproof, since the hypervisor is already preconfigured for nVidia passthrough.)
My problem is, that the GTX690 is giving me code43 in native windows, which means either the bios does not like the PLX chip or the PCIe slots i try are faulty. Interesting is, that the GTX 1080 works fine in these slots. Anyways, on the B150M board I am able to run GTX690 in native windows, so I just need to install a hypervisor and try to setup VMs. If that does not work, I will try to mod the card into quadro/tesla, that would yield quite interesting results considering the 2nd-hand price of GTX690.

2 Likes

That sounds like either there’s a driver issue with the 690, or the 690 is having hardware problems. Could be UEFI on the motherboard or the BIOS on the card too (just spitballing).

Could also be related to the way the chips are setup on the card. They may not be able to be split as they could be sharing the same VRAM pool or something

Have you tried asking at the Unraid forums. They can usually be helpful unless you are not conforming to their expected Unraid usage.

still I have not tested with the new moherboard.
In unraid forums, there is no record of succesfull passthrough of one GPU on a dual GPU card. On the otherhand there are records of people modding the GTX 690 into grid K2 and running vGPUs in xenserver.

hopefully I get to testing over weekend

So here are my findings:
On the asus b150m motherboard both gpu chips (gk104) are in separate iommu group (one gk104 + hdmi audio in one group) so no acs patch needed.
I did testing with Unraid 6.7.0 and iGPU is used as display output for Unraid. Tested are windows 10 VMs.
I am able to passthrough one gk104 to a VM and driver is ok. For some reason only the chip with 2xDVI and miniDP behaves well. The second chip with only 1xDVI-D does not detect LCD, but besides that driver is happy with it. I assume that is just some bug of this GPU, since I have not seen this behaviour on any other gtx 690.
When I do passthrough of both chips to different VMs, I get code43 in the VMs.

There is most probably problem with some shared resource such as fan control, which I do not know how to investigate further.

I will try with different bioses for the GPU and try modding resistors to make the card think it is Grid K2.

Any suggestions what to try is welcomed.

Any further luck with this? I’d love to get more GPU’s into one of my unraid servers using GPU passthrough, but if they won’t work simultaneously then it is a no-go .

Got suggested this yesterday:
Timestamped to Tesla to Grid modification.

1 Like

Yeah I’ve actually seen Jeff’s videos on getting multiple VM’s going with a few K2 cards he has physically altered. To me it isn’t worth taking it that far, I was just hoping to get more than about 4 GPU’s into a gaming VM server (really I can only get about 3, as I need other add in cards). I was also hoping to stay away from vGPU’s which I believe is how he is accomplishing it instead of using pass through.

I spent this weekend messing with a used GTX 690 from eBay. I found the UEFI ROMs on TechPowerup, you’ll need a pair as the GPUs each have their own. I used 80.04.4C.00.0E (GPU1) and 80.04.4C.00.0F (GPU2).

I can’t include links (yet?) so you’ll have to paste these after the techpowerup domain portion of the URL:

/vgabios/212271/212271 - GPU1
/vgabios/189094/189094 - GPU2

I actually flashed my card and so far so good but I’d probably recommend just passing those files with qemu/libvirt. I’m using Slackware as my host and libvirt (via virt-manager) so I’m not sure exactly how you’d do that in Unraid. I’m using the Q35 pc and OVMF.

In my host OS, the card now boots up and uses efifb properly so the new VBIOS does seem to work with EFI.

I had to use the ACS Override patch to get them out of the IOMMU group but I am actually able to use them separately with a Linux guest, even using the NVIDIA proprietary driver. However it seems like the VBIOS isn’t actually run properly in the guest as there is no output before the kernel load and only nouveau creates a framebuffer after the driver is loaded. When using the NVIDIA driver I have to launch X from the command line (through SSH) before any output is shown.

I’m seeing errors related to the BAR in dmesg, so again, I’m thinking the VBIOS isn’t being run for whatever reason. Here are the seemingly relevant lines:

[    0.553044] pci 0000:07:00.0: [10de:1188] type 00 class 0x030000                                                                                                                            
[    0.556116] pci 0000:07:00.0: reg 0x10: [mem 0x90000000-0x90ffffff]                                                                                                                         
[    0.558117] pci 0000:07:00.0: reg 0x14: [mem 0x800000000-0x807ffffff 64bit pref]                                                                                                            
[    0.560117] pci 0000:07:00.0: reg 0x1c: [mem 0x808000000-0x809ffffff 64bit pref]                                                                                                            
[    0.562117] pci 0000:07:00.0: reg 0x24: [io  0x7000-0x707f]                                                                                                                                 
[    0.565117] pci 0000:07:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref]                  
[    0.565172] pci 0000:07:00.0: Max Payload Size set to 128 (was 256, max 256)                
[    0.566691] pci 0000:07:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x16 link at 0000:00:02.6 (capable of 126.016 Gb/s with 8 GT/s x16 link)
...
[    0.670558] pci 0000:07:00.0: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window
...
[    0.722997] pci 0000:07:00.0: BAR 6: no space for [mem size 0x00040000 pref]                                                                                                                
[    0.723813] pci 0000:07:00.0: BAR 6: failed to assign [mem size 0x00040000 pref]
...

I’ll keep plugging away at this as there seems to be some promise here. I get Code 43 in Windows guests, which is ultimately what I want to be working (not much point in running a Linux guest on a Linux host for me personally). Maybe this will help you or others out, and someone knows something I don’t. It’d be awesome to get this working as my desktop machine is a Mini-ITX form factor and only has one PCIe slot period, so it’s this or nothing for me :wink:

2 Likes

still we do not know, what is the issue with the dual GPUs.
I went with PCIe bifurcation. This riser https://www.ebay.com/itm/Bifurcated-PCI-E-riser-x16-to-PCI-E-x4x4x4x4/223684389701?hash=item3414a0eb45:g:wh8AAOSw63tdwqnL

it gives me 4 PCIe 3.0 x4 slots for GPUs and other cards. I have a x399 board, so bifurcation is supported. Some high-end boards for AM4 or intel should have x4x4x4x4 on at least one slot too.

We need users like you to create this sort of documentation.