GPU Passthrough on PowerEdge R7525

[SteveiJobs]

Hello all!

I’m working on a weird issue that I can’t seem to diagnose.

I have a PowerEdge R7525 with 2x Epyc 7302 CPUs, 256G RAM, etc.

It also has a Quadro RTX 6000 server variant (direct from Dell). I am trying to get it to passthrough in ESXi 7u2 but the guest will not properly boot. I have turned on IOMMU in the BIOS and have toggled passthrough in ESXi but it errors out when trying to power on the guest. I was reading that this may be an SEV setting but I can’t find anything concrete.

Interestingly, ESXi will not allow me to configure SR-IOV, even though the card supports it.

Anybody have any pointers?

Thanks!

[burrissj15]

Hey,

So a few things.
First to run the card with SR-IOV to support vgpu profiles you need to install the Nvidia Host VIB. Then set the card to shared in the GPU settings in ESXi.

Next for Passthrough because the RTX6000 has more memory than 4 GBs you will need to update these two advanced settings on the VM.

pciPassthru.use64bitMMIO=“TRUE”
pciPassthru.64bitMMIOSizeGB = “128"

Where the 128 is the GPUs memory to the power of 2.

You will also have to reserve all the RAM on the VM that is running with the GPU attached.

Here is the VMware KB that goes over this: VMware Knowledge Base

Hope this helps

[SteveiJobs]

Thanks! I will try next time I am in the office.

Just a quick clarification, when you say power of two, do I take the VRAM and square it (24*24 = 576)?

Thanks!

[burrissj15]

Sorry for the semantic error, it should be times 2 not to the power of 2.

It should be 24*2 = 48