ESXi 6.5 w/ GPU passthrough on AMD Dell gaming desktop

Since I was tired of most of the B.S. coming from Nvidia (GPP, etc.) - and to a lesser extent Intel - I decided to buy a pure AMD system. So this was my solution:

My plan all along was to take this system and set up a Windows VM for gaming, and create other VMs (Linux, pfSense, etc.) for personal servers and educational use.

The system I got differs from the one in the video. For US$20 more, I got a Ryzen 7 1700 and 16GB of RAM (vs. the R5 1400 w/ 8GB in the video.) The hardware does have some disappointments. The NVMe port functions at PCIe Gen2 speeds, the two video slots are PCIe 3.0 x8, and it came with one stick of DDR4 16GB (apparently Dell doesn’t know what dual channel RAM means to a Ryzen system.) Also, it only has 2 memory slots and the included RX 580 is a reference model, with a baseline clock speed of 1266MHz and a wimpy blower style cooler.

Overclocking the CPU is possible ONLY in Windows with Ryzen Master; the BIOS is tightly locked down. In Windows 10, I can hit 3.7GHz easily, though 3.8GHz craps out under load. Is there any way to OC with software in other environments?

In spite of those limitations, I decided to keep the system and try putting ESXi 6.5U1 on it. I had to build a custom ISO with Realtek 8168 Gigabit NIC support, as well as a generic AHCI driver. Once that was accomplished, installation went smoothly.

Getting passthrough working was a little trickier. Although the GPU + HDMI audio controller were in an independent IOMMU group, the USB controllers were lumped in with a bunch of other hardware. Fortunately, the USB 3.0 controller was grouped with some non-essential devices and I was finally able to passthrough a USB keyboard and mouse instead of fooling with Synergy or VMware’s Remote Console.

I’m not sure if Linux supports this, but a neat feature of ESXi is that I can pass through the system’s sole GPU instead of having to have two GPUs, one for the OS and one for GPU passthrough. At system boot I see the ESXi console, but after the Windows VM starts the ESXi screen vanishes, and the Windows login appears.

So now I have a Windows 8.1 VM with GPU passthrough running games. The performance hit is a little worse than I’ve come to expect with VM gaming. In Assassin’s Creed Origins, I’m seeing 41 fps at Medium/1080p settings, whereas if I boot Win10 I get about 47 fps at 1080p/Medium. Of course I only have 6 threads assigned to the Windows 8 VM, whereas Win10 has access to all 16 threads. The Win8 Adrenalin drivers are also several months older than the Win10 drivers.

Since I’ve got this working fairly well with ESXi, I’m not sure if it’s worth the effort to try a Linux virtualization setup with this rig. Would Linux offer any particular advantages over ESXi in this situation? Would I have to add a 2nd GPU?

I also need to do something about the RAM. I have a single 16GB stick of DDR4 2400, and if I want to run more than 2 or 3 VMs I’ll need to expand. If I add a second stick from a non-Dell source, what are the odds that it’ll function in dual channel mode?

2 Likes

Personally, I’d stick with the ESXi setup since you have got it running well. Adding another stick of generic DDR2400 should work fine, in theory. You might find it better to buy 2 x matching 16GB sticks and sell the one included if you can’t match it. Some motherboards can be more fussy than others although I’ve generally not had too many problems mixing brands of RAM (even with different timings).

Hi,

I’m thinking about building a system like this with an AsRock Taichi x470 and ESXi 6.7 –

I’m curious, did you have to alter the VMX file for Windows at all? (Or add any flags under VM Options --> Advanced --> Configuration Parameters)?

Which video card did you say you are using? RX580? Do you know if this will work with an Nvidia card, or will I get the error 49 lockdown bullshit?

Thanks!

@AveryFreeman, this thread is 4 months old. You should start a new one.

Feel free to link this one and/or @ any of the posters if you want to get their attention.