I recently “migrated” my and my girlfriends pc into one machine, which is running 2 VMs as our desktops.
After 1-2 days of tweaking, the system is, more or less, running as it should. Except I have some performance issues/lag on one VM, where I think the issue is too few PCIe lanes. This is where I would need some assurance, so that I don’t have to overpurchase anything.
Our specs:
CPU: AMD Ryzen 9 3900X
MB: Asus Rog Strix x570 E-Gaming
RAM: 4 x 16GB G.Skill RipJaws V @3600MHz
GPUs: Radeon RX 5700xt, GTX 970
Drives: 2 x PCIe SSDs (Samsungs 970 Evo and 960 Evo) and 3 x Sata HDDs
Power Supply: EVGA Supernova 1000 P2
OS: Unraid
We are running 2 Windows 10 VMs with the GPUs and Drives passed through.
Now, if I understand correctly, I can use 20 PCI lanes with the 3900X, therefore I would be 4 lanes short (GPUs [2x8 lanes=16], SSDs [2x4 lanes=8]).
In theory, if I replace the two SSDs with one bigger SSD, which would be used by both of us, I should be fine right?
I’m also thinking of replacing the SSDs with 2 Sata SSDs to be completely on the safe side.
I would be glad to hear some opinions from you guys!
Cheers!
Bottom M.2 goes trough chipset, so assuming you don have anything else plugged in then:
x8+x8+x4 - 2 GPU and upper M.2 (20 lanes directly to CPU)
bottom M.2 - x4 through chipset
So it shouldn’t cause any obvious problems with PCIe allocation.
As for cause of lags, hard to say, as it depends how you configured VMs. However, past few months I saw few times that people had problems mixing 5700xt with 9xx gpu’s.
Also, if you don’t do proper CPU pinning then both VM might be using same cores. For example try to pin say 4 different cores for each VM, and check the results. Since its 3900x probably 4/4/4 would be good split (host/vm1/vm2), but you may tweak it. So 0-7 host and qemu io, 8-15 first VM, 16-23 second VM.
And if you use this machine only for VM-s then probably isolating CPUs for VMs will help too.
BTW, how did you resolved sound? With Scream? Sound output can cause stutters for sure, even with single VM. Its easy to check, just disable sound and check if lags go away.
And what about input? Are you passing 2 sets of USB with kb/mouse to each VM?
Looking at the manual, it lists 8x 8x 4x - 4x 4x as a valid configuration. Each of those should be the full listed speed(both slots are up to 16x, but 8x when both are populated)
I thought passing the M.2 and cleaning all the drivers in windows would be sufficient - turns out it isn’t.
I managed to get rid of the lags by setting up the M.2 as a btrfs drive, creating a new vdisk and doing a fresh windows install.
Right now the systems are running almost perfectly - only dealing with the AMD reset bug now.
Where did you find out that the bottom M.2 goes through the chipset? Does that mean that I could use 2 GPUs and 2 M.2 at full bandwidth?
Didn’t do anything special for sound. With the RX5700xt it works fine over HDMI/Displayport and with the gtx970 I experienced some stuttering, but there should be a fix for this (currently running audio through a docking station).
Currently passing through a temporary docking station for one machine and directly passing through USB ports and audio ports for the other machine.
I wanna get two USB-C docking stations/hubs with audio jacks but that stuff is expensive
Manual says that one of M.2 slots go through chipset:
3rd Gen AMD Ryzen™ Processors - 1 x M.2 Socket 3 with M Key, Type 2242/2260/2280/22110 storage devices support (PCIE 4.0 x 4 and SATA modes)
2nd Gen AMD Ryzen™/2nd and 1st Gen AMD Ryzen™ with Radeon™ Vega Graphics Processors - 1 x M.2 Socket 3 with M Key, Type 2242/2260/2280/22110 storage devices support (PCIE 3.0 x 4 and SATA modes)
AMD X570 Chipset: - 1 x M.2 socket 3 with M Key, Type 2242/2260/2280/22110 (PCIE 4.0 x 4 and SATA modes) storage devices support - 8 x SATA 6.0 Gb/s connectors - Support RAID 0, 1, 10
And I can’t see manual explicitly stating that the bottom one goes through chipset, however this is most common and probable scenario.
You can use 2 gpus at 8x and 2 M.2 at x4 with PCIe 3rd gen bandwidth, since you have 3900x. If this if “full” bandwidth available with your devices may be arguable, but can come be close to it.
For example x4 bandwith from chipset may be shared with other devices in it. And since Pcie 3rd gen has 8GT/s transfer rate (around 1GB/s per lane) then chipset has around 4GB/s bandwidth available.
So, how this will be allocated depends on other devices you have plugged in.
Ah, ok, you’re using HDMI. Nevermind then.
So you may want to pass whole USB controller to each VM instead just USB devices. But that depends on your IOMMU groups.
Also have in mind that USB from chipset shares bandwidth with one M.2 drives…
Anyway, you’re pretty much pushing to the limit what x570 has to offer, so yeah… there can be some issues.
I had a few stutters from different problems.
I had faulty USB cables, external powered USB cable which didnt like the powered USB Hub.
A few bad USB Hubs.
Some of the cheap PCie to USB Cards Produced stuttering.
A USB Soundcard was faulty too.
All in all over a few different Systems and OS .
By now i have the best experience with ESXI and Enterprise Gear as much as possible.
I would Always passtrough a whole USB Controller to an VM.
as much as possible.