@udmh I’m starting to “feel at home” with my new EPYC machine so I thought I’d go back to your original post and try and answer as many of your questions as I can.
Regarding virtualization : the motherboard’s BIOS exposes all relevant settings including SR-IOV. I’ve only used VMWare Workstation over Windows 10 but there’s nothing special to report.
The KSM32RD4/32MEI sticks are as nicely generic as you could want : nothing fancy, they don’t get hot, I haven’t had the slightest issue with them. They are good value, I’d say.
You are right to want to use a PCIe card for NVMe drives : the placement and routing of the on-board M.2 sockets isn’t ideal. Heatsinks would definitely be required and those would interfere with at least 3 of the PCIe slots. A bad idea if you want to use long GPU’s directly on the motherboard.
Speaking of which, you’ll need to be extra-careful if you’re building into a tower case : those PCIe slots are not built to support heavy GPU’s at all. Make sure to tighten those I/O brackets or find a more “creative” way like I did.
Also, keep in mind that if your GPU’s cover the X550’s heatsink you will definitely end-up with thermal throttling issues. This heatsink is barely sufficient on an unpopulated board. I think Asrock could have done a better job. I doubt it would see any air even in a bona fide server chassis with jumblowiematron fans.
Speaking of the downsides mentionned by @wendell , I didn’t run into any. The one that really got me in the feels is the total lack of ACPI S3 and S4 support. I got an e-mail back from Asrock saying that they had no plans to support that. Which makes sense for server hardware, but still, it would have made this board the perfect workstation board.
However, it’s not necessarily a big issue. I’ve been using it for a while now and if you leave it alone (as in : turn off the monitor and go to bed) it only draws 65 W. What’s more, at that speed the PSU fan doesn’t spin, and neither do the fans on an RTX 3090 FE. If you can live with the additional expense 65 Wh represents, you may not need to go for ThreadRipper Pro.
Perhaps another issue is that the IPMI’s remote KVM can’t show you the desktop unless its own VGA port is included as part of your monitor setup. This comes with its own set of annoyances : that port is not 4K, obviously, and if you use a KVM switch Windows will mess up your windows’ size and placement any time you switch away from the EPYC workstation. I’ve had to disable that VGA adapter in the BIOS (by setting “external GPU” in the BIOS instead of “auto”).
I don’t have any AMD GPU so I can’t rest that reset bug you mentioned, but I’ve had no issue with Nvidia cards.
All in all, the machine you’re envisioning is a very viable workstation. It’s as fast as you can imagine a 16 core / 128 GB / NVMe platform would be.
As for me, I think I’ll keep it as my workstation until ThreadRipper Pro is slightly easier to get my hands on. I really want to go with EPYC mainly because it’s cheaper and the motherboards are smaller, but that Gigabyte WRX80 motherboard looks like it does at least everything the ROMED8-2T does and it’s not too much bigger. We shall see.
If you have any question, don’t hesitate to ask.