Ryzen 2700x or Threadripper (any) for KVM with 1 GPU for host, 1 for Windows gaming VM

Hardware Build a PC
1

Greetings.

I’ve doing a bit of searching, but couldn’t seem to find anything similar.
Apologies if my search fu is bad :fearful:

Just to give a lil something about where I’m coming from:
Currently I have an FX8350, Crosshair V Formula-Z, some 16Gb 2400MHz DDR3 RAM, a HD7790 for Debian host running KVM with a R9 290X passed through to a Windows gaming VM.

I would like (at least in the beginning) to use both GPUs in a similar, but more modern, setup. Either Ryzen 7 2700x (and most likely a Crosshair Hero VII) or some form of Threadripper (not decided on motherboard yet). Motherboard must be excellent in terms of IOMMU groups (and other goodies).
It has come to my attention, though, that the 2700x only comes with 20-24 PCIe lanes, compared to Threadrippers 60+ lanes.

Will most likely want to use 2 (if possible) nvme SSDs in the setup, with one with the Linux boot/system/root/perhaps more, and the other passed through to the Windows VM.
Already this setup seems to be at odds with the 2700x and PCIe lanes.
Additional PCIe cards might occur.
I would like for the PCIe to run full on version 3.0 and not downgrade itself to either x8 3.0 or x16 2.0.
Currently, my old arsed FX8350 setup is running full on x16 2.0 (max for the motherboard) on both GPUs at the same time.

Would it be insane to go for a threadripper setup (for example 2920x or something) (price wise), but get all the PCIe lanes (& other features, and possibly whatever issues there might be with threadripper :wink: ), or would the performance difference be neglible to non existent on a 2700x setup?

Budget wise, I can deal with up to a threadripper 2950X, some 32Gb of fast’ish RAM, and any of the threadripper motherboards.

Also, would threadrippers “game” mode be of any use for such a setup, or is it just for a full on Windows only installation?

Additional: Will also be streaming on twitch some of the games played, both on Linux and Windows, through OBS on whichever OS I play on.

Any constructive or informational feedback is appreciated.

Regards.

2

If you want to use 2 NVMe drives, then go with Threadripper. The 1920x with the AsRock Taichi or Professional Gaming are good choices and pretty cheap relative to 2nd gen Threadripper.

Ryzen has 20 Lanes, 16 for PCIe slots and 4 for an NVMe drive, by design. To run 2 GPUs with 2 NVMe drives, you’d need a Threadripper system. You could run 2 GPUs at x8 speeds, an NVMe SSD and SATA SSD, however. I know you don’t really want to run them at x8 speed, but I can’t think of a situation in gaming where the FPS was impacted by using the GPU at x8 speed on PCI 3.0 (it’s gonna be basically the same speed as your current setup with PCIe 2.0 16x in terms of bandwidth). x8 speed has some slight implications for SLi setups, but not for single GPU scenarios

1 Like
3

I’m usual cost efficiency oriented so I’d do B450 + 2700x. Run the R9 290x in the first slot at x16, run the HD7790 in the second full length slot at x4 and use that as your boot gpu (https://www.redhat.com/archives/vfio-users/2016-August/msg00025.html). Run one NVME for boot, and a SATA ssd for passthrough.

4

It really depends on the budget for the most part.
Because you could also concider to go intel.
Something like the i9-9900K 8C / 16T and a Z390 board.
One of the benefits is that those intel cpu´s come with an igpu.
So you could use that for the host, only needing one gpu then.
Also intel generaly temps to work better with virtualization and hardware passtrough,
then AMD does currentlly.
Because iommu groupings on intel boards are better sometimes.
Allthough it does seems to work okayish on Ryzen, but on TR it can be a bit fidlly.

X299 might also be an option to concider aswell.

TR generally isnt that great for gaming.
On the AMD side for things Ryzen 2xxx is better for gaming.
But yeah like i said it kinda depends on your budget.

5

Pretty sure you’ll need to run x8/x8, as the way Ryzen PCIe lanes are counted, the 20 includes the 4 needed for the chipset(?). I.e., you don’t get 16+4 for the full length slots. You get x8/x8 for the full length and the other smaller chipset slots share the x4 allocated for the chipset.

That said, x8/x8 is fine and will not have any impact on the VAST majority of cards.

if you DO go threadripper, and are planning to do VFIO, you can probably negate the threadripper die to die penalty where game threads bounce around across dies (give only ONE die to the VM, or even 1 CCX only) by allocating only the specific cores within either a CCX or individual die to your VFIO virtual machine. So the TR vs Ryzen gaming concerns would be moot…

6

Nah, a lot of B450 motherboards like the Asrock Pro 4 have x16/x4 layouts for the two full length pcie slots.

7

Doesn’t he need a x370/x470 for 2 gpus?
As far as I remember, the b350/b450 only supports 1 gpu?

8

Nope, B350/450 support crossfire if i’m not mistaken.

SLI maybe not.

edit:
and in either case, no special support is required to run 2 GPUs independently of one another, so long as you have the slots they will fit in… performance may vary of course. Chipset support is only a thing if you’re trying to do crossfire or SLI. Or IOMMU.

9

Oh yeah, it was only sli/crossfire that was limited to X370/470.

But yeah, given OP wants two gpu’s, two nvme’s and maybe additional pci cards, he should look either threadripper or intel’s x299.

1 Like
10

It’s a tough choice.

I made it and went for X470 personally. I did agonise over it for months. I don’t regret my choice, but YMMV.

Sure, i’d like 2x M.2 NVME drives running at full speed, but the reality is i already have a bunch of SATA SSDs and i can get by with 1x NVME (i do have a second slot free though, albeit through chipset).

2x 8 PCIe 3.0 is enough for video and i have a USB add in card for VFIO.

So sure, i’m currently maxxed out but the cost is significantly less than threadripper. CPU cost mainly…

I’d think very carefully as to whether you are REALLY going to use those extra lanes, cores and m.2 slot because if you’re not, you’re spending quite a bit more money for nothing.

The money i saved can go on say, a bigger m.2 NVME drive rather than running 2 of them… or any one of a number of other things. Like a ryzen 2 upgrade next year, etc.

Not saying threadripper doesn’t have its place. Just be realistic with your use case because its a fair whack more dollars for something that may be outperformed by a 2700X in a lot of less-threaded workloads.

@Bollox : have you considered just running the windows gaming VM off cheaper/larger SATA SSD? Because if you are willing to do so, you don’t need threadripper and can save a bunch of money that you could then spend on a shit tonne more SATA SSD storage. Gaming doesn’t really demand NVME, and as someone currently running both NVME and SATA SSDs (850 EVO, 860 EVO and 970 EVO), for games and stuff like that you really are hard pressed to see ANY difference. Games aren’t generally IO bound. A lot of the loading time is stuff other than raw throughput from storage.

Carefully consider what additional PCIe cards you are likely to run(?), and whether they would even fit.

In my Define R6 case for example with 2x GPUs and a USB add-in card i’m pretty much out of space. If i was on threadripper i could put the two GPUs side by side with single-slot water blocks (to reclaim some physical space), but that’s also additional expense for a custom loop. With 2+ slot GPUs running air cooling you are likely to run out of physical space to install stuff almost as fast as PCIe lanes. No point having the lanes if you can’t physically fit stuff in them…

2c.

edit(lol, many):
“Game mode” with threadripper disables a die.

if you’re doing VFIO there is no need for it, just allocate the cores appropriately to the Windows VM in KVM’s configuration and you can prevent any cross-die thread bouncing that way. i.e., allocate the cores to Windows that are all in the same die or all in the same CCX even.

Game mode is a physical windows install only thing.

1 Like
11

Thanks for all the advices and feedback.

Intel is not, and will not be an option, since I strongly disagree with their R&D, innovation (or lack thereof), marketing and ethics towards the competition.
A note on the I9-9900k: It appears to me, that it only has 16 PCIe lanes, is “only” 8cores/16 threads, supports only DDR4-2666 RAM.
Compared to a comparable threadripper (even just price alone), this just seems…like a bad choice. Even more so against a much cheaper 2700X.

I have not had any troubles with an entirely AMD system and VFIO/IOMMU (other than the performance before the NPT linux kernel patch). Groupings have been perfect for me, and there have been little to no issues with passthrough.
My own experiences with virtualization on desktop systems, have been generally better with AMD, than Intel (Not counting xeons!), but your experiences might be different.

It’s tempting to go for the much cheaper 2700X setup, but I don’t really like the x8/x8 PCIE3.0 lanes for the GPUs, as we do not know what is in store for future cards (or my own unknown future use cases).
ATM I’m only gaming in 1080p, but who knows if that might change in the future.

SLI/Crossfire won’t be an issue, as it will only be one GPU per OS.

I have thought about using 1 normal SATA SSD instead of one nvme drive, and I don’t yet know how speeds are in everyday use with nvme, as I only have normal boring SATA SSD drives currently. This is definitely something I would have to look into.
Also, by nvme, I might mean m.2. Not sure about that. Just shows my lack of knowledge on this part :smile:

Good point on physical space, but the case it will go into is big and plentiful. Probably the motherboard will be the biggest physical space constraint, especially with a slot limited 370/470 board.
I could be throwing in a capture card for console/old computer capture, and possibly who knows.

Good to know about the die/CCX/game mode options.
And yes, I will be using VFIO, just like i currently am with my FX/990FX setup.

I will surely end up looking at a ton of articles/reviews/forum posts about all these aspects, because it IS a tough choice that I will think about over a long time.

Thanks again for your thoughts. I’m not sure if I’m any closer to a winner :confused:

Regards.

2 Likes
12

For me (after using it for general stuff) the big appeal of M.2/NVME is less cables.

For regular every day use, the difference between my 970 and 860 is pretty indistinguishable.

Sure, if you’re doing a heavily IO bound workload you’ll notice, but in general day to day stuff… same.

Put it this way - i haven’t been blown away by any magical improvement from the 970, having previously been using the 850 evo and 860 evo for primary drives previously.

In fact, my windows install is still on the 850 as primary. It doesn’t “feel” significantly slower if at all vs. the 970 under linux.

Benchmarks will show it for sure, as will heavy video editing, etc. But general “stuff” (including gaming)… for bulk storage i’d still recommend cheaper SATA SSDs for that. They’re still plenty fast.

Also… on x8 PCIe for video. I’ve noticed no difference with Vega 64 between x8 and x16. If you’re hitting the PCIe bus extensively it is because you don’t have enough VRAM. Solution to that is to buy enough VRAM on the card.

By the time PCIe 8x is insufficient you’ll likely want a new motherboard with larger memory support, PCIe 4.0, DDR5, etc. i suspect.

Again, maybe some edge case benchmark might be able to show a difference by saturating the PCIe bus to the card. But real world… not noticeable. And it means you save a lot of money to put towards other stuff (that you can get a benefit from today - e.g., more SSD storage), or the next round of hardware WHEN you need it.

1 Like
13

Differences between NVMe and SATA for general or light useage is minimal. In general, the differences between NVMe and SATA SSDs for most use cases is minimal, to the point where if you stuck them side by side you’d generally not notice, unless you ran specific tests. Unless you plan on doing something absurdly IO heavy, SATA SSDs are just fine.

Both Ryzen and Threadripper has all their Motherboard chipset items except the TPM chip and 1-2 USB ports (for the USB key) in the same IOMMU group (these are seperated presumably for security reasons). Threadripper will be necessary if you want to passthrough more than a GPU and the TPM chip’s USB ports to the Windows VM due to this. Otherwise, Ryzen is more than sufficient.

Also, x8 PCIe 3.0 speed is not going to bottleneck either of your GPUs. It would only come into play if you were doing SLi or Crossfire, which isn’t what you’re doing, so don’t worry about it.

2 Likes
14

Hey,
did you build your rig yet?
i am in a similar position and am looking for a build guide.
I am now on an old Xeon v3 and its time to upgrade :smiley:
hopefully you can share your build with us!

regards
n0s3

15

I’ve made and been using a similar build as my daily driver for the last month. If you want, you can check it out here

For my build post, you could put a 1x GPU for the host in one of the 1x slots, and the gaming VM’s GPU in the top slot

16

Pretty similar to mine.

X470 taichi
2700X
1x Vega 64
1x RX480 (swapped my second vega out to GF’s PC to make VFIO easier)
32 GB G.skill DDR4-3000
1x Samsung Evo 970 1 TB, 2x Samsung Evo 860 1 TB, 1x Samsung Evo 850 500 gb
1x USB add-in card

still need to get it working. lol.

17

I will mention that for passthrough, the USB ports tied to the TPM chip are the ones you’re going to be able to passthrough easily. For my setup, unfortunately, this means if the VM restarts the USB ports don’t get released properly so they can be reset, and thus causes a perpetual hang on the VM until the host reboots. I’m sure those running Linux with KVM and QEMU will be able to know more about why this is and possibly how to fix it, but I’m just guessing its motherboard chipset shenanigans

1 Like
18

I am making an attempt at this with Threadripper in the build log you can find here:

You can find my trials and tribulations there :rofl: Also, it sounds like you have more experience with Linux and KVM than I, so you could likely provide some valuable help to me as the build progresses forward!

Cheers!

19

I still haven’t build (or picked the parts yet) the rig yet. Still uhmming and ahwing over what it’ll be.
Currently I’m leaning towards TR4 in a Taichi X399, but still looking into memory (good speed 3200+ MHz and 64gigs, but seems not possible atm) and a possible AIO cooler.

Edit: Edited for multiple typos.