Opinions on a Linux + Software Development + Gaming with a Win10 VM and VFIO build?

Hardware Build a PC
1

Hey all! Thanks for taking the time, and also any advice.

Background

I haven’t built a PC in a really long time, so I hope I got most of this right. I feel like all my work needs checking by someone more knowledgeable.

What this PC is For

Software development in Linux. I run lots of containers and compile lots of things in lots of languages. I am dabbling in machine learning.

I’m a huge gamer, but I have a family so I’m usually not up on all the latest trends. It would be nice to be able to play VR games. I’d specifically like to play Cyberpunk 2077 at higher quality levels when it gets released. I want to run games either in Linux or using VFIO and a Win10 VM. Dual booting is not an option because I lose so much context when I reboot, and sometimes I just want to load a game for 15m or so. I really want to make sure this build works with VFIO!

The Build

I am swapping in an old GeForce GTX 660 while I wait for AMD and NVIDIA to release their new cards. I’m planning on an AMD card for Linux, and maybe an NVIDIA card for Win10? I know NVIDIA’s driver complains. How hard is that to work around?

Are there any other parts I should wait on due to upcoming releases?

pcpartpicker.com/list/sQgsNq

Type Item Price
CPU AMD Threadripper 3970X 3.7 GHz 32-Core Processor $1899.99 @ Amazon
CPU Cooler Noctua NH-U14S TR4-SP3 82.52 CFM CPU Cooler $79.90 @ Amazon
Motherboard MSI Creator TRX40 EATX sTRX4 Motherboard $669.99 @ Amazon
Memory G.Skill Ripjaws V 64 GB (4 x 16 GB) DDR4-3600 CL16 Memory $289.99 @ Newegg
Storage Samsung 970 Evo 500 GB M.2-2280 NVME Solid State Drive $99.99 @ Amazon
Storage Samsung 970 Evo 1 TB M.2-2280 NVME Solid State Drive $178.75 @ Amazon
Case Fractal Design Define 7 XL Light ATX Full Tower Case $209.99 @ Amazon
Power Supply Corsair AX 1000 W 80+ Titanium Certified Fully Modular ATX Power Supply $284.99 @ Corsair
Case Fan be quiet! Pure Wings 2 140 PWM 61.2 CFM 140 mm Fan $12.90 @ Amazon
Prices include shipping, taxes, rebates, and discounts
Total $3726.49
Generated by PCPartPicker 2020-06-16 14:57 EDT-0400
1 Like
2

IDK why not just have a work pc and a game pc? Then each one can abe tuned for what you want. Plus, still have a pc if something breaks on one.

3

That’s a great question. Generally speaking, I want my work PC to be as powerful as I can afford, but I also want my gaming PC to be as powerful as I can afford. VFIO is a great way for me to build one powerful machine and take advantage of it in different ways depending on what I’m doing at the moment. I’ve done the 2 separate build path, and it hasn’t really worked out for me. One of the machines always lags behind and I can’t get around to updating it.

As for redundancy, no worries there. Being a software engineer, I have several computers lying around that will do in a pinch.

2 Likes
4

You should be able to post links now, I bumped your user level. I also added the link back into the post.

Oh, this could be interesting to get working in a VM. Part of it is that VR can be really picky about USB chipsets and bandwidth, not only for the headset but also for the base stations, Also, there are USB controllers that do not support resetting correctly, just like AMD gpus.

At this point, it is pretty easy. Who knows, it may be patched to be impossible to work around at any point. Keep at least an old driver version or two around.

Really the best option at the moment, what with the reset bugs with AMD.

I would check to see if people have used this motherboard before for VFIO and if it has good IOMMU groups.

2 Likes
5

EDIT:

Thanks!

Derp. I didn’t even think about how this might be different for VR. I don’t even own any VR gear yet. I just thought that since I was building a powerful machine, it may as well support VR for when I do check it out. I would be comfortable punting on this since I’m not very serious about it. I’m way more interested in getting vanilla games working reliably. I have an entire backlog to chew through after all!

Great suggestion. I’ll go and do that and report back! Are there any boards that are known to be good threadripper boards and have good IOMMU groups? I’d rather taken the well-worn path if possible.

6

It is posible, its just another level of hoops to jump through.

Also, you might want to buy a Nvidia GPU with the virtuallink USB port.

I’m not up to date with that, sorry.

7

Well. I think I began designing this system without a full appreciation for the research that needs to go into selecting a motherboard specifically for VFIO. I’ve done my best to scour the internet, and the only real discussion I could find regarding this motherboard was on reddit. Amazingly, this person even has the same Geforce GTX 660 in place I’m planning to use (initially). It appears they have gotten it working.

Other than that, I searched around for TRX40 VFIO recommendations, and the Gigabyte boards kept popping up as pretty well supported (including this post, thank you @wittycurie!). That makes me wonder if I should instead be looking into the Gigabyte Auros Xtreme.

I’m struggling to put all of this together to make a decision. Trying to select a good motherboard is difficult enough after not having built a machine in so long, but adding in the VFIO requirements makes things extra complicated.

At least I think it’s down to the Auros Xtreme vs. the Creator. Opinions welcome!

8

@wendell also briefly displays the IOMMU groups in his review (link jumps to relevant time). I am new to VFIO, but if I understand correctly, “good IOMMU” is when devices have their own IOMMU group, and if I pause the video, I believe this is the case except for the USB controllers he highlights. Someone please tell me if I’ve misinterpreted this.

I think I’m narrowing in on the Creator. Since I haven’t done this in awhile, I’m trying to follow in the footsteps of people who know what they’re doing, and this motherboard seems safer since @wendell has done a couple builds with this board, and the reddit post I linked earlier suggests this is working.

Still open to opinions though :slight_smile:

9

Just because a USB controller is in a separate IOMMU group does not mean that it will work well for passthrough.

1 Like
10

I would second this. While PCI passthrough is fun to use and configure, it would be less of a hassle to have 2 purpose built machines. You could also look into ESXi which would make each system separate and more fault tolerant. The only drawback is the host can not reclaim resources.

2 Likes
11

Hey, thanks for weighing in. It really helps me to hear a variety of takes.

For my personal situation, I disagree. This is my current setup. Now both machines are out of date, and I have some money to upgrade one. Which one wins? I don’t want to have to choose.

That’s a good suggestion, thanks! But if I’m going to learn and tinker and support something, it’s going to be Linux. All my research suggests that if I get this working, I can keep it working indefinitely, and then repeat the same type of build going forward. Linux virtualization technology is only ever going to improve.

12

I am no expert on VFIO with Threadripper and VFIO in general but i’ve managed to build mine and it work really well. If you can pass-through a GPU and a usb pci card, its pretty solid setup. I mean thats what i run right now to be able to game on Tarkov and Division 2 ( damn anti-cheat) and i can say its even better than when i ran a dualboot. Your system will defenitly have enough memory/cpu cores to do that soooo… why not? Defenitly do your research for the iommu groups of motherboard tho, Trx40 mobos arent cheap, you dont want the wrong one and suffer a loss…

2 Likes
13

Ok, I think you could reconsider a few things.

First off, what do you need a Threadripper for? I understand it’s a cool, shiny toy and 64 threads is more than anyone would ever realistically need. But I find it hard to justify that much money for only a VFIO gaming/dev machine. A Ryzen 5 3950X costs a third of that price for half the cores, and there are some really neat x570 motherboards that support VFIO, slashing your up-front costs by at least 25%, like the Asus Crosshair VIII Hero or Aourus x570 xTreme.

Of course if you must have 8 RAM slots then the threadripper is awesome, but I see it as overkill for your use case - might be wise to spend those money on GPU / VR instead.

Second thing to consider is your screen - if you want a 4k experience you will need a 4k screen to go with that power, it will be a lot of wasted potential otherwise. A Radeon 5600XT could handle 99% of everything you throw at a 1080p screen at an acceptable framerate, so make sure to leave some room at the receiving end, be that VR or 4k screens.

The storage is top-of-the-line - but chances are you will not bottleneck those 970 Evo drives like, ever for your use case, so here some money could be saved as well.

Finally, the power supply is overkill too - A 700W platinum supply is all you should ever need these days with plenty of room to spare.

All in all I’d say you could shave around $1500-$2000 from that build without losing much extra. Your money, your system, your choice though. :slight_smile:

Software wise, I think your plan is sound, but I also think you might not need that VFIO passthrough. I thought I did, turns out Proton is good enough for pretty much all my needs. By all means though, explore it!

Right now, since you do ML, you will need a Cuda card, that means at least one nVidia. AMD cards tend to work better with Linux once support hits mainline kernels and distros, but are a bit rough those first few months…

Have a pleasant evening and happy hunting!

1 Like
14

I know the 3950X is good, but gaming while compiling code quickly sounds a bit out of its league. With a TR, OP can compile code with a significant amount of threads while gaming on the side with 8 or 16 threads.

15

I admit that some of it is because I’m a nerd, and I like having that many cores. But I will use them.

I am regularly compiling multiple things at once which will consume all cores available while at the same time running multiple services and VMs in the background. Having this many cores will regularly save me a non-trivial amount of time.

The 500GB one is for Windows and games, and I don’t really care about performance there beyond wanting games to load quickly. The 1TB drive is for Linux, and is where I’ll be doing the heavy lifting: regularly cloning many code repos for building, compiling and importing container images, vm images, chewing through large data sets, scientific computing, etc., etc. I don’t have your confidence that I won’t be utilizing the throughput.

Really? If I toss a few modern video cards into pcpartspicker (say, 2x AMD Radeon RX 5700 XT at ~250W each), the estimate is 973W. I was going to go even higher just to give me some headroom. Have I completely missed something?

Right now the plan is an NVIDIA GPU for the Win10 guest (my impression is that NVIDIA enjoys better support from games). For the host, I’m really torn because AMD has better support, and I want to support them for taking Linux seriously, but their ROCm platform doesn’t seem to be there yet. I’ll probably sit on this decision for awhile whereas for the Win10 GPU I’ll just buy whatever is reasonable for gaming.

Hey, thanks a lot for your input, and for trying to save me some money. You’ve made me think twice about some things which I really appreciate. I hope you have a pleasant evening as well.

16

Pretty sure you want > 700W power supply, yeah. Specially not knowing how RTX30xx will be for power. Trying to spec down to the last watt is a good way to end up wasting money. Fast M.2 is also good for loading games and also for quickly loading lots of files like compiling so I think thats a good pick, too.

EVGA has some real good deals on RTX cards in their B-Stock store. Dad got me a 2080Ti for rendering for real good price and has warranty etc.

Our Linux workstation has a GTX1070 and that RTX2080TI (it replaces 2x GTX1080Ti) and has a 1200W PS. Rendering it uses most of it (less now with just the 2080Ti). I bet ML will too?

Stuff like rendering and ML work the system hard for longer times, so I think having more cusion on PS and cooling is a good idea; thats how my dad builds ours.

17

Just making sure you’re not needlessly throwing away large sums of money because it was the “best” money could buy. If you truly feel you need it, I see no problem with getting one, it is your money. :slight_smile:

From my own testing with an NVMe drive, it does not notably increase compile speeds over my SATA drive. I have seen benchmarks that draw the same conclusion. In my case, the bottleneck is that CMake sometimes only single-threads a single compile, leaving the other cores idle while that packet is built (work heavily with Yocto and compiling Linux distro packages). I do notice a big difference with tarballs and large SQL databases though.

I believe an intel 660p or Kingston A2000 could save you a few bucks with negligible performance dips, and as for reliability, if you write to these drives constantly 24/7 in 5 years they will wear down, not a very likely scenario. NVMe is the way to go though, and there is nothing wrong with the samsung either apart from Bang For the Buck.

Sorry, I completely brainfarted here. You are pretty much spot on.

Threadripper platform with motherboard will draw around 300W, and a 2080 Ti will draw 280W. Together with an extra card that is around 850W, but you also want some headroom, so the 1000W makes sense.

I’m too used to work with the AM4 platform that draws a mere 150W and I was assuming lower-wattage cards as well, so my total was around the 550-600W range. My bad.

Yeah, pretty solid reasoning. I don’t see why you wouldn’t be able to use the Nvidia card for CUDA as well though, if it’s a really heavy task you’ll probably offload it to Amazon or something anyway. If you can go AMD, it will give you the better experience in the long run.

18

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.