Dual Xeon Workstation retirement... Options

Hardware Build a PC
1

So its been a long while since ive done a new build for myself… and after throwing together a cheap ryzen build for a friend, its made me realise how much performance you can get out of these modern CPUs!

So this is my current setup:
ASRock EP2C602-4L/D16
Dual Xeon 2670
64GB DDR3
8x 3.5" SATA HDDs
1x 2.5" SSD
2x NVMe SSDs (for VMs)
RTX 2080
OS: Unraid
VM1 : W10, 16GB DDR3, NVME(1) passthrough, 2080 Pasthrough
VM2: Ubuntu, 16GB DDR3, NVME(2) passthrough, 2080 Passthrough
Docker - lots. (including plex, which does quite a lot of transcoding)

Im torn between keeping a single system, and using a VM for my workstation needs, or to split of the NAS (Unraid) functionality onto dedicated hardware. I know full well my CPU is a bottleneck for my 2080, and thats even before i throw in the overhead of a VM!
NAS OS wise, im very happy with unraid. Does what it says on the tin.
Only thing that frustrates me is the lack of control available when it comes to VMs. You’re tied in their OS update schedule for any module updates, which with the pace of development with QEMU gets quite frustrating at times!

So my initial plan is to have 2 systems which looks a little like this:

NAS:
Mobo: ASRockRack X470D4U
CPU: Ryzen 3\5\7?
Memory: 32GB DDR4

I really like the look of that motherboard… means I dont need a cheap-o graphics card eating up a PCIe slot, enough SATA ports so I dont need a PCIe SATA controller, and NVMe too.

Workstation Requirements:
Id still like to play with VMs. So something with nice IOMMU groups. All NVMe slots getting their own IOMMU group would be ideal, which ive read isnt always the case.
Something which isnt going to be a bottleneck for my 2080.

Threadripper puts me off slightly as I had many a headache over poor performance in VMs due to NUMA and PCIe lane assignments. Although, thats maninly because of the PCIe slot placing and assignment of my old motherboard, so I dont think it would be as much an issue here.

So I guess my questions are…
Has anyone upgraded from a similar dual xeon system to ryzen\threadripper? Anything you would have done differently with hindsight?
For a workstation, would you lean towards a 2700x or a 2920x?
Or do I stick with a single system and go threadripper 2950X…

2 Likes
2

For the NAS, any Ryzen chips will be okay. R5 and above are probably overkill, to be honest.

I’d go with the 2200G to take advantage of hardware transcoding with Plex (assuming you have a Plex Pass?). If you’re not doing more than 6+ transcodes at one time, that should be plenty of power. The only problem that could come about is possible ECC compatibility problems with Raven Ridge. From what I’ve read before, it’ll run, but ECC might not be enabled because reasons(?)

2 Likes
3

Thank you for your reply!
I’ll be honest, I didn’t even think about the APUs.
So my next question…
If I did go the APU route, would I be able to “pass through” the iGPU to a docker container running Plex? (I do have Plex pass) I’ve never really thought about what hardware gets presented to docker containers (because I’ve never had to before!)

Going the APU route would definitely save some £ to put towards the workstation instead!

4

Think @SgtAwesomesauce would be the one to answer that. I believe Docker containers just run on bare metal, so it should be fine? The hardware isn’t isolated like it is with a VM.

5

I’m not entirely sure how the whole exposing containers to GPUs goes, but you shouldn’t have an issue with it, since it’s more of a namespace/file permissions issue, rather than an IOMMU thing. You’re running a container on the same kernel as the Host OS, so when you look at it that way, you’ve already got your drivers loaded, you just need to expose the endpoints to the container, if that makes sense.

6

Only true CPU that won’t bottleneck a 2080: i7-9800X with a extremely mild overclock, like 4.4Ghz all core. (if you’re pinning less cores, you might need to OC further.)

A good board is the Asrock X299 Taichi XE. It addresses the early concerns with X299 boards with a good PCIe layout for passthrough/unRAID.

Do not get a 9900K because there are still issues with Hardware ACS with the lanes directly from the CPU.

Don’t wait for Threadripper 2 because I guarantee you there will be many day one issues despite favorable IPC.

7

That sounds unlikely to me. Could you explain?

Personally I like separate machines for everything, not a big VM guy. But I’m also only running linux these days. I had a couple dual 2670 builds and I gotta say, a 2700X is running circles around them.

If network speed is important to you, you might wanna take a look at the X470 Taichi Ultimate. You’ll have to check IOMMU yourself though. (Not a VM guy, like I said.)

8

Good single core IPC means if he pins less cores, the effect of having less cores is balanced by better IPC per core.

9

I don’t think there is a percievable difference between a 9800X and a 2700X coupled with a 2080, to be honest. What is that based on?

1 Like
10

If you’re using a dual GPU setup (which the 2700X doesn’t have a iGPU) you only get x8 lanes per GPU in unRAID/VFIO. IOMMU through the chipset is also has no hardware ACS, so dual NIC boards still need Kernel ACS to separate the NICs, USB, etc on the chipset.

The 9800X wised up and gave people the same amount of PCI-E lanes across all Skylake-X refresh CPUs. Leaves room for NICs, USB controllers, and NVMe drives without going through the chipset, and still has proper ACS on the chipset lanes too.

Mesh is also I believe faster than Infinity Fabric for stuff like Looking Glass.

11

I think the route i’ll be taking is to build a dedicated NAS (UnRaid) box with a Ryzen 2400G. For the sake of plex transcoding, docker containers to support plex, i think it’ll be more than enough.

Plus at the moment, im only using my workstation VM in the evenings, so im powering an RTX2080 24/7 for no reason at the moment. Seems sensible to split off the 24/7 stuff from my workstation which i’ll only be using in the evenings.
The workstation can be a little more exciting.

So based on that logic, im about to pull the trigger on:

Ryzen 5 2400G
ASRock B450M PRO4
16GB DDR4
Fractal Design 804
Corsair CXM 450W
£410 (which seems very cheap!)

12

For NAS stuff, double check ECC support with the mobo. The 2400G doesn’t support ECC, so if you did want to go the ECC route, a 2600 and a GT 710 might be better for the NAS.

13

Replying here because it seems like its the most recent reply about gpu docker “passthrough”. I’m newish here so I don’t know if not tagging @billington.mark will present them a notification as well as @SgtAwesomesauce.

From what I know about docker “passthrough” when it comes to the gpu you should only have to add “–device /dev/dri” to your run command and ensure that the user running the process in the container is in the video group and the video group has the same gid as the video group on the host (or whatever group owns /dev/dri).

Check out Jessie Frazelle’s talks/github for more information on docker and gpu (she mostly is using it to pass through applications to the host Xserver, but I think it’s applicable to plex). Some of the plex repos on docker hub mention /dev/dri as well.

The only thing that I don’t think is supported yet is if you were running your containers in kubernetes there isn’t a way to tell the pod to pass a device path (as far as I’ve found yet).

2 Likes
14

Thank you for that!
Hardware arrives tomorrow so I’ll report back on the docker situation, and what I had to do to get it working.

As for ECC, I’m not too fussed as the data has dual parity redundancy.

2 Likes
15

Yep, we get notifications for mentions.

ECC is pretty unnecessary in general.

16

I meant notified without tagging. I figured you would be notified regardless since it was a reply to you but I wasn’t sure if anyone who’s posted in the thread or the OP would be if they weren’t specifically tagged.

Anyway, glad to know this has been sorted :slight_smile:

1 Like
17

So all the kit arrived, and the build booted without a hiccup.

BUT… it seems unraid doesnt seem include the APU driver for the 2400G in the kernel, so im not able to add the device to the plex docker.
Worth noting that it can actually transcode a 4k mkv to 720p on the fly (which my dual xeon setup couldnt!).

(forgive my noob linux knowledge if i say anything stupid from this point!)

From what I understand, as long as the host loads a driver for a device, you can add a device mapping to a docker container with something like ‘-device:/dev/dri:/dev/dri’.
It seems that unraid can see the device, but its not assigning a kenel driver to it:

‘lspci -v’ result:
38:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Raven Ridge [Radeon Vega Series / Radeon Vega Mobile Series] (rev c6) (prog-if 00 [VGA controller])
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Raven Ridge [Radeon Vega Series / Radeon Vega Mobile Series]
Flags: bus master, fast devsel, latency 0, IRQ 11
Memory at e0000000 (64-bit, prefetchable) [size=256M]
Memory at f0000000 (64-bit, prefetchable) [size=2M]
I/O ports at e000 [size=256]
Memory at fe300000 (32-bit, non-prefetchable) [size=512K]
[virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [64] Express Legacy Endpoint, MSI 00
Capabilities: [a0] MSI: Enable- Count=1/4 Maskable- 64bit+
Capabilities: [c0] MSI-X: Enable- Count=3 Masked-
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [200] Resizable BAR <?>
Capabilities: [270] Secondary PCI Express <?>
Capabilities: [2a0] Access Control Services
Capabilities: [2b0] Address Translation Service (ATS)
Capabilities: [2c0] Page Request Interface (PRI)
Capabilities: [2d0] Process Address Space ID (PASID)
Capabilities: [320] Latency Tolerance Reporting

Also, no /dev/dri folder exists on the host either, so as far as i can tell, no driver is loading.
From what ive seen, hardware transcoding does work in plex for linux, albeit undocumented. So im hoping it should just be a case of it just working if i manage to get the driver enabled on the host.

So questions…
How can i tell if there’s a modprobe kernel module i can enable for the APU driver? (This seems to be a thing on unraid for the intel iGPU).
Feature request wise for the UnRaid devs… Am I just asking for the Ryzen APU driver to be added as a kernel module?

18

Ive requested the drivers are added to unraid.
If there are any UnRaid users here who want to +1 the request, here’s the link: