HELP PLEASE - Motherboard + CPU for Home Server: Proxmox (host) + TrueNAS (guest) and other VMs: best choice for ECC, 4444, IOMMU, PCI lanes?

Hardware Hub Motherboards
, ,
1

I want to build a beefy server for media streaming, NVR from security cameras (yet to buy the cameras - so thinking in advance), TrueNAS, Home Assistant, and a bunch of coding and self-hosting stuff.

I followed advice from Wendel in this video:

and bought 4x: 118GB Optane drives to use as a special VDev.

I was originally leaning towards Intel due to QuickSync, but I think AMD will be a better choice for me due to:

  • normal cores (vs e-cores)
  • what seems like better energy efficiency in Ryzens (at least ones like 7900)
  • 4x4x4x4x bifurcation which is essential for the 4x Optane array Wendel-style, and 4444 bifurcation is absent from LGA1700 platform completely, as far as I can tell.

Since I will need 2x simultaneous tarnscodes/streams at most, I think lack of QuickSync will be fine: even if I fail to passthrough and use Ryzen’s iGPU, I think software transcode will do just fine for occasional 2x max simultaneous streaming, even if 4k → 1080p (please correct me if I am wrong).

ECC is a must - I have important files, and I want to do this project right. ECC is strongly recommended by the TrueNAS community, and Ryzen seems to be better in this regard due to more widespread full ECC support, as opposed to Intel, where it’s available only on W680 and W790 boards.

Since I want to be able to run everything on Proxmox, good IOMMU groupings are also essential.

My research shows that x670 motherboards are suboptimal in this regard due to them being essentially 2 daisy-chained B650, where second chipset tends to not have good IOMMU groupings whatsoever. Correct me if I am wrong please - I am basing my conclusions on this thread:
AM5 IOMMU VFIO - Best Motherboard - Hardware / Motherboards - Level1Techs Forums

So:

  1. What would you guys recommend for the CPU? I am thinking Ryzen 7900 (non-x) for good perf VS energy efficiency.

Conversely, If I find a better deal on 7900x specifically, is it possible to run it in eco mode under Proxmox/TrueNAS, or is it only supported on Windows?

  1. What would you guys recommend for a motherboard?

Need these:

  • ECC
  • Good IOMMU groupings
  • 4x4x4x4x bifurcation
  • Enough PCI slots/lanes for:

a. GPU (for NVR encoding of camera streams)
b. SATA HBA card to passthrough all HDDs to TrueNAS
c. PCI-to-M.2 card for a 4x Optane array ( 2x mirrors of 2) which is also what I need 4x4x4x4x bifurcation for.
d. 10Gbe NIC if the motherboard has a slow built-in NIC or a 10G built-in NIC with non-Intel chipset (it seems, from my research, that both Realtek and Broadcom chipsets are hated for 10G networking)

Is it possible to get these many lanes ^^ ? I am a little confused on what the number of lanes is for Ryzens, and how PCI slots compete with m.2 and integrated stuff.

E.g., how do internal NICs compete for lanes with PCI slots? If a MoBo has a shitty built-in Realtek NIC, does it mean that there are a few lanes wasted on that?

I would appreciate any and all help with this, especially the IOMMU groupings. I did a ton of research, but I am still fuzzy on that. I wish there was a simple-to-understand list of how MoBos rank in this regard. I’ve seen pastes of IOMMU data from various brands, but I am very new to this, and I have hard time interpreting whether it means the MoBo is a good choice or not.

Thank you in advance.

P.S. I was also hoping to maybe get into local LLM hobby and play around with FOSS ML/AI projects, but it looks like I would need at least a couple GPUs to run decently-sized models, and that would push me into a Threadripper territory due to PCI lane requirements, which I don’t think would be wise for me to spend money on (as well as several GPUs, although used 24GB 3090s are said to be perfect for this and came down in price a lot) considering that I am new to this and need to learn a lot.

2

ASUS ROG Strix B650E-E Gaming WIFI + 7900x Black Friday sale.

For anyone else who doesn’t need x8/x8 or similar PCIe bifurcation the ASUS ROG Strix B650E-F Gaming WIFI is the absolute best value B650(E) board period.

You get the 5.0 M.2, 5.0 PCIe x16 slot and 80Amp Mosfets way under $300

1 Like
3

Could you please link me to this sale?

Also, would it have 4x4x4x4x bifurcation, ECC, and necessary number of lanes and slots for all my stuff?

Thank you!

4

ECO mode is an OS agnostic, BIOS/UEFI setting.
Sounds like you want workstation/server PCIe count. If you’re willing to sacrifice a bit on power efficiency, you can save A LOT going with something older. Can’t speak to anything specific for IOMMU not being set up stupidly, except that a server board is likely to keep that in mind from the start. 1st/2nd gen EPYC could be good.
last I checked SuperMicro’s site (while looking at much older stuff) they specified which boards supported 4x4x4x4 bifurcation in a big list.

5

Don’t know the price; I’m sure it’s a lot. EDIT: Looks like $500-600 on eBay. Some of the early Epyc CPUs are already really cheap.
Has dual 10Gig

image
image1634Ă—919 158 KB

image
image738Ă—515 67.3 KB

https://www.supermicro.com/en/products/motherboard/h12ssl-nt
Looks like all slots support bifurcation down to 4 lanes.

6

Thanks, but that MoBo alone is at least $650 according to google, and up to $1000.
I know nothing about Epyc CPUs, but I think even older 7002 are expensive and probably very power hungry. Cost is the reason, like I mentioned, for my not considering Threadripper. Epycs are even higher league, I think.

With unlimited budget it would be easy to satisfy all reqs, for sure. :slight_smile:

7

How about this?
Manual says this one supports same bifurcatiuon. No 10Gig, but you have slots.

image
image1208Ă—749 237 KB

Not the most baller CPU, but… should handle things you care about plenty well.
EDIT:
image
image1634Ă—826 53.6 KB

2 Likes
8

Slight motherboard variation, double the RAM. Double check board manual. No cooler included with this one it seems. Looks like there are a lot of combos available.

image
image1255Ă—681 241 KB

1 Like
9

By the way, any SuperMicro board with a T in the model name, has 10Gig integrated.

10

Lower end server CPUs drop in price much quicker than high end desktop. Also, with Epyc, buy a board+CPU bundle to avoid vendor lock issues, A.K.A. PSB. And those bundles are probably already vendor locked, meaning the CPU will only work in a SuperMicro board. Part of why the bundles are cheaper.

11

Hi @Cat7Cat I wanted to point out that for the nvme m.2 you do not need a mobo that supports bifurcation as long as you purchase a riser card that has the plx chip on it. The plx chip does the lane splitting on the riser card, forgoing the need for bifurcation support in the mobo.
The two caveats to pay attention to when shopping for these sorts of boards are the chip. You want the PLX not PLC. The other gotcha is some of the boards are sata m.2 so make sure your buying an nvme m.2 like the one below. If the mobo you are getting supports bifurcation you can still get a plx based board and just not set the bifurcation in the mobo or you can save a little bit of cost and get a plc based board and let the mobo do the work.
A couple of other considerations about the riser board, lane splitting & the hardware you have called out as the current intended build.
Because these boards will operate best in 4x4x4x4, that means putting them in a full x16 slot. These PLX cards can be used in an x8 slot but you can potentially bandwidth starve the nvme drives if you get close to pci gen 3 x8 bandwidth work loads. Since the PLX cards AND the optane nvme drives are both pci gen 3, you could save yourself some $$ by going previous gen 5900x vs 7900. That would drop you back to ddr4 as well. The last 2 items to remember, skipping Intel cpu means giving up on really good igpu capabilities that AI and surveillance systems such as Blue Iris can use, not to mention the media transcoding. The last item to ponder, you mentioned energy efficiency and indicated AMD would be saving you in the long run. I do not know your intended 7x24 workload but remember that in most non-professional use cases the server is actually not being taxed/used and sits more or less at low use/idle. In these situations the data appears to indicate the e-core design is the real power saver. It is not such a big deal anymore but it has been a known issue that with amd in some cases the c-state caused issues so you had to disable it. Completely killing the power savings.
Anyway, good luck and congrats on whatever you build!
Here is a decent nvme pcie 3x4 w/4 slots that fits the bill.
$115+$28 shipping
https://www.aliexpress.us/item/3256801158360351.html?gatewayAdapt=glo2usa4itemAdapt

1 Like
12

32Cores… do you need that much cores… pcie 3.0…

Go atleast „Rome“ and i gues 8-16 real cores are plenty for what i Think you do.

There are a few epyc Rome cheapos on the eBay and boards to.

The Intel 1700 boards can do 4x4x4x4 and 8x8 but NOT 8x4x4

An 13700K goes over 5GHz on 8 P Cores and those E cores can handle docker quite well to.

Iam Running a Unraid Server this way.

There are a few boards with 4x m.2 …

sure they are connected over pcie 8x over the Chipset but that more than enough for 4x118 Optane

Heck Special vdev Right ? So mirror, aint that 1x Optane Speed ? Even 1xPciex4 m.2 Slot saturates that more than Double

but if i where you :smiley: i would go epyc its cheaper and would just use lesser cores.

13

You need a workstation/server board for that. EPYC Rome or upcoming EPYC Siena are cost effective platforms for premium home server that need a lot of lanes.

Consumer board has 16 lanes for the slots and chipset scraps. Wasting 4x M.2 for Optane on precious PCIe lanes doesn’t help if you want to use consumer board. You need those 16 chipset and CPU lanes wired in M.2 for other stuff like storage. And x16 bifurcation card is a thing for Threadripper or Xeons, not for AM5. AM5 and 1700 have one slot for the GPU and (basically) nothing else. You could plug the GPU into some x4 chipset slot…but don’t expect any high bandwidth, may work for selected use cases.

On-board NIC is usually wired to the chipset, but could be x4 from the CPU as well, dropping a CPU M.2 slot in the process. Depends on the board.

Good boards with 10Gbit NIC and x8/x8 don’t go <500€. So Server boards aren’t that much more and Rome is known for good power efficiency. Seen a forum member who has way lower idle on his Rome than I have on my 5900x.
AsRock Rack boards with 10Gbit and 1700/AM5 go for 700€+ nowadays…board prices are out of control for 2 years now and it won’t get any better.

B650 Strix or B650 ProArt are probably the most cost-effective boards with a homeserver use case feature set. I also like the W680 ProArt with IPMI and ECC, but that one is expensive.
But they all are still limited in having 24/28 lanes. You can build great servers with it, but it’s hard and requires careful planning and compromises.

A good home server has a good board (how much stuff you can plug in) and good memory (how much stuff can you run on it). And everything else is built around it.

1 Like
14

I get the impression he fucked off down a rabbit hole after seeing Epyc can indeed be had for cheap compared to a 7900X rig lol

1 Like
15

feature-rich consumer board prices are comparable to server boards nowadays…and if you find a niche cheap and maybe second-hand CPU that fits into the workload…

And Xeon Bronze or EPYC Rome/Siena certainly don’t have the clocks, but you can get them on a (premium) homeserver budget. 45Drives with the HL15 has a really crappy 6 core Xeon Bronze. Runs TrueNAS with 16x HDD perfectly fine and most containers just want some RAM and not CPU horsepower.

And EPYC Rome…DDR4 RDIMMs are dirt cheap compared to DDR5 UDIMMs. You save a whole lot if going 128GB. Done those calculations myself lately.

edit: Upside on consumer boards…you can cool the CPUs better in 1-2U form factor as there are a lot of low-profile aftermarket coolers available. EPYC 2U…yeah 40mm Dynatron fan with 8k rpm, no thank you :wink:

1 Like
16

If I didn’t already have my X79 era server, I’d be jonesing for for one of those bundles.

He didn’t say gaming, nor doing some maxxed out NVMe only cache server, AI (as far as I’m aware) just needs modest CPU and PCIe lanes out the ass if you want the most out of multi-GPU.
My server would probably do what he wants, except power efficiency I guess, and I paid $150 for the board, $50 for the pair of E5 2667 V2s. Doesn’t game worth a shit in modern garbage coded games; Doom 2016 though, is fine. I CPU encode for streaming… that’s fine. It has some goofy slot indexing shit though, and I know jack shit about IO-MMU on this thing, or this era stuff in general… otherwise, I wouldn’t hesitate to throw that option out there. Eh, what the hell, I’ll find a link for it too.
As an example. this is the later version that actually matches the manual. My older revision, they act like doesn’t exist.

image
image1311Ă—823 190 KB

17

No, but if you’re new to servers, you bring along your knowledge from other computers. And GHz is what’s important there. And high RAM clocks. Doesn’t translate well into (home) server space.

Sure, 12x Ryzen cores run circles around 16 cores of Rome. But unless you run a benchmarking server to run benchmarks, that’s not that critical in practice. Because you primarily want the lanes and the memory.

18

The 7900X will still slaughter them, but I’m seeing some good deals on second-hand 7F52’s which go to 3.9GHz turbo.

1 Like
19

Other than punctuation, I agree with that… i think, because punctuation.
EDIT: because EDIT

I believe he also mentioned speculation of lots of VMs; good luck running out of 32 cores with a 7551P.

Still considerably diminishing return on investment though.

20

No question. But you can’t get >28 lanes for your stuff. If you need them, there is no point comparing them because there is no choice. EPYC spends a lot of TDP on the I/O Die enabling the 128 lanes and memory. So there is a tax associated with it.

7900x is a great CPU. And if you can get your server to fit into the lane restrictions, it’s great and probably also overkill while being very power-efficient @65W TDP. My home server runs with a 5900x and it can punch way above its weight class. And was dirt cheap compared to EPYC with similar performance and 24 lanes were within my specifications :wink:

It’s comparing motorcycles to trucks. Bikes are really fast and cheap. And you can carry some stuff with your bike, but if you want to transport a couch, the bike just is the wrong kind of transport. And you can buy Trucks with a V10 300PS too…but “just a truck” often is totally fine.

1 Like