I need help deciding on my first secondhand enterprise server buy

[mtsmith]

Hello there. I have been spending a couple of weeks trying to price out some second hand server gear to serve as a home server. In this time, I have found out many things about enterprise hardware that I had no idea about. A non-complete list of things I believe I have found out:

• Server cases are loud, because they don’t have to be quiet
• Server PSUs are loud, so you can’t even quiet down a server if it uses a proprietary PSU
• U.2 NVME ports are expensive, as far as I can tell. Looks like at least 100 dollars a slot if you’re using an oculink backplane
• I have no idea how server case backplanes are connected, and can’t seem to find much information on if you can use a, for instance, dell backplane on your own case.
• Epyc 7001 used equipment is like 4 times cheaper than 7002
• Intel naming conventions are confusing.
• It’s hard to find benchmarks for old hardware
• It’s find to find what motherboard to even use.
• Help, I’m in over my head.

Originally, I thought I was going to pick up something like a supermicro supersystem 4U and silence it, but that doesn’t look to be a thing that can even be done, thanks to the proprietary PSUs.

I started looking to a gigabyte motherboard with 7 PCI-E slots, but it looks like it’s impossible to even install anything in the board because the space overlaps with the ram slots.

What I want:

• As many PCIEx16 slots as possible on a motherboard. This way I can use them for NVME, or for old Tesla cards if I feel like spinning up some LLMs or something.
• 32 or more cores. Epyc7501 or a dual intel system maybe?
• Hot swap 2.5" bays if possible, so I can load it up with some raid 6 arrays
• Enough space to watercool the processors, and hopefully silence the whole thing
• I want to spend somewhere around 500 < x < 3000 dollars

What this will be for:

• Giant Expandable Storage Array
• Maybe running Pihole, and/or/ PFsense, or
• Run VMs to do things – maybe game servers, etc
• Serve up media
• Serve as a VPN into my network (when I figure out how to do that)

After a couple of weeks of looking, I’ve come to the conclusion of confusion. I’m looking for suggestions for parts, and answers to some questions such as:

• What all can you use occulink for?
• What’s the main difference between minSAS, Occulink, sas, sata?
• What sort of other weird enterprise connectors are there that I should know about?
• How are backplanes usually connected? And are they universal between systems?
• Where’s a good place to just search a database of hardware to find things that I might want?
• What other cool things should I do with a home server?

My background: I’ve built a lot of gaming PCs. I know consumer hardware pretty well, but it turns out I know like nothing about enterprise hardware. I know my way around linux, etc.

Thanks a ton if you got it this far. Thanks even more if you leave a comment.

[edit] The main resource I’ve been looking at for boards/cpus currently is Ebay. If there are other good resources out there, please let me know!

[regulareel]

Ive been tempted to buy 2nd hand enterprise gear but apart from noise that you already knew, what about the eletricity overhead and cooling requirements?

[mtsmith]

Cooling – I plan on doing waterblocks for the CPUs, and putting everything into a larger 4U case with hopes that it will be easier to keep cool. If I do any power-hungry cards, I’ll also water-cool them. I have a huge 400mmx400mm radiator for this.

Power requirements – I’ll be running it on a 120AC circuit. If needed, I can make it have its own dedicated circuit.

[mtsmith]

After looking around ebay some more, it looks like engineering sample xeon scalable chips can be had for fairly cheap. Has anyone gone this route? Do you recommend for or against it?

Maybe looking at something like:
Motherboard: Gigabyte C621-WD12 Intel Xeon LGA 3647 Dual ECC DDR4 S/PDIF Out EEB SERVER WORK | eBay

Processors: SR37Q Intel Xeon Platinum 8173M 2.0GHz 28-Core LGA3647 Processor ***USED | eBay

I’m also super curious what one would do with a board like this: https://www.asrockrack.com/general/productdetail.asp?Model=ROME2D32GM-2T#Specifications It exposes 128 PCI-E lanes through Low profile SlimSAS (PCIe4.0x8) connectors. Is this meant for storage? Or are there backplanes out there that connect through slimsas to give you PCI-E slots/etc.

[rrubberr]

I personally replaced my Epyc 7713P system with a pair of 8173M chips; you could also go for the one-generation-newer 8273CL if you want slightly higher memory bandwidth and IPC.

The L prefix also lets you install more memory, 4.5TB per socket for L over 1.5TB for M.

It’s worth noting that these aren’t engineering samples (and you don’t want Skylake engineering samples anyway ). These are off-roadmap final stepping parts made for hyperscalers like Amazon and Facebook, and therefore have no drawbacks with motherboard compatibility.

One thing I’ve noticed is that the whole dual-socket system only draws 90 watts at the wall while idling, with 2 sockets, 12 DIMMs, 2 HDDs, 4 U.2 drives, a sound card, a FireWire card, a Thunderbolt card, etc.

This is compared to my 7713P machine that drew well over 140 watts at the wall with >70 watts reported for the CPU alone at idle; the system also had no hard disks or add-in cards and half the number of DIMMs and U.2 drives. Infinity fabric results in tons of power overhead, especially on Milan chips.

This would mainly be a consideration if you’re planning to run the system 24/7.

In terms of cooling, I had the 7713P on a 240mm AIO radiator and the system was nearly silent at idle, aside from some pump whine. The CPU never exceeded 65C at 240 watt cTDP.

The 8173M system is an HP Z6 G4 which is entirely silent at idle, but slightly louder at full tilt due to being air cooled. Cooling isn’t a huge concern for these large-footprint 165 watt processors. Remember, a 13900K easily takes 250 watts in a tiny little socket and much smaller silicon

One random tidbit is that the Skylake system supports Resizeable BAR but the Milan system did not… despite being a much newer platform.

[mtsmith]

Interesting. Thanks a ton for the feedback here. I learned some things