Xeon scalable 1st/2nd gen for homelab

[drbsg]

I am currently running a dual v3 Xeon system that I built myself which has been solid for the past three years, but I am feeling the itch to upgrade. I mostly use it for software development where more CPU frequency would help, but also use it for running EM and GR simulations which can require >512GB RAM.

There are a lot of cheap Epyc Rome CPUs and motherboards coming out of China which look interesting, but I know that people on here have had issues with getting cooling working, and with other workstation features. Which is why I took note when I found some second hand dual Xeon scalable workstations for sale on eBay.

There are some attractive machines like the Lenovo P920 available, and they have solved any cooling issues. Changing over a couple of CPUs and upgrading the PSU wouldn’t be a problem. But I am struggling to find any good value second hand or refurbished CPUs from this series.

So does anyone here use 1st/2nd gen Xeon scalable machines in a similar context? And how do you find them? Which of the lengthy list of SKUs is good value for a non-gaming enthusiast?

[rrubberr]

I personally made the mistake of going the Epyc Rome (then Milan) route, moving from a 64-core Rome ES chip to a 64-core 7713p. Frankly, AMD’s platform has too many shortcomings to be practical for my workstation use case, less so in cooling issues and more so in terms of feature set and hardware compatibility.

@drbsg

So does anyone here use 1st/2nd gen Xeon scalable machines in a similar context?

So I moved to a dual socket Skylake system, like you are considering. I wound up with an HP Z6 G4, as I have used numerous HP Z6xx series workstations in the past and have always had a great experience with them.

@drbsg

Which of the lengthy list of SKUs is good value for a non-gaming enthusiast?

In terms of cheap processors, chips like the 8180 and 8168 are inexplicably still sold for thousands of dollars.

You’ll want to look for “off roadmap” processors like the 8124M (18 cores, ~$250), 8173M (28 cores, ~$300), and 8273CL (28 cores, ~$700). To be clear, these are not engineering samples, but instead are specialized final stepping SKUs sold directly to enterprise customers.

The “M” designation means the processor supports 1.5TB of memory per socket, whereas “L” means 4.5TB per socket.

@drbsg

And how do you find them?

Do some research on machines from the big OEMs (Dell, Lenovo, HP) and check eBay! Also, if you have questions about hardware compatibility (like with the 8124M which requires 240 watts, not all boards will be compatible), check out this STH Forum thread.

[drbsg]

@rrubberr

In terms of cheap processors, chips like the 8180 and 8168 are inexplicably still sold for thousands of dollars.

Yes, this is my main reason for starting the thread. The prices do seem ridiculous, and it made me wonder whether there was any value in this platform for a hobbyist.

@rrubberr

You’ll want to look for “off roadmap” processors like the 8124M (18 cores, ~$250), 8173M (28 cores, ~$300), and 8273CL (28 cores, ~$700). To be clear, these are not engineering samples, but instead are specialized final stepping SKUs sold directly to enterprise customers.

I was aware of the 8124M from @Wendell’s coverage. I will see what I can find out about the others you list - they look interesting.

@rrubberr

Do some research on machines from the big OEMs (Dell, Lenovo, HP) and check eBay! Also, if you have questions about hardware compatibility (like with the 8124M which requires 240 watts, not all boards will be compatible), check out this STH Forum thread .

Sorry, my question as asked was ambiguous and means “What is your experience of using them?” in the local lingo. But thank you for the clear answer.

[rrubberr]

@drbsg

“What is your experience of using them?”

In terms of an architecture I don’t think anyone will argue that Skylake isn’t getting long in the tooth. As you can see from my own Passmark results for 64-core Rome ES, an EPYC 7713P, and Dual 8173M, the AMD processors should perform better in most workloads.

The performance discrepancy in actual software isn’t nearly what Passmark would make it out to be though; again with my own results for 64-core Rome ES, an EPYC 7713P, and Dual 8173M.

Then there are certain pieces of software, like Substance Painter or Mitsuba Render 2 (results below) that make serious use of AVX512, which tell a much different story, but those are obviously outliers:

In terms of platform, I was using the EPYC processors in an Asrock Rack motherboard, and they seem to have lost interest in the platform after Milan X was released. The last BIOS release was December of 2022.

HP still releases BIOS updates for the Z6, with the most recent being dated April 5th of this year. One of the more recent BIOS updates added Resizable BAR support, which is nice to have.

The Skylake server processors are officially Windows 11 supported, if that’s the sort of torture you want to subject yourself to

If you only need the machine for compute, you’ll be happy with either. I personally ended up missing creature comforts like Thunderbolt and FireWire for digital audio, more than two USB ports, quick boot times, VROC, etc. that the AMD system couldn’t provide.

[Arpiruk]

You have to consider the nature of the main application. The render example above is known to be highly optimized for intel architecture.

For, simulation use case that are memory bound, you would want to use as much bandwidth as possible. In this particular case, EPYC Rome is almost twice as fast than Naples while the Naples is slightly slower than Xeon SP gen I. Can’t say anything for gen II, never had a hand on it. But when your application can make use of AVX512, Xeon will be 1.5X faster than without it.

BUT, my two 8124m or 8173m cannot hold a candle to 7742 or 7b13. This is made possible through the 16X channel of 3200DDR4 vs 12X2666DDR4, and the giant cache, 10-times of that of the Xeon’s. And lastly, the double vector unit.

Look for the optimal n-core times all-core boost. In general, the scaling won’t goes beyond 2-4 core per channel. 2 for a very large case and 4 for an intermediate one. That means, you will not gain anything running simulation with 128 core on the dual socket 7742/7763/7b13. My scaling stops at 32 per socket. Running 64 per sockets is slower. Again mine is memory bound.

7F72 can be found around 3-4 hundred now which is very nice. You get 24x3.7GHz compare to 32x3.2GHz on 7742. Note that the cheapest 7742 is about 800 bucks. If I have time I will compare it with 7b13 and 7742 and report here.

I still don’t get the point why EPYC platform has limited use case compared to Xeon.
I do love the avx512 of gen 1, but the memory is too slow. I am thinking about upgrade the CPU to gen 2, but I highly doubt that it would worth the money.

The 3rd and the 4th gen is still excessively expensive.
If I had 4K USD, I would just skip to Genoa. I believe Any 32 core Genoa would be at least twice faster than 7B13. In memory bound problems of course.

Lastly, X11dsi-NT is just 300 buck now, but you need 500 for xeon board. A complete barebone of epyc gen I II III, cost a grand.
Plug 2X7F72 for 800, 16x16GB for 350 you have it ready for 2150. The same price I did componentwise for 8124m two years ago.

Oh a cooling solution for SP3 is 35 buck per unit, 1u 2u 4u pasive or active. Look at aliexpress.

[Arpiruk]

For thermal management of tower case, ditch the mainstream exhaust fan like be quitet or cooler master and grab one, or may be three from Delta.

The one above has 170CFM, which should be about twice more airflow than the commodity 120-mm fan.

[Styp]

Honestly, I would make a list with priorities.

• I would not go the epyc route, because it lakes a workstation board.
• if high memory is a must, go the ddr4 ecc route and chose the best platform.
• I run a c621-64l with a Xeon gold 6212u. Single core performance is not as good as a modern 12th or 13th generation, but it’s a tradeoff to run 24core computation tasks for 3 days. Wouldn’t want to miss that.
• 8100 vs 8200 skylake makes almost no difference, the frequency is within 100mhz across all matching sku s.
• intel has some advantages in the ddr4 era.
• the computing sweet spot is 4 cores per channel for ddr4.

I would fancy a high l3 cache chip, like a Milan or Rome but the absence of proper workstation boards with sleep support, on board sound and optimized for workstation cooling makes it a no no for me.