After many years of virtualising on a Type 2 hypervisor, it’s time for me to have a bit more fun!
Looking to build a powerful Proxmox machine without completely breaking the bank. Some of my requirements are as follows:
Quiet as I will be working in the same room
Not rack mounted
Somewhat power efficient as it’ll be online 24/7 and electricity prices are insane.
Use it as a NAS (Virtualised instance of TrueNas)
Use as Media Server (likely a linux VM running docker accessing the NAS Storage)
Be able to run multiple linux and windows VM’s concurrently (I work in the cybersec field so want a little playground)
I was initially looking at getting consume hardware (Ryzen 5800X or 5900x) - however since I’ll be using it as a NAS I would like ECC memory and thats very hard to come by (and expensive) for consumer CPUs and boards.
I accidentally stumbled across the AMD EPYC™ 7252 and ASRock EPYCD8/R32. The ATX form factor is ideal as I don’t have a rack, and the specs on the 7252 look pretty decent. Also the board has IPMI which is a definite bonus over consumer HW.
However I cant find a whole lot of info from people who have used these in a build. If anybody is familiar with these (or have any alternatives) I’d love to hear from you!
Whatever EPYC processor you consider, think 150W idle power, without any way of setting an ECO mode. The more RAM/cards/GPUs you add to it the more it will draw, 24/7
Power consumption aside, I am running a similar setup with a 24 core Rome/128GB of ram, dual GPU … not completely silent but bearable with all noctua fans (about 6 of them) i a bequiet case, what killed it for me and had me moving it into the attic was the heat generation, think 150W at idle and 350/450 watts when gaming worth of heat duly pumped out by the noctua fans in my room … it was pleasant in winter but very very unpleasant in any other season.
If you don’t need the 128 PCI express lanes that EpYC gives you, you may find the X570D4U-2L2T to be a better fit for your use case:
Power consumption with a Ryzen 3600 CPU is 29Watts at idle, 128W max…
Choosing hardware is such a difficult decision! There’ll be some downside with whatever route I go.
The reason I’m not going with a Ryzen CPU is purely because of the fact that I’ll be running a NAS on it. With the Ryzen CPU’s you need to use Unbuffered ECC, which is insanely hard to come by from what I’ve seen.
The need for ECC is really what is limiting me to either EPYC or XEON.
Here in Europe, I can buy both UDIMMs and RDIMMs for about 180€ incl. tax per 32GB module. Board availability for the AsRock Rack stuff is much more difficult. I’m glad I bought all the stuff last year when offer was high across the board. Intel is bringing W680 chipset that may be an equivalent, but availability is equally bad or non-existant and also requires DDR4 UDIMMs
I’m in europe as well, I feel it, I had to source the epyc parts in the us (ECC memory as well) …
It al boils down to less spend now and more electricity bill laters or the other way around …
that not considering pci lanes /expansion capabilities, mind me
Hi @CasperSec, I was wondering where you purchased your EPYC 7252. I am looking for two, either an EPYC 7702 or 7742. I want to build an excellent server to take over many duties my current desktop does and can’t do. I use Vmware Workstation Pro 16 for my hypervisor needs. Any equipment I purchase needs to be Linux compatible. The following are the rest of my requirements.
rack-mounted item
Somewhat power efficient as it’ll be online 24/7, and electricity prices are insane.
Use as Media Server (by creating a VM using Embey)
A hypervisor server
Be able to run multiple Linux and Windows virtual machines concurrently (I want a colossal playground)
Need a graphic card or two for Linux and Windows virtual machines
When I say I ‘stumbled across’ one, I mean that I ended up learning about the EPYC range yesterday. I wasn’t too familiar with AMD’s server specific offerings.
Provantage is a good place to source some of this stuff in the US. They’re a reseller, so shipments will come from a variety of distributors (and sometimes drop-shipped from the manufacturer), but they’ll work with individuals and not just businesses. They have good customer service from what I’ve seen, just be aware that it has a business logistics feel compared to a retailer, if you’re not used to that.
You didn’t mention what region you’re in, but Kingston KSM32Exx DIMMs are readily available in the US. That might open up some options for you.
As for EPYC, the 7252 has looked a little difficult to find recently, so you may need to step up to the 7262 or 7272 just due to stock. Be aware there’s a division in the SKUs for Rome, so those two are more different than they might appear at first glance.
The 7252 (8 core), 7272 (12 core), and 7282 (16 core) are “optimized for 4 channel” RAM. They only have two CCDs and the memory controllers are capped at 1333MHz (DDR4-2666), so filling more than 4 DIMM slots on the EPYCD8/R32 will increase capacity but not memory bandwidth (theoretical aggregate max 85.30GB/s). They are also permanently configured for a single NUMA domain. I suspect they’re slightly lower power draw because of all this, but I haven’t been able to find verification of that anywhere.
The 7262 (8 core) is part of the rest of the SKUs, with four CCDs and support for 8 memory channels at 1600MHz (DDR4-3200), max 204.80GB/s, and can be configured for separate NUMA domains if you have workloads that are sensitive to memory latency.
For boards, the EPYCD8/R32 is a PCIe Gen 3 platform. ROMED8 jumps up to PCIe Gen 4, if that matters for any future expansion you may be looking at during the lifetime of this machine. Other brands with ATX form factors are Supermicro and Tyan.
I think your initial choices are fine, this is more an FYI if you start having to think “what if” about some of these options, particularly due to supply constraints.
For a NAS, you want to limit power consumption, especially given today’s market prices. Hence you may want to investigate the AMD EPYC 3000 series SoC (system on chip). Gigabyte and Asrock Rack offer boards with these chips, ranging from a few 100 USD up to 2k USD depending on core count, board size and amount of connectivity. These boards come with CPU+cooler pre-installed/soldered in place so only require RAM, storage (SSD, etc) and PSU to get a working system.
That is, if you can find a vendor that a) carries these and b) sells to non-business end-consumers. I’m trying here in Europe but so far little luck
Funnily enough I actually started looking at the 7282 (16 Core). While the 8 core 8252 will probably suit my needs for now, I’m tempted to step up to a 16 core to future proof myself a little bit.
With regards to the memory controllers being capped at 1333MHz and being configured for a single NUMA domain - does this come with any very noticeable downsides? Memory stuff definitely isn’t be strongest area of knowledge and it’s been quite a few years since I’ve looked at any system architectures.
I own a 7252, which I ran as a placeholder in my main rig for about a year, while waiting for my 7443p. I ran it on a Supermicro H12SSL mainboard, and has recently moved it to an ASRock ROMED8U-2T with the intention of making it into a server. No complaints really, my experience with it is scattered through the AMD Epyc Milan Workstation … megathread. (see e.g. here and here)
Long story short, it did a decent job as a placeholder in my Work/Game/Lab/Daily driver -station for about a year, with 4 memory channels populated, as it is indeed a “4-channel optimized” model. ServeTheHome has a good writeup on what it means (essentially what @Quension described but more detailed).
I only recently started repurposing it as a server, so I haven’t really tested it for the type of use you are planning. My first server incarnation (built but not deployed) looks like this currently:
7252
ASRock ROMED8U-2T
4x 16Gb Samsung ECC dimms
2x Seagate Exos 10Tb
2x Micron 7200 MAX 1Tb
Old Intel 320 for booting
On top of this, the mainboard has an Intel X710-AT2 10GbE onboard.
I intend to run TrueNAS Scale in order to host both storage and VMs in the baremetal OS. Then I’ll run a few VMs but nothing heavy. So as you can see, I won’t use the IO capacity of the chip by far for now, which makes power draw a bit of a concern. I hope to put some NVMe storage in it over the next few years, and also upgrade my home network to 10Gbit.
It will replace an old Atom C2000 system (14w TDP), and the main reason I went with the 7252 is that I needed a CPU a year ago for my work machine, and now I have it left over. In a few days when I’m at home again I can check my idle power draw and post here. Though I’m a bit concerned with power draw relative to my planned usage.
I’m not sure it’s a cap really, although I’ve seen mentions from AMD that these chips are optimized for that speed. I do not know what it means though - AMD officially lists it as supporting 3200 “MHz” memory, and my testing with Intel MLC (Memory Latency Checker) suggested that DDR4-3200 was better in most scenarios. I believe I posted the results somewhere around here, but I can’t seem to find them now.
See above, you can configure it as any of DDR4-{2666, 2933, 3200}. The ability to have several NUMA nodes would help to make memory local to cores and thus processes, thus improving latency at the expense of bandwidth. Good for gaming and some compute workloads, but probably not important for the uses you describe.
Although, from your described use case I think its 8 cores are plenty, and I’d rather agree with those who suggest you could consider something even more power-efficient. I would probably not have gone with this a strong CPU unless I had one already.
Unless you want to run a gaming VM or two - then the 7282 would probably provide an edge, through more cores and the ability to split into several NUMA domains.
The power draw is definitely a concern but I’m not massively worried about the cost. The main worry I have would be the heat output since I will be spending a lot of time in the same room as it (I work from home) and don’t have A/C. Sadly I don’t have a separate space for it currently other than the Attic which probably wouldn’t be the best idea.
Going the Ryzen route would be a semi feasible solution for that - but then that comes with other drawbacks. The EPYC series offers a LOT more room for expansion down the road. It’s not too difficult to fill up the 24 PCIe lanes offered by Ryzen. Also, since the EPYC CPUs are all using the same socket I could easily upgrade to a higher core count in a few years without having to dish out for a new board.
With Ryzen I could go all out and get a 5950X (16 core) - but then I’ll basically be at the peak with little room to move up especially if AM4 gets killed off.
I agree with you that the 16 Cores is probably a bit overkill for my current needs but could be the best option for the long run. I guess my current state of mine is ‘get more than you need now - so you don’t need to upgrade in a year’. I’ve been in far too many situations where I got what I think I needed, and then 6-12 months later I was like “damn, I should have spent the extra few hundred bucks for the higher spec”.
Thanks for the information. I will look into Provantage. I already have a graphic card I can use for my new server. I need to find a source for the rest of the parts. It might be cheaper to purchase an already built server (or not much more) that a data center or enormous business is replacing with new equipment. @CasperSec, you might want to look into gear a data center or a company is replacing with new equipment.
Thanks for the suggestion, The problem with using ebay is it is full of scammers; it’s hard to separate the honest sellers from the ones who aren’t. Amazon is starting to get that way; it seems Amazon doesn’t check out the business they allow to advertise on their site. Since I live quite a ways from a huge recycling business, I guess the best option is to check out a considerable Data Center or company that is replacing their EPYC servers because their contracts are up.
Sure, I see your point, my reasoning was similar. I expect to use more PCIe lanes with time, and there is really nothing interesting between 24 and 128 lanes right now, that is also fairly modern. Although I’d still say 8 cores with SMT takes you far today for a server.
Socket SP3 definitely has a good range of upgrade options if one is willing to watch Ebay in the future. Did you find a particularly good deal on the EPYCD8/R32? For as @Quension points out, it won’t support Milan. But I can also see a point if it keeps the cost down, as it is now last-gen, and you will still have powerful Rome chips out there for future upgrades. Actually, now thinking about it, I realize I didn’t even look for used 1st/2nd gen EPYC boards, that I could’ve got for substantially cheaper than my second EPYC gen2/3 board.
I’m been looking at the 7302P (16 Core) and it’s really been tempting me. Thinking about pairing it up with a Supermicro MBD-H12SSL-I if I can find them both for a reasonable price.
Realistically it’s a bit overkill - but ‘go big or go home’ I guess. I think paying a few hundred bucks more for a whole lot more headroom isn’t the worst thing I could do.
The way I look at it, it’s a great way of advancing my career so in the long run it’ll be a good investment.
