Home Server on the Cheap with ECC support and minimal Power Usage

Hardware Build a PC
, ,
1

Good Evening,

I could need some help in picking the right parts for a Hardware Update of my Home Server.

I´ve had a pretty sketchy build so far, consisting of a Xeon LC3528, a used HP DL320G6 board, 12GB of DDR3 REG ECC Ram, 2 80GB HDDs and 2 8TB HDDs (each in ZFS mirror). Sketchy because i didnt use a Server Case, i didnt have a Case planned in at all when i bought the hardware, instead i reused an old ATX tower i´ve gotten for free. And powered all with a Be Quiet TFX 350W 80Plus gold, since with this gigantic Board, no normal ATX powersupply would fit in that case anymore…

I use Arch Linux, set up on ZFS as operating System, and as described above, i have my 4 HDDs configured in 2 ZFS mirrors with the 80Gb one for arch and the 8TB one for my Data. Just used SMB for Network access

My Problem with this build: although i purposely chose the lowest powered Xeon for that Platform that i could find, (it´s rated 35W TDP), that build draws 98-107W on Average, when booting up and the server Fans (Couldnt figure out how to get it to boot with normalt fans, as the board has proprietary 6 pin connectors) suck their full 2A, its well above 150W. Thats a little bit too much for a Machine i want to possibly 24/7 for maximum Data Safety… and i already configured anything that stupid board let me to maximum power efficiency.

So i figured it´s time for an upgrade, and since i heard Wendell say in Multiple Videos, that Ryzen supports ECC, and i´ve seen in a Der8auer Video that Zen2 can boot with insanely low voltage and power draw, i thought a ryzen 3 3100 would be ideal for my needs. (since its the smallest zen2 CPU available and i really dont need much horsepower)

I´ve quickly found that ASRock and Biostar seem to support ECC on seemingly all their boards, but i couldnt figure out wether the boards just “accept” ECC ram or whether i could actually use its added functionality. Recently i found an article on a Site called “Elefacts” (sorry its a german one…)
about their "NAS Advanced 2.0 Build)

where there´s a big discussion about the current state of ECC on ryzen.

In the article itsself it´s shown that as i already assumed, its the first case… No usable ECC functions in Linux (at least debian, but most times that uses a really old Kernel…)

That same site suggests an Intel build with a Fujitsu C246 Board and a Pentium G4500 instead if you need ecc. But that board is a little bit too expensive (140€) for me when i can´t even undervolt for further power savings, and with the G4500(T) drawing round 18W instead of Zen 2s 11W (both measured by the Author of that Site, for the zen2 number i used the Power draw of the new Renoir APU they also measured in an other article, so its comparable) in IDLE.

So, can anybody tell me if and how its possible to get ECC working on Ryzen (non Threadripper) on Linux? (Possibly with logging)

And if not, can you suggest me some hardware i could use for that build instead?
(including serverhardware of course, if its within price range)

I need:

  • 2 PCIe Slots, once x8 for my SAS controller, and once x4 for future Network Cards, if the board has only m.2 instead of a second x4 or greater PCIe slot, thats okay too, as it can be adapted to PCIe x4.
  • Less than 20W of power draw in IDLE (without storage)
  • ECC Ram support

Nice to have´s:

  • Maximum 300€ overall pricetag
  • Less than 10W of Powerdraw or capability of undervolting

Would be great if anybody could help me out here.

2

The 3100 seems to have support for ECC, and the CPU itself seems to draw around 18W when idle. Do note motherboard, disks and other will draw some as well, so a reasonable, non-tinkered build would be around 40W power draw from the wall when idle. No idea what undervolting will do for you, perhaps a 30% reduction at 25W-ish? 20W power draw from the wall I find unlikely, but not impossible. Depends on your skew. :slight_smile:

As for ECC support for AM4 in general - All CPUs without integrated graphics has ECC support, but only unbuffered ECC. That said, going cheap, here are my recommendations towards an upgrade kit:

Part Model Price Comment
CPU AMD Ryzen 3 3100 €92.95 Great value for price
Motherboard Gigabyte B550M Aorus Pro €107.64 Aorus is my go-to brand, B550 is better because less likely to fail than X570, better VRMs than B450. mATX allows for some flexibility while you work on getting something smaller
Memory 2 x Kingston Server Premier DIMM 8GB DDR4 2400MHz CL17 €88.30 AM4 only supports unbuffered ECC, make sure it is enabled in the BIOS - not all boards do that
System storage Intel Optane Memory 16GB, M.2 €19.90 A great system disk drive. Sure, not much storage but then you really don’t need much for the root file system on Linux. If you need more, bind-mount /usr/ to a directory on your ZFS volume
Total €308.79

Do note, this assumes you will keep the current case and drives. If you want to get rid of that bulky tower, here is an upgrade path to an even smaller build later on. The above components, plus:

Part Model Price Comment
Case U-NAS 810A €280 Price is approximate, costs a small fortune but worth it for the Wife Acceptance Factor - you can have this puppy on your desk should you want to
CPU Cooler Noctua NH-L9a-AM4 €36.71 Really good and silent cooler
SATA expansion card DeLOCK 10x SATA 6Gb/s €86.59 No need to worry about connectors on motherboard
PSU Fortron Flex ATX Netzteil FSP150-50FGBBB €76.44 Decent enough PSU, 150W, fanless, bronze efficiency
PSU Supermicro PWS-202-1H 200W €45.16 Cheaper alternative if you don’t care about noise
Total (fan PSU) €448.46
Total (no fan PSU) €479.74

Drives are not included, obviously. Good luck! :slight_smile:

3

Welcome to the forum and thanks for the question.

Note this is not an answer to your specific question but a personal perspective that may give you an alternative solution. Apologies if it seems critical, that is not the intention.

I’m really looking forward to mainstream adoption of DDR5* so that we can end the regular threads of “need ECC ram for ZFS”. I replied to a bunch of threads recently debating this.

To restate the position as debated:

  • ECC is useful for in- memory error correction. That means it protects against bit flips for data that resides in memory for long periods and is changed regularly, like big databases or scientific data sets.
  • ECC has no impact, at all, on ZFS data integrity and is of no benefit whatsoever to integrity of a File share data transfer where the data will not reside in memory, even for caching.
  • You are more likely to lose data through network transfer corruption or on the client machine. If the client doesn’t have ECC, then ECC on the server is of little value (for data integrity)
  • The main benefit of ECC for older hardware (DDR3) is that you can get RDIMMS super cheap and get 64GB + of memory for less than the price of 16GB ddr4.

Not looking to derail your plans but if you want low total system power, adding ECC will increase your overall power usage as it has a processing overhead.

On your specific specs:

I don’t think this is realistically achievable on x86. Your network card alone will draw 5W and the SAS controller will draw a few. CPU will be low double digits and the Southbridge will take approx 10W on a newer board with pcie4 support. Realistically your idle power budget should be 30 - 50W.

To go lower consider buying an appliance device with an ARM chip. Something like a qnap or Synology. However then you lose the benefits of ZFS.

Another perspective, I run an old thinkstation s30 as my Freenas box and it pulls approx 120W at idle and 150W typical load with 9 drives and a CCTV system. But It is recycled hardware that would have been scrapped and the 64GB rdimms cost lest than 16GB of ddr4.

I did the calculation that showed my power bill for the year is approx £130 for running this 24*7. Newer hardware would cost me ~£200 to £300 and maybe half my power bill. So a 5+ year ROI. Therefore I’ll keep running this until it breaks or I can get cheap parts that do the same thing with ultra low power.

The tech isn’t there yet though, maybe wait another generation or two.

Again just a personal view. Others will have different insights.

  • Article about the future of ddr5 and ECC for all:

https://www.anandtech.com/show/15912/ddr5-specification-released-setting-the-stage-for-ddr56400-and-beyond

Best of luck

4

First of: Size isnt a problem. I have a Chenbro SR-107 Plus (because of the hotswap bays), and that Case should have enough for almost all Form factors (i´ve already had E-ATX in there without a problem). And i´m really happy with its bulk, because this leaves me a ton of options for the future in a System thats anyways not planned to be moved very much.

I´m not planning on replacing any of my existing Hardware except for mainboard ram and CPU. My psu, CPU cooler and SAS card work alright. Btw the Sata card you´ve linked me there: (why should i use sata when i already have a SAS controller??? )I wouldn´t recommend to use this card at all, cause it has an obvious bandwith problem. 10x SATA 3 sounds nice and all,but this card is only connected to PCIE 2.0 x2. That means you have a maximum bandwith of 1Gbyte/s = 8Gbit/s. Each Sata Port claims to be 6Gbit/s which makes a total of 60Gbit/s needet, you see the problem? Also i already have system ssd´s. The 18W for Zen2 wasnt hot air, that was an actuall measurement from a site that i can not link over here because of forum policy. Its called Elefacts. (again, sry its in german)

I´ve already read that ECC appears to be CPU only and for APU restricted to the Pro Series, but Elefacts seemed to get no Difference what so ever in ECC compatibility between Raven Ridge and other CPUs without IGPU.

I´m not a real fan of Gigabyte desktop Boards for server Projects, as they all specifically exclude ECC in their memory Compatibility list. Also you mentioned X570 which i excluded on purpose from my “list of accepted Chipsets” in the first post, because of its high power consumption, and the fact that i can´t really make much use of PCIE 4.0.

And my biggest Problem with ECC on Ryzen wasn´t adressed. I need proof that the recommended Mainboard supports ECC on a hardware level (so far that i can log potential failures from my OS.) I know that basically every ASRock AM4 board will boot with ECC unbuffered ram, but that says nothing about ECC usage, which seems to be locked according to the Elefacts article i mentioned in the first Post. (Its title is “NAS Advanced 2.0” if anyone wants to look that up).

Thank you for your help and advice, but i didnt really have a problem with picking hardware for a general Ryzen based NAS more with determining wether a board “accepts” or “supports” ECC.

5

Fair enough, was only a suggestion from my part. No need to buy hardware you already have a decent alternative to! :slight_smile: I still recommend that Intel Optane disk though, SOOOO much better performance for the system as a whole. But, your call!

While this is true, this is only a problem if you connect 10 blazing fast SSDs in a RAID0 setting. Since this thing is built for mechanical harddrives that has a maximum transfer speed of around 1/5th of that, still only a problem if RAID0 (max theoretical 12 Gbit/s with mechs then). However, a RAID0 with 10 drives is extremely fragile, so not a practical problem at all. You will most likely never saturate that PCIe 2.0 lane with mechanical drives, and the use case specified was 8 drives + NVMe, not 10.

Always look at the big picture and purpose. There is a reason I recommend what I recommend. :slight_smile:

According to the specifications I have seen, yes, the 3100 combined with a B550 motherboard should allow for this, looking at the manual for the motherboard I posted, it says:

Only way to be sure about it is to try it out for yourself. Like AirstripOne suggested, you won’t really lose anything in this use case by going non-ECC though.

If you want to be really sure, go to MicroCenter or similar place and ask if you could try and pop some ECC in a Ryzen build just to confirm it works.

6

Thx for your Answer,

I dont really see why the introduction of DDR5 has anything to do with the whole ECC and ZFS topic.

Also i cant really get behind why you think ecc would have no Impact what so ever on ZFS data integrity. As soon as a Cached file in Memory becomes corrupted and something new is cached, the corrupted file is written on the Storage, which means my file is also corrupted by now.
OF COURSE, its really rare that something like this happens, but not rare enough that the OpenZFS teams specifically mentions in their FAQ that ZFS indeed works without ECC but they highly recommend it due to that fact.

You have a point in the whole “Corruption on its way to the machine”, but this problem can be easily solved by checking Hash sums before and after transfer. And also Network traffic is corrected by the crc, so it´s not that flawed.

You mentioned that ECC use more Power than non ECC Ram. That´s something i can´t confirm. I´ve tested that on DDR3 with multiple Platforms and never had a significant uplift in Powerdraw on average on same Voltage and Capacity.

Again i can Only link to the Elefacts article, the 11 or 18W aren´t a bunch of hot air, that´s what they measured right behind their powersupply With a whole renoir based System.
I know that i have to add the SAS controller and my HDDs but i specifically said below 20W (without HDDs). I plan on having round 60-65 W for the complete System in the end.

But if you´ve got some other architecture based chips (For example arm) that suits my needs (PCIe connectors, ECC and that stuff), i´m not married to x86 with arch. I´d take Arm over X86 every day (for its efficiency) if i knew a system that could fullfill all that.

7

btw i have to address that again, thank you so much for the time you invested in here so far!!!

8

If that is the case with undervolting it exceeds my expectations. It is unfortunately hard to find proper data points here. 40W peak load is not unrealistic for an undervolted system btw for a headless system, the numbers I gave are for stock settings.

9

Okay, that would be the first Board i´ve seen so far that actually mentions ECC in its manual, i´ve only seen it listed on the “Specs” page.

Also i know for a fact that it does indeed matter if i just go without ecc, as i said in the post above, is because of the way ZFS works, there´s a reason that the developers highly recommend ECC.

I´d love to go to a place like Microcenter, but here in Germany the Biggest “tech-seller” we have in non Urban areas are places called “media-markt” or “saturn”, which sell stuff like TV´s Washing Machines, Macs and prebuild PCs and Laptops. If you´re lucky you´ll find 1 or 2 GPUs an SSD or HDD and a PSU there, (each one only from one brand, no diversity here i think they have contracts that include only using this brand) most times this stuff is already outdated 1 or 2 Generations and i´ve never seen a cpu or mainboard there. The other Option to get Hardware without ordering it on amazon etc. is going to a little PC integrator, which most times have a bit more stuff and actual hardware in their stores, but most times wont allow you to just “test build” something because they dont have the budget to afford that, and i think i´ll have a hard time finding one that has ecc in stock, when i couldnt even find thermal pads a few months ago.

So if somebody actually tried out a board and can ensure me that it works with ecc that would really help me out here, because i dont really have the possibility to do that, and i dont really want to order a dozen boards on amazon, only to send back 11 of them and get blacklisted.

10

That system wasnt undervolted. Thats just Zen2 in its full glory xD, it goes down to insanely low voltages when idleing (sry if that misspelled), which der8auer shows in this Video:


Of course peak will be a lot higher (40W would be awesome for a 3100, but i think that will involve heavy tuning),but i dont have much utilisation on my server usually, so i think its really the most time idle.
11

BTW this is my source for the ECC on ZFS does matter Statement.
https://openzfs.github.io/openzfs-docs/Project%20and%20Community/FAQ.html

12

Then try a mom&pop store or a more niched franchise, any place that does PC repairs will do :slightly_smiling_face: If you buy the ECC memory since that will be part of the build regardless, then just ask to borrow a Ryzen Mobo and CPU in store even. Or even order through them, worst case they can put the parts in a PC and sell it.

Did a similar thing once with a local repairshop here in Sweden.

13

Thank you, thats a great idea, i think i´ll try that!

14

Not going to argue about the wattage since my knowledge here is limited, if ppl have gotten that low from the wall that is fantastic news. Only way to find out is to experiment.

@wendell might be something to make a video about: How low can you get from the wall with an undervolted Ryzen?

2 Likes
15

As stated at the beginning of my post, i was just presenting an alternative that may save you some money and stress. If you really think you need ECC then sure, but its useful for other readers to be aware that the link between ECC and ZFS is misleading. They are not required and the “strong recommendation” is only intended for enterprise environments where the developer is building a high integrity system that is likely to see a lot of in-memory processing. This is the key. If you are just building a basic file server, there will not be much in-memory processing. Every action will be a data transfer which has integrity checks built in.

Per the linked article - DDR5 will introduce ECC on all RAM. Therefore there will no longer be a debate about whether to get ECC on non-ECC, it will all be ECC.

Thats not how ARC works. In the read cache the data is never written back to the ZFS drive, it is just a recent copy of the data from the ZFS pool that can be read more quickly. For writing, the use of a SLOG is only for transient data, so there are data integrity checks in the write algorithm. Again, not data at rest in memory.

Yes agree -this makes my point. There are already data checks on data in transit. ECC does not help with data checks for data in transit. It only benefits data at rest in memory.

The referenced FAQ states 3 times that ECC is not required for ZFS, and is recommended for enterprise environments.

As I say, good luck with your search for a low power enterprise grade system, but hopefully you find the alternative perspectives useful.

1 Like
16

I never said that ECC is needed. So i dont really get why you stress that out so much. I only said that i want it for peace of mind.

You´re right i oversimplified the caching Algorithm of ZFS, but my point still remains. One of ZFS Core features is that whenever a file is accessed, zfs calculates a checksum of that file, and compares it to the one thats saved on the accessed Storage medium. Here comes the important Part: ZFS blindly relies on the System Memory that the calculated Checksum read from the System Memory is still correct after storing it there for comparision with the Checksum on the Storage. Because it doesnt have any way what so ever to handle broken Memory, which is discussed in this Paper.

If the 2 Checksums dont match, ZFS will attempt to repair the seemingly broken file, by using data from the Mirrored Storage medium, assuming that its checksum is correct. If its not, and the 2nd Device is faulty as well as the broken ram, but the first storage device was correct, your data gets bassically repaired broken.

To quote Matthew Ahrens, one of the ZFS Co-Founders:

“There’s nothing special about ZFS that requires/encourages the use of ECC RAM more so than
any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS.”

Anything i mentioned can be checked from the ZFS Wikipedia Page, an Oracle Blog i´ll link below and another Site i´ll link.

As some other source i´ve read mentioned(i know its a little bit over the top): “sure you can run zfs without ECC and never have any faulty data from it, but you can also drive with 120km/h through a traffic calmed area and never somebody over.” I know thats a little bit drastic, but the priciple is basically the same.

17

Short Update: the Autor from Elefacts told me that according to their users, ECC seems to work just fine on all Ryzen CPUs without integrated Graphics. That Problem he described seems to be caused only by the R3 2200 or the Athlon he was using in his other build. Though he hasn´t verified that yet. So i´ll stick with that Repairshop idea, and i´ll post my results (will be a little while till then though) over here if there´s nothing new posted that makes it obsolete.

18

I’m running a Ryzen 5 PRO 2400GE (35Watt) Asrock B450M4 PRO and 2666MHz DDR4 ECC memory on my HTPC. A 35Watt PRO APU might also work for you. I’m saying that because you don’t have a GPU listed, and I’m not sure if your Gigabyte motherboard can boot headless?

19

Well i didnt mention any board yet, but thats another topic to investigate for me, i didnt think there might be a problem at all to boot headless. I also considered the ASRock Pro4 Series, as they are the cheapest boards with a second PCIe Slot >=X4. Would it be possible to you to test if Matisse CPUs also work with ecc on that board? Or IF it can boot headless? Would be a great help, as i planned on going on the ATX version of that board before i considered cheaping out, going A320 and using a M.2 to PCIe x4 converter xD

20

@Pleytos
Oh, i´m sad you withdrew that Post, i think it was really helpful