[LOG] Low Power Servers for TrueNAS - the next move (includes power usage and spec)

[ChrisA]

Hello all!

Well, I’ve made some progress, hopefully in the right direction this time!

Following on from this thread, where I was trying to reduce the power usage of an existing machine that I was hoping to re-use (it wasn’t the right machine to use):

[SOLVED] (UK) The quest, for a lower power motherboard using my existing LGA2011 CPU/ECC REG RAM

Hello all, I hope you’re all well! Before I begin, I’m not expecting anyone to do the leg work for me, just on the off chance that someone has found a more energy efficient configuration.

Some background blurb

So I’ve got me a predicament, with Electric prices suddenly surging in the wrong direction, I’ve had to look at the Server I’ve been slowl…

I thought I’d re-arrange things, and make a more appropriate system. So the new one that will run daily consists of:

CPU: Intel G4560 (second hand for £22.50)
MOBO: Supermicro X11SSM-F (off a well known auction site, used but never taken out of box, around £160)
RAM: 16GB RAM ECC UNBUFF (£20 of a well known auction site)
PSU: Be Quiet 400W (£40)
SSD: Boot, some 128GB Samsung drive I found
HDD: Test pair of old 320GB 2.5 inch hard drives.

Thought I’d get some early observations of power usage early on…why not eh, it’s what life is for!

3W - power off (just some juice for IPMI I guess
13W - Powered on and running TrueNAS, 1 x SSD attached, No GPU installed (using onboard VGA when needed)
22W - when downloading/installing TrueNAS update
17W - Powered on and running TrueNAS, 1 x SSD, 2 x 2.5in HDD’s (no pool configured)
29-31W - during copy TO drives in mirrored arrangement (speed was a slow 60MB/s)

Screw up

screw up on my part - to check power usage I used a smart plug…which I borrowed from a device that was being set to turn on and off every few hours during the day. So of course, as I was in TrueNAS it turned the system off. Oops. No side effects present, system still stable.

Speed up the transfer for testing
Now I’ve striped the drives, I’m getting an expected 106-112MB/s with occasional drops down to 80MB/s (max’ing out the 1GB network connection). I just want to dump some data on there and have some perfect video files that are chunky to do the job.

31W - When copying files TO drives
36W - During replication from 2 x HDD’s (striped) to 1 x HDD (Striped, of course!).

16W - Idle, no hard drive activity (wow)
23W - During a scrub of the 2 x HDD’s (2.5in size)

Interesting to see how one (very old) HDD taken from an old laptop is a bit slower than the other

General console view during transfer

Ooooh, didn’t notice this IPMI integration before, that’s quite cool.

The board during testing

The HDD arrangement while testing, hehe

I’m going to do a bit more fiddling before I put it in a case and use decent HDD’s. Anyway, hope that’s of interest to someone.

[ChrisA]

Just a minor addition, but while replicating from 1 x 3.5in to 2 x 3.5in mirrored, power is at around 40W.

[SgtAwesomesauce]

@Novasty Low power NAS stuff you might be interested in here.

---

Excellent report, thanks for sharing this info.

I’m probably going to be looking for something fairly low-power in the next few months.

[Token]

Awesome report. I’m all about low power NAS after paying a rack mounts power bill for a few years.

[F1N]

Looking for a low power NAS/Storage solution myself as well the UK power bills atm hate my current Xeon Icelake and x299 Storage systems.

May look into the Pentiums and see what they have to offer especially the newer Comet lake and Alderlake ones

[KleerKut]

Those power numbers seem really good! I had an older generation that did pretty well with undervolting, but between Plundervolt and your current numbers it seems like things are good as-is. That CPU is from when Pentiums got their Hyperthreading back, nice find!

[ChrisA]

@SgtAwesomesauce

Excellent report, thanks for sharing this info.

I’m probably going to be looking for something fairly low-power in the next few months.

That’s really kind of you to say, you’re very welcome.

@Token

Awesome report. I’m all about low power NAS after paying a rack mounts power bill for a few years.

You’re very welcome

@F1N

Looking for a low power NAS/Storage solution myself as well the UK power bills atm hate my current Xeon Icelake and x299 Storage systems.

May look into the Pentiums and see what they have to offer especially the newer Comet lake and Alderlake ones

That’s what motivated me too (I’m in the UK also). The original X99 / Xeon E5-1650v4 was just too power hungry, though I think I just identified the problem as being the motherboard, along with other contributors.

It is irritating that Intel pulled low power consumer CPU’s that have ECC. I specifically chose this one (and the i3-9100) because they are ECC friendly.

@KleerKut

Those power numbers seem really good! I had an older generation that did pretty well with undervolting, but between Plundervolt and your current numbers it seems like things are good as-is. That CPU is from when Pentiums got their Hyperthreading back, nice find!

They’re not bad are they! The motherboard settings are stock at the moment, might turn off a few things like network boot though.

[ChrisA]

OVERNIGHT UPDATE

So I let it run over night with these settings

A bit of bad news, I always thought that HDD standby and advanced power management worked hand in hand, i.e. after 5 minutes (in the above case), the Advanced Power management setting would then work. I don’t think that’s the case.

I saw 15-16W after the 5 minute mark, as soon as I accessed the drives via Samba, they clearly spun up, going up to 40W and then stabilising at 27W. The drives are just sitting on my desk and as soon as I accessed the data via a network connection, I could feel the vibrations coming through the desk.

I’ve now set the HDD standby to “always on”, and so far 27W is holding.

I’ve turned off shares just as a precaution.

[ChrisA]

23W seems to be the resting power now.

[ChrisA]

I should have mentioned, these are NOT NAS drives being used, just very old ones from old computers, so there’s a chance they’re not clever enough to be instructed to do things!

[felixthecat]

I’d been trying to find info on this exact hardware, since I’m planning to free up my X10SRM-F and Xeon E5 for a dedicated virtualization server and replace it with an E3-1220 v5 or Pentium in the exact board you used .

The main takeaway from this is how much power usage has gotten better. I have an X9 Board and 4C/4T Xeon E3 v2 that uses exactly the same amount of power as my Xeon E5 v4 8C/16T. So I was a bit worried that going with an E3 V5 I wouldn’t gain much power savings in comparison to another E5 Build.

So thank you very much for this, it’s very helpful information.

[ChrisA]

ADDITIONAL

Seems like when transferring power is around 40W.
(from mirrored pair of 3.5in drives)

[Exard3k]

@ChrisA

I’ve now set the HDD standby to “always on”, and so far 27W is holding.

Wars have been waged on whether to spin-down or not
But spinning up the drive to 4-7k rpm takes a lot of energy every time. And for a server that’s running 24/7, every watt counts.

I really like the cheap hardware coming together in a really good basic server for <30W on x86.

Replacing the HDDs with an SSD that has good standy/sleep states, you can further improve on the strengths and reduce storage to mW in idle and avoid annoying (both power and noise) spin-ups.

[ChrisA]

I’m really glad to have helped you mate, it’s only fair to share these statistics. I hope your future plans go well

It is a capable board though, no doubt about that. I went with Supermicro because I’ve already got one and although the cost proper money, they seem to be a good investment and as reliable as you can get.

That is interesting about your power usage with the E3 vs E5.

[ChrisA]

Haha, I’ve seen those wars…a lot of casualties (though more self-confidence based loses!).

I’m really glad that it’s worked out this way, it was a bit of a gamble because these figures are easy to come by. When you consider I had a server (X99-E WS) that was using around 90-95W WITHOUT HDD’s, it’s just bonkers how much money I’ll save.

I’d love to get some SSD’s one day as that’ll really help. My only concern is how SSD’s tend to work or not work, without much warning that there’s a problem - though I’m sure my TrueNAS server would email me if there was a problem!

[Exard3k]

If you’re concerned about data integrity, you can use copies=2 in ZFS to get everything but protection from disk failure at the cost of 50% capacity, but allows for single disk operation, avoiding power bill for two drives.

[ChrisA]

ADDITIONAL COMMENT

Might be worth noting that there aren’t any case fans…as I’ve just been using the old motherboard box for testing!

[ChrisA]

Cheers for that, I never even knew it was a feature.

My current arrangement involves 3 servers, 2 will be always on (Server 1 & 2) and one snapshots to the other. Then the third will be turned on every week or so and will receive snapshots of the snapshots box…umm, if that makes sense!

[ChrisA]

COMPARISON

Just something that may help, I have an old i7-7700K/Quadro RTX 4000 (3 x 27in displays) with a NVME & 3.5in HDD. On balanced mode (in Win10) it hovers around 55-60W when I’m just webs browsing like this (and playing a youtube video in the background). If I select High Performance mode it goes up to 80W. If I do something a little intensive (Graphical work) it goes to around 120-130W.

Doing video editing it goes up to 150-160W

[Exard3k]

@ChrisA

Cheers for that, I never even knew it was a feature.

You can set it for the entire pool or on a dataset level. Use case is special, but sometimes “single-disk-raid1” can be worth it.

@ChrisA

My current arrangement involves 3 servers, 2 will be always on (Server 1 & 2) and one snapshots to the other. Then the third will be turned on every week or so and will receive snapshots of the snapshots box…umm, if that makes sense!

It does. I have a similar rotation and cross replication.

I tweaked my ZFS on my laptop a bit, so ZFS doesn’t wake up my NVMe every 5 seconds from deep slumber. The tunable vfs.zfs.txg.timeout="120" tells ZFS to keep data in memory for 2 minutes and only then flush it to disk. Only applies to writes, but it adds up over time. Doesn’t really work with HDDs, but squeezes out some more savings on SSDs. Also applies to your boot drive and TrueNAS likes to log a lot of stuff.

I’m not familiar with power tweaks on FreeBSD, but checking things like CPU governor, etc. may get you some gains too.