[SOLVED] (UK) The quest, for a lower power motherboard using my existing LGA2011 CPU/ECC REG RAM

[ChrisA]

Hello all,

I hope you’re all well!

Before I begin, I’m not expecting anyone to do the leg work for me, just on the off chance that someone has found a more energy efficient configuration.

Some background blurb

So I’ve got me a predicament, with Electric prices suddenly surging in the wrong direction, I’ve had to look at the Server I’ve been slowly preparing over the last few years

What’s got me with this, is the comparison to my snapshot server, which only draws around 60W (i3-9100 based build on SuperMicro board). The problem server is drawing around 200W, fair enough if does have 4 more drives than the i3-9100 setup, but I’d like the idle to be more like 80W if at all possible.

Just for a gauge, 50W in the UK, on 24 hours a day is around £125 for the year, so that thirsty server would cost me £500 a year. That’s around a 1/5th of my total energy cost

So I’m looking for a motherboard that I can transplant the other bits onto!

Current hardware:

Motherboard: Asus X99-E WS
CPU: E5-1650v4 (Socket LGA2011)
RAM: 8x16GB DDR4 2133 ECC REGISTERED
HDD: 10 x Spinning rust
SSD: 2 x for mirrored boot
NVME: 1x for Plugins/Jails

Nice things about current board:

• 10 SATA ports
• Lots of PCI’s
• NVME slot

Other things I have:

• A 4 port HBA LSI card (so the new board doesn’t need so many SATA ports)

Software and Usage:
OS: TrueNAS Core (eventually Scale I expect)
Jail: Emby, Syncthing
Main use: short and long term file storage

I’m happiest buying new, but I do appreciate it’s an old socket - after VERY slowly building it up with RAM, I’d be sad to let it go and start again.

I've tried a few things with the current board

Spun down hard drives to see how much they contribute, but we’re talking a 50W saving
The current board has the ability to reduce core count, so I tried it with no HT and just 4 cores.
I have fiddled and disabled
I haven’t pulled the RAM out yet, though I’ve seen some other users with the same quantity of RAM and same CPU with healthy power usage.
I checked CPU frequency in TrueNAS and it did reduce to 1200Mhz, so I’m guessing the culprit is elsewhere?

Any suggestions welcome!

And as People always like pictures, here it is before I added another 2 trayless units.

[ChrisA]

PSU

Something that I could try, and just means spending money on a reusable item, is the PSU - it’s currently a 750W unit, and I really don’t need that much. I could probably make do with a 350-400W unit perhaps? I recall that PSU’s are most efficient when they’re near their limit?

[Mach3.2]

Typically they peak in efficiency at around ~50% load. Double check the efficiency curve for the PSU you’re interested in.

[ChrisA]

Thank you for that

[xzpfzxds]

Is it mostly idle or running at full whack all day? Do you primarily need the single-thread speed or lots of cores?
- If the CPU is at 100% all day, maybe look at one of the -L Xeons. Ark says your CPU is 140W TDP. The 8-core Xeon E5-2608L v4 which fits the same socket has a 50W TDP - far slower single thread performance, but much more efficient.

Have you done any power measurements with different components?
- I’d imagine that 10 HDDs would be a significant proportion of the idle power, I think WD Reds (as an example) are 3 to 4W idle, so that could be 30W-40W just on HDDs. If you have a way to measure the power, try unplugging them or forcing them all to sleep (hdparm -Y) to see how much difference there is. Might be worthwhile to have a sleep timer on them.

Socket 2011 boards were basically designed for speed rather than efficiency, so I’d be surprised if changing the mainboard would make a big difference. It’s still a very capable system - (I had a Broadwell Xeon E3 (basically a E5 chopped in half) as a gaming rig until very recently, and it kept up even in the most recent titles), so I think it’d cost a lot to find a lower-power equivalent with the same features/expansion options.

[Trooper_ish]

So the goal is for lower power, but is the reason, to save money?
Either way, you might want to get a power measure to see what the system is using idle, and then kick off a scrub or something to see what it actually uses when busy.

As @xzpfzxds mentions, the HDD’s could draw a bunch of power at full, and there is no way to save money on that.
If (big if) all drives are active, they alone could take 150W themselves, along with the CPU.

If the reason to lower power is to run on a battery or something, the SSD’s and less data might be the way to go, but I suspect it is to save money.

In that case, if the CPU is plenty fast, you could Down Clock the CPU, and/or run it at less voltage?

Then you spend £0 extra to save money.

A new mobo would likely code more than £0

I also suspect, the CPU uses more power than the motherboard? So if you got a more efficient one, you might save electricity, even if it costs more to buy a CPU than the savings.

Also, Less Ram would use less electric, if you don’t need it all.

So in summary, I would get a “watt man” or similar power meter, run system at idle, run the system at full for top, then under clock+ lower voltage , and remove some ram to see any changes.

Just my two pence

[thro]

the motherboard is going to be one of the lower power consumption devices in the system.

trying to reduce power via the board is going to be far less effective than changing cpu etc.

go for the higher power consumption devices first.

also be realistic with cost saving via power.

save 10 watts? that’s like 0.1c per hour or less at most power consumption expenses. 2-3 cents per day. likely not worth spending money on component replacement.

also see @Trooper_ish above. maybe fewer larger hard drives. but again… if that costs several hundred dollars it may be money you never get back during the life of the box. and in terms of green you just created a bunch of e waste replacing the otherwise fine drives.

[ChrisA]

@xzpfzxds

Is it mostly idle or running at full whack all day? Do you primarily need the single-thread speed or lots of cores?

Thanks for commenting

It’s mostly idle, never seen it go over 25% CPU usage even during my version of heavy use. I don’t think I need multi-thread, apart from rare occasion when it might be using SyncThing, Emby and file transfer at the same time, but that will be incredibly rare.

@xzpfzxds

• If the CPU is at 100% all day, maybe look at one of the -L Xeons. Ark says your CPU is 140W TDP. The 8-core Xeon E5-2608L v4 which fits the same socket has a 50W TDP - far slower single thread performance, but much more efficient.

If I have to change CPU I certainly will, cheers for looking outside the box

@xzpfzxds

Have you done any power measurements with different components?

• I’d imagine that 10 HDDs would be a significant proportion of the idle power, I think WD Reds (as an example) are 3 to 4W idle, so that could be 30W-40W just on HDDs. If you have a way to measure the power, try unplugging them or forcing them all to sleep ( hdparm -Y ) to see how much difference there is. Might be worthwhile to have a sleep timer on them.

Yes I have. It hovers around 200W AT IDLE. You’re right that the HDD’s are around 50W, I’ve let them spin down and see the lower energy usage. However when I look at my other system (the i3-9100), I have 6 HDD’s in there and that’s 60W total at Idle. It would be nice to see a comparable power usage. I would like to use a sleep timer, but I don’t want that very bad lag that happens during the drives spin up. My final intention is to have 8 HDD’s that spin down/off (data at rest) and 2 HDD’s constantly spinning.

@xzpfzxds

Socket 2011 boards were basically designed for speed rather than efficiency, so I’d be surprised if changing the mainboard would make a big difference. It’s still a very capable system - (I had a Broadwell Xeon E3 (basically a E5 chopped in half) as a gaming rig until very recently, and it kept up even in the most recent titles), so I think it’d cost a lot to find a lower-power equivalent with the same features/expansion options.

Oh, that’s a downer May be my sensible step forward is to get the CPU you mentioned above. Thank you!

@Trooper_ish

So the goal is for lower power, but is the reason, to save money?

Yep, to save money Do expand “Some background blurb” in the first post, electric is costing quite a bit here.

@Trooper_ish

Either way, you might want to get a power measure to see what the system is using idle, and then kick off a scrub or something to see what it actually uses when busy.

Already done sir, do see above

@Trooper_ish

If the reason to lower power is to run on a battery or something, the SSD’s and less data might be the way to go, but I suspect it is to save money.

I’d love too, but that would cost quite a bit…well, an enormous amount of money. And I have the 4TB HDD’s already and would quite like to use them.

@Trooper_ish

In that case, if the CPU is plenty fast, you could Down Clock the CPU, and/or run it at less voltage?

Do check the “I’ve tried a few things with the current board” in the first post, perhaps I shouldn’t use those expanding thingys?!

@Trooper_ish

A new mobo would likely code more than £0

Yes, I’m probably foolishly hoping that the new board would be paid for by the old board, which is selling for around £200 these days.

@Trooper_ish

I also suspect, the CPU uses more power than the motherboard? So if you got a more efficient one, you might save electricity, even if it costs more to buy a CPU than the savings.

I’m going to look into this I think, thank you

@Trooper_ish

Also, Less Ram would use less electric, if you don’t need it all.

I did think about this, I’ve monitored power usage ever since I had 32GB installed, it seems not to have gone up much, but when I have the confidence I’ll remove it back down to 16GB and see what happens.

@Trooper_ish

Just my two pence

I always welcome your two pence, even one penny’s worth

[ChrisA]

@thro

the motherboard is going to be one of the lower power consumption devices in the system.

trying to reduce power via the board is going to be far less effective than changing cpu etc.

go for the higher power consumption devices first.

also be realistic with cost saving via power.

save 10 watts? that’s like 0.1c per hour or less at most power consumption expenses. 2-3 cents per day. likely not worth spending money on component replacement.

also see @Trooper_ish above. maybe fewer larger hard drives. but again… if that costs several hundred dollars it may be money you never get back during the life of the box. and in terms of green you just created a bunch of e waste replacing the otherwise fine drives.

Thank you. It does seem that the Motherboard isn’t the issue then - I just figured that as it’s high performance, it might be the culprit. I’m looking into another CPU that has a lower TDP.

UK electric Prices have gone up substantially, i think it’s equal to 40 cents per hour (USA) , or 60 Australian cents per hour…does that sound like a lot over there?

Just to summarise, I have 2 servers. One is 200W at idle, the other is 60W at idle. I would quite like two machines that are both 60W at idle (or thereabouts).

To put it another way, I have an office that’s 20 Sq. M./215 Sq. Ft. in area that I have to heat in winter.

Heating that room for 8-10 hours = Running the server for 24 hours.

I just don’t like that maths very much.

My final intention is to have 8xHDD’s at rest for most of the time (during the day at least), that should save some cost. Only 2 x HDD’s will be my “dropbox/Gdrive replacement”.

When I began building the servers 2 years ago, electric was cheap and it wasn’t a real factor, so I invested in an awful lot of RAM and HDD’s. I bought the extra RAM in case the server was hit heavily by other users - it’s not happened yet because I haven’t provided access, it’s the end goal though. But now electric prices have risen by 60-80%, I’m having to monitor my usage as boring as it is.

Imagine if your annual energy costs went from £2000 to £3500 (or $2500 to $4300), because that’s what has happened here. I think I’m doing the right thing by looking into my baseline energy usage.

Oh, in Australia that’s equal to a rise from $3500 Aussie Dollars to $6200 Aussie Dollars. I don’t know how much Aussie electric is, perhaps over $6000 Aussie dollars is a usual and acceptable cost?

[Trooper_ish]

@ChrisA

perhaps I shouldn’t use those expanding thingys?

Wow, perhaps I should go back to school and effin lern to reed gud…

And I hear you about leccy prices.
Here near Gatwick it’s gone up to 24p per kw hour (a bit less economy 7)

And there are like 8760 hours-ish a year.

So every watt is like 8.7 *24p per year.
So every extra watt, is £2.1 each over a year.
So a 5w pi, costs ten quid just sitting there.
Ten hard drives at 3w each is £63 just idle.
Just figures I kinda try to pay attention to, not to stress at, but a 100w idle device, gonna run me £200/year

But, it’s only like £63, and to replace them with SSD’s would cost a Lot more than that.

But I should have read the first post properly. My bad

[ChrisA]

Just one reason why I thought motherboard is the power hog is this post - same config almost, accept board (they later corrected to say same CPU as me… But 70W.

[Trooper_ish]

So it also has a v3 chip, and 4 less drives?

I mean, I totally hear you on downsizing; I switched down from a threadripper +32 gigs ram,12 drives, down to a Ryzen 3000g, 8 gigs of ram and 4 HDD’s.
I slimmed the content and compressed it to save space, and the machine does less.
Just emby, nas, pihole and router. No more rendering or running game servers and stuff.

Saved a bunch every year on power, for the cost of about £70 mobo, and £90 CPU.

It is not great tho, and no ECC

[wertigon]

Hmm… Found a few interesting options for that socket.

Model	Cores	Threads	Price	Release Quarter
E5-2603 v4	$213	6	6	Q1’16
E5-2609 v4	$306	8	8	Q1’16
E5-2620 v4	$419	8	16	Q1’16

I think either could fit your bill, E5-2603 will draw the least power but the 16 threads of the E5-2620 is really nice for virtualization purposes.

[KleerKut]

I would look at what settings can be changed and how much CPU you are using before making any hardware changes. It might not net as much as it did over a decade ago, but I still go into UEFI/BIOS when I first set up a machine and turn off all the things I don’t use like serial and parallel ports. Also unplug things you don’t regularly use like optical drives or USB stuff. If the machine mostly sits there and isn’t directly interacted with much then it doesn’t need anything but power and network to do its job.

Then again you could just go all in and have it mine crypto to offset expenses. Makes a lot more sense in colder climates where the heat is welcome.

If you don’t need all of those cores but like the single core performance, you can keep the same CPU and disable cores. I only play one game that is heavy on my machine, and there is no discernible FPS difference between 4 cores 8 threads and 3 cores 3 threads. The power drop was pretty significant - from around ~125W to under 90W in game. To me it makes sense if you are building a server with a used CPU to pick one that slightly overshoots your needs and dial it back for efficiency. That leaves you headroom to grow.

If you could find any way other than replacing the board to change voltages then that would be even better. I’m not sure it is worth a new motherboard in this case versus just selling it all off and buying into a different platform. I remember back in the day messing with some software that would allow you to change FSB, multiplier, and in some cases voltages, but I have no idea what that is like nowadays or what hardware it works on. I just remember getting a mythical 50% FSB overclock on a dual socket 604 machine one cold January morning.

[ChrisA]

@Trooper_ish

Wow, perhaps I should go back to school and effin lern to reed gud…

And I hear you about leccy prices.
Here near Gatwick it’s gone up to 24p per kw hour (a bit less economy 7)

And there are like 8760 hours-ish a year.

So every watt is like 8.7 *24p per year.
So every extra watt, is £2.1 each over a year.
So a 5w pi, costs ten quid just sitting there.
Ten hard drives at 3w each is £63 just idle.
Just figures I kinda try to pay attention to, not to stress at, but a 100w idle device, gonna run me £200/year

But, it’s only like £63, and to replace them with SSD’s would cost a Lot more than that.

But I should have read the first post properly. My bad

Ah don’t worry Troop, if anyone can be forgiven, you can

Wow, bad with you too eh, we’re paying pretty much 30p per KWh now (inc. 5% VAT), no Economy7 here, mores the pity. This is the thing, these constant power takers mount up quite easily, and I’d prefer to spend the money on beer…well, Rum may be (personally choice).

You’ve got almost the same figures as me - 50W 24 hours a day is around £125. I don’t mind that so much, but when it comes to 250W, that’s a big wad of cash for convenience.

[ChrisA]

@Trooper_ish

So it also has a v3 chip, and 4 less drives?

I mean, I totally hear you on downsizing; I switched down from a threadripper +32 gigs ram,12 drives, down to a Ryzen 3000g, 8 gigs of ram and 4 HDD’s.
I slimmed the content and compressed it to save space, and the machine does less.
Just emby, nas, pihole and router. No more rendering or running game servers and stuff.

Saved a bunch every year on power, for the cost of about £70 mobo, and £90 CPU.

It is not great tho, and no ECC

Ah, the guy changed his post - said that he meant to say E5-1650v4, which is same as me. He’s only got 2 fewer drives than me though.

Jeez, you had a beast eh. I’d really like to go the Ryzen route, if only they had proper ECC and not just “Works with, but does not use” type of compatibility.

[ChrisA]

Thank you @wertigon I’m going to look at those, probably the lowest thread one as I really don’t use much performance in my use case.

[ChrisA]

@KleerKut

I would look at what settings can be changed and how much CPU you are using before making any hardware changes. It might not net as much as it did over a decade ago, but I still go into UEFI/BIOS when I first set up a machine and turn off all the things I don’t use like serial and parallel ports. Also unplug things you don’t regularly use like optical drives or USB stuff. If the machine mostly sits there and isn’t directly interacted with much then it doesn’t need anything but power and network to do its job.

Then again you could just go all in and have it mine crypto to offset expenses. Makes a lot more sense in colder climates where the heat is welcome.

If you don’t need all of those cores but like the single core performance, you can keep the same CPU and disable cores. I only play one game that is heavy on my machine, and there is no discernible FPS difference between 4 cores 8 threads and 3 cores 3 threads. The power drop was pretty significant - from around ~125W to under 90W in game. To me it makes sense if you are building a server with a used CPU to pick one that slightly overshoots your needs and dial it back for efficiency. That leaves you headroom to grow.

If you could find any way other than replacing the board to change voltages then that would be even better. I’m not sure it is worth a new motherboard in this case versus just selling it all off and buying into a different platform. I remember back in the day messing with some software that would allow you to change FSB, multiplier, and in some cases voltages, but I have no idea what that is like nowadays or what hardware it works on. I just remember getting a mythical 50% FSB overclock on a dual socket 604 machine one cold January morning.

Cheers @KleerKut

Well, I’ve had a look all I can at the board settings, such as:

I’ve turned off all unused things that you’ve mentioned, to no real effect, or just a few watts here and there.

I’ve turned off a few cores as well as HyperThreading too. I might try for having only 2 Cores, but I think that’s pushing it for TrueNAS.

Yeah If needed I might have to spend money and move on, it’s a shame though having built it up to this level. I can get some reasonable money for all the RAM I guess.

I had thought about adjusting the voltages, but I don’t want to damage the CPU accidentally (not a proficient tweaker of the BIOS.

Really appreciate your comments though, thank you!

[ChrisA]

Update: Well I’ve just bought a Xeon E5-2603 v3 for £10, let’s see what happens eh!

[ChrisA]

Ah balls, may be I’ll let 6 Cores loose, 4 Cores doesn’t seem to cope with transcoding Emby to a mobile phone. I’ve got a funny feeling the same issue would happen during remote play too.