NVME power consumption when idle (zfs)

Quick question.

recently built my server (7773x) and i got 6x2tb NVMEs in it, 2 on the mobo and 4 on an asus expansion card. configured as ZFS raid10.

I’m considering using the holiday prices to get 4 more so i get a total of 10tb usable space with the speed that comes with it.

what I’m not sure of is the power consumption when idle and when busy while, and if ZFS effects it. since I’m running this server 24/7 at home, I am conscious of the extra wattage.

Does anyone have benchmarks/experience/numbers ?

p.s. if i add 4 more, they will also live on a second asus expansion card.

I’m pretty sure m.2 is limited to like 7w maximum, system fans can use more power than that

also its overall power consumption depends entirely on which specific NVME you have

your CPU getting pinned or pinged by something random will have more impact

be careful about adding too many, they only qualify those for single card solutions and with the mobo it came with, if you have like 7 in your system, the last 3 may fail to post

never heard of this issue, can you clarify ? I’m pretty sure I’ve seen systems with more than 7, but I’m open to being wrong and learn

Are you talking about the various lower power states that NVMe drives can enter? If so ZFS doesn’t do anything specifically to either enable or thwart that, that I’m currently aware of.

By default it’ll coalesce asynchronous writes and write them out every 5 seconds, which likely makes them even more efficient.

NVMe drives have crazy fast power transition latency between states. Note these are microseconds, not milliseconds.

These states do seem to also rely on the PCIe lane power state as well, but overall I would expect the drive to spend most of its time with very low draw.

The Linux nvme cli package may be able to query what state they are in. or not: NVME drives have no mechanism for reporting power consumption · Issue #518 · linux-nvme/nvme-cli · GitHub

there was a forum post on here where some of his cards were failing to post when all were in at the same time
to clarify I meant the 4 in one cards not the m.2 themselves

I thought it was 25 but I couldn’t find anything to back it up

To be fair, only U.2 or other “serious business” drives with a massive heatsink are going to pull that much. The Micron 9100 Max seems to go even higher even
They also tend to idle at a few watts.

Consumer M.2 drives seem to pull roughly ~6-10 watts max depending on the drive from what I’ve seen, though I think some of the recent gen will actually pull past 12.

Iirc most NVMes have idle/sleep power consumption in low triple digit milliwatts.

As for posting, I have a Rome system with a H12SSL-i, with two of those Asus cards plus two motherboard slots for a total of 10 drives, it posts and UEFI shell shows all of them. My pendrive crapped out so I didn’t get to install an OS yet.

If it’s server drives, they often don’t idle. My samsung drive has a power rating of about 8w or so, and this seems fairly typical for NVME under load, and is the same on their consumer drive of a previous generation of product when under load, but the server drive idles in the 0 state, and temps only vary by about 5~10c, compared to 40c+Δ with the consumer drive.
Similar behavior with a SKHynix server drive, except that one doesn’t throttle when it hits dangerous temperatures. :upside_down_face:

these are 980 pro drives, not server drives

Just look it up for yourself. smartctl will tell ya

[~]# smartctl -a /dev/nvme0n1
...
Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     8.49W       -        -    0  0  0  0        0       0
 1 +     4.48W       -        -    1  1  1  1        0     200
 2 +     3.18W       -        -    2  2  2  2        0    1000
 3 -   0.0400W       -        -    3  3  3  3     2000    1200
 4 -   0.0050W       -        -    4  4  4  4      500    9500
...
2 Likes

Micron 9300 :unamused:

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    25.00W       -        -    0  0  0  0      100     100
 1 +    24.00W       -        -    1  1  1  1      115     115
 2 +    23.00W       -        -    2  2  2  2      130     130
 3 +    22.00W       -        -    3  3  3  3      145     145
 4 +    21.00W       -        -    4  4  4  4      160     160
 5 +    20.00W       -        -    5  5  5  5      175     175
 6 +    19.00W       -        -    6  6  6  6      190     190
 7 +    18.00W       -        -    7  7  7  7      205     205
 8 +    17.00W       -        -    8  8  8  8      220     220
 9 +    16.00W       -        -    9  9  9  9      235     235
10 +    15.00W       -        -   10 10 10 10      250     250
11 +    14.00W       -        -   11 11 11 11      265     265
12 +    13.00W       -        -   12 12 12 12      280     280
13 +    12.00W       -        -   13 13 13 13      295     295
14 +    11.00W       -        -   14 14 14 14      310     310
15 +    10.00W       -        -   15 15 15 15      325     325

Intel Optane 905 :grimacing:

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    18.00W       -        -    0  0  0  0        0       0

seems like the Samsung 980 pro which are not enterprise drives of course, are fairly low on energy.

My decision on going with those drives was that even if i burn a drive a year, I’m ok with the cost and tradeoffs.

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     8.49W       -        -    0  0  0  0        0       0
 1 +     4.48W       -        -    1  1  1  1        0     200
 2 +     3.18W       -        -    2  2  2  2        0    1000
 3 -   0.0400W       -        -    3  3  3  3     2000    1200
 4 -   0.0050W       -        -    4  4  4  4      500    9500