I’m in the process of replacing my trusted TrueNAS box with a newer one based on an AMD Epyc 7302p, Supermicro H11SSL-i, 256GB RAM, LSI 3008 (SAS) in a Supermicro 846 case with a 24-bay SAS3 backplane. HDD-wise I’m starting with 8 Seagate Exos X20 20TB disks supposed to be in RaidZ2 and two Optane P4800X 750GB for the metadata special device.
Currently I’m running badblocks to check for disk reliability for the new Exos disks and this is where I found some peculiar performancy anomaly. Let me explain:
Currently there are 6 badblocks runs concurrently writing and reading back data to the disks. I have 5 of the 8 Exos X20 under test (the other 3 are already tested) and a 8TB WD drive that made problems in the past and also needs to be checked. The 8TB drive obviously is not as fast as the 20TB ones, but is a lot smaller and thus finishes a pass quicker. I monitor this with Grafana, here is a block of writes as an example:
As you can see, the performance is degrading as expected towards the end, but what is not expected is, that the write performance on the 20TB disks jumps by about 15-20mb/s as soon as the 8TB drive is finished writing (and starts reading again).
Here the relevant moment:
Performance jumps from ~170-190mb/s to ~205-220mb/s on the Exos drives.
At first i thought the backplane has reached it’s limit, but that could not be the case. The write block starts at ~290mb/s on those 5 Exos drives and ~200mb/s on the WD drive, which totals to ~1650mb/s. The moment the 8TB drive is in it’s last sectors I have 5x ~180mb/s + ~90mb/s from the 8TB drive, which is ~990mb/s.
The backplane is connected with two SAS3 cables for 96Gbps total bandwidth.
Can someone here point me in the right direction on where to look or might have an idea what’s causing this? Not that this is something to worry about, but it is peculiar nonetheless.
Oh, and I forget: This is only happening for writes, reads are totally fine. See here:
PS: System is a TrueNAS SCALE-22.12.0. I’ll repeat that with TrueNAS CORE once badblocks is done.
