Critique My ZFS Setup for a VM/Dev Workload Using 3.2TB PCIe 4.0 Optane

[LiKenun]

I appreciate the prompt feedback! I read many of the threads already but seemed to have missed some critical posts. Just to make sure I have a good foundation to work with, I went ahead and put the whole M.2-OCuLink-enclosure setup together for testing. This thread about PCIe bifurcation casted a lot of doubt that I could ever get it to work: A Neverending Story: PCIe 3.0/4.0/5.0 Bifurcation, Adapters, Switches, HBAs, Cables, NVMe Backplanes, Risers & Extensions - The Good, the Bad & the Ugly - Hardware / Motherboards - Level1Techs Forums.

Using this finally worked for me with 0 errors while putting the drive under load for 10 hours:

• Asus ROG Crosshair X670E Extreme
• Micro SATA Cables Passive PCIe Gen 4 M.2-to-OCuLink adapter on the Asus-provided Gen-Z.2 riser card (PCIe 5.0 side)
• LINKUP 25cm active OCuLink cable (OCU-8611N-025)
• ICY DOCK U.2-to-OCuLink enclosure (MB699VP-B V3)

This was after some trial and error with different M.2 slots and a 50cm version of the active OCuLink cable. The 50cm cable gave errors no matter which M.2 slot I used. 42 WHEA errors events in 30 minutes was the best I could get. The M.2 slot at the far end of the motherboard gave the most errors—27 events in 60 seconds.

This would be my first time sinking so much money into experimenting with this stuff and I was surprised how much of a difference a few centimeters could make.

@jode

What exactly is your use case? Your 905 or even your P1600X Optanes could suffice (for a lot less money).

@jode

Consider using your SSDs as a ZFS special device. Look at ZFS Metadata Special Device: Z and ZFS: Metadata Special Device - Real World Perf Demo for explanations and details

@jode

Your 6x SATA HDDs are intended as main storage medium and will determine the speed of your zfs pool. In a RaidZ2 configuration that would range between ~ 10MB/s to a theoretical max of ~800MB/s. The SSDs as SLOG, L2ARC, or special device can be used to overcome the performance drops that HDDs have helping them to stay at peak performance by read-/writing mostly large block size chunks of data. With an Optane P5800X in the mix I have the feeling that this is not the performance level you’re looking for.

I had been intending to test bcachefs well into my part orders and intended to get 2 × P5800X as a mirrored write cache and metadata device. I only started looking at ZFS a couple weeks ago and unfortunately, some further reading explained that the SLOG is not the equivalent of a write cache which I thought would have allow me to have the best of both high-capacity RAIDz2 and the speed/latency of Optane for the bursty workloads that I’d be throwing at it. An actual writeback cache feature for ZFS doesn’t seem to have gone anywhere. I’ll test out ZFS still, but cutting 96TB of storage in half with RAID1 is something I would like to avoid if I can help it.

@jode

I have been running an Optane SSD as single SLOG and L2ARC device for a while, but using two without redundancy does introduce risks I would not be willing to take.

I know the SLOG only helps synchronous writes and protects asynchronous writes with sync=always (at the cost of latency), but would a persistent L2ARC at least mask the read latency of QLC NAND over SATA for VM/dev workloads?

@jode

Do you have a plan for using your 16x PCIe Gen 5 lanes? Review your mobo capabilities and schematics. I assume your 16x PCIe Gen 5 lanes may be very hard to use. Look for CPU vs chipset connected PCIe lanes. I assume yours are connected directly to CPU (because Gen 5). If they were connected to the chipset, they would share bandwidth with the 6x SATA lanes on their way to the CPU/memory. Does the mobo support bifurcation? If not, do you have a PLX card to switch the PCIe lanes into usable PCIe lanes (e.g. 4x or 8x m.2 or similar number of PCIe slots).

Yeah. They are hard to use. I have been monitoring news of PCIe 5.0 HBAs/RAID cards, which I plan to get for hooking up the Optane SSDs. They’re never going to make PCIe 5.0 Optanes, so the next best thing is to aggregate a whole bunch of them together behind an HBA/RAID controller to better utilize the bandwidth. In the meantime, those M.2-OCuLink adapters will do.

For the Asus ROG Crosshair X670E Extrene, the chipset gets 4 × PCie 5.0 lanes worth of bandwidth, and after subtracting the bandwidth consumed by 2 Thunderbolt ports, that leaves 2 lanes worth shared by 2 × PCIe 4.0 x4 expansion slots and presumably a bunch of other things on the motherboard. Definitely not using those PCIe 4.0 slots especially since having tested 4K random read performance of the 905Ps on both PCIe 4.0 (chipset) and PCIe 5.0 (CPU) I found that the MBps dropped 7.89% on PCIe 4.0.