Critique My ZFS Setup for a VM/Dev Workload Using 3.2TB PCIe 4.0 Optane

Couple of thoughts:

  • Congrats on the Optane P5800X. This is quite an expensive device in your setup. What exactly is your use case? Your 905 or even your P1600X Optanes could suffice (for a lot less money).
  • Check out these observations about connecting Optane P5800X via m.2. Make sure to test your connection under load and check for errors
  • Consider using your SSDs as a ZFS special device. Look at ZFS Metadata Special Device: Z and ZFS: Metadata Special Device - Real World Perf Demo for explanations and details
  • I have been running an Optane SSD as single SLOG and L2ARC device for a while, but using two without redundancy does introduce risks I would not be willing to take.
  • Do you have a plan for using your 16x PCIe Gen 5 lanes? Review your mobo capabilities and schematics. I assume your 16x PCIe Gen 5 lanes may be very hard to use. Look for CPU vs chipset connected PCIe lanes. I assume yours are connected directly to CPU (because Gen 5). If they were connected to the chipset, they would share bandwidth with the 6x SATA lanes on their way to the CPU/memory. Does the mobo support bifurcation? If not, do you have a PLX card to switch the PCIe lanes into usable PCIe lanes (e.g. 4x or 8x m.2 or similar number of PCIe slots).
  • Your 6x SATA HDDs are intended as main storage medium and will determine the speed of your zfs pool. In a RaidZ2 configuration that would range between ~ 10MB/s to a theoretical max of ~800MB/s. The SSDs as SLOG, L2ARC, or special device can be used to overcome the performance drops that HDDs have helping them to stay at peak performance by read-/writing mostly large block size chunks of data. With an Optane P5800X in the mix I have the feeling that this is not the performance level you’re looking for.