I actually bought the wrong fans! I ordered A4x20 FLX instead of the PWM variant. So I reconnected the 40mm fans to the IcyDock Backplane with the low-noise adapter. If you can’t control the fan speed, it really doesn’t matter where you plug them in 
Also allowed me to use a spare 40mm fan on the (now) free header, to cool the on-board NIC. I’m back to 10Gbit networking again.
I set up VMs and containers for everyday use and my desktop and other stuff now has access to CephFS and block devices.
- Benchmarks:
Did some dolphin and fio stuff with 10G networking, measured at my desktop mounting a ext4 formatted RBD (ceph block device, like iSCSI LUN):
Sequential writes: solid 1.1GiB/s
Sequential reads: 850MiB/s (not sure why yet), folders with 50k files drop down to ~350MiB/s
random 4k reads QD=1 (fio) = 1.8k IOPS
same with 4k random writes QD=1 (fio) = 800 IOPS
Doesn’t look too shabby by Ceph/Cluster standards. Using all defaults, so there still are many best-practice parameters not touched yet.
Oh and be careful with intense random writes, Ceph will totally take all your CPU cycles. fio flags were16 threads and QD=32, results in this kind of VM utilization and 6k IOPS (12 cores, 2x NVMe to feed):
I narrowed down my shopping list now.
- Build and Parts for 2nd Node:
-
- CPU: Ryzen 7900 (king of threads/€? and 65W TDP requirement)
-
- Board: AsRock Rack EPYC4000D4U (Made a thread about it,looks like it’s the best that fits)
-
- 2x Micron 32GB ECC UDIMM
-
- Noctua LH-L9x65 CPU cooler
-
- Silverstone RM203-502
-
- Be Quiet Dark Power 13 750W Titanium PSU
-
- IcyDock 4xU.3 bay + 2x MCIO8i → OCulink cables
-
- ConnectX-4 25Gbit NIC
Along with 2xHDD I have (really have to fully migrate my ZFS pool before doing that, too lazy) and I probably order two more U.3 and the fans needed.
Feel free to comment on parts and build, always good to hear from other angles/perspectives. Intention was high-performance NVMe Ceph + headroom for virtualization via Proxmox while keeping costs and power bill low for this kind of platform.
Still not full HA and I will still have to run a Rocky VM for Ceph.
Transitioning to SFP and 25Gbit is another matter I have to face (and pay for) at some point. DACs and connecting the nodes is the least problem there. (Mikrotik) Switch and 2x >3m optical cables might be required.
I ignore it for the time being. Particularly because the drives won’t see passthrough in the long term and remain bare metal. Scrubs so far were all fine…so there is no rampant data corruption by jank cabling going on. But I keep monitoring this.