I gave up on wrangling the Dual Xeon NUMA nodes for now with no luck so had time to revisit the ThreadRipper Pro 24x core with quad T705 Gen 5 NVMe drives.
tl;dr;
I ran some tests with ktransformers.
The bad news: buffered disk read i/o still saturates well before the limit which the drives and mdadm raid array are potentially capable. The kernel kswapd0 pegs at 100% (it handles page cache, not just swap space which was basically disabled for this test).
The good news: I can hit just around 4.2 tok/sec generation running the 377GiB Q4_M quant overflowing 256GB system RAM on the 24 Core ThreadRipper Pro described above. The disk array puts out about 4GB/s average which is about the same as what a single Gen 5 T700 NVMe drive does on my local 9950X rig.
I can hit 1.2 tok/sec generation on the 665 GiB Q8_0 full fat model even too pulling almost 6GB/s through buffered i/o lol…
Details
I setup a new RAID0 striped XFS array attempting to tune it for smaller block random IOPS read loads. It is slightly faster 4k IOPS but worse performance sequential reads given I was targeting the i/o sizes discovered in the above explorations.
Direct 4k random read IOPS
# NOW
read: IOPS=4483k, BW=17.1GiB/s (18.4GB/s)(1026GiB/60001msec)
# BEFORE
read: IOPS=4352k, BW=16.6GiB/s (17.8GB/s)(996GiB/60001msec)
Direct 1M sequential read
# NOW
READ: bw=22.6GiB/s (24.3GB/s), 22.6GiB/s-22.6GiB/s (24.3GB/s-24.3GB/s), io=1359GiB (1460GB), run=60035-60035msec
# BEFORE
READ: bw=37.8GiB/s (40.6GB/s), 37.8GiB/s-37.8GiB/s (40.6GB/s-40.6GB/s), io=2268GiB (2435GB), run=60036-60036msec
I could probably try tuning other knobs, but the bottle neck does not seem to lie with the performance of the drives or mdadm array itself, but with the Linux Kernel Page Cache buffered i/o being unable to deliver more than 4~6GB/s read performance during inferencing when the model size exceeds RAM size.
Check out the btop screenshot showing kswapd0 pegging indicating we’re saturating buffered i/o pretty sure:
Also checkout the sar/sadf log showing md0 reads pegging out around 6 GB/s and averaging around 4GB/s as the generation continues.
Conclusion
So until they start soldering flash chips directly onto GPUs or if DeepSeek’s custom distributed networked NVMe filesystem gets integrated into an inference engine the best strategy is still as much RAM bandwidth crammed into a single NUMA node as you can get plus a single GPU.