Thanks @Sedanxio for the mention. Yep I’ve been working on a benchmark suite in this direction. It’ll probably be a while before I get hands on with a Blackwell system so your mileage may vary.
I got some great feedback after posting to reddit, x, here etc. There’s still a bit to do as well my full range of cards (Kepler-Volta) all ironed out as well. I’ll ping you when it’s ready, probably the next minor version on GitHub.
Kinda late reply but I have been doing a significant amount of benchmarking the past year across a variety of weird custom setups designed to absolutely maximize throughput for my given workloads.
The workload you are running radically impacts performance, down to drivers, libraries, engine, model, configuration, and data input. Even nightly builds of vllm, flashattention, llama, etc. can impact performance one way or the other.
From a pure hardware perspective the p2p bandwidth and latency test are one key thing I have been tracking in a git repo for all my command output every iteration of rig redesign, pcie topology changes, card additions, custom drivers, and major configuration breakthroughs with bios/grub/kernel/etc.
This one was actually a customized NVidia-open-driver that hijacks BAR1 to forcibly enable direct peer to peer communication between cards, inspired by the infamous George Hotz. This was combined with disabling PCIE ACS in the bios, ensuring the IOMMU was in passthrough, and a number of other tweaks/tunes.
The result was (depending on cards/topology/config) ~2.5x the bandwidth and 90% latency reduction in some cases. The Epyc system management interface (E_SMI) was used to confirm traffic was crossing the infinity fabric only, not bottlenecked by core compute dies.
this is really nifty, I should do a video on this that you’ve presented well. I have some adjacent information about iommu on xeon sPR refresh, and peer to peer in both nvidia and amd contexts… amd doesnt seem to suffer from the same bugs.
turns out there is some device grouping hints coming with major updates to iommu for these kinds of things.