Benchmarking Inference on NVIDIA Blackwell Systems

Hey folks, long-time lurker, first-time poster.

What are people using to benchmark LLM inference, particularly on NVIDIA Blackwell systems?

Mainly looking at real-world processes/impact like latency, tokens/sec, scaling across GPUs/NVLink, and power efficiency… not marketing benchmarks.

Any tools, scripts, or gotchas you’ve found useful would be appreciated.

Cheers!

1 Like

Hello and welcome.

I think this thread could interest you:

2 Likes

Thanks @Sedanxio for the mention. Yep I’ve been working on a benchmark suite in this direction. It’ll probably be a while before I get hands on with a Blackwell system so your mileage may vary.

1 Like

I still have to try it. As soon as I’ve got a little time I’ll take a look to your benchmarking system😉 first impression is that it looks promising👍

1 Like

I got some great feedback after posting to reddit, x, here etc. There’s still a bit to do as well my full range of cards (Kepler-Volta) all ironed out as well. I’ll ping you when it’s ready, probably the next minor version on GitHub.

1 Like

Kinda late reply but I have been doing a significant amount of benchmarking the past year across a variety of weird custom setups designed to absolutely maximize throughput for my given workloads.

The workload you are running radically impacts performance, down to drivers, libraries, engine, model, configuration, and data input. Even nightly builds of vllm, flashattention, llama, etc. can impact performance one way or the other.

From a pure hardware perspective the p2p bandwidth and latency test are one key thing I have been tracking in a git repo for all my command output every iteration of rig redesign, pcie topology changes, card additions, custom drivers, and major configuration breakthroughs with bios/grub/kernel/etc.

Here is one big breakthrough ~2 months ago:

This one was actually a customized NVidia-open-driver that hijacks BAR1 to forcibly enable direct peer to peer communication between cards, inspired by the infamous George Hotz. This was combined with disabling PCIE ACS in the bios, ensuring the IOMMU was in passthrough, and a number of other tweaks/tunes.

The result was (depending on cards/topology/config) ~2.5x the bandwidth and 90% latency reduction in some cases. The Epyc system management interface (E_SMI) was used to confirm traffic was crossing the infinity fabric only, not bottlenecked by core compute dies.

4 Likes

this is really nifty, I should do a video on this that you’ve presented well. I have some adjacent information about iommu on xeon sPR refresh, and peer to peer in both nvidia and amd contexts… amd doesnt seem to suffer from the same bugs.
turns out there is some device grouping hints coming with major updates to iommu for these kinds of things.

3 Likes