Benchmarking AI Box Performance?

Okay! Finally able to close the loop on this and post results.





  • The V100 is the Sweet Spot: The V100 (16GB) surprised me. Its performance hangs right up there with the much more expensive T40 and was a big improvement over the incrementally improving previous generations.

  • P40 > P100 for LLMs: The community consensus holds true here. If you specifically want to run Large Language Models, with Pascal, use P40.

  • M60 is a Whisper Beast: If you have a ton of audio transcription to do, the M60 is shockingly capable (beating even V100) and can be had for only $50.

  • Scaling is mostly Linear: Stacking cards doesn’t hit a wall of diminishing returns within a 4U chassis. More GPUs generally equal linear performance scaling, though if you mix generations, slower cards will bottleneck your faster ones in LLM setups.

  • CPU/Mobo Choice: Faster single-core CPU speeds help slightly for tasks like Whisper and Vision Transformers, but generally, any cheap X99 board and high-lane Xeon will feed these GPUs perfectly fine.

The complete set of graphs and findings are on my blog. Now that I have the setup and tooling, I’d love to benchmark more workloads, anything missing from my findings?

1 Like