Does anyone have AI performance comparison between 600W and 300W Max-Q versions of the 6000 cards?
I didn’t think there were multiple variants. The ones I found the 600W looks like the 2 fan “workstation” card in L1T videos, the 300W looks like a more server oriented blower style.
My current personal blower cards top out at 230W (RTX A5000).
Depends on your use case:
This is the latest llama.cpp build from last night. I can compare other inference engines or models if you like.
600W llama 3.1 8b q8:
root@three-TURIN2D24G-2L-500W:/nfs/ai/llama.cpp/build/bin# CUDA_VISIBLE_DEVICES=0 ./llama-bench -ngl 99 -m /nfs/models/gguf/Llama-3.1-8B-UltraLong-1M-Instruct.Q8_0.gguf
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, compute capability 12.0, VMM: yes
| model | size | params | backend | ngl | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| llama 8B Q8_0 | 7.95 GiB | 8.04 B | CUDA | 99 | pp512 | 14040.03 ± 22.08 |
| llama 8B Q8_0 | 7.95 GiB | 8.04 B | CUDA | 99 | tg128 | 165.10 ± 0.04 |
build: 03d9a77b8 (7278)
300W llama 3.1 8b q8:
root@three-TURIN2D24G-2L-500W:/nfs/ai/llama.cpp/build/bin# /usr/bin/nvidia-smi -i 0 -pl 300
Power limit for GPU 00000000:01:00.0 was set to 300.00 W from 600.00 W.
All done.
root@three-TURIN2D24G-2L-500W:/nfs/ai/llama.cpp/build/bin# CUDA_VISIBLE_DEVICES=0 ./llama-bench -ngl 99 -m /nfs/models/gguf/Llama-3.1-8B-UltraLong-1M-Instruct.Q8_0.gguf
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, compute capability 12.0, VMM: yes
| model | size | params | backend | ngl | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| llama 8B Q8_0 | 7.95 GiB | 8.04 B | CUDA | 99 | pp512 | 10093.25 ± 2206.31 |
| llama 8B Q8_0 | 7.95 GiB | 8.04 B | CUDA | 99 | tg128 | 155.64 ± 0.06 |
build: 03d9a77b8 (7278)
600W llama 3.3 70b q8:
root@three-TURIN2D24G-2L-500W:/nfs/ai/llama.cpp/build/bin# CUDA_VISIBLE_DEVICES=0 ./llama-bench -ngl 99 -m /nfs/models/gguf/Llama-3.3-70B-Instruct-Q8-unsloth-00001-of-00002.gguf
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, compute capability 12.0, VMM: yes
| model | size | params | backend | ngl | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| llama 70B Q8_0 | 69.82 GiB | 70.55 B | CUDA | 99 | pp512 | 1734.59 ± 19.37 |
| llama 70B Q8_0 | 69.82 GiB | 70.55 B | CUDA | 99 | tg128 | 20.54 ± 0.00 |
build: 03d9a77b8 (7278)
300W llama 3.3 70b q8:
root@three-TURIN2D24G-2L-500W:/nfs/ai/llama.cpp/build/bin# /usr/bin/nvidia-smi -i 0 -pl 300
Power limit for GPU 00000000:01:00.0 was set to 300.00 W from 600.00 W.
All done.
root@three-TURIN2D24G-2L-500W:/nfs/ai/llama.cpp/build/bin# CUDA_VISIBLE_DEVICES=0 ./llama-bench -ngl 99 -m /nfs/models/gguf/Llama-3.3-70B-Instruct-Q8-unsloth-00001-of-00002.gguf
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, compute capability 12.0, VMM: yes
| model | size | params | backend | ngl | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| llama 70B Q8_0 | 69.82 GiB | 70.55 B | CUDA | 99 | pp512 | 906.64 ± 4.29 |
| llama 70B Q8_0 | 69.82 GiB | 70.55 B | CUDA | 99 | tg128 | 19.29 ± 0.01 |
build: 03d9a77b8 (7278)
Token generation seems mostly unaffected by 300w vs 600w but prompt processing speed seems to scale.
Don’t forget if you like blower style stuff, you can 3d print an attachment and strap your own blowers to the datacenter card that’s effectively the same as the 600w workstation version with double flow-through fans.
But from other tests I’ve seen here
Max-Q 300W = Wokrstation 450W
So I’m not sure it’s correct to compare them by simply limiting the 600W version to 300 W as from other tests Max-Q looks much more efficient.
Same chip, same power, different cooling solution.
The much much larger factor is what software you are running, and what model you are running. You can rent these cards in various clouds for a couple bucks an hour and run your specific test cases to see if any of the power/cooling version matters.
Here are some real benchmarks from the nearby topic
Here’s what I get (~12K shapes):
mamf-finder.py --m_range 0 8192 256 --n_range 0 8192 256 --k_range 0 4096 256
600w: 423.2 TFLops (best median)
450w: 361.3 TFlops
300w: 260.2 TFlops
Wow, that suggests even the 450W setting on the WS doesn’t get close to the 300W setting on the Max-Q? Guess it really is a 600w card and it’s meant to be used that way, and the Max-Q sounds pretty darn efficient.
Seems like 600W WS actually needs more than 450 W on its default settings to catch up to a Max-Q. Seems you can tune it with tools like LACT to make it more efficient, but out of the box, the Max-Q just seems better optimized from the factory for those power levels.
This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.