Was curious how the bench results compared to my current 4x3090 setup, sharing results to help with comparison, skimmed the settings used for vllm serve and vllm bench from the video and post.
~~~
============ Serving Benchmark Result ============
Successful requests: 50
Failed requests: 0
Benchmark duration (s): 73.58
Total input tokens: 51200
Total generated tokens: 25600
Request throughput (req/s): 0.68
Output token throughput (tok/s): 347.93
Peak output token throughput (tok/s): 700.00
Peak concurrent requests: 50.00
Total token throughput (tok/s): 1043.80
---------------Time to First Token----------------
Mean TTFT (ms): 18778.79
Median TTFT (ms): 18961.10
P99 TTFT (ms): 34846.77
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 106.04
Median TPOT (ms): 105.78
P99 TPOT (ms): 137.75
---------------Inter-token Latency----------------
Mean ITL (ms): 106.04
Median ITL (ms): 76.39
P99 ITL (ms): 1343.31
~~~
The B70 seem like a pretty good deal, as long as their software works, that is.