OFFICIAL SPEED & THROUGHPUT LEADERBOARD
NVIDIA Hardware Benchmarks
Verified throughput (tokens/sec) and Time To First Token (TTFT) comparing TensorRT-LLM and vLLM against baseline vanilla PyTorch.
| Model | Hardware Cluster | Runtime Engine | Precision | Throughput | TTFT Latency | Speedup |
|---|---|---|---|---|---|---|
DeepSeek-R1 (671B MoE) 671B (37B active) | 8x NVIDIA B200 SXM NVLink | TensorRT-LLM | FP8 | 218.4 tok/s | 24.5 ms | 4.8x faster |
DeepSeek-R1 (671B MoE) 671B (37B active) | 8x NVIDIA H200 SXM (1128 GB) | TensorRT-LLM | FP8 | 142.6 tok/s | 38.2 ms | 4.1x faster |
Llama-3.1-70B-Instruct 70B | 1x NVIDIA H100 SXM5 80GB | TensorRT-LLM | FP8 | 165.2 tok/s | 14.8 ms | 3.9x faster |
Llama-3.1-70B-Instruct 70B | 1x NVIDIA H100 SXM5 80GB | vLLM | FP8 | 138.5 tok/s | 19.4 ms | 3.2x faster |
Llama-3.1-70B-Instruct 70B | 2x RTX 4090 24GB (48GB total) | TensorRT-LLM | INT4-AWQ | 54.8 tok/s | 42.1 ms | 2.8x faster |
Llama-3.1-8B-Instruct 8B | 1x RTX 4090 24GB | TensorRT-LLM | INT4-AWQ | 118.2 tok/s | 11.2 ms | 3.4x faster |
Llama-3.1-8B-Instruct 8B | 1x RTX 4090 24GB | Vanilla PyTorch | FP16 | 34.6 tok/s | 45 ms | 1x faster |