New: connect Claude & other AIs to GenAIList over MCP — research the catalog and contribute to the shared knowledge base. Learn how →
NVIDIA H200
Runs downloadable LLMs up to ~221B at Q4, ~782.1-932.1 tok/s on an 8B model.
What you can run
At 8k context, FP16 KV. Full-speed tok/s holds up to "max ctx"; beyond it the context spills to system RAM (CPU/PCIe offload) and drops to the degraded figure.
| Model size | FP16 | Q8 | Q4 |
|---|---|---|---|
| 3B |
584-696 t/s ≤1865k ctx · then ~4.9-5.8 (slow) |
1168-1392 t/s ≤1911k ctx · then ~9.7-11.6 (slow) |
2085.7-2485.7 t/s ≤1931k ctx · then ~17.4-20.7 (slow) |
| 7-8B |
219-261 t/s ≤856k ctx · then ~1.8-2.2 (slow) |
438-522 t/s ≤917k ctx · then ~3.7-4.4 (slow) |
782.1-932.1 t/s ≤944k ctx · then ~6.5-7.8 (slow) |
| 13B |
134.8-160.6 t/s ≤624k ctx · then ~1.1-1.3 (slow) |
269.5-321.2 t/s ≤703k ctx · then ~2.2-2.7 (slow) |
481.3-573.6 t/s ≤738k ctx · then ~4-4.8 (slow) |
| 32B |
54.8-65.3 t/s ≤245k ctx · then ~0.5-0.5 (slow) |
109.5-130.5 t/s ≤367k ctx · then ~0.9-1.1 (slow) |
195.5-233 t/s ≤421k ctx · then ~1.6-1.9 (slow) |
| 70B | won't run |
50.1-59.7 t/s ≤178k ctx · then ~0.4-0.5 (slow) |
89.4-106.5 t/s ≤272k ctx · then ~0.7-0.9 (slow) |
| 120B | won't run |
29.2-34.8 t/s ≤25k ctx · then ~0.2-0.3 (slow) |
52.1-62.1 t/s ≤186k ctx · then ~0.4-0.5 (slow) |
| 400B | won't run | won't run | won't run |
Estimate: tok/s ≈ bandwidth ÷ (params × bytes/param) × efficiency (NVIDIA). Cyan = fits ~129.7 GB usable. Beyond "ctx", KV spills to system RAM (CPU/PCIe offload) (rough degraded speed shown).
What else you need for a full build▶
Minimum supporting components for a desktop built around the NVIDIA H200. Guidance — exact needs depend on your case, other parts and how much you offload to system RAM.
Run popular models on the NVIDIA H200
Videos about the NVIDIA H200
H200 vs H100: Ultimate AI Inference GPU Comparison 2025
Nvidia Wouldn't Send Me This $30,000 GPU - H200 Holy $H!T
More hardware like the NVIDIA H200
About the NVIDIA H200
NVIDIA H200 is a discrete GPU from NVIDIA, 2024. For local genAI the numbers that matter are its 141 GB (how big a model + context fits) and 4800 GB/s bandwidth (generation speed). Plan for a ~1100W power supply.
Frequently asked questions
What size LLM can the NVIDIA H200 run?▶
With ~129.7 GB usable it runs up to ~221B at Q4 (~62.5B at FP16) at 8k context. Bigger spills into system RAM (CPU/PCIe offload) and slows sharply.
How many tokens/sec does the NVIDIA H200 generate?▶
Generation is bandwidth-bound. At 4800 GB/s an 8B model at Q4 runs ~782.1-932.1 tok/s (estimate).
Why does long context slow it down?▶
The KV cache for the context must also fit fast memory; once weights+KV exceed 129.7 GB it spills to system RAM (CPU/PCIe offload) and throughput collapses. Each row shows the max usable context.
What else do I need to run the NVIDIA H200?▶
Besides the card: a ≥1100W 80+ Gold power supply, at least 141GB of system RAM (2× the 141GB VRAM is ideal for CPU offload), a free PCIe x16 slot, any modern multi-core CPU, and a ≥1TB NVMe SSD for model weights.