New: connect Claude & other AIs to GenAIList over MCP — research the catalog and contribute to the shared knowledge base. Learn how →
NVIDIA A100 SXM4 80GB
Runs downloadable LLMs up to ~123.5B at Q4, ~332.2-396 tok/s on an 8B model.
What you can run
At 8k context, FP16 KV. Full-speed tok/s holds up to "max ctx"; beyond it the context spills to system RAM (CPU/PCIe offload) and drops to the degraded figure.
| Model size | FP16 | Q8 | Q4 |
|---|---|---|---|
| 3B |
248.1-295.7 t/s ≤1009k ctx · then ~4.9-5.8 (slow) |
496.2-591.3 t/s ≤1054k ctx · then ~9.7-11.6 (slow) |
886-1055.9 t/s ≤1075k ctx · then ~17.4-20.7 (slow) |
| 7-8B |
93-110.9 t/s ≤428k ctx · then ~1.8-2.2 (slow) |
186.1-221.7 t/s ≤489k ctx · then ~3.7-4.4 (slow) |
332.2-396 t/s ≤516k ctx · then ~6.5-7.8 (slow) |
| 13B |
57.2-68.2 t/s ≤281k ctx · then ~1.1-1.3 (slow) |
114.5-136.5 t/s ≤361k ctx · then ~2.2-2.7 (slow) |
204.5-243.7 t/s ≤396k ctx · then ~4-4.8 (slow) |
| 32B |
23.3-27.7 t/s ≤31k ctx · then ~0.5-0.5 (slow) |
46.5-55.4 t/s ≤153k ctx · then ~0.9-1.1 (slow) |
83.1-99 t/s ≤207k ctx · then ~1.6-1.9 (slow) |
| 70B | won't run |
21.3-25.3 t/s ≤6k ctx · then ~0.4-0.5 (slow) |
38-45.3 t/s ≤100k ctx · then ~0.7-0.9 (slow) |
| 120B | won't run | won't run |
22.1-26.4 t/s ≤15k ctx · then ~0.4-0.5 (slow) |
| 400B | won't run | won't run | won't run |
Estimate: tok/s ≈ bandwidth ÷ (params × bytes/param) × efficiency (NVIDIA). Cyan = fits ~73.6 GB usable. Beyond "ctx", KV spills to system RAM (CPU/PCIe offload) (rough degraded speed shown).
What else you need for a full build▶
Minimum supporting components for a desktop built around the NVIDIA A100 SXM4 80GB. Guidance — exact needs depend on your case, other parts and how much you offload to system RAM.
Run popular models on the NVIDIA A100 SXM4 80GB
Videos about the NVIDIA A100 SXM4 80GB
A100 SXM4 80GB NVLink NVIDIA Professional Graphics Card for Deep Learning #graphics #graphics card
4×NVIDIA A100 GPU Server Benchmark: Running Multi-Modal AI Models
More hardware like the NVIDIA A100 SXM4 80GB
About the NVIDIA A100 SXM4 80GB
NVIDIA A100 SXM4 80GB is a discrete GPU from NVIDIA on Ampere, 2020. For local genAI the numbers that matter are its 80 GB (how big a model + context fits) and 2039 GB/s bandwidth (generation speed). Plan for a ~800W power supply.
Frequently asked questions
What size LLM can the NVIDIA A100 SXM4 80GB run?▶
With ~73.6 GB usable it runs up to ~123.5B at Q4 (~34.5B at FP16) at 8k context. Bigger spills into system RAM (CPU/PCIe offload) and slows sharply.
How many tokens/sec does the NVIDIA A100 SXM4 80GB generate?▶
Generation is bandwidth-bound. At 2039 GB/s an 8B model at Q4 runs ~332.2-396 tok/s (estimate).
Why does long context slow it down?▶
The KV cache for the context must also fit fast memory; once weights+KV exceed 73.6 GB it spills to system RAM (CPU/PCIe offload) and throughput collapses. Each row shows the max usable context.
What else do I need to run the NVIDIA A100 SXM4 80GB?▶
Besides the card: a ≥800W 80+ Gold power supply, at least 80GB of system RAM (2× the 80GB VRAM is ideal for CPU offload), a free PCIe x16 slot, any modern multi-core CPU, and a ≥1TB NVMe SSD for model weights.