New: connect Claude & other AIs to GenAIList over MCP — research the catalog and contribute to the shared knowledge base. Learn how →
AMD Instinct MI325X
Runs downloadable LLMs up to ~410B at Q4, ~843.8-1031.3 tok/s on an 8B model.
What you can run
At 8k context, FP16 KV. Full-speed tok/s holds up to "max ctx"; beyond it the context spills to system RAM (CPU/PCIe offload) and drops to the degraded figure.
| Model size | FP16 | Q8 | Q4 |
|---|---|---|---|
| 3B |
630-770 t/s ≤3479k ctx · then ~4.2-5.1 (slow) |
1260-1540 t/s ≤3525k ctx · then ~8.4-10.3 (slow) |
2250-2750 t/s ≤3545k ctx · then ~15-18.3 (slow) |
| 7-8B |
236.3-288.8 t/s ≤1663k ctx · then ~1.6-1.9 (slow) |
472.5-577.5 t/s ≤1724k ctx · then ~3.2-3.9 (slow) |
843.8-1031.3 t/s ≤1751k ctx · then ~5.6-6.9 (slow) |
| 13B |
145.4-177.7 t/s ≤1270k ctx · then ~1-1.2 (slow) |
290.8-355.4 t/s ≤1349k ctx · then ~1.9-2.4 (slow) |
519.2-634.6 t/s ≤1384k ctx · then ~3.5-4.2 (slow) |
| 32B |
59.1-72.2 t/s ≤649k ctx · then ~0.4-0.5 (slow) |
118.1-144.4 t/s ≤771k ctx · then ~0.8-1 (slow) |
210.9-257.8 t/s ≤824k ctx · then ~1.4-1.7 (slow) |
| 70B |
27-33 t/s ≤287k ctx · then ~0.2-0.2 (slow) |
54-66 t/s ≤501k ctx · then ~0.4-0.4 (slow) |
96.4-117.9 t/s ≤595k ctx · then ~0.6-0.8 (slow) |
| 120B | won't run |
31.5-38.5 t/s ≤348k ctx · then ~0.2-0.3 (slow) |
56.3-68.8 t/s ≤509k ctx · then ~0.4-0.5 (slow) |
| 400B | won't run | won't run |
16.7-20.4 t/s ≤14k ctx · then ~0.1-0.1 (slow) |
Estimate: tok/s ≈ bandwidth ÷ (params × bytes/param) × efficiency (AMD). Cyan = fits ~235.5 GB usable. Beyond "ctx", KV spills to system RAM (CPU/PCIe offload) (rough degraded speed shown).
What else you need for a full build▶
Minimum supporting components for a desktop built around the AMD Instinct MI325X. Guidance — exact needs depend on your case, other parts and how much you offload to system RAM.
Run popular models on the AMD Instinct MI325X
Videos about the AMD Instinct MI325X
Is the Nvidia Monopoly Over? Fine-tuning our Liquid AI LLMs on AMD MI325X
Massive AMD Instinct MI325X 2TB HBM3e Server by Gigabyte
More hardware like the AMD Instinct MI325X
About the AMD Instinct MI325X
AMD Instinct MI325X is a discrete GPU from AMD, 2024. For local genAI the numbers that matter are its 256 GB (how big a model + context fits) and 6000 GB/s bandwidth (generation speed). Plan for a ~1400W power supply.
Frequently asked questions
What size LLM can the AMD Instinct MI325X run?▶
With ~235.5 GB usable it runs up to ~410B at Q4 (~115.5B at FP16) at 8k context. Bigger spills into system RAM (CPU/PCIe offload) and slows sharply.
How many tokens/sec does the AMD Instinct MI325X generate?▶
Generation is bandwidth-bound. At 6000 GB/s an 8B model at Q4 runs ~843.8-1031.3 tok/s (estimate).
Why does long context slow it down?▶
The KV cache for the context must also fit fast memory; once weights+KV exceed 235.5 GB it spills to system RAM (CPU/PCIe offload) and throughput collapses. Each row shows the max usable context.
What else do I need to run the AMD Instinct MI325X?▶
Besides the card: a ≥1400W 80+ Gold power supply, at least 256GB of system RAM (2× the 256GB VRAM is ideal for CPU offload), a free PCIe x16 slot, any modern multi-core CPU, and a ≥1TB NVMe SSD for model weights.