New: connect Claude & other AIs to GenAIList over MCP — research the catalog and contribute to the shared knowledge base. Learn how →
AMD Instinct MI300X 192GB
Runs downloadable LLMs up to ~305B at Q4, ~745.3-910.9 tok/s on an 8B model.
What you can run
At 8k context, FP16 KV. Full-speed tok/s holds up to "max ctx"; beyond it the context spills to system RAM (CPU/PCIe offload) and drops to the degraded figure.
| Model size | FP16 | Q8 | Q4 |
|---|---|---|---|
| 3B |
556.5-680.2 t/s ≤2581k ctx · then ~4.2-5.1 (slow) |
1113-1360.3 t/s ≤2627k ctx · then ~8.4-10.3 (slow) |
1987.5-2429.2 t/s ≤2647k ctx · then ~15-18.3 (slow) |
| 7-8B |
208.7-255.1 t/s ≤1214k ctx · then ~1.6-1.9 (slow) |
417.4-510.1 t/s ≤1275k ctx · then ~3.2-3.9 (slow) |
745.3-910.9 t/s ≤1302k ctx · then ~5.6-6.9 (slow) |
| 13B |
128.4-157 t/s ≤910k ctx · then ~1-1.2 (slow) |
256.8-313.9 t/s ≤990k ctx · then ~1.9-2.4 (slow) |
458.7-560.6 t/s ≤1025k ctx · then ~3.5-4.2 (slow) |
| 32B |
52.2-63.8 t/s ≤424k ctx · then ~0.4-0.5 (slow) |
104.3-127.5 t/s ≤546k ctx · then ~0.8-1 (slow) |
186.3-227.7 t/s ≤600k ctx · then ~1.4-1.7 (slow) |
| 70B |
23.9-29.2 t/s ≤107k ctx · then ~0.2-0.2 (slow) |
47.7-58.3 t/s ≤321k ctx · then ~0.4-0.4 (slow) |
85.2-104.1 t/s ≤415k ctx · then ~0.6-0.8 (slow) |
| 120B | won't run |
27.8-34 t/s ≤168k ctx · then ~0.2-0.3 (slow) |
49.7-60.7 t/s ≤329k ctx · then ~0.4-0.5 (slow) |
| 400B | won't run | won't run | won't run |
Estimate: tok/s ≈ bandwidth ÷ (params × bytes/param) × efficiency (AMD). Cyan = fits ~176.6 GB usable. Beyond "ctx", KV spills to system RAM (CPU/PCIe offload) (rough degraded speed shown).
What else you need for a full build▶
Minimum supporting components for a desktop built around the AMD Instinct MI300X 192GB. Guidance — exact needs depend on your case, other parts and how much you offload to system RAM.
Run popular models on the AMD Instinct MI300X 192GB
Videos about the AMD Instinct MI300X 192GB
The AMD Instinct MI300X GPU can handle the Meta LLaMa 60B parameter model on a single GPU. #amd
AMD MI300X: LLM Setup Was SHOCKINGLY FAST!
More hardware like the AMD Instinct MI300X 192GB
About the AMD Instinct MI300X 192GB
AMD Instinct MI300X 192GB is a discrete GPU from AMD on CDNA 3, 2023. For local genAI the numbers that matter are its 192 GB (how big a model + context fits) and 5300 GB/s bandwidth (generation speed). Plan for a ~1150W power supply.
Frequently asked questions
What size LLM can the AMD Instinct MI300X 192GB run?▶
With ~176.6 GB usable it runs up to ~305B at Q4 (~86B at FP16) at 8k context. Bigger spills into system RAM (CPU/PCIe offload) and slows sharply.
How many tokens/sec does the AMD Instinct MI300X 192GB generate?▶
Generation is bandwidth-bound. At 5300 GB/s an 8B model at Q4 runs ~745.3-910.9 tok/s (estimate).
Why does long context slow it down?▶
The KV cache for the context must also fit fast memory; once weights+KV exceed 176.6 GB it spills to system RAM (CPU/PCIe offload) and throughput collapses. Each row shows the max usable context.
What else do I need to run the AMD Instinct MI300X 192GB?▶
Besides the card: a ≥1150W 80+ Gold power supply, at least 192GB of system RAM (2× the 192GB VRAM is ideal for CPU offload), a free PCIe x16 slot, any modern multi-core CPU, and a ≥1TB NVMe SSD for model weights.