New: connect Claude & other AIs to GenAIList over MCP — research the catalog and contribute to the shared knowledge base. Learn how →
Intel Gaudi 3
Runs downloadable LLMs up to ~199.5B at Q4, ~479-594.6 tok/s on an 8B model.
What you can run
At 8k context, FP16 KV. Full-speed tok/s holds up to "max ctx"; beyond it the context spills to system RAM (CPU/PCIe offload) and drops to the degraded figure.
| Model size | FP16 | Q8 | Q4 |
|---|---|---|---|
| 3B |
357.7-444 t/s ≤1682k ctx · then ~3.9-4.8 (slow) |
715.3-888 t/s ≤1728k ctx · then ~7.7-9.6 (slow) |
1277.4-1585.7 t/s ≤1748k ctx · then ~13.8-17.1 (slow) |
| 7-8B |
134.1-166.5 t/s ≤765k ctx · then ~1.5-1.8 (slow) |
268.3-333 t/s ≤826k ctx · then ~2.9-3.6 (slow) |
479-594.6 t/s ≤853k ctx · then ~5.2-6.4 (slow) |
| 13B |
82.5-102.5 t/s ≤551k ctx · then ~0.9-1.1 (slow) |
165.1-204.9 t/s ≤630k ctx · then ~1.8-2.2 (slow) |
294.8-365.9 t/s ≤665k ctx · then ~3.2-4 (slow) |
| 32B |
33.5-41.6 t/s ≤199k ctx · then ~0.4-0.5 (slow) |
67.1-83.3 t/s ≤321k ctx · then ~0.7-0.9 (slow) |
119.8-148.7 t/s ≤375k ctx · then ~1.3-1.6 (slow) |
| 70B | won't run |
30.7-38.1 t/s ≤141k ctx · then ~0.3-0.4 (slow) |
54.7-68 t/s ≤235k ctx · then ~0.6-0.7 (slow) |
| 120B | won't run | won't run |
31.9-39.6 t/s ≤150k ctx · then ~0.3-0.4 (slow) |
| 400B | won't run | won't run | won't run |
Estimate: tok/s ≈ bandwidth ÷ (params × bytes/param) × efficiency (Intel). Cyan = fits ~117.8 GB usable. Beyond "ctx", KV spills to system RAM (CPU/PCIe offload) (rough degraded speed shown).
What else you need for a full build▶
Minimum supporting components for a desktop built around the Intel Gaudi 3. Guidance — exact needs depend on your case, other parts and how much you offload to system RAM.
Run popular models on the Intel Gaudi 3
Videos about the Intel Gaudi 3
Intel's Big AI Bet: Gaudi 3 ft. Supermicro -- System Tour, Tested Pytorch & More
Intel Gaudi 3 AI Accelerator with 24x 200GbE #ai
More hardware like the Intel Gaudi 3
About the Intel Gaudi 3
Intel Gaudi 3 is a discrete GPU from Intel, 2024. For local genAI the numbers that matter are its 128 GB (how big a model + context fits) and 3700 GB/s bandwidth (generation speed). Plan for a ~1300W power supply.
Frequently asked questions
What size LLM can the Intel Gaudi 3 run?▶
With ~117.8 GB usable it runs up to ~199.5B at Q4 (~56.5B at FP16) at 8k context. Bigger spills into system RAM (CPU/PCIe offload) and slows sharply.
How many tokens/sec does the Intel Gaudi 3 generate?▶
Generation is bandwidth-bound. At 3700 GB/s an 8B model at Q4 runs ~479-594.6 tok/s (estimate).
Why does long context slow it down?▶
The KV cache for the context must also fit fast memory; once weights+KV exceed 117.8 GB it spills to system RAM (CPU/PCIe offload) and throughput collapses. Each row shows the max usable context.
What else do I need to run the Intel Gaudi 3?▶
Besides the card: a ≥1300W 80+ Gold power supply, at least 128GB of system RAM (2× the 128GB VRAM is ideal for CPU offload), a free PCIe x16 slot, any modern multi-core CPU, and a ≥1TB NVMe SSD for model weights.