New: connect Claude & other AIs to GenAIList over MCP — research the catalog and contribute to the shared knowledge base. Learn how →
NVIDIA GeForce RTX 4090
Runs downloadable LLMs up to ~32.5B at Q4, ~164.3-195.8 tok/s on an 8B model.
What you can run
At 8k context, FP16 KV. Full-speed tok/s holds up to "max ctx"; beyond it the context spills to system RAM (CPU/PCIe offload) and drops to the degraded figure.
| Model size | FP16 | Q8 | Q4 |
|---|---|---|---|
| 3B |
122.6-146.2 t/s ≤222k ctx · then ~4.9-5.8 (slow) |
245.3-292.3 t/s ≤268k ctx · then ~9.7-11.6 (slow) |
438-522 t/s ≤288k ctx · then ~17.4-20.7 (slow) |
| 7-8B |
46-54.8 t/s ≤35k ctx · then ~1.8-2.2 (slow) |
92-109.6 t/s ≤96k ctx · then ~3.7-4.4 (slow) |
164.3-195.8 t/s ≤123k ctx · then ~6.5-7.8 (slow) |
| 13B | won't run |
56.6-67.5 t/s ≤46k ctx · then ~2.2-2.7 (slow) |
101.1-120.5 t/s ≤81k ctx · then ~4-4.8 (slow) |
| 32B | won't run | won't run |
41.1-48.9 t/s ≤10k ctx · then ~1.6-1.9 (slow) |
| 70B | won't run | won't run | won't run |
| 120B | won't run | won't run | won't run |
| 400B | won't run | won't run | won't run |
Estimate: tok/s ≈ bandwidth ÷ (params × bytes/param) × efficiency (NVIDIA). Cyan = fits ~22.1 GB usable. Beyond "ctx", KV spills to system RAM (CPU/PCIe offload) (rough degraded speed shown).
What else you need for a full build▶
Minimum supporting components for a desktop built around the NVIDIA GeForce RTX 4090. Guidance — exact needs depend on your case, other parts and how much you offload to system RAM.
Real-world measurements (2)▶
| Model | Quant | Runtime | tok/s | Ctx | Source |
|---|---|---|---|---|---|
| Gemma 4 26B | Q4_K_M | Ollama | 149.6 | 4k | blog ↗ |
| Gemma 4 31B | Q4_K_M | Ollama | 7.8 | 4k | blog ↗ |
Run popular models on the NVIDIA GeForce RTX 4090
Videos about the NVIDIA GeForce RTX 4090
LLMs on RTX 4090 Laptop vs Desktop 🤯 not even close!
4090 Local AI Server Benchmarks
More hardware like the NVIDIA GeForce RTX 4090
About the NVIDIA GeForce RTX 4090
NVIDIA GeForce RTX 4090 is a discrete GPU from NVIDIA on Ada Lovelace, 2022. For local genAI the numbers that matter are its 24 GB (how big a model + context fits) and 1008 GB/s bandwidth (generation speed). Plan for a ~850W power supply.
Frequently asked questions
What size LLM can the NVIDIA GeForce RTX 4090 run?▶
With ~22.1 GB usable it runs up to ~32.5B at Q4 (~9.5B at FP16) at 8k context. Bigger spills into system RAM (CPU/PCIe offload) and slows sharply.
How many tokens/sec does the NVIDIA GeForce RTX 4090 generate?▶
Generation is bandwidth-bound. At 1008 GB/s an 8B model at Q4 runs ~164.3-195.8 tok/s (estimate).
Why does long context slow it down?▶
The KV cache for the context must also fit fast memory; once weights+KV exceed 22.1 GB it spills to system RAM (CPU/PCIe offload) and throughput collapses. Each row shows the max usable context.
What else do I need to run the NVIDIA GeForce RTX 4090?▶
Besides the card: a ≥850W 80+ Gold power supply, at least 24GB of system RAM (2× the 24GB VRAM is ideal for CPU offload), a free PCIe x16 slot, any modern multi-core CPU, and a ≥1TB NVMe SSD for model weights.