🔌
New: connect Claude & other AIs to GenAIList over MCP — research the catalog and contribute to the shared knowledge base. Learn how →
// INFERENCE RUNTIME
llama.cpp
High-performance LLM inference in C/C++ with broad hardware and quantization support. The de-facto standard for CPU-first and Metal/CUDA offload inference. Developed by Georgi Gerganov and the ggml-org community.
- Platforms
- Windows, macOS, Linux, Android
- Backends
- CUDA, Metal, ROCm, Vulkan, SYCL, CPU
- Quant formats
- GGUF (Q2_K, Q3_K_M, Q4_0, Q4_K_M, Q5_K_M, Q6_K, Q8_0, IQ1_M, IQ2_XXS, IQ3_XXS, F16, BF16)
Videos about llama.cpp
How to Run Local LLMs with Llama.cpp: Complete Guide
Run AI Models Locally with llama.cpp
What Is Llama.cpp? The LLM Inference Engine for Local AI
FAQ
What platforms does llama.cpp support?▶
llama.cpp runs on Windows, macOS, Linux, Android with backends CUDA, Metal, ROCm, Vulkan, SYCL, CPU.
What quant formats does llama.cpp support?▶
llama.cpp supports GGUF (Q2_K, Q3_K_M, Q4_0, Q4_K_M, Q5_K_M, Q6_K, Q8_0, IQ1_M, IQ2_XXS, IQ3_XXS, F16, BF16).