🔌
New: connect Claude & other AIs to GenAIList over MCP — research the catalog and contribute to the shared knowledge base. Learn how →
// INFERENCE RUNTIME
vLLM
High-throughput LLM serving engine with PagedAttention for efficient KV cache management. Designed for production deployment with continuous batching and OpenAI-compatible API. Best-in-class for multi-user server throughput.
- Platforms
- Linux (primary), macOS (experimental)
- Backends
- CUDA, ROCm, CPU (OpenVINO)
- Quant formats
- GPTQ, AWQ, FP8, BF16, SqueezeLLM, Marlin
Videos about vLLM
vLLM: Easily Deploying & Serving LLMs
What is vLLM? Efficient AI Inference for Large Language Models
How the vLLM inference engine works?
FAQ
What platforms does vLLM support?▶
vLLM runs on Linux (primary), macOS (experimental) with backends CUDA, ROCm, CPU (OpenVINO).
What quant formats does vLLM support?▶
vLLM supports GPTQ, AWQ, FP8, BF16, SqueezeLLM, Marlin.