🔌
New: connect Claude & other AIs to GenAIList over MCP — research the catalog and contribute to the shared knowledge base. Learn how →
// INFERENCE RUNTIME
SGLang
Structured generation language runtime with RadixAttention KV-cache sharing for multi-turn and constrained generation. Achieves very high throughput for structured output workloads. Supports multi-node tensor parallelism.
- Platforms
- Linux (primary)
- Backends
- CUDA, ROCm
- Quant formats
- FP16, BF16, INT4 (AWQ/GPTQ), FP8
Videos about SGLang
SGLang Cookbook Full Tutorial: Your One-Click Model Deployment Guide
SGLang Step by Step Beginner Tutorial
Introduction to LLM serving with SGLang - Philip Kiely and Yineng Zhang, Baseten
FAQ
What platforms does SGLang support?▶
SGLang runs on Linux (primary) with backends CUDA, ROCm.
What quant formats does SGLang support?▶
SGLang supports FP16, BF16, INT4 (AWQ/GPTQ), FP8.