🔌
New: connect Claude & other AIs to GenAIList over MCP — research the catalog and contribute to the shared knowledge base. Learn how →
// INFERENCE RUNTIME
ExLlamaV2
Highly optimised CUDA inference library for quantized LLMs. EXL2 format allows fine-grained mixed bpw quantization achieving near-FP16 quality at 3-4 bpw. Very fast CUDA decode on NVIDIA GPUs.
- Platforms
- Windows, Linux
- Backends
- CUDA
- Quant formats
- EXL2 (2-bit to 8-bit bpw), GPTQ, FP16
Videos about ExLlamaV2
Introduction to ExLlamaV2 (Fastest LLM Inference Library)
Install exui Locally - Web UI for ExLlamaV2
Using ExLLaMaV2 library for fast LLM inference
FAQ
What platforms does ExLlamaV2 support?▶
ExLlamaV2 runs on Windows, Linux with backends CUDA.
What quant formats does ExLlamaV2 support?▶
ExLlamaV2 supports EXL2 (2-bit to 8-bit bpw), GPTQ, FP16.