Local inference runtimes

Highly optimised CUDA inference library for quantized LLMs. EXL2 format allows fine-grained mixed bpw quantization achie

KTransformers

Heterogeneous CPU/GPU inference for very large MoE models.

KoboldCpp

Windows, macOS, Linux

A single-file llama.cpp-based executable adding KoboldAI-compatible API, story mode, and character cards on top of llama

LM Studio

Windows, macOS, Linux

Desktop GUI for discovering, downloading, and running local LLMs. Includes an OpenAI-compatible local server and chat in

MLX

macOS

Apple's array framework for machine learning on Apple Silicon. MLX-LM provides fast inference and fine-tuning for transf

Ollama

Windows, macOS, Linux

User-friendly local LLM runner with a REST API and model library. Ships with llama.cpp under the hood and exposes an Ope

SGLang

Linux (primary)

Structured generation language runtime with RadixAttention KV-cache sharing for multi-turn and constrained generation. A

TensorRT-LLM

Linux (NVIDIA only)

NVIDIA's production LLM inference library built on TensorRT. Enables highly optimised engine compilation for maximum thr

llama.cpp

Windows, macOS, Linux, Android

High-performance LLM inference in C/C++ with broad hardware and quantization support. The de-facto standard for CPU-firs

text-generation-webui

Windows, macOS, Linux

Feature-rich browser-based interface for running LLMs locally. Supports many backends and quantization formats, plus ext

vLLM

Linux (primary), macOS (experimental)

High-throughput LLM serving engine with PagedAttention for efficient KV cache management. Designed for production deploy