// INFERENCE RUNTIME

TensorRT-LLM

NVIDIA's production LLM inference library built on TensorRT. Enables highly optimised engine compilation for maximum throughput on NVIDIA data-center and consumer GPUs. Supports advanced features like in-flight batching and speculative decoding.
Platforms
Linux (NVIDIA only)
Backends
CUDA (TensorRT)
Quant formats
FP16, BF16, INT8, FP8, INT4, AWQ, SmoothQuant, GPTQ
TensorRT-LLM website ↗

Videos about TensorRT-LLM

TensorRT LLM 1.0 Livestream: New Easy-To-Use Pythonic Runtime

How-To Install TensorRT Locally to Optimize and Serve Any Model

From model weights to API endpoint with TensorRT LLM: Philip Kiely and Pankaj Gupta

FAQ

What platforms does TensorRT-LLM support?▶

TensorRT-LLM runs on Linux (NVIDIA only) with backends CUDA (TensorRT).

What quant formats does TensorRT-LLM support?▶

TensorRT-LLM supports FP16, BF16, INT8, FP8, INT4, AWQ, SmoothQuant, GPTQ.

→ All runtimes→ Hardware