Quickstart

Rust acceleration for LiteLLM in under a minute. One install, one import line, zero other code changes.

1. Install

# Using uv (recommended)
uv add fast-litellm

# Or using pip
pip install fast-litellm

Prebuilt wheels for Linux, macOS, and Windows. Python 3.8–3.13. No Rust toolchain needed.

2. Enable acceleration

Import fast_litellm before litellm:

import fast_litellm  # patches LiteLLM's hot paths
import litellm

response = litellm.completion(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Hello"}],
)

3. Run under the LiteLLM proxy

Create a wrapper so the patch applies before the proxy loads:

# app.py
import fast_litellm
from litellm.proxy.proxy_server import app

Then preload so all workers inherit the accelerated components:

gunicorn app:app --preload -w 4 \
  -k uvicorn.workers.UvicornWorker -b 0.0.0.0:4000

Full systemd, Docker, and config-file walkthroughs are in the proxy acceleration guide.

4. Verify it's working

Fast LiteLLM exposes per-component metrics you can scrape to confirm the Rust paths are active and to watch fallback behaviour. If a component errors repeatedly, the circuit breaker disables it automatically — your requests keep succeeding on Python.