Everything Fast LiteLLM does
A focused Rust acceleration layer for the LiteLLM hot path — with the safety rails to run it in production and the honesty to tell you where Python still wins.
Acceleration
Rust implementations of LiteLLM hot-path components, compiled via PyO3.
Rust connection pool
A DashMap-backed, lock-free HTTP connection pool replaces LiteLLM's per-provider bookkeeping. 3.2× faster — the single biggest win under concurrent load.
Learn more →Rust rate limiter
Atomic-counter rate limiting: 1.6× faster and, more importantly, 42× less memory when tracking thousands of unique API keys or tenants.
Learn more →Rust token counting
tiktoken-rs backs large-text and batch tokenization for a 1.5–1.7× speedup. Small inputs stay on Python automatically, where FFI overhead would otherwise dominate.
Learn more →Drop-in integration
No rewrites, no new services. One import line and the LiteLLM API is unchanged.
One-line activation
Add import fast_litellm before import litellm. The package monkey-patches LiteLLM at import time — nothing else in your code changes.
Learn more →Prebuilt wheels
Wheels for Linux (x86_64, aarch64), macOS (x86_64, ARM64), and Windows (x86_64). Rust is not required to install. Python 3.8–3.13.
Learn more →Proxy-friendly
Works with the LiteLLM proxy under gunicorn/uvicorn via a preload wrapper module so workers inherit the accelerated components.
Learn more →Production safety
Designed so a Rust component can never make a bad night worse.
Automatic fallback
Any exception on a Rust path transparently falls back to the original Python implementation on the same call.
Learn more →Circuit breaker
After 10 errors in a Rust component, that component disables itself for the remainder of the process — no repeated failures on the hot path.
Learn more →Feature flags & canary
Every component can be disabled or rolled out by percentage via env vars, e.g. FAST_LITELLM_BATCH_TOKEN_COUNTING=canary:10.
Learn more →Performance monitoring
Per-component metrics are exposed for scraping so you can watch acceleration behaviour in production.
Learn more →See the honest numbers
Every claim above is backed by reproducible benchmarks — including the components where Python is faster and Rust is off by default.