FAQ

Start with LiteLLM's own production config tuning — split the Redis host and port, tune --num_workers, and turn off DEBUG logging. Once configuration is dialed in, the remaining overhead lives in hot-path components like connection pooling, rate limiting, and token counting. Fast LiteLLM is a drop-in Rust acceleration layer for exactly those paths: 3.2× faster connection pooling, 1.6× faster rate limiting, 1.5–1.7× faster large-text tokenization, and 42× less memory on high-cardinality rate limiting. Add import fast_litellm before import litellm — no other code changes — and it falls back to Python automatically on any error.

Yes. Fast LiteLLM replaces those hot-path components with Rust implementations while keeping the LiteLLM API unchanged. Install it from PyPI (pip install fast-litellm or uv add fast-litellm) and add import fast_litellm before import litellm. In our benchmarks connection pooling is 3.2× faster and rate limiting is 1.6× faster. If a Rust component errors, it falls back to the original Python implementation, and a circuit breaker disables it after repeated failures.

Fast LiteLLM's Rust rate limiter uses 42× less memory on high-cardinality workloads — the case where you have many unique API keys or users, each tracked separately. This is its strongest unique result. Memory is measured as steady-state RSS after running through 1,000+ unique keys. It is a drop-in: add import fast_litellm before import litellm, no code changes, with automatic Python fallback.

Both use Rust to speed up LiteLLM, and they take different shapes. LiteLLM ships its own Rust sidecar as a built-in option. Fast LiteLLM is an independent, alternative drop-in accelerator: you install it from PyPI and activate it with a single import fast_litellm line — no separate process to run. We publish honest, reproducible benchmarks for our own components (3.2× connection pooling, 1.6× rate limiting, 42× less memory at high cardinality) and we do not publish head-to-head numbers against the sidecar. Note the same honest caveat applies to any Rust layer here: for small-text tokenization and routing, the cost of crossing the Python↔Rust boundary (FFI overhead) can make Python faster. Pick based on your workload; both are legitimate choices.

When your workload is dominated by small-text tokenization or routing. For short messages, the cost of crossing the Python↔Rust FFI boundary dominates the actual work, so Python is faster — the author has been candid that bindings overhead often dominates in these cases. Fast LiteLLM detects small-text tokenization and prefers Python automatically, and routing acceleration is off by default. The clear wins are connection pooling, rate limiting, large-text tokenization, and high-cardinality memory usage; if none of those are your bottleneck, you may see little benefit.