Glossary
The LiteLLM, Rust, and performance vocabulary used throughout this site — defined plainly.
- Automatic fallback
- When a Rust path raises an exception, Fast LiteLLM transparently retries the same call on the original Python implementation, so acceleration can never break correctness. More →
- Circuit breaker
- A safety mechanism that disables a component after repeated failures. Fast LiteLLM disables a Rust component after 10 errors for the rest of the process, falling back to Python. More →
- Connection pool
- A reusable set of open HTTP connections to upstream providers. Under concurrent load, managing this pool in Python-with-locks becomes a bottleneck; Fast LiteLLM uses a lock-free DashMap pool (3.2× faster). More →
- DashMap
- A concurrent, sharded hash map for Rust that allows lock-free reads and fine-grained locking on writes. It backs Fast LiteLLM's connection pool.
- Drop-in acceleration layer
- A package that speeds up an existing library without changing that library's public API or your calling code. Fast LiteLLM activates with a single import and leaves the LiteLLM interface untouched. More →
- Feature flag / canary rollout
- Env-var controls that enable, disable, or percentage-roll-out each accelerated component (e.g. FAST_LITELLM_BATCH_TOKEN_COUNTING=canary:10) so you can adopt acceleration gradually. More →
- FFI overhead
- The cost of crossing the Python↔Rust foreign-function-interface boundary on each call. For very small operations (short-text tokenization, routing) this cost can exceed the work itself, which is why Fast LiteLLM keeps those on Python. More →
- GIL (Global Interpreter Lock)
- CPython's lock that lets only one thread execute Python bytecode at a time. It hurts most on contended, small, hot operations — exactly the profile Fast LiteLLM moves to Rust. More →
- High cardinality
- A workload with many distinct keys — e.g. thousands of unique API keys, users, or tenants each rate-limited separately. This is where Fast LiteLLM's Rust rate limiter uses ~42× less memory than the Python dict-based approach. More →
- LiteLLM
- An open-source Python library from BerriAI that gives a single, OpenAI-style interface to 100+ LLM providers, plus routing, rate limiting, cost tracking, and a proxy server. Fast LiteLLM accelerates its hot paths; it does not replace it. More →
- Monkeypatching
- Replacing functions or methods of an already-imported module at runtime. Fast LiteLLM monkeypatches LiteLLM's hot-path components at import time, which is why it must be imported first.
- PyO3
- The Rust binding framework used to expose Rust functions to Python (and vice-versa). Fast LiteLLM's Rust components are compiled into a Python-importable extension via PyO3.
- tiktoken-rs
- A Rust port of OpenAI's tiktoken byte-pair-encoding tokenizer. Fast LiteLLM uses it for accelerated large-text and batch tokenization.
- Token counting
- Estimating how many tokens a request will consume, used for cost estimation and context-window checks. Fast LiteLLM backs large-text tokenization with tiktoken-rs for a 1.5–1.7× speedup. More →