Long-document & batch processing
For: Teams building RAG ingestion or batch LLM pipelines
Pipelines that tokenize long documents for cost estimation and context checks spend real CPU on tokenization. Fast LiteLLM backs large-text tokenization with tiktoken-rs for a 1.5–1.7× speedup, and keeps short inputs on Python where FFI overhead would otherwise dominate.
The problem
- →Pre-flight tokenization of long documents adds latency
- →Batch jobs spend measurable CPU in tokenization
- →Naive Rust everywhere would slow down short inputs
How Fast LiteLLM helps
tiktoken-rs for big texts
Large-text and batch tokenization run 1.5–1.7× faster in Rust.
Automatic small-input routing
Fast LiteLLM detects small texts and keeps them on Python, avoiding FFI overhead.
No API change
Tokenization calls stay identical; only the implementation underneath changes.
Honest caveat: For workloads dominated by many short strings, the win is small — this use case is about large texts and batches.
See if it fits your workload
Open source, MIT, one import line. Reproduce the benchmarks on your own traffic before you commit.