Want to read

TokTier: Exact Stateful Tokenization for Agentic LLM Serving

Artificial Intelligence Computer Science

TokTier addresses LLM serving inefficiencies by providing exact stateful tokenization for agentic workloads. It handles session continuations via incremental repair and new requests through GPU-accelerated full tokenization. The system guarantees output identity with reference tokenizers, achieving significant latency reductions and throughput improvements over existing stateless or cache-based baselines in production environments.

Read the paper

Key takeaways

TokTier manages two-mode workloads: frequent session continuations with small appends and rare full-context rebuilds.
Incremental repair processes appended text in 0.5 to 1.1 milliseconds, up to 437 times faster than Hugging Face tokenization.
GPU full tokenization encodes one million characters in 0.87 milliseconds, outperforming previous CPU methods by 23.4 times.
The system integrates with vLLM to lower median time to first token by 16 to 34 percent and P99 latency by 23 percent.
A four-core CPU repair pool combined with one GPU sustains 1,821 requests per second under a 50 millisecond P99 objective.
Validation campaigns across 17 tokenizer families and 12.4 terabytes of text confirm zero divergence from reference implementations.

Reflections

No notes on this one yet. I add reflections as I finish or revisit a paper.