Jiejing 269a61f7c5 serve: exact fast prompt tokenization (fastokens or added-token segment cache), checked against tokenizer.encode ...

Agentic prompts are re-tokenized in full every turn; tokenizer.encode takes ~280 ms p50 on AgentX prompts
(p50 163K tokens) and dominates TTFT once the prefix cache hits. serve/prompt_tokenizer.py picks fastokens if
installed, else a dependency-free segment cache that splits at added tokens (where HF already encodes pieces
independently) and reuses earlier turns' pieces; both are enabled only after matching tokenizer.encode on
startup probes, and a sampled background re-check switches to hf on any mismatch.

Suggested by whn09 in review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YNgkNrJL5oR1pZqVYzZMHW

2026-10-23 03:50:06 +00:00

0 lines

0 B

Python

Raw Blame History

The file is empty.