Jiejing 269a61f7c5 serve: exact fast prompt tokenization (fastokens or added-token segment cache), checked against tokenizer.encode ... Agentic prompts are re-tokenized in full every turn; tokenizer.encode takes ~280 ms p50 on AgentX prompts (p50 163K tokens) and dominates TTFT once the prefix cache hits. serve/prompt_tokenizer.py picks fastokens if installed, else a dependency-free segment cache that splits at added tokens (where HF already encodes pieces independently) and reuses earlier turns' pieces; both are enabled only after matching tokenizer.encode on startup probes, and a sampled background re-check switches to hf on any mismatch. Suggested by whn09 in review. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YNgkNrJL5oR1pZqVYzZMHW		2026-10-23 03:50:06 +00:00
..
__init__.py	serve: exact fast prompt tokenization (fastokens or added-token segment cache), checked against tokenizer.encode	2026-10-23 03:50:06 +00:00
test_prompt_tokenizer.py	serve: exact fast prompt tokenization (fastokens or added-token segment cache), checked against tokenizer.encode	2026-10-23 03:50:06 +00:00