serve/openai_server.py exposes /v1/chat/completions (streaming, tools,
usage) on top of LLMEngine: one engine process per GPU (attention DP + MoE
EP, the layout of examples/benchmark.py), prompts rendered with the
checkpoint's own encoding.py, conversation-affinity routing, a watchdog
that fails open requests if an engine rank dies, and /deepinfer/metrics.
serve/server.sh runs it as a process group so a stop also reaps the
engine's helper processes, which otherwise keep the GPUs.
It integrates the host prefix cache from deepinfer/hicache when present
(--hicache-size-gb, --hicache-state-slots, per-request "prefix_cache":
false bypass; serve/check_prefix_cache.py is the A/B check) and serves
without it otherwise. It also sets DEEPINFER_PREFILL_ANY_ATTEMPT_INTERVAL
to 2 by default (--prefill-any-interval): with the engine default of 100,
a rank whose peers are idle admits a waiting prompt only every 100 decode
steps, a ~1.4 s TTFT floor on every request in a serving setting.
Used to drive the InferenceX AgentX replay (aiperf) on 4x GB300.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TNVW4sqoaPqbNtSrgpA2zK
The open-source engine has no KV reuse across requests, so every turn of a
multi-turn conversation prefills its whole prompt. This adds deepinfer/hicache:
- Pinned host pools organized in pages (one page = one 128-token KV block)
plus slots for the SWA state that the production KV store persists
(select_state_slice_to_store / load_state_from_store_to_prefill_cache).
Pages are indexed by a hash chain over the prompt, LRU evicted, written
through at the end of every prefill (engine hook pe_do_save_kvcache).
- Transfers use one small CUDA kernel copying pages between GPU and pinned
host memory through UVA pointers on the engine stream ("kernel" I/O
method, ~180 GiB/s on GB300); completion is tracked with CUDA events.
- A hit is capped so that at least yoco keep_tokens remain to recompute
(bounded replay rebuilds the decoder SWA state from that suffix), then
snapped to the nearest 128-aligned position with a stored state; states
are stored at the last few boundaries of a prompt so short follow-ups hit.
The sequence is pre-allocated, filled from host and scheduled with
num_cached_tokens set (HostPrefixCache.try_resume).
- Engine: read-only Sequence.block_table binding (to copy a finished
prefill's blocks out) and its stub.
Validated on 4x GB300 with DeepSeek-V4.1-Flash through an OpenAI-compatible
server (separate PR): greedy outputs byte-identical with and without the
cache over 8 multi-turn requests; InferenceX AgentX replay at concurrency 1
served 224 requests per hour vs 99, TTFT p50 0.59 s vs 8.9 s, with 89-96%
of prompt tokens served from the cache. tests/hicache/test_host_cache.py
covers the cache logic against a fake engine.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TNVW4sqoaPqbNtSrgpA2zK
DeepEP now records an HTTPS DeepJIT URL, so build_vendor no longer needs the git@github.com rewrite.
Co-authored-by: Shaoyuan Chen <chensy@deepseek.com>
