c6faea3bc2
serve/openai_server.py exposes /v1/chat/completions (streaming, tools, usage) on top of LLMEngine: one engine process per GPU (attention DP + MoE EP, the layout of examples/benchmark.py), prompts rendered with the checkpoint's own encoding.py, conversation-affinity routing, a watchdog that fails open requests if an engine rank dies, and /deepinfer/metrics. serve/server.sh runs it as a process group so a stop also reaps the engine's helper processes, which otherwise keep the GPUs. It integrates the host prefix cache from deepinfer/hicache when present (--hicache-size-gb, --hicache-state-slots, per-request "prefix_cache": false bypass; serve/check_prefix_cache.py is the A/B check) and serves without it otherwise. It also sets DEEPINFER_PREFILL_ANY_ATTEMPT_INTERVAL to 2 by default (--prefill-any-interval): with the engine default of 100, a rank whose peers are idle admits a waiting prompt only every 100 decode steps, a ~1.4 s TTFT floor on every request in a serving setting. Used to drive the InferenceX AgentX replay (aiperf) on 4x GB300. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TNVW4sqoaPqbNtSrgpA2zK |
||
|---|---|---|
| .. | ||
| check_prefix_cache.py | ||
| openai_server.py | ||
| README.md | ||
| server.sh | ||
Serving DeepInfer + host prefix cache (local additions, not part of upstream)
Pieces
serve/openai_server.py— OpenAI-compatible/v1/chat/completions(streaming,tools, usage) on top of the offline engine: one engine process per GPU (attention DP + MoE EP), prompts rendered with the checkpoint's ownencoding.py.serve/server.sh start|stopruns it as a process group (stop reaps the engine helpers, which otherwise keep the GPUs).deepinfer/hicache/— host-memory prefix cache (SGLang HiCache style): pinned host pools organized in pages (one page = one 128-token KV block) plus SWA state slots, hash-chain index, LRU eviction, write-through at the end of every prefill, "kernel" transfers (a CUDA kernel copying pages through UVA pointers on the engine stream,page_copy.cu, JIT-built on first use). Needs the read-onlySequence.block_tablebinding added incsrc/engine/pybind_sequence.cpp(rebuilddeepinfer._Cwithbash scripts/build.sh build).tests/hicache/test_host_cache.py— cache logic against a fake engine (needs one GPU).serve/check_prefix_cache.py— A/B on a live server: each turn sent with"prefix_cache": falseand with the cache, greedy; compares outputs and TTFT.
Run
export DEEPINFER_ACCELERATOR=nvidia DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017
serve/server.sh start --dp 4 --port 8888 # cache on: 32 GiB pages + 4096 states per rank
serve/server.sh start --dp 4 --port 8888 --hicache-size-gb 0 # cache off
python serve/check_prefix_cache.py --port 8888 --turns 3
curl -s localhost:8888/deepinfer/metrics | grep hicache # per-rank hit/store counters
serve/server.sh stop
Knobs: --hicache-size-gb, --hicache-state-slots, --routing affinity|least-loaded (affinity keeps a
conversation's turns on one rank, keyed by the first user message; required for hits since the cache is
per rank). The cache requires decoder bounded replay (--yoco-keep-tokens > 0, the default 140).
How a hit works
A new prompt's page hashes are matched from the start; the match is capped so that at least keep_tokens
tokens remain to be recomputed (the decoder SWA state is rebuilt from that suffix by bounded replay), then
snapped down to the nearest 128-aligned position that has a stored SWA state. The sequence is allocated up
front, pages and state are copied in on the engine stream, num_cached_tokens is set, and the sequence is
enqueued; the scheduler prefills only the suffix. Generated tokens are not stored at decode end (they are
stored when they come back as part of the next turn's prompt).
AgentX
The server speaks what InferenceX's AgentX replay (aiperf, --endpoint /v1/chat/completions --streaming --use-server-token-count) needs; prefill-any-interval matters there, since a single lane keeps most ranks idle.