mirror of
https://github.com/deepseek-ai/DeepInfer.git
synced 2026-10-31 00:36:20 +08:00
4159328eb5
Hard-wired enable_eplb=True rearranged experts every 30 steps and after every prefill step; under multi-turn traffic that fires almost continuously and each rearrangement stalls all ranks for ~0.3 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TNVW4sqoaPqbNtSrgpA2zK
68 lines
4.1 KiB
Markdown
68 lines
4.1 KiB
Markdown
# Serving DeepInfer + host prefix cache (local additions, not part of upstream)
|
|
|
|
## Pieces
|
|
|
|
- `serve/openai_server.py` — OpenAI-compatible `/v1/chat/completions` (streaming, `tools`, usage) on top of the
|
|
offline engine: one engine process per GPU (attention DP + MoE EP), prompts rendered with the checkpoint's own
|
|
`encoding.py`. `serve/server.sh start|stop` runs it as a process group (stop reaps the engine helpers, which
|
|
otherwise keep the GPUs).
|
|
- `deepinfer/hicache/` — host-memory prefix cache (SGLang HiCache style): pinned host pools organized in pages
|
|
(one page = one 128-token KV block) plus SWA state slots, hash-chain index, LRU eviction, write-through at the
|
|
end of every prefill, "kernel" transfers (a CUDA kernel copying pages through UVA pointers on the engine
|
|
stream, `page_copy.cu`, JIT-built on first use). Needs the read-only `Sequence.block_table` binding added in
|
|
`csrc/engine/pybind_sequence.cpp` (rebuild `deepinfer._C` with `bash scripts/build.sh build`).
|
|
- `tests/hicache/test_host_cache.py` — cache logic against a fake engine (needs one GPU).
|
|
- `serve/check_prefix_cache.py` — A/B on a live server: each turn sent with `"prefix_cache": false` and with
|
|
the cache, greedy; compares outputs and TTFT.
|
|
|
|
## Run
|
|
|
|
```bash
|
|
export DEEPINFER_ACCELERATOR=nvidia DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017
|
|
serve/server.sh start --dp 4 --port 8888 # cache on: 32 GiB pages + 4096 states per rank
|
|
serve/server.sh start --dp 4 --port 8888 --hicache-size-gb 0 # cache off
|
|
python serve/check_prefix_cache.py --port 8888 --turns 3
|
|
curl -s localhost:8888/deepinfer/metrics | grep hicache # per-rank hit/store counters
|
|
serve/server.sh stop
|
|
```
|
|
|
|
Knobs: `--hicache-size-gb`, `--hicache-state-slots`, `--routing affinity|least-loaded` (affinity keeps a
|
|
conversation's turns on one rank, keyed by the first user message; required for hits since the cache is
|
|
per rank). The cache requires decoder bounded replay (`--yoco-keep-tokens > 0`, the default 140).
|
|
|
|
EPLB expert rearrangement is off by default (`--eplb-interval 0`). Every rearrangement stalls all ranks for about
|
|
0.3 s, and with `--eplb-after-prefill 1` it fires after nearly every step under multi-turn traffic; enable it with
|
|
`--eplb-interval 30` only for decode-heavy workloads whose expert load is skewed.
|
|
|
|
## How a hit works
|
|
|
|
A new prompt's page hashes are matched from the start; the match is capped so that at least `keep_tokens`
|
|
tokens remain to be recomputed (the decoder SWA state is rebuilt from that suffix by bounded replay), then
|
|
snapped down to the nearest 128-aligned position that has a stored SWA state. The sequence is allocated up
|
|
front, pages and state are copied in on the engine stream, `num_cached_tokens` is set, and the sequence is
|
|
enqueued; the scheduler prefills only the suffix. Generated tokens are not stored at decode end (they are
|
|
stored when they come back as part of the next turn's prompt).
|
|
|
|
## AgentX
|
|
|
|
The server speaks what InferenceX's AgentX replay (aiperf, `--endpoint /v1/chat/completions --streaming
|
|
--use-server-token-count`) needs; `prefill-any-interval` matters there, since a single lane keeps most ranks idle.
|
|
|
|
## Prefill/decode split (`pd_server.py`)
|
|
|
|
`serve/pd_server.py` runs the same OpenAI-compatible frontend over two engines on two hosts: a prompt-only
|
|
engine (PE, `--role pe`, host prefix cache on) and a decode-only engine (CE, `--role ce`, user-facing).
|
|
Per request the CE frontend renders the prompt, picks a CE rank and POSTs `/prefill` to the PE; the PE
|
|
prefills, exports the sequence's KV blocks and SWA state (`deepinfer/hicache/pd_transfer.py`) plus the first
|
|
sampled token, pushes the bytes to the CE rank over TCP and the CE admits the sequence straight into decode
|
|
(`Scheduler.add_prompted_sequence`).
|
|
|
|
```bash
|
|
serve/pd_run.sh start <pe-host> <ce-host> # PE HTTP 8889, CE HTTP 8888, CE KV receivers 8900+rank
|
|
python serve/check_pd.py --ref http://<single-engine-host>:8888 --pd http://<ce-host>:8888 # token-identical?
|
|
serve/pd_run.sh stop <pe-host> <ce-host>
|
|
```
|
|
|
|
`serve/nccl_xtray_test.py` reports the transport and bandwidth NCCL picks between the two hosts.
|
|
Results on 4+4 GB300 vs a single 4-GPU tray: `doc/benchmark_gb300.md`.
|