# Serving DeepInfer + host prefix cache (local additions, not part of upstream)

## Pieces

- `serve/openai_server.py` — OpenAI-compatible `/v1/chat/completions` (streaming, `tools`, usage) on top of the
  offline engine: one engine process per GPU (attention DP + MoE EP), prompts rendered with the checkpoint's own
  `encoding.py`. `serve/server.sh start|stop` runs it as a process group (stop reaps the engine helpers, which
  otherwise keep the GPUs).
- `deepinfer/hicache/` — host-memory prefix cache (SGLang HiCache style): pinned host pools organized in pages
  (one page = one 128-token KV block) plus SWA state slots, hash-chain index, LRU eviction, write-through at the
  end of every prefill, "kernel" transfers (a CUDA kernel copying pages through UVA pointers on the engine
  stream, `page_copy.cu`, JIT-built on first use). Needs the read-only `Sequence.block_table` binding added in
  `csrc/engine/pybind_sequence.cpp` (rebuild `deepinfer._C` with `bash scripts/build.sh build`).
- `tests/hicache/test_host_cache.py` — cache logic against a fake engine (needs one GPU).
- `serve/check_prefix_cache.py` — A/B on a live server: each turn sent with `"prefix_cache": false` and with
  the cache, greedy; compares outputs and TTFT.

## Run

```bash
export DEEPINFER_ACCELERATOR=nvidia DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017
serve/server.sh start --dp 4 --port 8888                       # cache on: 32 GiB pages + 4096 states per rank
serve/server.sh start --dp 4 --port 8888 --hicache-size-gb 0   # cache off
python serve/check_prefix_cache.py --port 8888 --turns 3
curl -s localhost:8888/deepinfer/metrics | grep hicache         # per-rank hit/store counters
serve/server.sh stop
```

Knobs: `--hicache-size-gb`, `--hicache-state-slots`, `--routing affinity|least-loaded` (affinity keeps a
conversation's turns on one rank, keyed by the first user message; required for hits since the cache is
per rank). The cache requires decoder bounded replay (`--yoco-keep-tokens > 0`, the default 140).

EPLB expert rearrangement is off by default (`--eplb-interval 0`). Every rearrangement stalls all ranks for about
0.3 s, and with `--eplb-after-prefill 1` it fires after nearly every step under multi-turn traffic; enable it with
`--eplb-interval 30` only for decode-heavy workloads whose expert load is skewed.

## How a hit works

A new prompt's page hashes are matched from the start; the match is capped so that at least `keep_tokens`
tokens remain to be recomputed (the decoder SWA state is rebuilt from that suffix by bounded replay), then
snapped down to the nearest 128-aligned position that has a stored SWA state. The sequence is allocated up
front, pages and state are copied in on the engine stream, `num_cached_tokens` is set, and the sequence is
enqueued; the scheduler prefills only the suffix. Generated tokens are not stored at decode end (they are
stored when they come back as part of the next turn's prompt).

## AgentX

The server speaks what InferenceX's AgentX replay (aiperf, `--endpoint /v1/chat/completions --streaming
--use-server-token-count`) needs; `prefill-any-interval` matters there, since a single lane keeps most ranks idle.

## Prefill/decode split (`pd_server.py`)

`serve/pd_server.py` runs the same OpenAI-compatible frontend over two engines on two hosts: a prompt-only
engine (PE, `--role pe`, host prefix cache on) and a decode-only engine (CE, `--role ce`, user-facing).
Per request the CE frontend renders the prompt, picks a CE rank and POSTs `/prefill` to the PE; the PE
prefills, exports the sequence's KV blocks and SWA state (`deepinfer/hicache/pd_transfer.py`) plus the first
sampled token, pushes the bytes to the CE rank over TCP and the CE admits the sequence straight into decode
(`Scheduler.add_prompted_sequence`).

```bash
serve/pd_run.sh start <pe-host> <ce-host>          # PE HTTP 8889, CE HTTP 8888, CE KV receivers 8900+rank
python serve/check_pd.py --ref http://<single-engine-host>:8888 --pd http://<ce-host>:8888   # token-identical?
serve/pd_run.sh stop  <pe-host> <ce-host>
```

`serve/nccl_xtray_test.py` reports the transport and bandwidth NCCL picks between the two hosts.
Results on 4+4 GB300 vs a single 4-GPU tray: `doc/benchmark_gb300.md`.