At AgentX c256 on one GB300 tray, 30 vs 2 gives +9% tok/s/GPU and 2.2x P90
interactivity, for +0.4 s TTFT p50. Use 2 for low TTFT.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YNgkNrJL5oR1pZqVYzZMHW
Without rearrangement they are fixed copies of experts 0-31 and only add expert weight per rank.
Decode SPS +4.8% / +3.9% at 4 / 16 requests per rank on GB300 (examples/benchmark.py, 128K context),
matching whn09's B200 numbers; 6 GiB less model memory per GPU.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YNgkNrJL5oR1pZqVYzZMHW
Agentic prompts are re-tokenized in full every turn; tokenizer.encode takes ~280 ms p50 on AgentX prompts
(p50 163K tokens) and dominates TTFT once the prefix cache hits. serve/prompt_tokenizer.py picks fastokens if
installed, else a dependency-free segment cache that splits at added tokens (where HF already encodes pieces
independently) and reuses earlier turns' pieces; both are enabled only after matching tokenizer.encode on
startup probes, and a sampled background re-check switches to hf on any mismatch.
Suggested by whn09 in review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YNgkNrJL5oR1pZqVYzZMHW
Hard-wired enable_eplb=True rearranged experts every 30 steps and after every
prefill step; under multi-turn traffic that fires almost continuously and
each rearrangement stalls all ranks for ~0.3 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TNVW4sqoaPqbNtSrgpA2zK
Hard-wired enable_eplb=True rearranged experts every 30 steps and after every
prefill step; under multi-turn traffic that fires almost continuously and
each rearrangement stalls all ranks for ~0.3 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TNVW4sqoaPqbNtSrgpA2zK
A prompt-only engine tray (PE) and a decode-only engine tray (CE)
behind the OpenAI-compatible frontend from serve/openai_server.py:
- serve/pd_server.py: --role pe / --role ce. The CE frontend renders
the prompt, picks a CE rank (affinity or least-loaded routing) and
POSTs /prefill to the PE. The PE prefills with the host prefix cache
on, exports the prompt's KV blocks and the full SWA state in the
pe_do_save_kvcache hook, streams the first sampled token back in the
HTTP response and pushes the KV bytes to the CE rank over TCP. The CE
rank admits the sequence into decode via Scheduler.add_prompted_sequence
and reclaims PE sequences through Scheduler.free_finished_seqs.
- deepinfer/hicache/pd_transfer.py: export/admit of KV pages and state
through the hicache page-copy kernels. The CE keeps the pinned receive
buffers alive until the admit copy kernels finish (the kernels read
them by raw pointer; the caching host allocator otherwise reused the
memory while the copy was queued -> cudaErrorLaunchFailure).
- serve/pd_run.sh start/stop/logs over ssh; serve/check_pd.py compares
outputs against a single-engine server token by token;
serve/nccl_xtray_test.py reports the cross-host NCCL transport.
Measured on 4 GB300 (PE) + 4 GB300 (CE) against one 4-GPU tray with
the AgentX replay, outputs identical: ITL p90 stays within 8.6-15 ms
from conc 8 to 256 (single tray 12-98 ms), P90 interactivity 66-117
tok/s/user where the single tray falls to 10; per total chip PD is
behind up to conc 64 and ahead at 256 (doc/benchmark_gb300.md).
Depends on the host prefix cache (#144), the OpenAI server (#147) and
the scheduler bindings (add_prompted_sequence / free_finished_seqs),
which are included as the first three commits.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LFPG29daR1FhchJBTjiEKr
Two bindings a prefill/decode split needs from the scheduler:
- add_prompted_sequence(seq): put a sequence whose KV blocks and state
were filled by the caller (received from a prompt-only engine)
straight into the prompted queue, without cloning. The sequence holds
prompt + first generated token, with everything but the last token
computed; the checks assert that layout.
- free_finished_seqs(): expose the existing helper that moves finished
sequences out of running and returns them without freeing blocks or
state, so a prompt-only engine can export the prefilled KV first and
free the sequence itself afterwards.
No scheduling behaviour changes for engines that do not call them.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LFPG29daR1FhchJBTjiEKr
serve/openai_server.py exposes /v1/chat/completions (streaming, tools,
usage) on top of LLMEngine: one engine process per GPU (attention DP + MoE
EP, the layout of examples/benchmark.py), prompts rendered with the
checkpoint's own encoding.py, conversation-affinity routing, a watchdog
that fails open requests if an engine rank dies, and /deepinfer/metrics.
serve/server.sh runs it as a process group so a stop also reaps the
engine's helper processes, which otherwise keep the GPUs.
It integrates the host prefix cache from deepinfer/hicache when present
(--hicache-size-gb, --hicache-state-slots, per-request "prefix_cache":
false bypass; serve/check_prefix_cache.py is the A/B check) and serves
without it otherwise. It also sets DEEPINFER_PREFILL_ANY_ATTEMPT_INTERVAL
to 2 by default (--prefill-any-interval): with the engine default of 100,
a rank whose peers are idle admits a waiting prompt only every 100 decode
steps, a ~1.4 s TTFT floor on every request in a serving setting.
Used to drive the InferenceX AgentX replay (aiperf) on 4x GB300.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TNVW4sqoaPqbNtSrgpA2zK
The open-source engine has no KV reuse across requests, so every turn of a
multi-turn conversation prefills its whole prompt. This adds deepinfer/hicache:
- Pinned host pools organized in pages (one page = one 128-token KV block)
plus slots for the SWA state that the production KV store persists
(select_state_slice_to_store / load_state_from_store_to_prefill_cache).
Pages are indexed by a hash chain over the prompt, LRU evicted, written
through at the end of every prefill (engine hook pe_do_save_kvcache).
- Transfers use one small CUDA kernel copying pages between GPU and pinned
host memory through UVA pointers on the engine stream ("kernel" I/O
method, ~180 GiB/s on GB300); completion is tracked with CUDA events.
- A hit is capped so that at least yoco keep_tokens remain to recompute
(bounded replay rebuilds the decoder SWA state from that suffix), then
snapped to the nearest 128-aligned position with a stored state; states
are stored at the last few boundaries of a prompt so short follow-ups hit.
The sequence is pre-allocated, filled from host and scheduled with
num_cached_tokens set (HostPrefixCache.try_resume).
- Engine: read-only Sequence.block_table binding (to copy a finished
prefill's blocks out) and its stub.
Validated on 4x GB300 with DeepSeek-V4.1-Flash through an OpenAI-compatible
server (separate PR): greedy outputs byte-identical with and without the
cache over 8 multi-turn requests; InferenceX AgentX replay at concurrency 1
served 224 requests per hour vs 99, TTFT p50 0.59 s vs 8.9 s, with 89-96%
of prompt tokens served from the cache. tests/hicache/test_host_cache.py
covers the cache logic against a fake engine.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TNVW4sqoaPqbNtSrgpA2zK
DeepEP now records an HTTPS DeepJIT URL, so build_vendor no longer needs the git@github.com rewrite.
Co-authored-by: Shaoyuan Chen <chensy@deepseek.com>
serve/openai_server.py exposes /v1/chat/completions (streaming, tools,
usage) on top of LLMEngine: one engine process per GPU (attention DP + MoE
EP, the layout of examples/benchmark.py), prompts rendered with the
checkpoint's own encoding.py, conversation-affinity routing, a watchdog
that fails open requests if an engine rank dies, and /deepinfer/metrics.
serve/server.sh runs it as a process group so a stop also reaps the
engine's helper processes, which otherwise keep the GPUs.
It integrates the host prefix cache from deepinfer/hicache when present
(--hicache-size-gb, --hicache-state-slots, per-request "prefix_cache":
false bypass; serve/check_prefix_cache.py is the A/B check) and serves
without it otherwise. It also sets DEEPINFER_PREFILL_ANY_ATTEMPT_INTERVAL
to 2 by default (--prefill-any-interval): with the engine default of 100,
a rank whose peers are idle admits a waiting prompt only every 100 decode
steps, a ~1.4 s TTFT floor on every request in a serving setting.
Used to drive the InferenceX AgentX replay (aiperf) on 4x GB300.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TNVW4sqoaPqbNtSrgpA2zK
The open-source engine has no KV reuse across requests, so every turn of a
multi-turn conversation prefills its whole prompt. This adds deepinfer/hicache:
- Pinned host pools organized in pages (one page = one 128-token KV block)
plus slots for the SWA state that the production KV store persists
(select_state_slice_to_store / load_state_from_store_to_prefill_cache).
Pages are indexed by a hash chain over the prompt, LRU evicted, written
through at the end of every prefill (engine hook pe_do_save_kvcache).
- Transfers use one small CUDA kernel copying pages between GPU and pinned
host memory through UVA pointers on the engine stream ("kernel" I/O
method, ~180 GiB/s on GB300); completion is tracked with CUDA events.
- A hit is capped so that at least yoco keep_tokens remain to recompute
(bounded replay rebuilds the decoder SWA state from that suffix), then
snapped to the nearest 128-aligned position with a stored state; states
are stored at the last few boundaries of a prompt so short follow-ups hit.
The sequence is pre-allocated, filled from host and scheduled with
num_cached_tokens set (HostPrefixCache.try_resume).
- Engine: read-only Sequence.block_table binding (to copy a finished
prefill's blocks out) and its stub.
Validated on 4x GB300 with DeepSeek-V4.1-Flash through an OpenAI-compatible
server (separate PR): greedy outputs byte-identical with and without the
cache over 8 multi-turn requests; InferenceX AgentX replay at concurrency 1
served 224 requests per hour vs 99, TTFT p50 0.59 s vs 8.9 s, with 89-96%
of prompt tokens served from the cache. tests/hicache/test_host_cache.py
covers the cache logic against a fake engine.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TNVW4sqoaPqbNtSrgpA2zK
