At AgentX c256 on one GB300 tray, 30 vs 2 gives +9% tok/s/GPU and 2.2x P90
interactivity, for +0.4 s TTFT p50. Use 2 for low TTFT.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YNgkNrJL5oR1pZqVYzZMHW
Without rearrangement they are fixed copies of experts 0-31 and only add expert weight per rank.
Decode SPS +4.8% / +3.9% at 4 / 16 requests per rank on GB300 (examples/benchmark.py, 128K context),
matching whn09's B200 numbers; 6 GiB less model memory per GPU.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YNgkNrJL5oR1pZqVYzZMHW
Agentic prompts are re-tokenized in full every turn; tokenizer.encode takes ~280 ms p50 on AgentX prompts
(p50 163K tokens) and dominates TTFT once the prefix cache hits. serve/prompt_tokenizer.py picks fastokens if
installed, else a dependency-free segment cache that splits at added tokens (where HF already encodes pieces
independently) and reuses earlier turns' pieces; both are enabled only after matching tokenizer.encode on
startup probes, and a sampled background re-check switches to hf on any mismatch.
Suggested by whn09 in review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YNgkNrJL5oR1pZqVYzZMHW
Hard-wired enable_eplb=True rearranged experts every 30 steps and after every
prefill step; under multi-turn traffic that fires almost continuously and
each rearrangement stalls all ranks for ~0.3 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TNVW4sqoaPqbNtSrgpA2zK
DeepEP now records an HTTPS DeepJIT URL, so build_vendor no longer needs the git@github.com rewrite.
Co-authored-by: Shaoyuan Chen <chensy@deepseek.com>
serve/openai_server.py exposes /v1/chat/completions (streaming, tools,
usage) on top of LLMEngine: one engine process per GPU (attention DP + MoE
EP, the layout of examples/benchmark.py), prompts rendered with the
checkpoint's own encoding.py, conversation-affinity routing, a watchdog
that fails open requests if an engine rank dies, and /deepinfer/metrics.
serve/server.sh runs it as a process group so a stop also reaps the
engine's helper processes, which otherwise keep the GPUs.
It integrates the host prefix cache from deepinfer/hicache when present
(--hicache-size-gb, --hicache-state-slots, per-request "prefix_cache":
false bypass; serve/check_prefix_cache.py is the A/B check) and serves
without it otherwise. It also sets DEEPINFER_PREFILL_ANY_ATTEMPT_INTERVAL
to 2 by default (--prefill-any-interval): with the engine default of 100,
a rank whose peers are idle admits a waiting prompt only every 100 decode
steps, a ~1.4 s TTFT floor on every request in a serving setting.
Used to drive the InferenceX AgentX replay (aiperf) on 4x GB300.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TNVW4sqoaPqbNtSrgpA2zK
