coffee/DeepInfer
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepInfer.git synced 2026-10-30 07:47:10 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'fork/openai-server' into pd-split-fix

Browse Source
This commit is contained in:
Jiejing 2026-10-25 21:37:48 +00:00
commit 823b798934
2 changed files with 7 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
serve/README.md
View File

@ -36,6 +36,11 @@ EPLB expert rearrangement is off by default (`--eplb-interval 0`). Every rearran
the 32 redundant expert slots (`DEEPINFER_OVERRIDE_NUM_EXTRA_EXPERTS=0`): without rearrangement they are fixed copies
of experts 0-31, and removing them speeds decode up by 4-5%.
`--prefill-any-interval` (default 2) trades TTFT for decode speed. A prompt step pauses decode on every DP rank, and
with 2 almost every arriving request triggers one. That keeps TTFT low; the engine default of 100 adds a ~1.4 s
floor. At high concurrency, 30-100 gives faster decode and more throughput at the cost of TTFT: on B200 AgentX c256,
interval 100 gave 255K vs 211K tok/s/GPU and interactivity p50 119 vs 55, with TTFT p50 2.5 vs 0.7 s.
## Prompt tokenization
Agentic clients resend the whole conversation every turn, and `tokenizer.encode` needs ~1.7 ms per 1K tokens on a

3
serve/openai_server.py
View File

@ -67,7 +67,8 @@ def parse_args() -> argparse.Namespace:
type=int,
default=2,
help="DEEPINFER_PREFILL_ANY_ATTEMPT_INTERVAL: a rank with a waiting prompt gets a prefill step at most this "
"many decode steps later even if other ranks are idle (engine default 100 => ~1.4 s TTFT floor)",
"many decode steps later even if other ranks are idle (engine default 100 => ~1.4 s TTFT floor). A prompt step "
"pauses decode on every rank, so at high concurrency a larger value (30-100) trades TTFT for decode speed",
)
p.add_argument(
"--eplb-interval",