coffee/DeepInfer
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepInfer.git synced 2026-10-30 07:47:10 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge openai-server (#147) into pd-split: no redundant experts with EPLB off

Browse Source
This commit is contained in:
Jiejing 2026-10-25 21:36:07 +00:00
commit 754bcb5b43
2 changed files with 7 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
serve/README.md
View File

@ -32,7 +32,9 @@ per rank). The cache requires decoder bounded replay (`--yoco-keep-tokens > 0`,
EPLB expert rearrangement is off by default (`--eplb-interval 0`). Every rearrangement stalls all ranks for about
0.3 s, and with `--eplb-after-prefill 1` it fires after nearly every step under multi-turn traffic; enable it with
`--eplb-interval 30` only for decode-heavy workloads whose expert load is skewed.
`--eplb-interval 30` only for decode-heavy workloads whose expert load is skewed. With EPLB off the server also drops
the 32 redundant expert slots (`DEEPINFER_OVERRIDE_NUM_EXTRA_EXPERTS=0`): without rearrangement they are fixed copies
of experts 0-31, and removing them speeds decode up by 4-5%.
## Prompt tokenization

4
serve/openai_server.py
View File

@ -123,6 +123,10 @@ def worker_main(args: argparse.Namespace, dp_rank: int, in_q: mp.Queue, out_q: m
# 128-boundaries behind the end of a prompt (see HostPrefixCache.state_lookback_blocks).
os.environ.setdefault("DEEPINFER_PREFILL_STORE_EXTRA_SIZE", "512")
os.environ.setdefault("DEEPINFER_PREFILL_ANY_ATTEMPT_INTERVAL", str(args.prefill_any_interval))
if args.eplb_interval == 0:
# The 32 redundant expert slots only pay off with EPLB rearrangement; without it they are fixed copies of
# experts 0-31 and only add expert weight to read per rank (decode +4-5% without them on B200 and GB300).
os.environ.setdefault("DEEPINFER_OVERRIDE_NUM_EXTRA_EXPERTS", "0")
setup_common_env(model_name=args.model, sampler_force_topk=1024, set_override_experts=True, enable_eplb=False)
if args.eplb_interval > 0:
os.environ["DEEPINFER_EXPERT_REARRANGEMENT_STEP_INTERVAL"] = str(args.eplb_interval)