coffee/DeepInfer
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepInfer.git synced 2026-10-30 15:57:20 +08:00
Code Issues Packages Projects Releases Wiki Activity

serve: EPLB rearrangement off by default, --eplb-interval / --eplb-after-prefill flags

Browse Source 
Hard-wired enable_eplb=True rearranged experts every 30 steps and after every
prefill step; under multi-turn traffic that fires almost continuously and
each rearrangement stalls all ranks for ~0.3 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TNVW4sqoaPqbNtSrgpA2zK
This commit is contained in:
Jiejing 2026-10-22 17:17:22 +00:00
parent aaf1e62c9e
commit d7844a221d
2 changed files with 24 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
serve/README.md
View File

@ -30,6 +30,10 @@ Knobs: `--hicache-size-gb`, `--hicache-state-slots`, `--routing affinity|least-l
conversation's turns on one rank, keyed by the first user message; required for hits since the cache is
per rank). The cache requires decoder bounded replay (`--yoco-keep-tokens > 0`, the default 140).
EPLB expert rearrangement is off by default (`--eplb-interval 0`). Every rearrangement stalls all ranks for about
0.3 s, and with `--eplb-after-prefill 1` it fires after nearly every step under multi-turn traffic; enable it with
`--eplb-interval 30` only for decode-heavy workloads whose expert load is skewed.
## How a hit works
A new prompt's page hashes are matched from the start; the match is capped so that at least `keep_tokens`

21
serve/openai_server.py
View File

@ -67,6 +67,22 @@ def parse_args() -> argparse.Namespace:
help="DEEPINFER_PREFILL_ANY_ATTEMPT_INTERVAL: a rank with a waiting prompt gets a prefill step at most this "
"many decode steps later even if other ranks are idle (engine default 100 => ~1.4 s TTFT floor)",
)
p.add_argument(
"--eplb-interval",
type=int,
default=0,
help="DEEPINFER_EXPERT_REARRANGEMENT_STEP_INTERVAL: rearrange experts every N steps; 0 (default) disables EPLB. "
"Each rearrangement stalls every rank (~0.3 s on B200/GB300), which under online traffic costs far more than "
"the balance it buys",
)
p.add_argument(
"--eplb-after-prefill",
type=int,
default=0,
choices=[0, 1],
help="DEEPINFER_REARRANGE_AFTER_PREFILL_IN_NORMAL_ENGINE: also rearrange after every prefill step (only with "
"--eplb-interval > 0); with agentic traffic nearly every step batch contains a prefill",
)
return p.parse_args()
@ -92,7 +108,10 @@ def worker_main(args: argparse.Namespace, dp_rank: int, in_q: mp.Queue, out_q: m
# 128-boundaries behind the end of a prompt (see HostPrefixCache.state_lookback_blocks).
os.environ.setdefault("DEEPINFER_PREFILL_STORE_EXTRA_SIZE", "512")
os.environ.setdefault("DEEPINFER_PREFILL_ANY_ATTEMPT_INTERVAL", str(args.prefill_any_interval))
setup_common_env(model_name=args.model, sampler_force_topk=1024, set_override_experts=True, enable_eplb=True)
setup_common_env(model_name=args.model, sampler_force_topk=1024, set_override_experts=True, enable_eplb=False)
if args.eplb_interval > 0:
os.environ["DEEPINFER_EXPERT_REARRANGEMENT_STEP_INTERVAL"] = str(args.eplb_interval)
os.environ["DEEPINFER_REARRANGE_AFTER_PREFILL_IN_NORMAL_ENGINE"] = str(args.eplb_after_prefill)
llm = create_llm(
args.model,