diff --git a/serve/README.md b/serve/README.md index ba46282..ba9add5 100644 --- a/serve/README.md +++ b/serve/README.md @@ -36,10 +36,18 @@ EPLB expert rearrangement is off by default (`--eplb-interval 0`). Every rearran the 32 redundant expert slots (`DEEPINFER_OVERRIDE_NUM_EXTRA_EXPERTS=0`): without rearrangement they are fixed copies of experts 0-31, and removing them speeds decode up by 4-5%. -`--prefill-any-interval` (default 2) trades TTFT for decode speed. A prompt step pauses decode on every DP rank, and -with 2 almost every arriving request triggers one. That keeps TTFT low; the engine default of 100 adds a ~1.4 s -floor. At high concurrency, 30-100 gives faster decode and more throughput at the cost of TTFT: on B200 AgentX c256, -interval 100 gave 255K vs 211K tok/s/GPU and interactivity p50 119 vs 55, with TTFT p50 2.5 vs 0.7 s. +`--prefill-any-interval` (default 30) trades TTFT for decode speed. A prompt step pauses decode on every DP rank, and +a rank with a waiting prompt forces one at most this many decode steps later. The default favours throughput. With 2, +almost every arriving request triggers a prompt step, which keeps TTFT low but slows decode at high concurrency. The +engine default of 100 adds a ~1.4 s TTFT floor. AgentX at c256 on one GB300 tray (20-minute runs): + +| interval | tok/s/GPU | TTFT p50 / p90 | ITL p50 / p90 | P90 interactivity | +|---|---|---|---|---| +| 2 | 258K | 1.26 / 4.53 s | 14.3 / 31.5 ms | 32 | +| 30 | 283K | 1.62 / 6.51 s | 10.0 / 14.1 ms | 71 | +| 100 | 276K | 3.00 / 14.95 s | 8.1 / 13.5 ms | 74 | + +B200 shows the same trend: at c256, interval 100 gave 255K vs 211K tok/s/GPU for interval 2. ## Prompt tokenization @@ -71,7 +79,8 @@ stored when they come back as part of the next turn's prompt). ## AgentX The server speaks what InferenceX's AgentX replay (aiperf, `--endpoint /v1/chat/completions --streaming ---use-server-token-count`) needs; `prefill-any-interval` matters there, since a single lane keeps most ranks idle. +--use-server-token-count`) needs. At low concurrency, use `--prefill-any-interval 2`: a single lane keeps most +ranks idle, and the default adds a few hundred ms of TTFT. ## Prefill/decode split (`pd_server.py`) diff --git a/serve/openai_server.py b/serve/openai_server.py index b11f94e..4bfd939 100644 --- a/serve/openai_server.py +++ b/serve/openai_server.py @@ -65,10 +65,10 @@ def parse_args() -> argparse.Namespace: p.add_argument( "--prefill-any-interval", type=int, - default=2, + default=30, help="DEEPINFER_PREFILL_ANY_ATTEMPT_INTERVAL: a rank with a waiting prompt gets a prefill step at most this " - "many decode steps later even if other ranks are idle (engine default 100 => ~1.4 s TTFT floor). A prompt step " - "pauses decode on every rank, so at high concurrency a larger value (30-100) trades TTFT for decode speed", + "many decode steps later even if other ranks are idle. A prompt step pauses decode on every rank, so a larger " + "value gives faster decode and more throughput at the cost of TTFT. 30 favours throughput; use 2 for low TTFT", ) p.add_argument( "--eplb-interval",