Merge remote-tracking branch 'fork/openai-server' into pd-split-fix

# Conflicts:
#	serve/README.md

serve/README.md

@@ -36,10 +36,18 @@ EPLB expert rearrangement is off by default (`--eplb-interval 0`). Every rearran
 the 32 redundant expert slots (`DEEPINFER_OVERRIDE_NUM_EXTRA_EXPERTS=0`): without rearrangement they are fixed copies
 of experts 0-31, and removing them speeds decode up by 4-5%.
 
-`--prefill-any-interval` (default 2) trades TTFT for decode speed. A prompt step pauses decode on every DP rank, and
-with 2 almost every arriving request triggers one. That keeps TTFT low; the engine default of 100 adds a ~1.4 s
-floor. At high concurrency, 30-100 gives faster decode and more throughput at the cost of TTFT: on B200 AgentX c256,
-interval 100 gave 255K vs 211K tok/s/GPU and interactivity p50 119 vs 55, with TTFT p50 2.5 vs 0.7 s.
+`--prefill-any-interval` (default 30) trades TTFT for decode speed. A prompt step pauses decode on every DP rank, and
+a rank with a waiting prompt forces one at most this many decode steps later. The default favours throughput. With 2,
+almost every arriving request triggers a prompt step, which keeps TTFT low but slows decode at high concurrency. The
+engine default of 100 adds a ~1.4 s TTFT floor. AgentX at c256 on one GB300 tray (20-minute runs):
+
+| interval | tok/s/GPU | TTFT p50 / p90 | ITL p50 / p90 | P90 interactivity |
+|---|---|---|---|---|
+| 2 | 258K | 1.26 / 4.53 s | 14.3 / 31.5 ms | 32 |
+| 30 | 283K | 1.62 / 6.51 s | 10.0 / 14.1 ms | 71 |
+| 100 | 276K | 3.00 / 14.95 s | 8.1 / 13.5 ms | 74 |
+
+B200 shows the same trend: at c256, interval 100 gave 255K vs 211K tok/s/GPU for interval 2.
 
 ## Prompt tokenization
 
@@ -71,7 +79,8 @@ stored when they come back as part of the next turn's prompt).
 ## AgentX
 
 The server speaks what InferenceX's AgentX replay (aiperf, `--endpoint /v1/chat/completions --streaming
---use-server-token-count`) needs; `prefill-any-interval` matters there, since a single lane keeps most ranks idle.
+--use-server-token-count`) needs. At low concurrency, use `--prefill-any-interval 2`: a single lane keeps most
+ranks idle, and the default adds a few hundred ms of TTFT.
 
 ## Prefill/decode split (`pd_server.py`)
 

serve/openai_server.py

@@ -65,10 +65,10 @@ def parse_args() -> argparse.Namespace:
     p.add_argument(
         "--prefill-any-interval",
         type=int,
-        default=2,
+        default=30,
         help="DEEPINFER_PREFILL_ANY_ATTEMPT_INTERVAL: a rank with a waiting prompt gets a prefill step at most this "
-        "many decode steps later even if other ranks are idle (engine default 100 => ~1.4 s TTFT floor). A prompt step "
-        "pauses decode on every rank, so at high concurrency a larger value (30-100) trades TTFT for decode speed",
+        "many decode steps later even if other ranks are idle. A prompt step pauses decode on every rank, so a larger "
+        "value gives faster decode and more throughput at the cost of TTFT. 30 favours throughput; use 2 for low TTFT",
     )
     p.add_argument(
         "--eplb-interval",