Jahidul Hossain Mekat e25a404e2c Don't let a preempted request block a chunked prefill that holds blocks ... A request that is preempted for recompute was pushed to the front of `waiting`. _schedule_prefill takes `waiting` in order and stops at the first sequence it cannot allocate, and a preempted request usually does not fit right away: it was preempted because it needed a block more than it had, and recomputing it takes that block as well. If a chunked prefill is in progress at that moment (it is back at the front of the queue holding the KV blocks of its whole prompt), the preempted request lands in front of it and ends the scan before the prefill is reached. The prefill never gets its next chunk, so it never finishes and never releases the blocks the preempted request is waiting for, and the engine stops making progress (or keeps recomputing the requests that are left, which are preempted again for the same reason). Decode steps between two chunks of a prompt are all it takes. They happen when the cache usage is above the prefill watermark, and on a data-parallel rank whenever the other ranks do not agree to a prompt step. Queue the preempted request ahead of the fresh requests, as before, but behind the sequences that already hold blocks. They only need compute to finish. A sequence that cannot be scheduled yet (a multimodal request that is still being preprocessed, but already owns its blocks as in examples/benchmark.py) is not one of them: _schedule_prefill stops at it, so the preempted request stays in front of it.		2026-10-20 14:49:40 +00:00
..
engine	Don't let a preempted request block a chunked prefill that holds blocks	2026-10-20 14:49:40 +00:00
graph_timer	initial commit	2026-10-20 12:09:42 +08:00
utils	initial commit	2026-10-20 12:09:42 +08:00
asc_utils.cpp	initial commit	2026-10-20 12:09:42 +08:00
cuda_utils.cpp	initial commit	2026-10-20 12:09:42 +08:00
cuda_utils.hpp	initial commit	2026-10-20 12:09:42 +08:00
pybind.cpp	initial commit	2026-10-20 12:09:42 +08:00