3 Commits

Author SHA1 Message Date
Jahidul Hossain Mekat
b6ddc56a0c
Release the KV blocks and state slot of requests aborted while waiting
A request that is aborted (or fails multimodal processing / structured output
setup) while it sits in `waiting` is popped by the scheduler and handed back as
an ignored sequence. Nothing released what it still held. A long prompt that is
prefilled in chunks is moved back to `waiting` after each chunk with the KV
blocks of the whole prompt (and its state slot) already allocated, so aborting
it between two chunks leaked those blocks for the lifetime of the engine, and
so did a fork that pinned its parent's blocks. Every such abort shrinks the
cache until the engine can no longer admit new requests.

_schedule_prefill and the waiting sweep in _schedule_decode now release the
sequence's resources when they drop it.

free_seq_nolock also reset nothing after freeing the state slot, so releasing a
sequence a second time (a preempted request that is aborted had already given
its slot back) freed the slot twice and tripped the block manager's
"ref_count corrupted" check. It now clears state_id, which makes the release
idempotent like the block table already was.

Tests drive the real scheduler through schedule / free / process_outputs on a
CPU process group. A decode step in the middle of a prefill is forced through
decode_step, which is what a data-parallel rank sees when the other ranks do
not agree to a prompt step.
2026-10-20 14:49:40 +00:00
Jiashi Li
8b16149741 update Q&A documentation 2026-10-20 17:06:57 +08:00
Jiashi Li
d585e6542c initial commit 2026-10-20 12:09:42 +08:00