Jahidul Hossain Mekat 6a934611d5 Let a chunked prefill continue above the prefill watermark ... A prompt that is prefilled in chunks reserves the KV blocks of its whole prompt with its first chunk and goes back to `waiting` until its next chunk. NORMAL engines only try a prompt step while the cache usage is at most GPU_CACHE_PREFILL_WATERMARK (0.9), which is meant to keep fresh prompts from filling the cache. The blocks the prompt reserved count in that usage, and only its remaining chunks can release them, so it was never scheduled again as long as they alone kept the usage above the watermark. A prompt that needs more than 90% of the cache by itself (a long context on a cache that barely holds it) left the engine idle for good, with the request unfinished. Skip the watermark when the request at the front of `waiting` is a chunked prefill in progress. It needs no new blocks. The test prefills a 37-token prompt that takes all 10 usable blocks of a small cache.		2026-10-20 14:49:40 +00:00
..
distributed	initial commit	2026-10-20 12:09:42 +08:00
engine	Let a chunked prefill continue above the prefill watermark	2026-10-20 14:49:40 +00:00
models	initial commit	2026-10-20 12:09:42 +08:00
ve	initial commit	2026-10-20 12:09:42 +08:00
worker	initial commit	2026-10-20 12:09:42 +08:00
__init__.py	initial commit	2026-10-20 12:09:42 +08:00
conftest.py	initial commit	2026-10-20 12:09:42 +08:00
test_cuda_sync_guards.py	initial commit	2026-10-20 12:09:42 +08:00