Sporadic EFA transfer errors (~0.015% of puts with Mooncake 83f8f877) tripped the breaker on the first failure
and dropped every queued put for 30 s. Retry the failed objects once (puts and gets); only a failure that persists
counts and trips the breaker. New counters: l3_put_retried, l3_get_retried.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Store transfer that cannot complete blocks for Mooncake's fixed 60 s timeout. Open a breaker on the first
transfer failure or a get slower than DEEPINFER_HICACHE_L3_SLOW_S (5 s); while open, skip reads and drop queued
puts; close after DEEPINFER_HICACHE_L3_COOLDOWN_S (30 s, doubling on consecutive trips, capped at 10 min).
Mooncake <= 0.3.13 queues at most 1024 TCP transfers per peer, fewer than one long prompt's batch put; default
MC_TCP_MAX_QUEUED_TRANSFERS_PER_PEER / MC_TCP_MAX_PENDING_ADMISSIONS_PER_PEER to 65535 (the default since
kvcache-ai/Mooncake#3710) unless set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stored page groups (DEEPINFER_HICACHE_L3_GROUP_PAGES consecutive pages, default 4) and SWA
states are written behind to a Mooncake Store straight from the registered pinned pools
(batch_put_from_multi_buffers on a background thread; host slots stay pinned until the put
returns). A host miss is extended from the Store: batch_is_exist finds the longest group run
and the deepest boundary with a state (host or Store), batch_get_into_multi_buffers fills fresh
host slots, and the existing H2D path runs unchanged. Keys are namespaced and carry a versioned
layout fingerprint (pages under dsv4kv/<fp>/g<G>/, states under state/<fp>/, as discussed in
kvcache-ai/Mooncake#4803). Pages are rank-independent, so prefixes are shared across DP ranks,
instances and restarts. Any Store error falls back to the host-only path.
tests/hicache/test_host_cache_l3.py: a second cache hits from L3 with exact bytes, no re-puts,
an object lost between exists() and get() falls back cleanly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DeepEP now records an HTTPS DeepJIT URL, so build_vendor no longer needs the git@github.com rewrite.
Co-authored-by: Shaoyuan Chen <chensy@deepseek.com>
The open-source engine has no KV reuse across requests, so every turn of a
multi-turn conversation prefills its whole prompt. This adds deepinfer/hicache:
- Pinned host pools organized in pages (one page = one 128-token KV block)
plus slots for the SWA state that the production KV store persists
(select_state_slice_to_store / load_state_from_store_to_prefill_cache).
Pages are indexed by a hash chain over the prompt, LRU evicted, written
through at the end of every prefill (engine hook pe_do_save_kvcache).
- Transfers use one small CUDA kernel copying pages between GPU and pinned
host memory through UVA pointers on the engine stream ("kernel" I/O
method, ~180 GiB/s on GB300); completion is tracked with CUDA events.
- A hit is capped so that at least yoco keep_tokens remain to recompute
(bounded replay rebuilds the decoder SWA state from that suffix), then
snapped to the nearest 128-aligned position with a stored state; states
are stored at the last few boundaries of a prompt so short follow-ups hit.
The sequence is pre-allocated, filled from host and scheduled with
num_cached_tokens set (HostPrefixCache.try_resume).
- Engine: read-only Sequence.block_table binding (to copy a finished
prefill's blocks out) and its stub.
Validated on 4x GB300 with DeepSeek-V4.1-Flash through an OpenAI-compatible
server (separate PR): greedy outputs byte-identical with and without the
cache over 8 multi-turn requests; InferenceX AgentX replay at concurrency 1
served 224 requests per hour vs 99, TTFT p50 0.59 s vs 8.9 s, with 89-96%
of prompt tokens served from the cache. tests/hicache/test_host_cache.py
covers the cache logic against a fake engine.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TNVW4sqoaPqbNtSrgpA2zK
