coffee/DeepInfer
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepInfer.git synced 2026-10-28 14:57:09 +08:00
Code Issues Packages Projects Releases Wiki Activity

Neuron: skip the unused candidate-block source when the filter is vacuous (#26)

Browse Source 
With runtime_max_model_len <= candidate_topk_blocks * candidate_block_size
every consumer already skips the candidate filter, so the source layer's block
top-k feeds nothing; below 16384 positions it is not even defined (fewer
blocks than candidate_topk_blocks, assert in _topk_candidate_block).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
whn09 2026-10-23 12:42:25 +08:00
parent 5dfa098dca
commit 7d897aedb2
1 changed files with 9 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
deepinfer/model_executor/models/v4_modules/index_attn.py
View File

@ -98,6 +98,15 @@ class MQA(nn.Module):
candidate_cfg = shared_topology.candidate_source_cfg[candidate_source] if candidate_source is not None else None
self.is_candidate_topk_source = shared_topology is not None and shared_topology.is_candidate_source_layer(self.layer_idx)
self.use_candidate_topk = candidate_source is not None and not self.is_candidate_topk_source
if (
PLATFORM.is_neuron
and self.is_candidate_topk_source
and candidate_cfg is not None
and cfg.runtime_max_model_len <= candidate_cfg.candidate_topk_blocks * candidate_cfg.candidate_block_size
):
# Neuron: every consumer skips the vacuous filter (below), so nothing reads the candidate blocks; and with
# fewer blocks than candidate_topk_blocks the block top-k would not be defined.
self.is_candidate_topk_source = False
if (
PLATFORM.is_neuron
and self.use_candidate_topk