coffee/DeepInfer
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepInfer.git synced 2026-10-31 08:46:21 +08:00
Code Issues Packages Projects Releases Wiki Activity
DeepInfer/deepinfer/neuron_ops
History
whn09 7f558a3775 neuron: workers bind memory to their own core's NUMA node; prompt chunks run the attention half as one graph; glibc trims freed heap early (TTFT 796 -> 720 ms on the bench prompt, jitter +-25 -> +-3 ms) 
- NUMA: each worker sees one core, so which_numa(local_rank) failed on all but local rank 0 and every worker
  inherited its MPOL_BIND to node 1: all host memory landed on one node and a run could be OOM-killed.
- DEEPINFER_NEURON_WHOLE_ATTN_MIN_ROWS=512: no cuts inside the attention half of a >=512-row step, only before
  the MoE (its collectives are another group kind).
- MALLOC_TRIM_THRESHOLD_/MALLOC_TOP_PAD_ defaults: compile-time host memory per rank stops growing ~15 GB.
  (MALLOC_ARENA_MAX=2 bounds it too but costs ~50 ms TTFT.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 12:10:31 +08:00
..
nki neuron: e4m3 KV-cache codes through the hardware conversion in an NKI kernel (neuron_ops/nki/fp8_encode.py; 512x512 0.63 -> 0.11 ms, tokens identical) 2026-10-29 00:21:02 +08:00
vendor neuron: single-request prompt chunks attend densely over their own SWA ring and compressed blocks, masks from positions (DEEPINFER_NEURON_PREFILL_DENSE, check mode DEEPINFER_NEURON_PREFILL_DENSE_CHECK) 2026-10-28 09:09:34 +08:00
__init__.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
fp8_guard.py Neuron: tiny model on a NeuronCore matches the reference (issues #1, #2, #4) 2026-10-21 13:22:59 +08:00
fp8.py Neuron: full DeepSeek-V4.1-Flash load + decode region graphs at EP=64 (issues #21, #25) 2026-10-23 01:39:22 +08:00
neuron_compile.py neuron: e4m3 KV-cache codes through the hardware conversion in an NKI kernel (neuron_ops/nki/fp8_encode.py; 512x512 0.63 -> 0.11 ms, tokens identical) 2026-10-29 00:21:02 +08:00
NEURON_PORT.md Neuron: tiny model on a NeuronCore matches the reference (issues #1, #2, #4) 2026-10-21 13:22:59 +08:00
nki_attention.py neuron: single-request prompt chunks attend densely over their own SWA ring and compressed blocks, masks from positions (DEEPINFER_NEURON_PREFILL_DENSE, check mode DEEPINFER_NEURON_PREFILL_DENSE_CHECK) 2026-10-28 09:09:34 +08:00
nki_gemm.py neuron: fp4 expert GEMMs decode their weight in SBUF on both cores (blockquant.fp4_linear_fused_2core, DEEPINFER_NEURON_FP4_FUSED=1): 512x5120x4608 0.527 -> 0.447 ms, TTFT 825 -> 807 ms; tokens differ by rounding 2026-10-28 21:24:41 +08:00
nki_hc.py neuron: mHC collapse through torch by default (DEEPINFER_NEURON_HC_TORCH=collapse); fix Neuron bind_cpu core index 2026-10-27 09:18:56 +08:00
nki_norm.py Neuron: full DeepSeek-V4.1-Flash load + decode region graphs at EP=64 (issues #21, #25) 2026-10-23 01:39:22 +08:00
nki_topk.py Neuron: nkilib rotational_topk for the sampler; single-host shared engram tables (#27) 2026-10-23 07:23:32 +08:00
op_census.py neuron: sampling flags per request batch, NUMA-local CPU binding, sync/op-site tracing 2026-10-26 19:04:53 +08:00
regions.py neuron: workers bind memory to their own core's NUMA node; prompt chunks run the attention half as one graph; glibc trims freed heap early (TTFT 796 -> 720 ms on the bench prompt, jitter +-25 -> +-3 ms) 2026-10-29 12:10:31 +08:00
sampling_mirror.py neuron: compiled decode verification behind DEEPINFER_NEURON_COMPILE_VERIFY (default off) 2026-10-26 21:33:34 +08:00
static_moe.py neuron: prompt chunks up to 1024 rows keep only the coarse region splits (TTFT 942 -> 813 ms, step 414 -> 406 ms, tokens identical); prefill grouped MoE without host sync (DEEPINFER_NEURON_MOE_PREFILL_CAPACITY, off: 942 -> 1000 ms) 2026-10-28 17:00:00 +08:00
transfer_to_neuron.py Neuron: static decode graphs, static-shape MoE, trn2 e4m3 fp8 (issues #9, #12, #24) 2026-10-22 11:59:37 +08:00