coffee/DeepInfer
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepInfer.git synced 2026-10-31 00:36:20 +08:00
Code Issues Packages Projects Releases Wiki Activity
DeepInfer/deepinfer/model_executor/layers/tilelang_ops/neuron
History
whn09 cc90926a1c Neuron: full DeepSeek-V4.1-Flash loads and prefills on 64 NeuronCores (EP=64) 
Device-memory fixes found bringing up the released checkpoint (16.45 GiB of
weights per 24 GB core):
- Engram: the Neuron EngramBuffer keeps only this rank's SF rows, so skip the
  all-gather of the replicated SF table (~3 GB per layer, allocated on the
  device -> "Failed to allocate resource").
- gather_tl: index_select / advanced indexing of an int8 table allocates
  several times the table on the device (+4.9 GiB for the 1.3 GB embedding);
  gather through F.embedding on an int32 / fp16 view instead (bit-exact, no
  extra memory).
- deep_gemm FP8/FP4 GEMMs: dequantize the weight in N chunks of 32M elements
  and one expert at a time instead of whole projections / all local experts
  in fp32 (OOM on a fragmented pool); bit-identical results.
- transfer_to_neuron: import torch.utils._device explicitly (missing when
  torch_neuronx is not auto-loaded).

examples/neuron/offline_inference_neuron.py: offline inference in basic mode
(text prompts, no MTP / EPLB, fixed KV pool, max_model_len >= 2048 * 8 for
the candidate-block filter), one process per NeuronCore.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 14:57:51 +08:00
..
__init__.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
_unsupported.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
candidate_blocks.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
compress.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
compute_seqlens_k_zos.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
dflash_copy.py Neuron basic mode — part 2: deep_ep, sampling kernels, fixes 2026-10-20 18:57:27 +08:00
dflash.py Neuron basic mode — part 2: deep_ep, sampling kernels, fixes 2026-10-20 18:57:27 +08:00
flash_mla_utils.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
fused_sample.py Neuron basic mode — part 2: deep_ep, sampling kernels, fixes 2026-10-20 18:57:27 +08:00
fused_transform.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
gather_engram_tokens_zos.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
gather_input_ids_zos.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
gather.py Neuron: full DeepSeek-V4.1-Flash loads and prefills on 64 NeuronCores (EP=64) 2026-10-21 14:57:51 +08:00
get_kv_index.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
image_embed_ops.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
kv_index_cat.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
kv_quant.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
load_paged_kv_cache.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
norm.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
prepare_cp_mirror.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
prepare_v4_decode_metadata.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
random_sampling.py Neuron basic mode — part 2: deep_ep, sampling kernels, fixes 2026-10-20 18:57:27 +08:00
rstd_1ulp.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
sampler.py Neuron basic mode — part 2: deep_ep, sampling kernels, fixes 2026-10-20 18:57:27 +08:00
schedule_mtp_varlen.py Neuron basic mode — part 2: deep_ep, sampling kernels, fixes 2026-10-20 18:57:27 +08:00
state_cache.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
store_kv_cache.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00
store_mtp_info.py Neuron basic mode — part 2: deep_ep, sampling kernels, fixes 2026-10-20 18:57:27 +08:00
structure_output.py Neuron basic mode — part 2: deep_ep, sampling kernels, fixes 2026-10-20 18:57:27 +08:00
sum_smaller_probs.py Neuron basic mode — part 2: deep_ep, sampling kernels, fixes 2026-10-20 18:57:27 +08:00
swa_index.py Add AWS Trainium (Neuron) platform, basic mode — part 1 2026-10-20 18:34:51 +08:00