loopyt 1e04aea7ea feat: add SM90 (Hopper) support for DeepSeek-V4.1 inference ... DeepInfer targets Blackwell: the vendored FlashMLA fork only ships SM100 kernels (`CUDA_ARCHS = ["sm_100a", "sm_103a"]`, every entry point asserts `arch.is_sm100f()`), and several DeepGEMM attention APIs plus the whole quantization recipe assume FP4 tensor cores, packed UE8M0 scales and per-32 blocks. None of that exists on Hopper, so this branch adds an SM90 backend and gets an 8x H20 node to produce tokens end to end. Scope and status: the pipeline runs (prefill, decode, sampling) and matches the reference on the first token, but numerics still diverge from position ~12 onward, so the output is not yet correct. H20-DEBUG-PLAN.md tracks the remaining work with the exact reproduction case; see H20-SM90-FINDINGS.md for the analysis. Quantization - `Platform.sf_pack_factor` separates two meanings that `fp8_sf_type` conflated: the GEMM scale type (FP32 on SM90) and the number of scales packed per element (1, instead of 4 UE8M0 exponents per int32). KV caches keep the ue8m0 byte layout. - Requantize FP4 experts and ue8m0 dense weights to FP8 E4M3 with FP32 per-128 scales while loading (`hugging_face/weights.py`, `h20_ops/quant.py`), so the runtime sees a plain FP8 checkpoint: `mxfp4` is disabled at runtime, expert weights are repacked as 2D, and `use_fp4_sparse_kv` is off since the FP4 cache kernels do not exist on Hopper. - GEMM recipes switch to `(1, 128, 128)` with `disable_ue8m0_cast=True`, which is the only shape SM90 DeepGEMM accepts. Attention - `h20_ops/sparse_mla_sm90.py` adapts TileLang's `examples/deepseek_v32` sparse MLA kernel (which is Hopper-targeted) to the V4.1 layout: RoPE lives inside the 512-wide latent, so the V3.2 `dim`/`tail` split is removed and QK/PV are both 512. Measured on H20: 104 TFLOP/s, bf16-noise accuracy. It takes no causal mask, because the indices are already causal -- deriving causality from the row number drops every index once the KV buffer is offset, which produced an all-NaN layer 0. - `h20_ops/attn_sm90.py` re-implements `prefill_attn_varlen_sparse` and `decode_mla_mqa_sparse` with the same signatures as `attention_backend.py`, and registers them through `patch_imports`, so call sites are untouched and the FlashMLA-only module is never imported. - Softmax normalisation clamps `sumexp`, so a padded row with no valid index yields zeros instead of 0/0 and cannot poison the whole hidden state. Engram - `h20_ops/engram_local.py` replaces `deep_ep.EngramBuffer.fetch`, which needs NCCL GIN (RDMA) and is unavailable on a single node, with a distributed gather over NVLink built from two `all_to_all`s. Row ownership follows the C++ layout (`owner = idx // num_local`). Verified against a full reference table on 8 ranks. Other Blackwell assumptions removed - DeepSelect's top-k kernel faults at launch on SM90, so `h20_ops/deep_select_sm90.py` reproduces its contract with `torch.topk` (end truncation, OOB fill, in-place output views). The shared-layer candidate-block optimisation is disabled because `fp8_fp4_sparse_mqa_logits` has no SM90 implementation, and the dense MQA-logits path is used instead; its fp32 output is narrowed for the block-max kernel. - Decode-time index scoring uses `_decode_p_sum_sm90`: Hopper's paged kernel hard-requires 64-token pages while the cache uses 128, and the pages are strided views that cannot be split without a copy. - `mega_mhc` / `mega_moe`, the FlashAttention-4 CuTe DSL kernels and the fused norm+rope+cast kernel are all SM100-only and take a Hopper fallback. - `allow_cuda_sync()` is used for the few places that genuinely need a host read (all-to-all sizing, the ViT fallback, the NaN diagnostic) instead of disabling the sync guard globally. Vendored submodules (not part of this commit; submodule gitlinks are unchanged) - `vendor/DeepSelect/setup.py`: re-enable the `sm_90a` gencode that is commented out, otherwise top-k fails with "no kernel image". Scripts: `h20_attn_check.py`, `h20_engram_check.py`, `h20_topk_check.py`, `h20_smoke_check.py`, `h20_real_requant_check.py` are unit checks that run on 8x H20; `h20_ask.py` is a one-prompt harness and `ref_next_token.py` produces the official reference next-token for arbitrary ids, so both sides can be compared on identical input.		2026-10-21 15:18:56 +08:00
..
__init__.py	initial commit	2026-10-20 12:09:42 +08:00
base.py	initial commit	2026-10-20 12:09:42 +08:00
cache_engine.py	feat: add SM90 (Hopper) support for DeepSeek-V4.1 inference	2026-10-21 15:18:56 +08:00
cuda_graph_runner.py	initial commit	2026-10-20 12:09:42 +08:00
decode_perf_cache.py	initial commit	2026-10-20 12:09:42 +08:00
dflash_cuda_graph_runner.py	initial commit	2026-10-20 12:09:42 +08:00
model_runner.py	initial commit	2026-10-20 12:09:42 +08:00
runtime_config.py	initial commit	2026-10-20 12:09:42 +08:00
worker.py	initial commit	2026-10-20 12:09:42 +08:00
zos_status.py	initial commit	2026-10-20 12:09:42 +08:00