`requant_expert_fp4` expanded the UE8M0 scale with `_expand_blocks`, which repeats *both*
axes. That is right for dense weights, whose scales are `(n_out/32, k/32)`, but expert scales
are already per-output-channel: `(n_out, k/32)`. Expanding both axes therefore treated
`(2304, 160)` as `(72, 5)` and handed every one of the 2304 output channels the scales of the
first 72 rows, scrambling each expert's weights.
This is what the layer comparison was pointing at: `shared_expert` matched the reference
(cos 0.9987) because it goes through `requant_dense_fp8`, which uses `_expand_blocks`
correctly, while `combined` -- routed experts plus shared -- was off by 3.6e-1.
Verified against the authoritative implementation, `inference/convert.py`'s
`cast_e2m1fn_to_e4m3fn`, on layer 0 / expert 370:
w1 cos 0.8936 -> 1.0000 rel_err 5.35e-1 -> 2.84e-2
w3 cos 0.8960 -> 0.9997 rel_err 5.26e-1 -> 2.69e-2
w2 cos 0.8964 -> 0.9996 rel_err 5.20e-1 -> 2.65e-2
The residual ~2.7% is the per-32 -> per-128 requantization loss, not a defect; the reference
is lossless because it folds the relative per-32 offset into the FP8 values.
End to end on the 11-token chat prompt:
before: [17607, 41175, 389, 666] "中国的首都是**"
after: [17607, 2793, 3139, 666, 6127, 666, 320, 1] "中国的首都是**北京**。"
ref: [17607, 2793, 3139, 6127, 320] "中国的首都是北京。"
The answer is now correct and the first three tokens match the reference exactly. The remaining
difference is markdown emphasis (`**`), so some numerical drift still moves the top-1 on the
occasional position.
Also adds two checks that would have caught this immediately:
`scripts/h20_expert_weight_check.py` diffs our requantization against DeepSeek's own converter,
and `scripts/h20_quant_check.py` pins the E2M1 table, nibble order and UE8M0 bias. The nibble
order and table turn out to match the reference; only the scale broadcast was wrong.
Narrows the SM90 numerical divergence to layer 0's FFN. The sparse MLA attention
kernel is not the problem: at layer 0 its output matches the reference with
cos=0.99897 / rel_err=4.7e-2, while the full layer output is already at 12.7% and the
error compounds to cos=0.865 by layer 11, which finally flips the top-1 token.
Tooling
- `h20_ops/debug_dump.py`: opt-in capture (`DEEPINFER_DUMP_LAYERS=<dir>`), last
position only, reduced over `hc_mult`, written to `.npz` at exit.
`DEEPINFER_DUMP_TOKENS=<n>` gates on the real prompt length, because the engine
runs a dummy profile forward with a different token count before it allocates the
KV cache and that dump would otherwise be mistaken for the real one.
- `scripts/ref_layer_dump.py`: the same anchors from the reference model via forward
hooks (block output, attention sub-layer output, embedding), plus the final
logits and greedy token.
- `scripts/compare_layers.py`: cosine similarity, relative L2 error and norm ratio
per layer, reporting the first layer above a tolerance.
Two traps this had to fix before the numbers meant anything
- `decoder_layer` returns the *sub-layer* output; the residual stream lives in
`residual` (`HCLazyPostData`, hc_mult=4). Comparing without merging them through
`hc_post` produced a bogus cos of 0.02 at every layer.
- `DEEPINFER_DEBUG_NAN` and the dump both need a host copy inside the forward, so
both take `allow_cuda_sync()` rather than fighting the engine's sync guard.
Also documents the reproduction case and the findings in H20-DEBUG-PLAN.md.
DeepInfer targets Blackwell: the vendored FlashMLA fork only ships SM100 kernels
(`CUDA_ARCHS = ["sm_100a", "sm_103a"]`, every entry point asserts `arch.is_sm100f()`),
and several DeepGEMM attention APIs plus the whole quantization recipe assume
FP4 tensor cores, packed UE8M0 scales and per-32 blocks. None of that exists on
Hopper, so this branch adds an SM90 backend and gets an 8x H20 node to produce
tokens end to end.
Scope and status: the pipeline runs (prefill, decode, sampling) and matches the
reference on the first token, but numerics still diverge from position ~12 onward,
so the output is not yet correct. H20-DEBUG-PLAN.md tracks the remaining work with
the exact reproduction case; see H20-SM90-FINDINGS.md for the analysis.
Quantization
- `Platform.sf_pack_factor` separates two meanings that `fp8_sf_type` conflated:
the GEMM scale type (FP32 on SM90) and the number of scales packed per element
(1, instead of 4 UE8M0 exponents per int32). KV caches keep the ue8m0 byte layout.
- Requantize FP4 experts and ue8m0 dense weights to FP8 E4M3 with FP32 per-128
scales while loading (`hugging_face/weights.py`, `h20_ops/quant.py`), so the
runtime sees a plain FP8 checkpoint: `mxfp4` is disabled at runtime, expert
weights are repacked as 2D, and `use_fp4_sparse_kv` is off since the FP4 cache
kernels do not exist on Hopper.
- GEMM recipes switch to `(1, 128, 128)` with `disable_ue8m0_cast=True`, which is
the only shape SM90 DeepGEMM accepts.
Attention
- `h20_ops/sparse_mla_sm90.py` adapts TileLang's `examples/deepseek_v32` sparse MLA
kernel (which is Hopper-targeted) to the V4.1 layout: RoPE lives inside the
512-wide latent, so the V3.2 `dim`/`tail` split is removed and QK/PV are both 512.
Measured on H20: 104 TFLOP/s, bf16-noise accuracy. It takes no causal mask,
because the indices are already causal -- deriving causality from the row number
drops every index once the KV buffer is offset, which produced an all-NaN layer 0.
- `h20_ops/attn_sm90.py` re-implements `prefill_attn_varlen_sparse` and
`decode_mla_mqa_sparse` with the same signatures as `attention_backend.py`, and
registers them through `patch_imports`, so call sites are untouched and the
FlashMLA-only module is never imported.
- Softmax normalisation clamps `sumexp`, so a padded row with no valid index yields
zeros instead of 0/0 and cannot poison the whole hidden state.
Engram
- `h20_ops/engram_local.py` replaces `deep_ep.EngramBuffer.fetch`, which needs
NCCL GIN (RDMA) and is unavailable on a single node, with a distributed gather
over NVLink built from two `all_to_all`s. Row ownership follows the C++ layout
(`owner = idx // num_local`). Verified against a full reference table on 8 ranks.
Other Blackwell assumptions removed
- DeepSelect's top-k kernel faults at launch on SM90, so `h20_ops/deep_select_sm90.py`
reproduces its contract with `torch.topk` (end truncation, OOB fill, in-place
output views). The shared-layer candidate-block optimisation is disabled because
`fp8_fp4_sparse_mqa_logits` has no SM90 implementation, and the dense MQA-logits
path is used instead; its fp32 output is narrowed for the block-max kernel.
- Decode-time index scoring uses `_decode_p_sum_sm90`: Hopper's paged kernel
hard-requires 64-token pages while the cache uses 128, and the pages are strided
views that cannot be split without a copy.
- `mega_mhc` / `mega_moe`, the FlashAttention-4 CuTe DSL kernels and the fused
norm+rope+cast kernel are all SM100-only and take a Hopper fallback.
- `allow_cuda_sync()` is used for the few places that genuinely need a host read
(all-to-all sizing, the ViT fallback, the NaN diagnostic) instead of disabling
the sync guard globally.
Vendored submodules (not part of this commit; submodule gitlinks are unchanged)
- `vendor/DeepSelect/setup.py`: re-enable the `sm_90a` gencode that is commented
out, otherwise top-k fails with "no kernel image".
Scripts: `h20_attn_check.py`, `h20_engram_check.py`, `h20_topk_check.py`,
`h20_smoke_check.py`, `h20_real_requant_check.py` are unit checks that run on 8x
H20; `h20_ask.py` is a one-prompt harness and `ref_next_token.py` produces the
official reference next-token for arbitrary ids, so both sides can be compared on
identical input.
