Adds `record_step` so every decode step is kept instead of just the first, a `--force_ids` mode
to `ref_next_token.py` that mirrors `generate.py`'s incremental loop, and four fixes to the dump
path without which the data was unusable:
- filenames carry `_rank<R>_pid<P>`: with data parallelism every engine is its own
single-process group and reports RANK=0, so eight processes wrote the same file and the last
one won;
- decode reads row 0, not the last row: the batch is padded and the padding rows hold logits for
placeholder tokens, so the last row was a constant `top1=69, margin=0.0528`;
- only the driver rank records: a follower only joins the embedding all-gather, so its logits are
placeholders that got recorded dozens of times and buried the real steps;
- `DEEPINFER_DUMP_TOKENS` gating plus `profiling()` now cover both profile paths.
The conclusion itself comes from a black-box four-way comparison rather than the dumps, because
per-step dumps still do not line up with the emitted token sequence (`max_tokens=7` produced 75
sampler calls). At the same 14-token context:
reference prefill 6127 (北京) correct
reference decode 6127 (北京) correct
our prefill 6127 (北京) correct
our decode 666 (**) wrong
So prefill is right and the remaining flip is introduced by decode alone. Note the reference's
own prefill and decode margins differ (3.112 vs 1.638), so the two paths are not expected to be
identical -- KV cache goes through a quantized round trip -- but they should not disagree on the
direction.
Remaining suspects are therefore narrowed to decode-only state: the KV cache round trip (P4.3)
and the indexer scored in torch versus the TileLang prefill kernel (P4.4), which have never
been diffed against each other.
Comparing final logits was impossible before: `ref_layer_dump.py` read `logits[0, -1]`, but the
head already collapsed the sequence axis (`x = x[:, -1]`), so logits are `[b, vocab]` and that
index silently stored a 0-dimensional scalar. Every earlier dump had an unusable
`final_row_logits`.
With that fixed, the comparison reframes the remaining drift. On the same 14 tokens:
prefill top-1 = 6127 (北京) margin(6127 - 666) = +0.256
reference top-1 = 6127 (北京) margin = +3.112
Our prefill picks the right token; the error only compresses the margin, by ~12x. But the
end-to-end run, which reaches that context via decode steps, picks 666 (`**`) instead. Same
context, different path, different answer -- so the remaining drift is in **decode**, not in the
per-128 quantization. The quantization floor is still 2.7e-2 and still irreducible
(`H20-DRIFT-ANALYSIS.md` has the DeepGEMM assertion that forces `gran_k == 128` on SM90), but it
is not what flips the token.
Two traps had to be handled to get numbers that mean anything:
- `execute_model_profile_run` runs a full forward + sampler on synthetic tokens. The layer dumps
were filtered by their prompt-length gate; logits are not, because their row count is the
number of *selected* tokens (1). The first attempt captured that step and produced cos = 0.34.
`debug_dump.profiling()` now suspends capture for the whole profile step.
- With `max_tokens=1` both a prompt and a decode step run, and only the prompt one is comparable
with a prefill-only reference. They are dumped as `logits_prompt` / `logits_decode`.
Also re-runs the layer comparison now that the expert-scale fix has landed: layer 0 went from
1.27e-1 to 2.88e-2 and layer 39 sits at 2.00e-1, so the error curve flattened by an order of
magnitude.
`h20_quant_check.py` cast to `torch.float4_e2m1fn_x2` to cross-check the code table, nibble order
and quantize/dequantize round trip. That cast kernel exists neither on SM90 nor on the CPU -- it
raises a device-side assert on H20 and `copy_kernel not implemented` on CPU -- so the check could
not run at all on the hardware it is meant for.
The E2M1 table and the unpack order now come from `inference/convert.py` (inlined, with the
source noted), and quantization picks the nearest table entry instead of casting. Still pure
integer/table work, so it runs anywhere and the round trip reports the same 2.77e-2 that the
expert-weight diff attributes to the per-32 -> per-128 requantization.
No behaviour change. `CONTRIBUTING.md` requires `ruff check --no-cache` and
`ruff format --check --no-cache` to pass, and the SM90 branch predates that requirement:
72 findings, all in the files added here.
Most of it is mechanical (`ruff format`), plus:
- annotations for the helper signatures (`torch.Tensor` / `int` / `Generator` / return types);
hooks are typed as `Callable[[nn.Module, tuple, object], None]` rather than left bare;
- two `if`/`else` assignments become ternaries (SIM108), and the index read in
`h20_real_requant_check.py` uses a context manager (SIM115).
Also corrects the `sparse_mla_fwd` docstring, which still described the old
`0 <= idx <= q_position` masking contract. Passing `is_causal=True` there is what produced the
all-NaN layer 0: indices address the assembled `[swa cache | current swa | compressed]` buffer,
so the row number is not a valid causal bound and the engine must pass False.
Consolidates the whole bring-up into one document so the next person does not have to
rediscover it: timeline by phase, every trap with symptom/root cause/fix, the full change
inventory, a reproduction guide, and an explicit TODO for the remaining numerical drift.
The three traps worth reading first are methodological, not code bugs, because each of them
made the comparison numbers meaningless rather than wrong:
- the reference wraps prompts in a chat template, so identical *text* is not identical *input*;
- `decoder_layer` returns the sub-layer output, not the residual stream, so comparing without
`hc_post` showed cos=0.02 at every layer;
- the engine runs a dummy profile forward before allocating the KV cache, so an ungated dump
captures the wrong token count.
Also records the drift TODO with measured numbers: the expert weights are at 2.7e-2 rel_err,
which is the per-32 -> per-128 requantization floor rather than a defect, but the reference
converter is lossless so P3.6 could still recover it if SM90 DeepGEMM accepts an e8m0 per-32
expert scale. P4/P5 in H20-DEBUG-PLAN.md are now cross-referenced to the journal.
`requant_expert_fp4` expanded the UE8M0 scale with `_expand_blocks`, which repeats *both*
axes. That is right for dense weights, whose scales are `(n_out/32, k/32)`, but expert scales
are already per-output-channel: `(n_out, k/32)`. Expanding both axes therefore treated
`(2304, 160)` as `(72, 5)` and handed every one of the 2304 output channels the scales of the
first 72 rows, scrambling each expert's weights.
This is what the layer comparison was pointing at: `shared_expert` matched the reference
(cos 0.9987) because it goes through `requant_dense_fp8`, which uses `_expand_blocks`
correctly, while `combined` -- routed experts plus shared -- was off by 3.6e-1.
Verified against the authoritative implementation, `inference/convert.py`'s
`cast_e2m1fn_to_e4m3fn`, on layer 0 / expert 370:
w1 cos 0.8936 -> 1.0000 rel_err 5.35e-1 -> 2.84e-2
w3 cos 0.8960 -> 0.9997 rel_err 5.26e-1 -> 2.69e-2
w2 cos 0.8964 -> 0.9996 rel_err 5.20e-1 -> 2.65e-2
The residual ~2.7% is the per-32 -> per-128 requantization loss, not a defect; the reference
is lossless because it folds the relative per-32 offset into the FP8 values.
End to end on the 11-token chat prompt:
before: [17607, 41175, 389, 666] "中国的首都是**"
after: [17607, 2793, 3139, 666, 6127, 666, 320, 1] "中国的首都是**北京**。"
ref: [17607, 2793, 3139, 6127, 320] "中国的首都是北京。"
The answer is now correct and the first three tokens match the reference exactly. The remaining
difference is markdown emphasis (`**`), so some numerical drift still moves the top-1 on the
occasional position.
Also adds two checks that would have caught this immediately:
`scripts/h20_expert_weight_check.py` diffs our requantization against DeepSeek's own converter,
and `scripts/h20_quant_check.py` pins the E2M1 table, nibble order and UE8M0 bias. The nibble
order and table turn out to match the reference; only the scale broadcast was wrong.
Narrows the SM90 numerical divergence to layer 0's FFN. The sparse MLA attention
kernel is not the problem: at layer 0 its output matches the reference with
cos=0.99897 / rel_err=4.7e-2, while the full layer output is already at 12.7% and the
error compounds to cos=0.865 by layer 11, which finally flips the top-1 token.
Tooling
- `h20_ops/debug_dump.py`: opt-in capture (`DEEPINFER_DUMP_LAYERS=<dir>`), last
position only, reduced over `hc_mult`, written to `.npz` at exit.
`DEEPINFER_DUMP_TOKENS=<n>` gates on the real prompt length, because the engine
runs a dummy profile forward with a different token count before it allocates the
KV cache and that dump would otherwise be mistaken for the real one.
- `scripts/ref_layer_dump.py`: the same anchors from the reference model via forward
hooks (block output, attention sub-layer output, embedding), plus the final
logits and greedy token.
- `scripts/compare_layers.py`: cosine similarity, relative L2 error and norm ratio
per layer, reporting the first layer above a tolerance.
Two traps this had to fix before the numbers meant anything
- `decoder_layer` returns the *sub-layer* output; the residual stream lives in
`residual` (`HCLazyPostData`, hc_mult=4). Comparing without merging them through
`hc_post` produced a bogus cos of 0.02 at every layer.
- `DEEPINFER_DEBUG_NAN` and the dump both need a host copy inside the forward, so
both take `allow_cuda_sync()` rather than fighting the engine's sync guard.
Also documents the reproduction case and the findings in H20-DEBUG-PLAN.md.
DeepInfer targets Blackwell: the vendored FlashMLA fork only ships SM100 kernels
(`CUDA_ARCHS = ["sm_100a", "sm_103a"]`, every entry point asserts `arch.is_sm100f()`),
and several DeepGEMM attention APIs plus the whole quantization recipe assume
FP4 tensor cores, packed UE8M0 scales and per-32 blocks. None of that exists on
Hopper, so this branch adds an SM90 backend and gets an 8x H20 node to produce
tokens end to end.
Scope and status: the pipeline runs (prefill, decode, sampling) and matches the
reference on the first token, but numerics still diverge from position ~12 onward,
so the output is not yet correct. H20-DEBUG-PLAN.md tracks the remaining work with
the exact reproduction case; see H20-SM90-FINDINGS.md for the analysis.
Quantization
- `Platform.sf_pack_factor` separates two meanings that `fp8_sf_type` conflated:
the GEMM scale type (FP32 on SM90) and the number of scales packed per element
(1, instead of 4 UE8M0 exponents per int32). KV caches keep the ue8m0 byte layout.
- Requantize FP4 experts and ue8m0 dense weights to FP8 E4M3 with FP32 per-128
scales while loading (`hugging_face/weights.py`, `h20_ops/quant.py`), so the
runtime sees a plain FP8 checkpoint: `mxfp4` is disabled at runtime, expert
weights are repacked as 2D, and `use_fp4_sparse_kv` is off since the FP4 cache
kernels do not exist on Hopper.
- GEMM recipes switch to `(1, 128, 128)` with `disable_ue8m0_cast=True`, which is
the only shape SM90 DeepGEMM accepts.
Attention
- `h20_ops/sparse_mla_sm90.py` adapts TileLang's `examples/deepseek_v32` sparse MLA
kernel (which is Hopper-targeted) to the V4.1 layout: RoPE lives inside the
512-wide latent, so the V3.2 `dim`/`tail` split is removed and QK/PV are both 512.
Measured on H20: 104 TFLOP/s, bf16-noise accuracy. It takes no causal mask,
because the indices are already causal -- deriving causality from the row number
drops every index once the KV buffer is offset, which produced an all-NaN layer 0.
- `h20_ops/attn_sm90.py` re-implements `prefill_attn_varlen_sparse` and
`decode_mla_mqa_sparse` with the same signatures as `attention_backend.py`, and
registers them through `patch_imports`, so call sites are untouched and the
FlashMLA-only module is never imported.
- Softmax normalisation clamps `sumexp`, so a padded row with no valid index yields
zeros instead of 0/0 and cannot poison the whole hidden state.
Engram
- `h20_ops/engram_local.py` replaces `deep_ep.EngramBuffer.fetch`, which needs
NCCL GIN (RDMA) and is unavailable on a single node, with a distributed gather
over NVLink built from two `all_to_all`s. Row ownership follows the C++ layout
(`owner = idx // num_local`). Verified against a full reference table on 8 ranks.
Other Blackwell assumptions removed
- DeepSelect's top-k kernel faults at launch on SM90, so `h20_ops/deep_select_sm90.py`
reproduces its contract with `torch.topk` (end truncation, OOB fill, in-place
output views). The shared-layer candidate-block optimisation is disabled because
`fp8_fp4_sparse_mqa_logits` has no SM90 implementation, and the dense MQA-logits
path is used instead; its fp32 output is narrowed for the block-max kernel.
- Decode-time index scoring uses `_decode_p_sum_sm90`: Hopper's paged kernel
hard-requires 64-token pages while the cache uses 128, and the pages are strided
views that cannot be split without a copy.
- `mega_mhc` / `mega_moe`, the FlashAttention-4 CuTe DSL kernels and the fused
norm+rope+cast kernel are all SM100-only and take a Hopper fallback.
- `allow_cuda_sync()` is used for the few places that genuinely need a host read
(all-to-all sizing, the ViT fallback, the NaN diagnostic) instead of disabling
the sync guard globally.
Vendored submodules (not part of this commit; submodule gitlinks are unchanged)
- `vendor/DeepSelect/setup.py`: re-enable the `sm_90a` gencode that is commented
out, otherwise top-k fails with "no kernel image".
Scripts: `h20_attn_check.py`, `h20_engram_check.py`, `h20_topk_check.py`,
`h20_smoke_check.py`, `h20_real_requant_check.py` are unit checks that run on 8x
H20; `h20_ask.py` is a one-prompt harness and `ref_next_token.py` produces the
official reference next-token for arbitrary ids, so both sides can be compared on
identical input.
