The note still ended at "next: P4" with an open checklist. Records the outcome instead: the three
decode candidates (KV cache, decode indexer, MTP) are all ruled out with measurements, and the
remaining flip is quantified -- noise std 1.63 against a reference margin of 1.638 at the one
position that differs, out of eight.
Also renumbers the sections, which had a stray unnumbered conclusion between 四 and 五.
Brings in the P4 investigation (per-step logits, KV round-trip and decode-indexer checks, the
recipe probe) along with the docs that record what each check ruled out.
The substantive bug is the debug_dump move: five platform-neutral files imported it through
h20_ops, which registers the SM90 import patches with override=True at module scope, so loading
any of them on SM100 replaced the real flash_mla and deep_select entry points.
The dump hooks live in five platform-neutral files -- `sampler.py`, `mlp.py`, `deepseek.py`,
`decoder_layer.py`, `model_runner.py` -- and all of them imported
`deepinfer.h20_ops.debug_dump`. That is wrong on every platform except SM90: `h20_ops/__init__.py`
calls `extend_import_patches(override=True, ...)` at module scope, so importing it from shared code
installs the SM90 import shims globally. On an SM100 machine that replaces
`flash_mla.decode_mla_mqa_sparse`, `flash_mla.prefill_attn_varlen_sparse` and
`deep_select.interface.topk` with the Hopper implementations, and on NPU it would do the same
before the Ascend shims get a chance to matter.
`debug_dump` itself has nothing platform-specific in it -- `os`, `pathlib`, `numpy`, `torch` --
so it moves to `deepinfer/debug_dump.py` and the five imports follow. Verified that importing it
no longer pulls `h20_ops` into `sys.modules`, and re-ran the end-to-end check on H20: same output
token for token, `[17607, 2793, 3139, 666, 6127, 666, 320, 1]`.
This bug is already in the open PR, so it needs to land before that PR is worth reviewing further.
P4.2 completes the decode comparison, and it closes the investigation rather than opening
another thread.
The one prerequisite was `DEEPINFER_USE_NEXTN_DECODE=0`. Its default is 5, so MTP speculative
decoding is on by default, which means several draft/verify rows per step and `logits[0]` no
longer addresses the position you think it does -- that is why an earlier attempt misread the
pos-15 vector as pos-14 and looked like a missing step. With MTP off, one row per step, clean
alignment. Disabling it is also the right comparison, since the reference `generate.py` has no
MTP at all.
Per position, 11-token prompt, 8 decode steps:
pos ours ref ours margin ref margin d margin
11 17607 17607 10.695 9.861 +0.835 (prefill)
12 2793 2793 8.448 10.328 -1.880
13 3139 3139 15.276 11.946 +3.330
14 666 6127 1.480 1.638 -0.157 <-- only flip
15 6127 6127 15.950 15.109 +0.842
16 666 666 20.118 17.874 +2.244
17 320 320 9.829 10.402 -0.573
18 1 1 12.262 11.091 +1.171
d margin: mean +0.711, std 1.628, max 3.330. Seven of eight positions are right. The single
flip sits where the reference's own margin is 1.638, against 9.9-17.9 everywhere else -- the
noise (std 1.63) and that margin are the same size, so it is a coin toss. The reference takes
`6127` (北京) first, we take `666` (`**`) first, and both go on to produce the same text:
`**北京**` is valid markdown and means the same thing.
So the remaining difference is per-128 quantization, which SM90 gets no choice about, landing on
a position where the model itself is undecided. Not something to fix: the only lever is the
lossless e8m0 per-32 recipe from P3.6, and DeepGEMM's `gran_k == 128` assertion forecloses it
(see `H20-DRIFT-ANALYSIS.md`).
Also rules out two more suspects. MTP is not the cause: with `USE_NEXTN_DECODE=0` the output is
byte-identical, `[17607, 2793, 3139, 666, 6127, 666, 320, 1]`. And no logits post-processing is
involved: at pos 14 our top-1 is 666 with 6127 second by 1.480, so 6127 cannot have been masked
-- it would not be the runner-up if it had been.
Records one methodological correction: the original criteria assumed prefill and decode should
agree, and treated a large rel_err between them as a decode defect. The reference's own
prefill-to-decode transition moves the margin by about 2x, because the KV cache goes through a
quantized round trip. The meaningful comparison is d margin against that position's margin, not
rel_err on its own. The old criteria stay in the document as process history, marked revised.
Two closed-loop checks, both clean, which finally retires the two standing suspects.
P4.3 KV cache. The repo's own store/load tests pass for the fp32-scale index layout SM90 uses
(6 each). Running every layout gives 48 passed / 12 failed, and every failure is a ue8m0 index
layout -- `FP4_INDEX_KV_FORMAT` or `MXFP8_INDEX_KV_FORMAT`, neither of which SM90 selects. Those
failures are in the test's reference, not the kernel: it reinterprets ue8m0 bytes as
`PLATFORM.fp8_sf_type`, which is float32 here, so `index_copy_` hits a dtype mismatch.
`scripts/h20_kv_roundtrip.py` then closes the loop without any reference at all -- write known
BF16 through the paged cache, read it back, compare. Both SM90 layouts, block 32 and 64, 1/233/
1029 tokens:
V41_FORMAT (main/swa KV) rel 0.0266-0.0295 cos 0.9996
FP8_FLOAT32_SF_INDEX_KV_FORMAT (132B) rel 0.0266-0.0327 cos 0.9996
That is the fp8 floor, the same order as the 2.7e-2 on weights, so the cache is not where the
decode error comes from and the 132-byte layout question from P3.9 is closed.
One trap: the first version of the script compared raw fp8 values against BF16 and reported
rel=126. Index layouts are deliberately *not* dequantized by `load_paged_kv_cache` -- the
consumer is a kernel that wants bytes plus scales. Multiplying by the scale gives 0.027. The
127x was the test, not the implementation.
P4.4 the decode indexer. `_decode_p_sum_sm90` reimplements DeepGEMM's paged MQA logits in torch
because Hopper's kernel requires 64-token pages; nothing else exercised it. Compared against a
float64 reference that shares no code -- explicit per-token offsets instead of strided views --
it agrees to rel~1e-7 / cos 1.0 across page sizes 32 and 64, boundary-crossing lengths,
multi-page contexts and 32 heads. The `q_sf[..., :1]` dequantization is also sound: head_dim is
128, equal to the granularity, so one scale per (row, head) is correct.
What this changes. `logits_prompt` from the max_tokens=1 run puts the prefill error at cos
0.9846 with an absolute error of 3.30 on the winning token, against a reference margin of 3.112:
the margin is thin because the error is the same size as the margin, not by coincidence. The
reference's own prefill-to-decode transition also costs about 2x margin (3.112 -> 1.638) purely
from the quantized KV round trip. Our 0.256 halved would still be positive, yet it flips, so
roughly 0.5-1.0 of logit-level error in decode remains unexplained -- now narrowed to attention
over the cached main KV and the SWA-window / compressed-position mapping.
Also corrects an earlier overstatement: "the difference is introduced by decode" does not mean
decode has a discrete bug, and P4.3/P4.4 show its two main suspects do not.
One more dump trap worth recording: with max_tokens > 1, `logits_prompt` is contaminated too.
mysteps6 (max_tokens=7) gives cos 0.68 and a 16.5 top-1 error for the same input where mylogits3
(max_tokens=1) gives 0.9846 and 3.30. Prefill logits are only trustworthy from a
max_tokens=1 run.
Adds `record_step` so every decode step is kept instead of just the first, a `--force_ids` mode
to `ref_next_token.py` that mirrors `generate.py`'s incremental loop, and four fixes to the dump
path without which the data was unusable:
- filenames carry `_rank<R>_pid<P>`: with data parallelism every engine is its own
single-process group and reports RANK=0, so eight processes wrote the same file and the last
one won;
- decode reads row 0, not the last row: the batch is padded and the padding rows hold logits for
placeholder tokens, so the last row was a constant `top1=69, margin=0.0528`;
- only the driver rank records: a follower only joins the embedding all-gather, so its logits are
placeholders that got recorded dozens of times and buried the real steps;
- `DEEPINFER_DUMP_TOKENS` gating plus `profiling()` now cover both profile paths.
The conclusion itself comes from a black-box four-way comparison rather than the dumps, because
per-step dumps still do not line up with the emitted token sequence (`max_tokens=7` produced 75
sampler calls). At the same 14-token context:
reference prefill 6127 (北京) correct
reference decode 6127 (北京) correct
our prefill 6127 (北京) correct
our decode 666 (**) wrong
So prefill is right and the remaining flip is introduced by decode alone. Note the reference's
own prefill and decode margins differ (3.112 vs 1.638), so the two paths are not expected to be
identical -- KV cache goes through a quantized round trip -- but they should not disagree on the
direction.
Remaining suspects are therefore narrowed to decode-only state: the KV cache round trip (P4.3)
and the indexer scored in torch versus the TileLang prefill kernel (P4.4), which have never
been diffed against each other.
Comparing final logits was impossible before: `ref_layer_dump.py` read `logits[0, -1]`, but the
head already collapsed the sequence axis (`x = x[:, -1]`), so logits are `[b, vocab]` and that
index silently stored a 0-dimensional scalar. Every earlier dump had an unusable
`final_row_logits`.
With that fixed, the comparison reframes the remaining drift. On the same 14 tokens:
prefill top-1 = 6127 (北京) margin(6127 - 666) = +0.256
reference top-1 = 6127 (北京) margin = +3.112
Our prefill picks the right token; the error only compresses the margin, by ~12x. But the
end-to-end run, which reaches that context via decode steps, picks 666 (`**`) instead. Same
context, different path, different answer -- so the remaining drift is in **decode**, not in the
per-128 quantization. The quantization floor is still 2.7e-2 and still irreducible
(`H20-DRIFT-ANALYSIS.md` has the DeepGEMM assertion that forces `gran_k == 128` on SM90), but it
is not what flips the token.
Two traps had to be handled to get numbers that mean anything:
- `execute_model_profile_run` runs a full forward + sampler on synthetic tokens. The layer dumps
were filtered by their prompt-length gate; logits are not, because their row count is the
number of *selected* tokens (1). The first attempt captured that step and produced cos = 0.34.
`debug_dump.profiling()` now suspends capture for the whole profile step.
- With `max_tokens=1` both a prompt and a decode step run, and only the prompt one is comparable
with a prefill-only reference. They are dumped as `logits_prompt` / `logits_decode`.
Also re-runs the layer comparison now that the expert-scale fix has landed: layer 0 went from
1.27e-1 to 2.88e-2 and layer 39 sits at 2.00e-1, so the error curve flattened by an order of
magnitude.
`h20_quant_check.py` cast to `torch.float4_e2m1fn_x2` to cross-check the code table, nibble order
and quantize/dequantize round trip. That cast kernel exists neither on SM90 nor on the CPU -- it
raises a device-side assert on H20 and `copy_kernel not implemented` on CPU -- so the check could
not run at all on the hardware it is meant for.
The E2M1 table and the unpack order now come from `inference/convert.py` (inlined, with the
source noted), and quantization picks the nearest table entry instead of casting. Still pure
integer/table work, so it runs anywhere and the round trip reports the same 2.77e-2 that the
expert-weight diff attributes to the per-32 -> per-128 requantization.
No behaviour change. `CONTRIBUTING.md` requires `ruff check --no-cache` and
`ruff format --check --no-cache` to pass, and the SM90 branch predates that requirement:
72 findings, all in the files added here.
Most of it is mechanical (`ruff format`), plus:
- annotations for the helper signatures (`torch.Tensor` / `int` / `Generator` / return types);
hooks are typed as `Callable[[nn.Module, tuple, object], None]` rather than left bare;
- two `if`/`else` assignments become ternaries (SIM108), and the index read in
`h20_real_requant_check.py` uses a context manager (SIM115).
Also corrects the `sparse_mla_fwd` docstring, which still described the old
`0 <= idx <= q_position` masking contract. Passing `is_causal=True` there is what produced the
all-NaN layer 0: indices address the assembled `[swa cache | current swa | compressed]` buffer,
so the row number is not a valid causal bound and the engine must pass False.
Consolidates the whole bring-up into one document so the next person does not have to
rediscover it: timeline by phase, every trap with symptom/root cause/fix, the full change
inventory, a reproduction guide, and an explicit TODO for the remaining numerical drift.
The three traps worth reading first are methodological, not code bugs, because each of them
made the comparison numbers meaningless rather than wrong:
- the reference wraps prompts in a chat template, so identical *text* is not identical *input*;
- `decoder_layer` returns the sub-layer output, not the residual stream, so comparing without
`hc_post` showed cos=0.02 at every layer;
- the engine runs a dummy profile forward before allocating the KV cache, so an ungated dump
captures the wrong token count.
Also records the drift TODO with measured numbers: the expert weights are at 2.7e-2 rel_err,
which is the per-32 -> per-128 requantization floor rather than a defect, but the reference
converter is lossless so P3.6 could still recover it if SM90 DeepGEMM accepts an e8m0 per-32
expert scale. P4/P5 in H20-DEBUG-PLAN.md are now cross-referenced to the journal.
`requant_expert_fp4` expanded the UE8M0 scale with `_expand_blocks`, which repeats *both*
axes. That is right for dense weights, whose scales are `(n_out/32, k/32)`, but expert scales
are already per-output-channel: `(n_out, k/32)`. Expanding both axes therefore treated
`(2304, 160)` as `(72, 5)` and handed every one of the 2304 output channels the scales of the
first 72 rows, scrambling each expert's weights.
This is what the layer comparison was pointing at: `shared_expert` matched the reference
(cos 0.9987) because it goes through `requant_dense_fp8`, which uses `_expand_blocks`
correctly, while `combined` -- routed experts plus shared -- was off by 3.6e-1.
Verified against the authoritative implementation, `inference/convert.py`'s
`cast_e2m1fn_to_e4m3fn`, on layer 0 / expert 370:
w1 cos 0.8936 -> 1.0000 rel_err 5.35e-1 -> 2.84e-2
w3 cos 0.8960 -> 0.9997 rel_err 5.26e-1 -> 2.69e-2
w2 cos 0.8964 -> 0.9996 rel_err 5.20e-1 -> 2.65e-2
The residual ~2.7% is the per-32 -> per-128 requantization loss, not a defect; the reference
is lossless because it folds the relative per-32 offset into the FP8 values.
End to end on the 11-token chat prompt:
before: [17607, 41175, 389, 666] "中国的首都是**"
after: [17607, 2793, 3139, 666, 6127, 666, 320, 1] "中国的首都是**北京**。"
ref: [17607, 2793, 3139, 6127, 320] "中国的首都是北京。"
The answer is now correct and the first three tokens match the reference exactly. The remaining
difference is markdown emphasis (`**`), so some numerical drift still moves the top-1 on the
occasional position.
Also adds two checks that would have caught this immediately:
`scripts/h20_expert_weight_check.py` diffs our requantization against DeepSeek's own converter,
and `scripts/h20_quant_check.py` pins the E2M1 table, nibble order and UE8M0 bias. The nibble
order and table turn out to match the reference; only the scale broadcast was wrong.
Narrows the SM90 numerical divergence to layer 0's FFN. The sparse MLA attention
kernel is not the problem: at layer 0 its output matches the reference with
cos=0.99897 / rel_err=4.7e-2, while the full layer output is already at 12.7% and the
error compounds to cos=0.865 by layer 11, which finally flips the top-1 token.
Tooling
- `h20_ops/debug_dump.py`: opt-in capture (`DEEPINFER_DUMP_LAYERS=<dir>`), last
position only, reduced over `hc_mult`, written to `.npz` at exit.
`DEEPINFER_DUMP_TOKENS=<n>` gates on the real prompt length, because the engine
runs a dummy profile forward with a different token count before it allocates the
KV cache and that dump would otherwise be mistaken for the real one.
- `scripts/ref_layer_dump.py`: the same anchors from the reference model via forward
hooks (block output, attention sub-layer output, embedding), plus the final
logits and greedy token.
- `scripts/compare_layers.py`: cosine similarity, relative L2 error and norm ratio
per layer, reporting the first layer above a tolerance.
Two traps this had to fix before the numbers meant anything
- `decoder_layer` returns the *sub-layer* output; the residual stream lives in
`residual` (`HCLazyPostData`, hc_mult=4). Comparing without merging them through
`hc_post` produced a bogus cos of 0.02 at every layer.
- `DEEPINFER_DEBUG_NAN` and the dump both need a host copy inside the forward, so
both take `allow_cuda_sync()` rather than fighting the engine's sync guard.
Also documents the reproduction case and the findings in H20-DEBUG-PLAN.md.
DeepInfer targets Blackwell: the vendored FlashMLA fork only ships SM100 kernels
(`CUDA_ARCHS = ["sm_100a", "sm_103a"]`, every entry point asserts `arch.is_sm100f()`),
and several DeepGEMM attention APIs plus the whole quantization recipe assume
FP4 tensor cores, packed UE8M0 scales and per-32 blocks. None of that exists on
Hopper, so this branch adds an SM90 backend and gets an 8x H20 node to produce
tokens end to end.
Scope and status: the pipeline runs (prefill, decode, sampling) and matches the
reference on the first token, but numerics still diverge from position ~12 onward,
so the output is not yet correct. H20-DEBUG-PLAN.md tracks the remaining work with
the exact reproduction case; see H20-SM90-FINDINGS.md for the analysis.
Quantization
- `Platform.sf_pack_factor` separates two meanings that `fp8_sf_type` conflated:
the GEMM scale type (FP32 on SM90) and the number of scales packed per element
(1, instead of 4 UE8M0 exponents per int32). KV caches keep the ue8m0 byte layout.
- Requantize FP4 experts and ue8m0 dense weights to FP8 E4M3 with FP32 per-128
scales while loading (`hugging_face/weights.py`, `h20_ops/quant.py`), so the
runtime sees a plain FP8 checkpoint: `mxfp4` is disabled at runtime, expert
weights are repacked as 2D, and `use_fp4_sparse_kv` is off since the FP4 cache
kernels do not exist on Hopper.
- GEMM recipes switch to `(1, 128, 128)` with `disable_ue8m0_cast=True`, which is
the only shape SM90 DeepGEMM accepts.
Attention
- `h20_ops/sparse_mla_sm90.py` adapts TileLang's `examples/deepseek_v32` sparse MLA
kernel (which is Hopper-targeted) to the V4.1 layout: RoPE lives inside the
512-wide latent, so the V3.2 `dim`/`tail` split is removed and QK/PV are both 512.
Measured on H20: 104 TFLOP/s, bf16-noise accuracy. It takes no causal mask,
because the indices are already causal -- deriving causality from the row number
drops every index once the KV buffer is offset, which produced an all-NaN layer 0.
- `h20_ops/attn_sm90.py` re-implements `prefill_attn_varlen_sparse` and
`decode_mla_mqa_sparse` with the same signatures as `attention_backend.py`, and
registers them through `patch_imports`, so call sites are untouched and the
FlashMLA-only module is never imported.
- Softmax normalisation clamps `sumexp`, so a padded row with no valid index yields
zeros instead of 0/0 and cannot poison the whole hidden state.
Engram
- `h20_ops/engram_local.py` replaces `deep_ep.EngramBuffer.fetch`, which needs
NCCL GIN (RDMA) and is unavailable on a single node, with a distributed gather
over NVLink built from two `all_to_all`s. Row ownership follows the C++ layout
(`owner = idx // num_local`). Verified against a full reference table on 8 ranks.
Other Blackwell assumptions removed
- DeepSelect's top-k kernel faults at launch on SM90, so `h20_ops/deep_select_sm90.py`
reproduces its contract with `torch.topk` (end truncation, OOB fill, in-place
output views). The shared-layer candidate-block optimisation is disabled because
`fp8_fp4_sparse_mqa_logits` has no SM90 implementation, and the dense MQA-logits
path is used instead; its fp32 output is narrowed for the block-max kernel.
- Decode-time index scoring uses `_decode_p_sum_sm90`: Hopper's paged kernel
hard-requires 64-token pages while the cache uses 128, and the pages are strided
views that cannot be split without a copy.
- `mega_mhc` / `mega_moe`, the FlashAttention-4 CuTe DSL kernels and the fused
norm+rope+cast kernel are all SM100-only and take a Hopper fallback.
- `allow_cuda_sync()` is used for the few places that genuinely need a host read
(all-to-all sizing, the ViT fallback, the NaN diagnostic) instead of disabling
the sync guard globally.
Vendored submodules (not part of this commit; submodule gitlinks are unchanged)
- `vendor/DeepSelect/setup.py`: re-enable the `sm_90a` gencode that is commented
out, otherwise top-k fails with "no kernel image".
Scripts: `h20_attn_check.py`, `h20_engram_check.py`, `h20_topk_check.py`,
`h20_smoke_check.py`, `h20_real_requant_check.py` are unit checks that run on 8x
H20; `h20_ask.py` is a one-prompt harness and `ref_next_token.py` produces the
official reference next-token for arbitrary ids, so both sides can be compared on
identical input.
