mirror of
https://github.com/deepseek-ai/DeepInfer.git
synced 2026-10-31 08:46:21 +08:00
3c15952e76
Narrows the SM90 numerical divergence to layer 0's FFN. The sparse MLA attention kernel is not the problem: at layer 0 its output matches the reference with cos=0.99897 / rel_err=4.7e-2, while the full layer output is already at 12.7% and the error compounds to cos=0.865 by layer 11, which finally flips the top-1 token. Tooling - `h20_ops/debug_dump.py`: opt-in capture (`DEEPINFER_DUMP_LAYERS=<dir>`), last position only, reduced over `hc_mult`, written to `.npz` at exit. `DEEPINFER_DUMP_TOKENS=<n>` gates on the real prompt length, because the engine runs a dummy profile forward with a different token count before it allocates the KV cache and that dump would otherwise be mistaken for the real one. - `scripts/ref_layer_dump.py`: the same anchors from the reference model via forward hooks (block output, attention sub-layer output, embedding), plus the final logits and greedy token. - `scripts/compare_layers.py`: cosine similarity, relative L2 error and norm ratio per layer, reporting the first layer above a tolerance. Two traps this had to fix before the numbers meant anything - `decoder_layer` returns the *sub-layer* output; the residual stream lives in `residual` (`HCLazyPostData`, hc_mult=4). Comparing without merging them through `hc_post` produced a bogus cos of 0.02 at every layer. - `DEEPINFER_DEBUG_NAN` and the dump both need a host copy inside the forward, so both take `allow_cuda_sync()` rather than fighting the engine's sync guard. Also documents the reproduction case and the findings in H20-DEBUG-PLAN.md. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| attn_sm90.py | ||
| debug_dump.py | ||
| deep_select_sm90.py | ||
| engram_local.py | ||
| quant.py | ||
| sparse_mla_sm90.py | ||
| vit_attn.py | ||