mirror of
https://github.com/deepseek-ai/DeepInfer.git
synced 2026-11-02 17:56:22 +08:00
39b086d781
The references were importable and numerically validated, but nothing pointed the model at them: the forward pass still called the compiled kernels, which raise ModuleNotFoundError on a host without TileLang. This adds the routing layer and swaps the call sites. `deepinfer/tk_compat/` exposes the *compiled* signatures -- out-buffers, in-place conventions, argument order -- because that is what the 25 call sites are written against, and adapts to the reference's shape internally. Selection is `DEEPINFER_KERNELS=auto|cuda|ref`; `auto` follows whether TileLang imports. The adapters are not pass-throughs. Each difference was found by running both backends: * `per_token_cast` takes `num_per_channels=n`, the reference takes `block_size=(1, n)`. * `apply_rotary` takes a concatenated cos|sin cache and rotates in place returning None; the reference takes cos and sin separately and returns a new tensor. The adapter splits, calls, and copies back, because the call site relies on the in-place contract. * `engram_gate` takes the pre-multiplied weight and `(eps, clamp_value)`; the reference needs the two weights separately and `(clamp_value, eps)`. Since a product cannot be un-multiplied, `fused_weight` returns the pair on the reference path and the gate unpacks it. * `mhc_pre_big_fuse_fwd` has no single reference: its five steps are composed from the reference's own components. * `mhc_reduce_partials_and_rmsnorm_fwd` likewise; rebuilt from the reference's formula. `moe_topk_gate(force_random=...)` raises rather than silently ignoring the flag. The reference has no equivalent, and dropping it would route tokens differently while looking healthy. Verification, all on the real call signatures: 18 cases compared between `cuda` and `ref` modes, `ALL_MATCH`, 12 bitwise identical including every quantised byte and every routing index; plus a smoke run that executes all 23 APIs with TileLang blocked, `ALL_CALLS_OK`. End to end still has not completed, and the reason is not this change: six GPUs on the host are held by other containers (82 GiB of 139 each), so the 8-way EP run OOMs in allocate_userbuffer and the surviving ranks wait at a barrier. Recorded in the doc along with the command to rerun it.
106 lines
1.4 KiB
Plaintext
106 lines
1.4 KiB
Plaintext
watch_*
|
|
deepinfer.tar.gz
|
|
tracing/
|
|
sweeps/
|
|
# Cross-node lock and temp files written when refreshing the golden on multiple nodes
|
|
tests/models/offline_decode_ppl.json.lock
|
|
tests/models/offline_decode_ppl.json.tmp
|
|
export_only_prof_dir/
|
|
benchmark_results*.jsonl
|
|
# Byte-compiled / optimized / DLL files
|
|
__pycache__/
|
|
*.py[cod]
|
|
*$py.class
|
|
|
|
# C extensions
|
|
*.so
|
|
|
|
# Distribution / packaging
|
|
.Python
|
|
/build/
|
|
develop-eggs/
|
|
dist/
|
|
downloads/
|
|
eggs/
|
|
.eggs/
|
|
lib/
|
|
lib64/
|
|
parts/
|
|
sdist/
|
|
var/
|
|
wheels/
|
|
share/python-wheels/
|
|
*.egg-info/
|
|
.installed.cfg
|
|
*.egg
|
|
MANIFEST
|
|
|
|
# Installer logs
|
|
pip-log.txt
|
|
pip-delete-this-directory.txt
|
|
|
|
# Unit test / coverage reports
|
|
htmlcov/
|
|
.tox/
|
|
.nox/
|
|
.coverage
|
|
.coverage.*
|
|
.cache
|
|
nosetests.xml
|
|
coverage.xml
|
|
*.cover
|
|
*.py,cover
|
|
.hypothesis/
|
|
.pytest_cache/
|
|
cover/
|
|
|
|
# Logs
|
|
*.log
|
|
|
|
# clangd
|
|
compile_commands.json
|
|
.clangd
|
|
|
|
# Environments
|
|
.env
|
|
.venv
|
|
env/
|
|
venv/
|
|
|
|
# IDE
|
|
.idea/
|
|
.vscode/
|
|
.DS_Store
|
|
|
|
# Linter / type checker caches
|
|
.mypy_cache/
|
|
.dmypy.json
|
|
dmypy.json
|
|
.ruff_cache/
|
|
.pkl_memoize_py3/
|
|
.pyre/
|
|
.pytype/
|
|
|
|
# Results
|
|
*.csv
|
|
|
|
# Python pickle files
|
|
*.pkl
|
|
|
|
# vim swap files
|
|
*.swo
|
|
*.swp
|
|
|
|
# Platform extension module artifacts (deepinfer._C); the directory currently holds only .so
|
|
# symlinks (already covered by *.so above) and is ignored explicitly so that non-.so files
|
|
# cannot slip in later
|
|
/deepinfer/_so/
|
|
|
|
# Generated patch + local debug dumps
|
|
h20-sm90.diff
|
|
*.npy
|
|
dump_debug/
|
|
.github-pr-body-*.md
|
|
.py.import-orig
|
|
.py.orig
|