mirror of
https://github.com/deepseek-ai/DeepInfer.git
synced 2026-10-31 00:36:20 +08:00
docs: correct what importing flag_gems actually does
Two files claimed that importing `flag_gems` registers ATen implementations process-wide, and used that to justify vendoring. The conclusion was right for the wrong reason, and the reason was wrong enough to distort planning: it made any broader FlagGems integration look reckless. Measured instead of assumed. Hashing `torch._C._dispatch_dump_table` for `aten::rms_norm`, `aten::silu`, `aten::softmax`, `aten::topk`, `aten::lerp` before and after `import flag_gems`: every table unchanged. The package `__init__` creates the `aten_lib` handle and imports the op modules; registration happens only when `enable()` or `only_enable(include=[...])` is called. And `only_enable(include=["rms_norm"])` changes exactly `aten::rms_norm` and nothing else. So the blast radius lives in `enable()`, not in the import -- which is what makes a wider integration tractable rather than risky. Vendoring still stands, for two reasons that survive the correction: the package root star-imports every op module and loads Triton TLE, which dlopen's `libflagcx` (a native dependency this deployment does not otherwise need); and the released kernel file has no internal imports, so the package bought nothing for that operator. Both files now say this, and say that it is a correction, so the next reader does not rediscover it.
This commit is contained in:
parent
6d337a82f0
commit
3486b93ac9
20
deepinfer/h20_ops/vendor/flashmla_sparse.py
vendored
20
deepinfer/h20_ops/vendor/flashmla_sparse.py
vendored
@ -39,13 +39,19 @@
|
||||
#
|
||||
# Why vendored instead of depending on `flag_gems`:
|
||||
#
|
||||
# 1. `flag_gems/__init__.py` registers ATen implementations process-wide
|
||||
# (`torch.library.Library("aten", "IMPL")` then `runtime.replace_customized_ops`). Importing
|
||||
# the package to reach one kernel would silently swap operators under the whole engine, which
|
||||
# changes numerics of code that has nothing to do with attention. This module imports only
|
||||
# torch and triton, so it cannot do that.
|
||||
# 2. The released 5.0.2 file has no `flag_gems` internal imports at all, so there is nothing to
|
||||
# gain from the dependency.
|
||||
# 1. Import cost and a native dependency. `flag_gems/__init__.py` star-imports every op module and
|
||||
# pulls in Triton TLE, which loads `libflagcx` through ctypes -- an .so this deployment does not
|
||||
# otherwise need. A single attention kernel should not drag that in. This module imports only
|
||||
# torch and triton.
|
||||
# 2. This file has no `flag_gems` internal imports at all, so there was nothing to gain from the
|
||||
# package for this operator.
|
||||
#
|
||||
# Correction, recorded because an earlier version of this header said the opposite: importing
|
||||
# `flag_gems` does *not* register ATen implementations. It creates the `aten_lib` handle and imports
|
||||
# the op modules, but registration happens only when `enable()` or `only_enable(include=[...])` is
|
||||
# called. Measured by hashing `torch._C._dispatch_dump_table` before and after `import flag_gems`:
|
||||
# unchanged for `aten::rms_norm`, `aten::silu`, `aten::softmax`, `aten::topk`. So a blanket
|
||||
# operator swap was never a risk of merely importing the package -- the risk lives in `enable()`.
|
||||
#
|
||||
# Why 5.0.2 rather than FlagGems master: master's copy is ~1200 lines with a TLE fast path gated on
|
||||
# triton internals, and its accumulation is fp16. This one is 329 lines, accumulates in fp32, and
|
||||
|
||||
@ -9,11 +9,20 @@ operator; it is now vendored, verified against the TileLang kernel, and wired in
|
||||
|
||||
`flagos-ai/FlagGems`, file `src/flag_gems/fused/flashmla_sparse.py`, function
|
||||
`flash_mla_sparse_fwd`. Vendored as `deepinfer/h20_ops/vendor/flashmla_sparse.py` from the **PyPI
|
||||
5.0.2 wheel**, which has no `flag_gems` internal imports at all -- only torch and triton. That
|
||||
matters: importing `flag_gems` itself registers ATen implementations process-wide
|
||||
(`torch.library.Library("aten", "IMPL")` then `runtime.replace_customized_ops`), so depending on the
|
||||
package to reach one kernel would swap operators under the whole engine. The file header records the
|
||||
source md5 and the two local deviations.
|
||||
5.0.2 wheel**. Two reasons, plus a correction to what this file said previously:
|
||||
|
||||
* Import cost: `flag_gems/__init__.py` star-imports every op module and pulls in Triton TLE, which
|
||||
loads `libflagcx` through ctypes -- an .so this deployment does not otherwise need. One attention
|
||||
kernel should not drag that in.
|
||||
* The released file has no `flag_gems` internal imports at all (only torch and triton), so the
|
||||
package bought nothing for this particular operator.
|
||||
|
||||
Correction: importing `flag_gems` does **not** register ATen implementations, contrary to the earlier
|
||||
wording here. It creates the `aten_lib` handle; registration happens only in `enable()` or
|
||||
`only_enable(include=[...])`. Measured by hashing `torch._C._dispatch_dump_table` before and after
|
||||
`import flag_gems`: unchanged for `aten::rms_norm`, `aten::silu`, `aten::softmax`, `aten::topk`. The
|
||||
blast radius lives in `enable()`, not in the import. That distinction is what makes a broader
|
||||
FlagGems integration tractable -- see `H20-FLAGGEMS-PLAN.md`.
|
||||
|
||||
Master's copy (~1200 lines, TLE fast path, fp16 accumulate) also works and was verified first; 5.0.2
|
||||
is used instead because it is a release, is 329 lines, accumulates in fp32, and already guards the
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user