coffee/DeepInfer
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepInfer.git synced 2026-10-31 00:36:20 +08:00
Code Issues Packages Projects Releases Wiki Activity
46 Commits 2 Branches 0 Tags Apache-2.0 15 MiB
Python 89.1%
C++ 10.6%
Shell 0.3%
Go to file
loopyt f10b3a7052 docs(h20): the missing 16 kernels are the small half of the TileLang problem 
I had been reporting the port as "16 kernels left". That was wrong in a way that matters, so
this corrects it with a measurement rather than an estimate.

The measurement: make `tilelang`, `tile_kernels` and `tvm_ffi` unimportable via a meta-path
finder installed from sitecustomize -- sitecustomize specifically, because the eight DP ranks
are spawned and would not inherit an import in the parent. Then try to import the engine.
It fails at import, before any kernel is reached: `hugging_face/weights.py` pulls in
`layers/linear.py`, whose module body has `import tile_kernels`.

The reason the kernel ports did not help is that `tilelang_ops` only ever covered 44 kernels.
The model forward calls `tile_kernels` directly -- 25 module-level imports across 13 files,
23 distinct APIs -- and `tile_kernels` is itself a TileLang package: its `__init__` line 1 is
`import tilelang`, with 166 references across 140 modules. Keeping tile_kernels keeps
TileLang.

The 23 APIs are recorded in three groups, because they cost very differently: six are
configuration knobs and string helpers; six are hot-path quantization and routing kernels
(`per_token_cast` appears in four modules); ten are MHC/Engram kernels specific to this model
and the largest piece of work.

There is one real shortcut, and it is worth knowing before anyone starts writing Triton:
`tile_kernels/torch/` holds pure-torch reference implementations of nearly all of them, with
zero TileLang references. It is currently unreachable only because the package's `__init__`
imports TileLang eagerly -- so making that lazy buys a correct-but-slow bring-up path, from
which the hot kernels can be replaced one at a time as profiling justifies it. `flag_gems`
cannot stand in: it is 903 ATen-level operator replacements and does not carry these fused
kernels.
2026-10-22 15:28:00 +08:00
csrc initial commit 2026-10-20 12:09:42 +08:00
deep_cluster style(sm90): satisfy ruff and add the missing type annotations 2026-10-21 17:09:08 +08:00
deepinfer feat(h20): port the four MTP kernels, so speculative decoding no longer needs TileLang 2026-10-22 14:16:33 +08:00
doc update Q&A documentation 2026-10-20 17:06:57 +08:00
eval initial commit 2026-10-20 12:09:42 +08:00
examples initial commit 2026-10-20 12:09:42 +08:00
scripts test(h20): catch a factory that returns nothing before eight ranks do 2026-10-22 14:16:41 +08:00
tests initial commit 2026-10-20 12:09:42 +08:00
vendor initial commit 2026-10-20 12:09:42 +08:00
.clang-format initial commit 2026-10-20 12:09:42 +08:00
.gitignore test(sm90): decode FP4 by table lookup in the quant check 2026-10-21 17:15:16 +08:00
.gitmodules initial commit 2026-10-20 12:09:42 +08:00
build.sh initial commit 2026-10-20 12:09:42 +08:00
CONTRIBUTING.md initial commit 2026-10-20 12:09:42 +08:00
H20-DEBUG-PLAN.md fix(debug): move debug_dump out of h20_ops so other platforms can import it 2026-10-21 22:37:49 +08:00
H20-DRIFT-ANALYSIS.md docs: record that plain-text prompting answers correctly 2026-10-21 22:50:57 +08:00
H20-PORTING-JOURNAL.md docs: record the two-branch split and the upstream PR link 2026-10-21 17:16:46 +08:00
H20-SM90-FINDINGS.md feat: add SM90 (Hopper) support for DeepSeek-V4.1 inference 2026-10-21 15:18:56 +08:00
H20-TRITON-PLAN.md test(h20): check what the model actually says, in a file rather than on the command line 2026-10-22 10:53:01 +08:00
LICENSE initial commit 2026-10-20 12:09:42 +08:00
Makefile initial commit 2026-10-20 12:09:42 +08:00
pyproject.toml test(h20): compare two hosts' Triton builds, not just two backends on one host 2026-10-22 10:53:36 +08:00
README.md initial commit 2026-10-20 12:09:42 +08:00
requirements-ascend.txt initial commit 2026-10-20 12:09:42 +08:00
requirements-dev.txt initial commit 2026-10-20 12:09:42 +08:00
requirements-nvidia.txt initial commit 2026-10-20 12:09:42 +08:00
requirements.txt initial commit 2026-10-20 12:09:42 +08:00
setup.py initial commit 2026-10-20 12:09:42 +08:00
TODO.md docs(h20): the missing 16 kernels are the small half of the TileLang problem 2026-10-22 15:28:00 +08:00

README.md

DeepInfer

DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.

Supported models:

  • DeepSeek-V4.1-Flash

Supported features:

  • Optimized kernels, model and sampler runtime
  • Parallelism: Attention DP / balanced CP, MoE EP + EPLB
  • DSpark and confidence-scheduled verification
  • SWA bounded replay
  • Asynchronous scheduling (zero-overhead batch scheduler)
  • Batch-invariant inference
  • Deterministic sampling
  • Structured outputs
Feature details and configuration
  • Optimized kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels.
  • DSpark and confidence-scheduled verification: accelerate decoding with draft tokens and use confidence estimates to schedule verification. Set DEEPINFER_ENABLE_MTP_VARLEN=1; DEEPINFER_USE_NEXTN_DECODE controls the maximum draft length (default: 5).
  • Decoder SWA bounded replay: reduce decoder prefill work by replaying only a bounded suffix. Set DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 for the default V4.1 settings; the retained suffix must cover the SWA window plus the maximum backtracking length.
  • Asynchronous scheduling: overlap CPU batch scheduling with device execution. Enabled by default.
  • Batch-invariant inference: keep model results identical across different batch compositions. Enable with DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only).
  • Deterministic sampling: produce identical outputs for identical input tokens and sampling parameters. Set DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0 and pass a fixed seq_seed to SamplingParams (NVIDIA only).
  • Structured outputs: constrain generation to a regex, JSON schema, or EBNF grammar. Pass so_args={"schema_type": ..., "schema": ...} to SamplingParams; see examples.

Supported hardware:

  • NVIDIA SM100 GPUs
  • Huawei Ascend 950 NPUs

Required for a complete inference service (not included in this release):

  • A KV cache system
  • VE/Prefill/Decode disaggregation (most required engine functionality is already implemented)
  • Load balancing across data-parallel replicas for both prefill and decode

Getting started

Prerequisites

  • Linux
  • GCC >= 13.0
  • Python 3.12
  • CUDA 13 (NVIDIA only)
  • CANN 9.2 (Ascend only)

Build from source

Before building DeepInfer, please create and activate an empty Python 3.12 environment.

# Use `ascend` instead for the Ascend platform
# Keep it exported in the environment where you run DeepInfer
export DEEPINFER_ACCELERATOR=nvidia
./build.sh

Offline inference

The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, is downloaded into the Hugging Face cache on the first run; set DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 to run from a local copy instead.

python examples/offline_inference.py

See examples for details.

Performance benchmarks

Prefill Decode
Prefill: non-cached input throughput vs. latency Decode: output throughput vs. per-user speed

Measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see run_sweep.sh.

Evaluation benchmarks

See eval/README.md.

License

DeepInfer is released under the Apache License 2.0. See LICENSE for details.

Acknowledgments

DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. We thank their developers.

Citation

@misc{deepinfer2026,
  title = {DeepInfer: High-performance, large-scale inference framework for DeepSeek models},
  author = {DeepSeek-AI},
  year = {2026},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/deepseek-ai/DeepInfer}},
}