coffee/DeepInfer
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepInfer.git synced 2026-10-31 08:46:21 +08:00
Code Issues Packages Projects Releases Wiki Activity
28 Commits 2 Branches 0 Tags Apache-2.0 15 MiB
Python 89.1%
C++ 10.6%
Shell 0.3%
Go to file
loopyt 0d144cbce4 feat(ops): report fallbacks at use time, and record why the sampler stays put 
The import-time line only counts kernels the selected backend does not export, which says nothing
about whether a running configuration reaches them. `_FallbackNotice` wraps each inherited kernel
and prints once, on first call, so an end-to-end run states the answer itself. On 8xH20 with
speculative decoding off that answer is a single line:

    [ops] kernel 'fused_sample_tl' used from 'cuda': 'triton' does not implement it

Eighteen of the nineteen kernels on that path are Triton and produce exactly the same tokens as the
pure TileLang run. The nineteenth is left there on purpose.

A probe of the engine's own sampling parameters (`DEEPINFER_PROBE_SAMPLING=1`, which stages an
async D2H copy and reports at exit so it does not synchronise inside the forward) settles what
`fused_sample` is being asked to do:

    temperatures: [1.0, 1.0]   top_ps: [0.99, 0.99]   top_ks: [-1, -1]

That is a real nucleus sample over 4096 candidates, not an argmax, so the draw matters. Two things
stand in the way of reproducing it bit for bit, both internal to TileLang: `T.rng_init`/`T.rng_rand`
are a Philox stream implemented in its C++ codegen, so a Triton kernel draws a different stream and
every sampled token at temperature > 0 would change; and the baseline pins the layout of its LSE
accumulator precisely to fix the summation order, after which probabilities are quantised into a
2^31-scaled uint32 CDF where a one-ulp difference can move a column across a boundary.

Swapping it anyway would be a behaviour change reported as a kernel swap. The per-kernel fallback
exists for exactly this case, so the sampler stays on TileLang and the rest of the path does not.
`triton/__init__.py` records this so the gap is not mistaken for an oversight, along with the order
in which porting it later would have to proceed: a Philox that matches the stream exactly, then the
radix filter, then the fixed-point CDF and the two-level search.
2026-10-22 08:05:05 +08:00
csrc initial commit 2026-10-20 12:09:42 +08:00
deep_cluster style(sm90): satisfy ruff and add the missing type annotations 2026-10-21 17:09:08 +08:00
deepinfer feat(ops): report fallbacks at use time, and record why the sampler stays put 2026-10-22 08:05:05 +08:00
doc update Q&A documentation 2026-10-20 17:06:57 +08:00
eval initial commit 2026-10-20 12:09:42 +08:00
examples initial commit 2026-10-20 12:09:42 +08:00
scripts feat(triton): port fused_transform, compress, prepare_v4 and both metadata kernels 2026-10-22 07:47:24 +08:00
tests initial commit 2026-10-20 12:09:42 +08:00
vendor initial commit 2026-10-20 12:09:42 +08:00
.clang-format initial commit 2026-10-20 12:09:42 +08:00
.gitignore test(sm90): decode FP4 by table lookup in the quant check 2026-10-21 17:15:16 +08:00
.gitmodules initial commit 2026-10-20 12:09:42 +08:00
build.sh initial commit 2026-10-20 12:09:42 +08:00
CONTRIBUTING.md initial commit 2026-10-20 12:09:42 +08:00
H20-DEBUG-PLAN.md fix(debug): move debug_dump out of h20_ops so other platforms can import it 2026-10-21 22:37:49 +08:00
H20-DRIFT-ANALYSIS.md docs: record that plain-text prompting answers correctly 2026-10-21 22:50:57 +08:00
H20-PORTING-JOURNAL.md docs: record the two-branch split and the upstream PR link 2026-10-21 17:16:46 +08:00
H20-SM90-FINDINGS.md feat: add SM90 (Hopper) support for DeepSeek-V4.1 inference 2026-10-21 15:18:56 +08:00
H20-TRITON-PLAN.md feat(ops): report fallbacks at use time, and record why the sampler stays put 2026-10-22 08:05:05 +08:00
LICENSE initial commit 2026-10-20 12:09:42 +08:00
Makefile initial commit 2026-10-20 12:09:42 +08:00
pyproject.toml feat(ops): add a selectable Triton backend, and measure the real kernel surface 2026-10-22 00:04:27 +08:00
README.md initial commit 2026-10-20 12:09:42 +08:00
requirements-ascend.txt initial commit 2026-10-20 12:09:42 +08:00
requirements-dev.txt initial commit 2026-10-20 12:09:42 +08:00
requirements-nvidia.txt initial commit 2026-10-20 12:09:42 +08:00
requirements.txt initial commit 2026-10-20 12:09:42 +08:00
setup.py initial commit 2026-10-20 12:09:42 +08:00

README.md

DeepInfer

DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.

Supported models:

  • DeepSeek-V4.1-Flash

Supported features:

  • Optimized kernels, model and sampler runtime
  • Parallelism: Attention DP / balanced CP, MoE EP + EPLB
  • DSpark and confidence-scheduled verification
  • SWA bounded replay
  • Asynchronous scheduling (zero-overhead batch scheduler)
  • Batch-invariant inference
  • Deterministic sampling
  • Structured outputs
Feature details and configuration
  • Optimized kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels.
  • DSpark and confidence-scheduled verification: accelerate decoding with draft tokens and use confidence estimates to schedule verification. Set DEEPINFER_ENABLE_MTP_VARLEN=1; DEEPINFER_USE_NEXTN_DECODE controls the maximum draft length (default: 5).
  • Decoder SWA bounded replay: reduce decoder prefill work by replaying only a bounded suffix. Set DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 for the default V4.1 settings; the retained suffix must cover the SWA window plus the maximum backtracking length.
  • Asynchronous scheduling: overlap CPU batch scheduling with device execution. Enabled by default.
  • Batch-invariant inference: keep model results identical across different batch compositions. Enable with DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only).
  • Deterministic sampling: produce identical outputs for identical input tokens and sampling parameters. Set DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0 and pass a fixed seq_seed to SamplingParams (NVIDIA only).
  • Structured outputs: constrain generation to a regex, JSON schema, or EBNF grammar. Pass so_args={"schema_type": ..., "schema": ...} to SamplingParams; see examples.

Supported hardware:

  • NVIDIA SM100 GPUs
  • Huawei Ascend 950 NPUs

Required for a complete inference service (not included in this release):

  • A KV cache system
  • VE/Prefill/Decode disaggregation (most required engine functionality is already implemented)
  • Load balancing across data-parallel replicas for both prefill and decode

Getting started

Prerequisites

  • Linux
  • GCC >= 13.0
  • Python 3.12
  • CUDA 13 (NVIDIA only)
  • CANN 9.2 (Ascend only)

Build from source

Before building DeepInfer, please create and activate an empty Python 3.12 environment.

# Use `ascend` instead for the Ascend platform
# Keep it exported in the environment where you run DeepInfer
export DEEPINFER_ACCELERATOR=nvidia
./build.sh

Offline inference

The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, is downloaded into the Hugging Face cache on the first run; set DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 to run from a local copy instead.

python examples/offline_inference.py

See examples for details.

Performance benchmarks

Prefill Decode
Prefill: non-cached input throughput vs. latency Decode: output throughput vs. per-user speed

Measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see run_sweep.sh.

Evaluation benchmarks

See eval/README.md.

License

DeepInfer is released under the Apache License 2.0. See LICENSE for details.

Acknowledgments

DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. We thank their developers.

Citation

@misc{deepinfer2026,
  title = {DeepInfer: High-performance, large-scale inference framework for DeepSeek models},
  author = {DeepSeek-AI},
  year = {2026},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/deepseek-ai/DeepInfer}},
}