mirror of
https://github.com/deepseek-ai/DeepInfer.git
synced 2026-10-31 00:36:20 +08:00
afbfdb6c7a
`fused_sample` for the configuration the engine builds: 4096 columns after the force_topk cut,
sampling on, rejection off. With one tile the baseline's two-level search collapses -- the coarse
offset *is* the token id -- so this implements log-softmax, the top-k/top-p cut as a threshold plus
the tie rule, renormalisation, a fixed-point cumulative distribution and one Philox draw.
The draw matches cuRAND bit for bit (that is what the previous commit established). The rest does
not all match, and the split is measured rather than claimed:
engine's configuration (top_p=0.99, 2644 kept columns) tokens identical on every row
narrow cuts (top_k=128, top_p=0.5) ~90% of rows
Kept sets match column for column and log-probabilities agree to ~1e-5 in every case. What differs
is the token, and the cause is a discontinuity in the baseline rather than a defect here: the draw
is `d % total` with `total = sum(floor(p_i * 2^31))`, and the baseline's rejection limit
`(2^32-1) // total * total` steps from `2 * total` to `1 * total` exactly at `total == 2^31` -- which
`total` sits next to, because the probabilities sum to one. On the near side almost no draw is
rejected, on the far side about half are, and each rejection moves to the next draw. A few units of
rounding error in `total`, which is a reduction order that cannot match, therefore change the token
outright. Clamping `total` under 2^31 does not help: the baseline crosses in either direction too,
so it only relocates the disagreement (30 rows instead of 11).
Four bugs found on the way, all worth recording because each looked like something else:
- Triton evaluates `//`, `%` and `<=` on **uint32 operands with signed arithmetic**, so 4294967295
behaves as -1. That made `limit` come out as `2^32 - total`, flipping the rejection decision.
Fixed by doing the whole cumulative distribution in int64.
- Nested runtime `if`s produced a threshold that was not the one assigned inside the branch; the
cuts are now combined with `tl.where` instead.
- The threshold was taken from the probability array and compared against logits. The sort has to be
of the logits (equivalent, since the map is monotone -- the mass is recomputed from the sorted
logits rather than sorting twice).
- Renormalising by division rather than through the log domain, as the baseline does, moved the
token agreement from 76% to 91% before the remaining gap was understood.
Also gives the A/B harness a fourth case element: an agreement floor for outputs whose answer is a
random draw, so the rate is reported rather than compared for equality -- a comparison that can pass
without testing anything would be worse than none.
|
||
|---|---|---|
| csrc | ||
| deep_cluster | ||
| deepinfer | ||
| doc | ||
| eval | ||
| examples | ||
| scripts | ||
| tests | ||
| vendor | ||
| .clang-format | ||
| .gitignore | ||
| .gitmodules | ||
| build.sh | ||
| CONTRIBUTING.md | ||
| H20-DEBUG-PLAN.md | ||
| H20-DRIFT-ANALYSIS.md | ||
| H20-PORTING-JOURNAL.md | ||
| H20-SM90-FINDINGS.md | ||
| H20-TRITON-PLAN.md | ||
| LICENSE | ||
| Makefile | ||
| pyproject.toml | ||
| README.md | ||
| requirements-ascend.txt | ||
| requirements-dev.txt | ||
| requirements-nvidia.txt | ||
| requirements.txt | ||
| setup.py | ||
DeepInfer
DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.
Supported models:
- DeepSeek-V4.1-Flash
Supported features:
- Optimized kernels, model and sampler runtime
- Parallelism: Attention DP / balanced CP, MoE EP + EPLB
- DSpark and confidence-scheduled verification
- SWA bounded replay
- Asynchronous scheduling (zero-overhead batch scheduler)
- Batch-invariant inference
- Deterministic sampling
- Structured outputs
Feature details and configuration
- Optimized kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels.
- DSpark and confidence-scheduled verification: accelerate decoding with draft tokens and use confidence estimates to schedule verification. Set
DEEPINFER_ENABLE_MTP_VARLEN=1;DEEPINFER_USE_NEXTN_DECODEcontrols the maximum draft length (default: 5). - Decoder SWA bounded replay: reduce decoder prefill work by replaying only a bounded suffix. Set
DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compactandDEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140for the default V4.1 settings; the retained suffix must cover the SWA window plus the maximum backtracking length. - Asynchronous scheduling: overlap CPU batch scheduling with device execution. Enabled by default.
- Batch-invariant inference: keep model results identical across different batch compositions. Enable with
DEEPINFER_BATCH_INVARIANT=1(NVIDIA only). - Deterministic sampling: produce identical outputs for identical input tokens and sampling parameters. Set
DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0and pass a fixedseq_seedtoSamplingParams(NVIDIA only). - Structured outputs: constrain generation to a regex, JSON schema, or EBNF grammar. Pass
so_args={"schema_type": ..., "schema": ...}toSamplingParams; see examples.
Supported hardware:
- NVIDIA SM100 GPUs
- Huawei Ascend 950 NPUs
Required for a complete inference service (not included in this release):
- A KV cache system
- VE/Prefill/Decode disaggregation (most required engine functionality is already implemented)
- Load balancing across data-parallel replicas for both prefill and decode
Getting started
Prerequisites
- Linux
- GCC >= 13.0
- Python 3.12
- CUDA 13 (NVIDIA only)
- CANN 9.2 (Ascend only)
Build from source
Before building DeepInfer, please create and activate an empty Python 3.12 environment.
# Use `ascend` instead for the Ascend platform
# Keep it exported in the environment where you run DeepInfer
export DEEPINFER_ACCELERATOR=nvidia
./build.sh
Offline inference
The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, is
downloaded into the Hugging Face cache on the first run; set DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 to
run from a local copy instead.
python examples/offline_inference.py
See examples for details.
Performance benchmarks
| Prefill | Decode |
|---|---|
 |
 |
Measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see run_sweep.sh.
Evaluation benchmarks
See eval/README.md.
License
DeepInfer is released under the Apache License 2.0. See LICENSE for details.
Acknowledgments
DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. We thank their developers.
Citation
@misc{deepinfer2026,
title = {DeepInfer: High-performance, large-scale inference framework for DeepSeek models},
author = {DeepSeek-AI},
year = {2026},
publisher = {GitHub},
howpublished = {\url{https://github.com/deepseek-ai/DeepInfer}},
}