mirror of https://github.com/deepseek-ai/DeepInfer.git synced 2026-11-01 01:06:23 +08:00

Go to file

Jahidul Hossain Mekat 6a934611d5 Let a chunked prefill continue above the prefill watermark A prompt that is prefilled in chunks reserves the KV blocks of its whole prompt with its first chunk and goes back to `waiting` until its next chunk. NORMAL engines only try a prompt step while the cache usage is at most GPU_CACHE_PREFILL_WATERMARK (0.9), which is meant to keep fresh prompts from filling the cache. The blocks the prompt reserved count in that usage, and only its remaining chunks can release them, so it was never scheduled again as long as they alone kept the usage above the watermark. A prompt that needs more than 90% of the cache by itself (a long context on a cache that barely holds it) left the engine idle for good, with the request unfinished. Skip the watermark when the request at the front of `waiting` is a chunked prefill in progress. It needs no new blocks. The test prefills a 37-token prompt that takes all 10 usable blocks of a small cache.		2026-10-20 14:49:40 +00:00
csrc	Let a chunked prefill continue above the prefill watermark	2026-10-20 14:49:40 +00:00
deep_cluster	initial commit	2026-10-20 12:09:42 +08:00
deepinfer	initial commit	2026-10-20 12:09:42 +08:00
doc	update Q&A documentation	2026-10-20 17:06:57 +08:00
eval	initial commit	2026-10-20 12:09:42 +08:00
examples	initial commit	2026-10-20 12:09:42 +08:00
scripts	initial commit	2026-10-20 12:09:42 +08:00
tests	Let a chunked prefill continue above the prefill watermark	2026-10-20 14:49:40 +00:00
vendor	initial commit	2026-10-20 12:09:42 +08:00
.clang-format	initial commit	2026-10-20 12:09:42 +08:00
.gitignore	initial commit	2026-10-20 12:09:42 +08:00
.gitmodules	initial commit	2026-10-20 12:09:42 +08:00
build.sh	initial commit	2026-10-20 12:09:42 +08:00
CONTRIBUTING.md	initial commit	2026-10-20 12:09:42 +08:00
LICENSE	initial commit	2026-10-20 12:09:42 +08:00
Makefile	initial commit	2026-10-20 12:09:42 +08:00
pyproject.toml	initial commit	2026-10-20 12:09:42 +08:00
README.md	initial commit	2026-10-20 12:09:42 +08:00
requirements-ascend.txt	initial commit	2026-10-20 12:09:42 +08:00
requirements-dev.txt	initial commit	2026-10-20 12:09:42 +08:00
requirements-nvidia.txt	initial commit	2026-10-20 12:09:42 +08:00
requirements.txt	initial commit	2026-10-20 12:09:42 +08:00
setup.py	initial commit	2026-10-20 12:09:42 +08:00

README.md

DeepInfer

DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.

Supported models:

DeepSeek-V4.1-Flash

Supported features:

Optimized kernels, model and sampler runtime
Parallelism: Attention DP / balanced CP, MoE EP + EPLB
DSpark and confidence-scheduled verification
SWA bounded replay
Asynchronous scheduling (zero-overhead batch scheduler)
Batch-invariant inference
Deterministic sampling
Structured outputs

Feature details and configuration

Optimized kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels.
DSpark and confidence-scheduled verification: accelerate decoding with draft tokens and use confidence estimates to schedule verification. Set DEEPINFER_ENABLE_MTP_VARLEN=1; DEEPINFER_USE_NEXTN_DECODE controls the maximum draft length (default: 5).
Decoder SWA bounded replay: reduce decoder prefill work by replaying only a bounded suffix. Set DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 for the default V4.1 settings; the retained suffix must cover the SWA window plus the maximum backtracking length.
Asynchronous scheduling: overlap CPU batch scheduling with device execution. Enabled by default.
Batch-invariant inference: keep model results identical across different batch compositions. Enable with DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only).
Deterministic sampling: produce identical outputs for identical input tokens and sampling parameters. Set DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0 and pass a fixed seq_seed to SamplingParams (NVIDIA only).
Structured outputs: constrain generation to a regex, JSON schema, or EBNF grammar. Pass so_args={"schema_type": ..., "schema": ...} to SamplingParams; see examples.

Supported hardware:

NVIDIA SM100 GPUs
Huawei Ascend 950 NPUs

Required for a complete inference service (not included in this release):

A KV cache system
VE/Prefill/Decode disaggregation (most required engine functionality is already implemented)
Load balancing across data-parallel replicas for both prefill and decode

Getting started

Prerequisites

Linux
GCC >= 13.0
Python 3.12
CUDA 13 (NVIDIA only)
CANN 9.2 (Ascend only)

Build from source

Before building DeepInfer, please create and activate an empty Python 3.12 environment.

# Use `ascend` instead for the Ascend platform
# Keep it exported in the environment where you run DeepInfer
export DEEPINFER_ACCELERATOR=nvidia
./build.sh

Offline inference

The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, is downloaded into the Hugging Face cache on the first run; set DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 to run from a local copy instead.

python examples/offline_inference.py

See examples for details.

Performance benchmarks

Prefill	Decode

Measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see run_sweep.sh.

Evaluation benchmarks

See eval/README.md.

License

DeepInfer is released under the Apache License 2.0. See LICENSE for details.

Acknowledgments

DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. We thank their developers.

Citation

@misc{deepinfer2026,
  title = {DeepInfer: High-performance, large-scale inference framework for DeepSeek models},
  author = {DeepSeek-AI},
  year = {2026},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/deepseek-ai/DeepInfer}},
}