coffee/DeepInfer
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepInfer.git synced 2026-11-01 01:06:23 +08:00
Code Issues Packages Projects Releases Wiki Activity
4 Commits 2 Branches 0 Tags Apache-2.0 15 MiB
Python 89.1%
C++ 10.6%
Shell 0.3%
Go to file
akash6murali 2afdb3d2d8 test: add unit tests for SamplingParams, CacheConfig, ParallelConfig, SchedulerConfig 
The config and sampling-parameter classes had zero test coverage despite
containing non-trivial validation logic that is exercised on every engine
startup.  This PR adds two focused, CPU-only test files that cover all
_verify_args paths and several construction edge cases.

tests/test_sampling_params.py (47 tests)
  Covers every validation rule in SamplingParams._verify_args:
  - temperature: non-negative check, greedy-mode clamp (< SAMPLING_EPS
    → temperature=SAMPLING_EPS, top_p=1.0, top_k=-1).
  - top_p: exclusive lower bound (> 0.0) and upper bound (<= 1.0).
  - top_k: -1 (disabled) or [1, INT32_MAX]; zero and other negatives rejected.
  - max_tokens: >= 1 and <= INT32_MAX; notably max_tokens=0 now raises
    ValueError (the original code silently coerced 0 → 1 before validation).
  - top_logprobs: [0, 1000].
  - seq_seed: auto-generation, explicit value, uint64 masking, non-int raises.
  - logprobs deprecation: logprobs=False emits no warning; logprobs=True
    emits DeprecationWarning matching "logprobs.*deprecated".
  - Optional fields: golden_token_ids, so_args, ignore_eos, first_n_token_no_eos,
    eos_id, prompt_logprobs.
  - Greedy mode: temperature=0 overrides top_k and top_p; non-zero preserves them.

tests/test_config.py (23 tests)
  CacheConfig:
  - Valid construction, boundary values (utilization=1.0), lookahead_tokens,
    deferred-field initialisation (num_gpu_blocks etc. start as None).
  - Rejects gpu_memory_utilization > 1.0 (existing check).
  - Two tests marked xfail(strict=True) for the missing lower-bound guard
    (0.0 and negative values currently pass silently).  These will flip to
    XPASS once PR fix/cache-config-validation is merged.

  ParallelConfig:
  - Auto-derivation of data_parallel_size when 0 is passed (min 1).
  - world_size, data_parallel_rank, model_parallel_size, engram derivation.
  - AssertionError when (cp * dp) is not divisible by ep.
  - AssertionError when engine_rank is out of range.

  SchedulerConfig:
  - max_num_batched_tokens auto-default (>= max(max_model_len, 4096)).
  - ValueError when explicit max_num_batched_tokens < max_model_len and
    chunked_prefill=False; passes when chunked_prefill=True.
  - ValueError for prefill_attempt_interval <= 0 and
    prefill_any_attempt_interval <= 0.
  - runtime_max_model_len >= max_model_len.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-10-20 14:28:11 -04:00
csrc initial commit 2026-10-20 12:09:42 +08:00
deep_cluster initial commit 2026-10-20 12:09:42 +08:00
deepinfer initial commit 2026-10-20 12:09:42 +08:00
doc update Q&A EN doc 2026-10-20 19:03:13 +08:00
eval initial commit 2026-10-20 12:09:42 +08:00
examples initial commit 2026-10-20 12:09:42 +08:00
scripts initial commit 2026-10-20 12:09:42 +08:00
tests test: add unit tests for SamplingParams, CacheConfig, ParallelConfig, SchedulerConfig 2026-10-20 14:28:11 -04:00
vendor initial commit 2026-10-20 12:09:42 +08:00
.clang-format initial commit 2026-10-20 12:09:42 +08:00
.gitignore initial commit 2026-10-20 12:09:42 +08:00
.gitmodules initial commit 2026-10-20 12:09:42 +08:00
build.sh initial commit 2026-10-20 12:09:42 +08:00
CONTRIBUTING.md initial commit 2026-10-20 12:09:42 +08:00
LICENSE initial commit 2026-10-20 12:09:42 +08:00
Makefile initial commit 2026-10-20 12:09:42 +08:00
pyproject.toml initial commit 2026-10-20 12:09:42 +08:00
README.md initial commit 2026-10-20 12:09:42 +08:00
requirements-ascend.txt initial commit 2026-10-20 12:09:42 +08:00
requirements-dev.txt initial commit 2026-10-20 12:09:42 +08:00
requirements-nvidia.txt initial commit 2026-10-20 12:09:42 +08:00
requirements.txt initial commit 2026-10-20 12:09:42 +08:00
setup.py initial commit 2026-10-20 12:09:42 +08:00

README.md

DeepInfer

DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.

Supported models:

  • DeepSeek-V4.1-Flash

Supported features:

  • Optimized kernels, model and sampler runtime
  • Parallelism: Attention DP / balanced CP, MoE EP + EPLB
  • DSpark and confidence-scheduled verification
  • SWA bounded replay
  • Asynchronous scheduling (zero-overhead batch scheduler)
  • Batch-invariant inference
  • Deterministic sampling
  • Structured outputs
Feature details and configuration
  • Optimized kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels.
  • DSpark and confidence-scheduled verification: accelerate decoding with draft tokens and use confidence estimates to schedule verification. Set DEEPINFER_ENABLE_MTP_VARLEN=1; DEEPINFER_USE_NEXTN_DECODE controls the maximum draft length (default: 5).
  • Decoder SWA bounded replay: reduce decoder prefill work by replaying only a bounded suffix. Set DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 for the default V4.1 settings; the retained suffix must cover the SWA window plus the maximum backtracking length.
  • Asynchronous scheduling: overlap CPU batch scheduling with device execution. Enabled by default.
  • Batch-invariant inference: keep model results identical across different batch compositions. Enable with DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only).
  • Deterministic sampling: produce identical outputs for identical input tokens and sampling parameters. Set DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0 and pass a fixed seq_seed to SamplingParams (NVIDIA only).
  • Structured outputs: constrain generation to a regex, JSON schema, or EBNF grammar. Pass so_args={"schema_type": ..., "schema": ...} to SamplingParams; see examples.

Supported hardware:

  • NVIDIA SM100 GPUs
  • Huawei Ascend 950 NPUs

Required for a complete inference service (not included in this release):

  • A KV cache system
  • VE/Prefill/Decode disaggregation (most required engine functionality is already implemented)
  • Load balancing across data-parallel replicas for both prefill and decode

Getting started

Prerequisites

  • Linux
  • GCC >= 13.0
  • Python 3.12
  • CUDA 13 (NVIDIA only)
  • CANN 9.2 (Ascend only)

Build from source

Before building DeepInfer, please create and activate an empty Python 3.12 environment.

# Use `ascend` instead for the Ascend platform
# Keep it exported in the environment where you run DeepInfer
export DEEPINFER_ACCELERATOR=nvidia
./build.sh

Offline inference

The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, is downloaded into the Hugging Face cache on the first run; set DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 to run from a local copy instead.

python examples/offline_inference.py

See examples for details.

Performance benchmarks

Prefill Decode
Prefill: non-cached input throughput vs. latency Decode: output throughput vs. per-user speed

Measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see run_sweep.sh.

Evaluation benchmarks

See eval/README.md.

License

DeepInfer is released under the Apache License 2.0. See LICENSE for details.

Acknowledgments

DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. We thank their developers.

Citation

@misc{deepinfer2026,
  title = {DeepInfer: High-performance, large-scale inference framework for DeepSeek models},
  author = {DeepSeek-AI},
  year = {2026},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/deepseek-ai/DeepInfer}},
}