coffee/deepseek-recipe
1
0
Fork 0
mirror of https://github.com/deepseek-ai/deepseek-recipe.git synced 2026-10-28 17:17:08 +08:00
Code Issues Packages Projects Releases Wiki Activity
deepseek-recipe/static/tokenizers/v4/README.md
icrim 57b9c84242 Initial commit 
Co-authored-by: xinchengxx <xinchengxx@deepseek.com>
Co-authored-by: XieJiSS <xiejiss@deepseek.com>
Co-authored-by: yqaty <jiayu@deepseek.com>
Co-authored-by: zhu-he <hezhu@deepseek.com>
Co-authored-by: season-guo <seasonguo@deepseek.com>
2026-09-10 18:07:01 +08:00

1.2 KiB
Raw Blame History

V4 tokenizer

tokenizer.json is a modified copy of the Hugging Face file deepseek-ai/DeepSeek-V4-Pro/tokenizer.json with the image tokens reassigned.

Local changes

id this file upstream
129264 <｜image｜> special=true <｜image2｜> special=false
129279 <｜image2｜> special=false <｜image｜> special=true

All other JSON entries match the comparison revision below.

Provenance and license

  • Comparison revision: b5968e9190ef611bbf34a7229255be88a0e937c1, verified on 2026-09-10.
  • Local SHA-256: 97d2f31b020d18b5aee5c9b3d5b4efb10ea210f3fe3f7dffe3f1cd90542d6b19.
  • Upstream license: MIT, Copyright (c) 2023 DeepSeek.
  • Preserved license notice: ../LICENSE.

The original import revision was not recorded; the comparison revision identifies the upstream JSON used to verify the changes above.