DeepSeek published its production inference engine, minus the parts that make it a service
DeepInfer is Apache 2.0 and powers DeepSeek's own serving, but its README lists no KV cache, no prefill-decode disaggregation and no replica load balancing.
Vlad Makarovreviewed and published
The repository deepseek-ai/DeepInfer took its initial commit at 04:09 UTC on 20 October and closed the day with three commits, the last at 11:03 UTC. It carries the Apache 2.0 licence, and by the following morning it had drawn more than eleven thousand stars.
The first line of the README explains why: "DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services." The second sentence does the qualifying work. "This open-source release provides offline inference with the core features needed for serving at scale." Both statements are true, and the distance between them is the substance of the release. DeepSeek has published an engine, not a server, and it says so in a way that is easy to read past.
One model, two accelerator families
The scope is narrow in a way the announcement does not advertise. There is exactly one supported model, DeepSeek-V4.1-Flash, with the October checkpoint DeepSeek-V4.1-Flash-1017 as the default, fetched into the Hugging Face cache on first run. No earlier DeepSeek generation, no dense-model path, no generic loader.
| Item | Value in this release |
|---|---|
| Licence | Apache 2.0 |
| Supported model | DeepSeek-V4.1-Flash only |
| Default checkpoint | deepseek-ai/DeepSeek-V4.1-Flash-1017 |
| Accelerators | NVIDIA SM100 GPUs, Huawei Ascend 950 NPUs |
| Build requirements | Linux, GCC 13.0 or newer, Python 3.12, CUDA 13 or CANN 9.2 |
| Commits at publication | Three, all on 20 October 2026 |
The feature list is where the engineering lives. Optimised kernels from DeepSeek's own published projects, FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, plus custom TileLang kernels. Attention data parallelism with balanced context parallelism, and mixture-of-experts expert parallelism with EPLB. A speculative decoding path the README calls DSpark with confidence-scheduled verification, switched on with DEEPINFER_ENABLE_MTP_VARLEN=1, draft length capped at five by default. SWA bounded replay, which replays only a bounded suffix of the prefill: DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact, with DEEPINFER_YOCO_PREFILL_KEEP_TOKENS defaulting to 140 for V4.1. A zero-overhead batch scheduler that overlaps CPU scheduling with device execution, on by default. Batch-invariant inference and deterministic sampling, both NVIDIA only. Structured outputs constrained by regex, JSON schema or EBNF.
Read as a list of techniques, that is a serious document. Several of these appeared in DeepSeek papers before; having them in runnable code is different from having them described.
What the README says is missing
Halfway down the page, under a heading no marketing department would have written, is the section that matters most. "Required for a complete inference service (not included in this release)." Three items follow: a KV cache system; VE, prefill and decode disaggregation, with the note that "most required engine functionality is already implemented"; and load balancing across data-parallel replicas for both prefill and decode.
None of the three is peripheral. A KV cache system is what lets a server reuse computation across turns and across requests that share a prefix, the difference between a demo and a viable endpoint. Disaggregation splits prefill and decode into separate pools so each runs on hardware suited to it. Replica load balancing is what turns many engines into one service. Without them, what ships is a fast single-process engine that reads prompts and writes tokens.
Commenters noticed the omission rather than the features. One, writing as entrope, argued that part of DeepSeek's hosting advantage sits in the missing layer: "Part of DeepSeek's hosting advantage, and thus a remaining part of their moat, is migrating KV cache from VRAM to host RAM to nonvolatile storage, and presumably routing requests to where KV cache has migrated." The comment ends: "That infrastructure isn't part of this open-source release."
That is a fair reading, and it is not a scandal. DeepSeek labelled the gap itself, in a section a reader cannot miss. The parts that generalise across deployments went out; the parts tuned to DeepSeek's own cluster topology stayed in.
Two charts, no scores
The performance section is two PNG files, a prefill Pareto chart of non-cached input throughput against latency and a decode chart of output throughput against per-user speed. The only numbers in prose describe the testbed: "Measured on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain." A sweep script at examples/sweep/run_sweep.sh is offered for reproduction.
No table of scores, no comparison against vLLM or SGLang, no statement of where the axes end. As of writing, nobody else has published a replication, which leaves the framework's performance claim resting on two images committed by its authors. The obvious question in the discussion thread, how this compares with the engines it descends from, the repository does not answer.
The acknowledgments answer part of it. DeepInfer "was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM." This is a fork-derived internal engine that grew inside a serving organisation, not a clean-room stack, and the novel material is the delta on top.
One commenter, cmrdporcupine, put the consequence better than the README does: "People will take the novel bits here and port them to the above." That is the likely outcome. The value outside DeepSeek is less a server to deploy than a reference implementation that vLLM and SGLang maintainers can read as working code.
The Ascend line, one variable away
Buried in the hardware list is the detail with the longest reach: alongside NVIDIA SM100 GPUs, the release supports Huawei Ascend 950 NPUs. One environment variable selects the target, export DEEPINFER_ACCELERATOR=nvidia or ascend, with a separate requirements file and CANN 9.2 in place of CUDA 13.
One serving codebase targeting both vendors' current-generation parts is unusual for a lab of DeepSeek's size, and the decision reads as insurance. Whether it reflects a production footprint that already spans both or preparation for one that might, the repository does not say. The Ascend path is also narrower: batch-invariant inference and deterministic sampling are marked NVIDIA only.
Running it
Set the accelerator variable, run ./build.sh inside a Python 3.12 environment, then python examples/offline_inference.py. Weights arrive from the Hub unless DEEPINFER_HF_MODEL_DIR points at a local copy. Anyone without an SM100 or an Ascend 950 can read the code but cannot run the charts.
What the release is worth
DeepSeek has now published a model card, a Rust and Python library called deepseek-recipe that renders conversations into prompts and tokens, and the engine that serves the model in production: a stack assembled in public, with the operational layer around it still held back.
For inference engineers the kernels and scheduling tricks are worth the read, and the blunt "not included" section is worth copying as a practice. For anyone hoping to stand up a DeepSeek endpoint this week, the README's own list is the answer: three components short, and the missing ones are what make a cluster behave like a service.
