DeepSeek open sources DeepInfer, the inference framework it says powers DeepSeek V4.1 Flash in production
馃殌 Introducing DeepInfer, our in-house inference framework, now fully open source! Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We鈥檙e sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable. 馃敆 t.co/JGAik1kgk0 1/4
Context
Inference engines decide how cheaply and quickly a model can be served, so releasing a production stack lets others run DeepSeek models closer to the way DeepSeek does.
DeepSeek is a Chinese AI lab that releases open weight models and sells API access. The claims about DeepInfer's speed and its role in production are DeepSeek's own, and outside coverage says this first release supports only V4.1 Flash on data center hardware.
- GitHub: deepseek ai DeepInfer
- GitHub: deepseek ai deepseek recipe
- Pasquale Pillitteri: DeepSeek releases DeepInfer, the engine it uses to serve its models
- DeepSeek on X: Inside DeepInfer
- Advance Intelligence: DeepSeek releases V4.1 Flash 1017
- Advance Intelligence: The Information reports DeepSeek and Huawei are collaborating on AI chip designs
Deep dive
On October 20, DeepSeek posted that it is introducing DeepInfer, its in house inference framework, now fully open source. It said DeepInfer powers DeepSeek V4.1 Flash at scale with high throughput and low latency, and that it is sharing its production stack to help the community lower compute costs, reduce energy use and make AI more affordable.
In the same thread, DeepSeek listed high performance kernels and an optimized inference engine, distributed inference and speculative decoding, and SWA bounded replay and structured outputs. It said inference code for future DeepSeek models and ongoing optimizations will land in the same repository. It also open sourced deepseek recipe, which it described as its production token encoding and decoding component, aligned with the processing behind the official DeepSeek API so community deployments behave the same way.
A review by developer Pasquale Pillitteri said the repository is under the Apache 2.0 license, that the README says it was built mostly on top of vLLM with ideas from SGLang and TensorRT LLM, and that this first release runs one model, V4.1 Flash, in offline mode. He wrote that it needs Nvidia SM100 GPUs or Huawei Ascend 950 chips, which means data center hardware rather than a laptop.
What to expect next
Watch whether DeepSeek adds online serving and its next models to DeepInfer, as it said it would, and whether cloud providers adopt it to match DeepSeek's own prices. Support for Huawei Ascend chips is notable given a secondhand report this week, covered in the October 19 deck, that DeepSeek and Huawei are collaborating on chip designs. Independent benchmarks against vLLM and SGLang will show whether the speed claims hold outside DeepSeek's own setup.