DeepSeek Opened Its Inference Engine. Should You Run It Yourself?

Imagine a software company that uses DeepSeek to help its customers search technical documents. It already pays for an API, the connection through which its application sends questions and receives answers. Now DeepSeek has published the software that performs much of the underlying model computation. Could the company run that software and keep more of the money?

For some infrastructure teams, that is a reasonable experiment. For an application team expecting a replacement API server after installation, the public release leaves substantial work. DeepInfer makes an important part of the system inspectable and reusable; whether operating it is worthwhile depends on everything around that part.

DeepSeek published the engine on October 20. Duncan Riley’s SiliconANGLE report put the release in a useful frame: serving powerful models at an affordable price is an engineering problem of its own. The distinction matters because a model’s capabilities, the speed of its computation and the reliability of the service selling access to it are different things.

What DeepSeek Actually Released

A model’s weights are the learned numerical values used to generate an answer. An inference engine loads those values, runs the calculations and coordinates the hardware. A hosted service adds the system that receives customers’ requests, manages shared capacity and returns answers reliably.

The public DeepInfer repository supplies an offline inference engine for DeepSeek-V4.1-Flash. Offline here means running submitted work directly through the engine; it does not mean that the model has lost its capabilities or that a first run never downloads anything. The example downloads the default Flash-1017 checkpoint—the files containing the model—when needed, and can instead use files already available locally.

Its published hardware targets are NVIDIA SM100 GPUs and Huawei Ascend 950 accelerators. This is a release for specific data center equipment. It does not establish support for a reader’s existing laptop or an older GPU merely because both can run some other DeepSeek software.

DeepInfer’s code carries the Apache 2.0 license. The Flash-1017 model weights have their own MIT license. Those permissions make reuse possible, but they do not supply machines, an operating team or a price advantage over a hosted provider. A procurement decision still needs a hardware and service plan.

There is also a practical benefit for people who never deploy it. Engineers can examine how DeepSeek organizes calculations and coordinates work, then decide whether particular techniques belong in their existing engines. An open implementation can be useful without becoming the default production system for every reader.

The Diagram Is Bigger Than the Download

DeepSeek’s architecture diagram shows requests moving through several services and a shared store of previously computed information. The public engine sits within that wider design. The README lists a cache system, the complete arrangement separating processing stages and routing across engine replicas among the missing service components. It also says most engine functionality needed for stage separation is already implemented. The remaining gap includes connecting those functions into an operating service.

The first processing stage, prefill, works through the input. Decode then produces the answer in successive pieces called tokens. Separating the two can let an operator allocate hardware according to different workloads. It also creates a coordination job: the result of processing a question must reach the machines generating its answer.

A key-value cache preserves internal model calculations that can be reused when a later request begins with the same material. In our hypothetical document application, repeatedly sending the same reference text creates an opportunity to reuse work. The engine’s ability to process a request is different from operating a shared cache that finds, retains and retrieves that work across machines.

Routing matters for a similar reason. A service receiving a mixture of short questions and long document sessions has to distribute those requests. An efficient calculation inside one batch does not guarantee that the next customer reaches an available machine promptly. Someone must supply and operate that surrounding system.

That missing work is conditional. A team running scheduled offline evaluations may not need a public service at all. An established inference provider may already have caching and routing. Such teams can evaluate DeepInfer as a component, whereas a small application company may be taking on an unfamiliar operational responsibility.

Read the Speed Curve as a Choice

The official decode chart plots total output throughput per GPU against the speed experienced by each user. Moving toward higher total throughput on this curve means accepting lower per-user speed. The largest number is therefore not automatically the best operating point for an interactive product.

The chart also states its conditions: a 131,072-token context, a particular arrangement across 64 GPUs, three speculative draft tokens and a 75% token acceptance assumption. Speculative decoding proposes possible next tokens and checks them with the main model. Its benefit depends partly on how much proposed work is accepted.

The reproduction scripts make that assumption especially important. The benchmark’s raw throughput uses full-acceptance accounting, while the plotting workflow takes an acceptance-rate parameter. A derived chart value should not be confused with directly measured delivery of useful answers to customers. The script and settings are part of the result. The point labels count simultaneous requests per parallel replica, rather than the total number of users across the cluster.

DeepSeek separately benchmarks prefill on four SM100 GPUs and decode on 64. These are company measurements of particular stages. They do not establish the response time of a complete customer service, equal performance on Ascend or savings against a competing engine under a matched workload.

For the document application, the question is whether customers get acceptable answers within the required wait. More total tokens may help when many customers are active together. It may be a poor trade if a long answer becomes too slow to use. The test needs both the busiest expected traffic and the quieter periods that still incur hardware costs.

Outside Experiments Show Why Configuration Matters

Public contributor work offers a useful check on the idea that installation automatically reproduces the headline curve. In an open pull request, GitHub user kzjeef published attempts on four GB300 and four B200 GPUs, with scripts, tables and descriptions of additional service components.

The contributor reported prefill results below approximate values read from DeepSeek’s chart. The GB300 write-up attributed much of one gap to inactive fast network links. The B200 write-up still recorded a gap with network ports active and left the explanation open. A plausible diagnosis on one machine did not settle the other result.

These are contributor reports, not an accepted upstream performance verdict. TechReadly has not run these hardware trials. The contributor’s branch includes extra changes, and its four-device decode arrangement differs from the official wider arrangement. The evidence supports investigating the whole configuration; it does not support declaring DeepInfer universally slower or defective.

The same work is a positive counterexample to dismissing the release as unusable. The contributor describes adding a server, caching and separated processing stages. Integration is possible. It takes code and measurement, and the pull request was still open when checked on October 26.

Another open proposal, by sicnuyudidi, concerns older SM90 H20 hardware. Its author explicitly describes an initial design document rather than an implementation. A proposed port can be worth following without becoming a supported installation path today. Teams should make their trial decision using working support for the machines they can actually obtain.

A Familiar API Format Does Not Finish the Service

DeepSeek also released deepseek-recipe, a separate toolkit that translates requests into model prompts and parses generated output into response formats. It supports interfaces including Chat Completions, Responses and Messages, reducing the work of connecting different application conventions to a DeepSeek backend.

The model’s Flash-1017 documentation explains why a maintained conversion toolkit matters: the release uses its own method for turning conversations into model input rather than supplying a conventional chat template. The way messages, thinking and tool interactions are represented is part of a deployment’s behavior. Loading the weights is not the entire integration.

Recipe’s documentation is explicit that the caller supplies inference, HTTP transport and tool execution. It can represent a model’s request to call a tool; it does not thereby perform that action for the application. A team still has to connect the result to its own tools and permissions.

Similarly, accepting a JSON schema in a request does not itself guarantee that the backend enforces that structure during generation. Recipe passes the relevant constraints along, while enforcement depends on the backend. Even correctly structured JSON can contain an incorrect answer. Format compatibility and task quality need separate checks.

A Trial That Can Answer the Business Question

The first trial should resemble the work the team expects to operate. For the hypothetical document product, that means representative documents, repeated prefixes, short follow-up questions and enough simultaneous users to expose waiting. It also means retaining a comparison with the service already meeting those needs.

The repository’s evaluation tools provide a useful discipline: generation and scoring are separate, raw outputs are saved, and configuration accompanies the results. A team can reassess an answer without rerunning every calculation. The supplied benchmark scores are reference checks for a deployment, rather than promises about a customer’s application.

Capacity should be counted across the whole service. DeepSeek’s recommendation of 64 GPUs for prefill plus 64 for decode describes its smallest efficient deployment arrangement, rather than a hard minimum for every runnable offline experiment. Likewise, its reported revenue per GPU-hour is revenue under stated pricing conditions. It is not the operator’s profit, rental rate or total cost.

A provider with suitable machines, recurring demand and existing service infrastructure has a credible reason to investigate. An application team with intermittent traffic may obtain more value by studying the implementation while continuing to buy access. The useful result is a working service that meets the team’s needs at an acceptable total cost; owning the engine is one possible route to that result.