The Model Label Is Only Half the Interface
Prime Intellect announced Prime Inference on October 2, offering serverless endpoints and reserved capacity for hosted open models. Its GLM-5.3 deployment runs on NVIDIA Blackwell infrastructure. For developers choosing where to run a long-running agent, the announcement raises a question that the model name cannot answer: how will the service handle the work between one decision and the next?
I would treat the endpoint as part of the model choice. An agent can accumulate repository history, tool outputs and conversational turns across many requests. The inference engine, routing layer and serving hardware affect whether that history can be reused, how long a new turn waits, and whether generated tool calls reach the application in a usable form. None of those properties can be established from the weights' capability scores alone.
A developer choosing a host is choosing how the model will take part in a workflow. A dependable tool response and a prompt that starts processing promptly can both matter, even when the model itself has not changed. The service offering deserves attention at that level, rather than as a place to compare one advertised token speed with another.
The service's own documentation also makes an important distinction. Prime runs hosted models on its infrastructure, while gateway models pass requests to an external provider. Both use the same API key and endpoint. GLM-5.3 is listed as hosted, but the common interface does not mean every model in the catalog uses the same machines or serving path.
Prime's GLM-5.3 endpoint lists a 1,048,576-token context window and text input and output. That limit describes what a request can contain, rather than promising fast execution at its maximum length. Open weights make another deployment possible; they do not make every deployment equivalent. Even a comparison that holds the model version fixed must account for the host's configuration.
What Returning Work Does to an Inference Cluster
A long-running coding agent sends more than the latest user instruction. Its next prompt may include earlier file contents, decisions and tool results. Much of that input can repeat the previous request, even when the agent needs a new response. Recomputing the unchanged part consumes resources without giving the application new information.
In model serving, request processing has two phases. Prompt processing, usually called prefill, evaluates the input. Token generation, or decode, produces the response one token at a time. The key-value cache, shortened to KV cache, stores intermediate attention state that can be reused. It is neither a saved answer nor a durable record of what the agent has learned.
vLLM's automatic prefix caching reuses that state when a request begins with the same token sequence as earlier work. Its documentation identifies repeated questions about a long document and conversations with shared history as useful examples. The gain comes from skipping repeated prompt computation. Caching does not shorten the generation of new output tokens, and it helps little when there is no matching prefix or output generation dominates the request.
For an application developer, that distinction makes prompt construction consequential. A changing field inserted early in the request can break the match before the repeated history begins. Keeping genuinely stable material together can preserve more reuse. This is an inference from prefix matching, rather than a claim that one prompt layout works everywhere. Current information still belongs in the request; an agent should not keep obsolete instructions or file contents just to protect a cache hit.
Placement matters as well. NVIDIA Dynamo's routing documentation describes a choice that combines available cached prefixes with active worker load. Sending all returning work to the worker with the most relevant cache could leave it overloaded. Sending work to an idle worker can require additional prompt computation. A worker with a useful cache can therefore lose the routing decision to one with less cached history.
That is why a cache hit is not a complete performance verdict. A returning request can avoid some computation and still wait behind other work. A different placement can spend more resources yet begin sooner. The router has to balance those costs rather than follow an unconditional rule that a session must stay in one place.
The application sees this as time to first token: the pause before a response starts. It sees decode behavior as the pace of the response afterward. Those delays can move differently. A fast continuation after the first token says little about how long the agent waited to get there, particularly when new sessions arrive alongside returning ones.
Latency Metrics Versus Agent Task Completion
Prime reports nearly 40% lower p90 inter-token latency in its tests after separating prefill and decode GPU pools. Its described workload mixes returning, long-context sessions with cold arrivals; it uses SemiAnalysis's AgentX replay harness with additional uncached requests. Dynamo coordinates the workers, vLLM executes the model, and computed KV state moves between the pools.
The p90 figure is the 90th percentile of the delay between generated tokens, describing the slower end of the measured distribution. It does not mean an agent finishes its task 40% sooner. The distinction matters because a full task also contains prompt processing, waiting, external tool execution and potentially further attempts. A shorter interval during one part of that sequence can be valuable without changing every other part.
vLLM's disaggregated-prefill documentation explains the rationale independently of Prime's results. When prompt processing shares resources with generation, a new prefill job can increase the delay experienced by a request already decoding. Separate instances allow operators to tune time to first token and inter-token latency separately. The documentation also explicitly cautions that disaggregation does not improve throughput.
Separation creates another responsibility: moving the computed state from prompt workers to generation workers. Operators must allocate resources to both pools and make the transfer work efficiently. A smooth stream is therefore one service objective, alongside the time required to start it and the amount of work the hardware can serve. Copying an architecture diagram cannot settle the tradeoff for a different workload.
AgentX helps make a relevant workload reproducible. Its methodology replaces original prompts, code and tool payloads from opt-in Claude Code sessions with deterministic synthetic content. The replay preserves request lengths, shared prefixes, timing and dependency branches. It can examine how a serving system handles a returning agent without publishing the original code or conversation.
Its scope is equally clear: AgentX measures serving performance, not answer quality. A replay cannot establish whether GLM-5.3 chose the right file to edit or completed a debugging task correctly. Using a third-party harness also does not turn Prime's customized run into an independently reproduced measurement of Prime's service.
The methodology offers another useful warning about comparison. Client concurrency is not a fixed request batch; one agent can branch into several requests. In a closed-loop replay, faster clients progress farther during the measurement window, so the request mix can vary. Throughput needs to be read together with first-token delay and interactivity, rather than reduced to one attractive speed number.
A deployment comparison also needs explicit reasoning settings. Prime's GLM-5.3 documentation says reasoning is always enabled, with low, high and max effort; max is the default. Comparing hosts while leaving their effort settings implicit can mix a configuration difference into what appears to be a hardware or serving comparison. A team should record the chosen setting and verify outcomes alongside response time.
Tool Serialization and Structural Correctness
The most revealing turn in Prime's account is about tool responses. According to the company, its serving path initially could discard calls to undeclared tools, produce unusable arguments, or change literal strings during parsing. One example converted the literal entity sequence < into <, altering content. Prime describes adding structural-tag support and fixing parsing behavior.
I consider that evidence as relevant to endpoint selection as the latency result. An agent depends on the complete API response, including tool names and arguments. If that boundary changes the content or drops an action, the application may have to recover before it can continue. The model's ability to propose an action and the service's ability to deliver it are related, but separately testable.
Grammar-constrained generation is one way to control the response's structure. XGrammar, an open library integrated into vLLM, restricts which output tokens are allowed by a grammar. A schema can constrain the shape of a tool call instead of leaving the application to repair arbitrary text after generation.
The guarantee depends on the actual configuration. vLLM's current tool-calling documentation distinguishes when structural tags become active from when a tool's parameter schema is enforced. Strictness settings and server policy matter; an automatic tool-choice request does not universally impose the same constraints. General framework support cannot establish the exact behavior of a particular hosted endpoint.
Even correctly applied constraints leave a separate question unanswered. A file path can be a valid string while naming the wrong file. A command can fit the declared argument schema while doing something the user never authorized. Structural validity does not prove that an action is useful, correct or permitted.
vLLM assigns callers responsibility for defining appropriate tools, providing context and handling execution. The application must still validate the final response and enforce its own permissions. A host's reliable serialization makes that job possible; it does not complete it. A low parsing-error rate should therefore remain separate from the rate of successful, authorized tasks.
Choosing an Endpoint for the Whole Session
Prime's launch earns a workload-specific trial because it addresses identifiable problems around a hosted model. It does not establish that the service is the fastest or cheapest choice for every agent. Its documentation sets pricing, limits and supported features per model, so neither a shared API nor an architecture description supplies a like-for-like commercial comparison.
A useful trial should include both a returning session and new work arriving beside it. The returning session reveals how response times change as history grows and is reused. Cold arrivals test a different condition: prompt computation that cannot rely on the same existing state. Results from one condition should not silently stand in for the other.
Keep the agent task and model settings comparable, then record the pause before the response, its generation pace and the time until the task passes an acceptance check. An API client may not expose the provider's internal cache state. Response timing alone therefore cannot prove why a request was slow or diagnose the exact routing architecture behind it. It can still reveal whether the service meets the application's needs.
Tool responses deserve the same attention. Inspect the names, types and literal strings that reach the application, including streamed responses if the agent uses them. Verify that the proposed action is appropriate before executing it. There is no universal number of calls that qualifies every deployment; the cases should reflect the tools and failure costs of the intended work.
My judgment is that the endpoint belongs in the evaluation, even after a team has chosen GLM-5.3. A model's capabilities set one boundary; the host's handling of history, contention and tool responses sets another. Prime has provided a concrete reason to examine both. I would adopt a service when the complete session meets the required responsiveness and verified outcome, rather than let a context-window limit or a fast token stream make that decision alone.