I used to hear “same weights” as a promise of sameness.
If two providers offered the same open-weight model, the buying decision seemed pleasantly mechanical. Compare token price, latency, regions and uptime. The underlying intelligence was the fixed item in the spreadsheet; the cloud vendor was a delivery service.
That assumption has become difficult to defend. A model repository contains tensors and instructions, not a finished API. Between those files and the answer that reaches an application sit a numerical format, an inference engine, sampling defaults, context and output limits, a chat template, a tool-call parser and whatever patches the provider applied last Tuesday.
Artificial Analysis’s Endpoint Accuracy Index now measures the consequences. Its first results showed that endpoints carrying the same model name could diverge sharply on reasoning, long-context recall and tool use. Simon Willison had documented the same frustration a year earlier after following provider differences down to a serving-framework bug.
The conclusion is uncomfortable but useful: an open model and a hosted version of that model are not the same procurement object. The endpoint deserves its own version record.
A reference is not the truth
The Endpoint Accuracy Index begins with a self-hosted reference deployment. Artificial Analysis runs the official weights at the model developer’s recommended precision and follows the published serving recipe. It then runs the same evaluation against serverless API providers.
The initial index gave equal weight to three areas: native tool calling, scientific reasoning and long-context recall. It repeated tests and published 95 percent confidence intervals. An endpoint counted as being at reference parity when the interval around its result included the reference result.
That last phrase matters. Reference parity does not mean the model is correct. It means the provider reproduced the behavior of one carefully configured deployment within the uncertainty of the test. If the underlying model answers a science question incorrectly, an endpoint can match it perfectly and still be wrong.
Nor does a result below the reference prove that a provider is universally bad. An endpoint may make an explicit trade: lower precision for lower cost, a smaller context limit for higher throughput, or a shorter output cap to keep queues manageable. Those can be reasonable products. The problem begins when the trade is invisible and the model name implies nothing changed.
The index is also a snapshot. Artificial Analysis dates its measurements because endpoints change. Any article that turns the table into a permanent vendor league is misusing the evidence.
A token ceiling can remove the answer
The GLM-5.2 results supplied the cleanest example. Artificial Analysis found that restrictive output limits cut off some responses before the model completed its reasoning. On the scientific-reasoning subset, the most restrictive endpoints scored half the reference result or less.
The weights were available. The provider simply did not allow the same amount of generation.
An output cap is easy to dismiss as a quota, separate from “model quality.” For reasoning systems, that distinction breaks down. If the model uses generated tokens to work through a problem, a hard ceiling can interrupt the process that produces the answer. The provider has effectively selected a smaller reasoning budget even when the client asks for the highest supported effort.
This does not mean longer is always better. A model can waste tokens and still be wrong. It means the cap is part of the model experience and should be disclosed as prominently as context length and price.
I would want to know both the advertised maximum and the limit actually enforced on my account tier. I would also test whether the endpoint returns a visible truncation signal. A half-finished answer that looks complete is worse than a clean error.
Tool calling exposes every seam
The gpt-oss-120b comparison showed a different failure mode. Artificial Analysis reported that some hosted endpoints scored 22 percent on its native tool-calling subset, against 37 percent for the reference. It also observed endpoints producing fewer reasoning tokens at the same requested setting.
Tool use is especially sensitive because the model’s generation is not the final product. A provider must recognize the model’s tool syntax, translate it into the API’s schema, preserve arguments and deliver a structure the client can execute. A parser error can turn a semantically correct choice into a failed call. A chat template can place instructions where the model does not expect them. A runtime can mishandle an effort parameter.
Willison’s 2025 investigation supplied a concrete instance of the last problem. Microsoft said one Azure result was caused by an older vLLM version that did not respect reasoning_effort, so requests fell back to medium. The company fixed it. That episode should not be recycled as proof that the current Azure endpoint is broken. Its value is that it reveals how ordinary the cause can be. The mysterious intelligence deficit was a software-version bug.
This is the part that changes my buying assumption. Even knowing the quantization is not enough. Two providers can use similar weights and precision while differing in templates, runtime versions and parser behavior. The API surface is part of inference.
Some providers matched the reference
The initial DeepSeek V4 Pro results prevent the story from becoming a cheap indictment of hosting companies. Most measured endpoints were at reference parity, and DeepSeek’s first-party endpoint scored slightly above the reference in that snapshot.
Parity is achievable. Providers that follow the serving recipe closely can deliver behavior that is statistically indistinguishable from the self-hosted baseline on the measured tasks. In some runs, sampling variation or a well-tuned serving system may even place an endpoint above it.
That is encouraging, but it does not restore permanence. A provider can update a kernel, change a quantization or fix a parser without changing the public model string. The gpt-oss incident improved after a software update; another endpoint could regress after an optimization. The capability belongs to a dated configuration, not a logo.
This is not unique to AI. Database drivers, browser engines and compiler versions all affect software behavior. We learned to version them because “Postgres” or “Chrome” was never specific enough for debugging. Model APIs are approaching the same maturity, only with more of the stack hidden from the customer.
What an endpoint record should contain
I do not think every company should respond by self-hosting. Running a large model well is difficult, and a managed endpoint may still be the cheapest and most reliable choice. The fix is to buy the service as the service it actually is.
For a production endpoint, I would ask for a record containing the base-weight revision, numerical precision or quantization, inference runtime and major version, supported effort settings, effective context and output limits, chat template, tool parser, and date of the last material change. Some providers will consider parts of that stack proprietary. In that case, they can at least supply a stable endpoint version, publish compatibility promises and notify customers before behavior-changing updates.
My own regression suite would travel with the application. It would contain representative tool calls, long documents, expected refusal behavior and tasks with machine-checkable results. I would run it before switching providers and again when the endpoint version changes. A public benchmark can reveal the category of risk; only the application’s tests can say whether it matters to that application.
Price comparisons should carry the same endpoint specificity. The cheapest token is not cheap if a parser failure triggers three retries or an output cap prevents task completion. Cost per accepted task is still the number that matters.
Give the served model its own name
Open weights remain valuable. They make the reference possible, allow independent hosting and reduce dependence on a single vendor. None of this evidence argues otherwise.
What open weights do not provide is automatic reproducibility across every runtime and provider. The repository gives us a common starting point, not a guarantee that all roads lead to the same answer.
I would like providers to acknowledge that in the product name. Keep the familiar model family, but attach a serving revision that changes when precision, limits, templates or runtimes change materially. Give customers a changelog and a period in which the previous revision remains available. Make “same endpoint” mean something testable.
Until then, the model name is a capability hypothesis. The endpoint—the whole dated, configured, occasionally buggy endpoint—is the product we are actually buying.