Introducing Sterling: Inference from verified providers.
The model you’re paying for might not be the one answering you. Every API model is served by a provider. Some serve a smaller version. Some break tool calls.
Sterling tests all providers and routes to the verified ones.
You can now download Sterling’s verification reports as JSON, CSV or PDF.
Every provider behind a model is re-tested every few hours, and the report keeps every run: how each one scored on each check, whether it got better or worse over time, and when its price or limits changed.
Pull any date range, or give it to your agent and ask which provider changed and when.
Explore the reports: sterling.mira.network/verifi…
A 95% success rate sounds pretty good.
Until your agent has to succeed ten times in a row.
Assuming ten independent steps, each with a 95% probability of success:
`P(all succeed) = 0.95^10 ≈ 59.9%`
So a workflow built from individually reliable steps can have a surprisingly low end-to-end success rate.
Real agents are more complicated. Failures may be correlated, retries can recover some steps, and one bad result can affect everything downstream.
But that's exactly why evaluating agent reliability only at the individual model-call level is insufficient.
The interesting metric is whether the workflow actually finishes its job.
Consider two models that answer 85 out of 100 benchmark questions correctly.
Their scores are identical.
But they might disagree on completely different questions, respond differently to tool-use instructions, or generate very different distributions of completions.
Aggregate accuracy compresses all of that into one number.
If you're evaluating model capability, benchmark scores are useful.
If you're evaluating whether an API behaves like a reference model, you need a different experiment.
Paper: proceedings.iclr.cc/paper_fi…
At ICLR 2025, researchers introduced Model Equality Testing.
Instead of comparing aggregate benchmark scores, they collected response samples from API endpoints and reference models, then used statistical two-sample tests to compare their output distributions.
They examined 31 commercial Llama endpoints during summer 2024.
11 produced distributions statistically distinguishable from the reference weights.
That doesn't establish why the endpoints differed, or whether they still differ today.
But it demonstrates why benchmark scores alone are insufficient evidence of model equivalence.
Two inference endpoints score 85% on the same benchmark.
Are they serving the same model?
There's no way to establish that from the score alone.
A benchmark measures performance on a particular set of tasks. Model identity testing asks whether two endpoints produce equivalent output distributions.
Here's why that distinction matters. 🧵
1,000 inference runs.
`temperature: 0`
80 distinct completions.
That's what Thinking Machines Lab observed when testing Qwen3-235B with greedy decoding.
The interesting culprit wasn't randomness in token sampling.
It was batch-dependent numerical behavior in the inference kernels.
Different server loads can change batching and numerical results, occasionally flipping token selection.
When the researchers used batch-invariant kernels, all 1,000 completions were identical.
A useful reminder that reproducibility is an inference-systems property, not just a sampling parameter.
thinkingmachines.ai/blog/def…
Here's a strange finding from long-context research:
You can make information harder for a model to retrieve without removing or changing it.
Just move it to the middle of a long prompt.
In *Lost in the Middle*, researchers found that several language models performed best when relevant information appeared near the beginning or end of their input.
Move that information toward the middle, and performance often deteriorated.
Same information. Different position. Different results.
The practical implication for long-context evaluations is interesting:
Don't just measure retrieval accuracy at 32K, 64K, or 128K tokens.
Measure it across positions within each context length.
A single long-context benchmark score can hide a surprisingly uneven reliability profile.
If you're deploying through a third-party inference API, model selection is only half the decision.
You also need to evaluate how faithfully the provider serves it.
That's the problem Sterling is built around: independently testing provider behavior and using verification results to inform routing.
Explore the results: sterling.mira.network/verifi…
Here's the interesting part.
After providers made implementation fixes, Artificial Analysis updated the benchmark results.
Several previously underperforming endpoints improved substantially.
The issue wasn't necessarily the underlying model's intelligence. The deployment itself was part of the problem.
This is why benchmarking a model and benchmarking an endpoint aren't the same thing.
A model's behavior depends on more than its weights.
The serving stack has to handle:
• Chat-template formatting
• Tokenization and special tokens
• Reasoning configuration
• Tool-call serialization
• Sampling and decoding
A mismatch in how requests are processed can affect downstream model behavior, even when the endpoint advertises the correct model ID.
Same model. Same benchmark. Different API providers. Different results.
When Artificial Analysis tested gpt-oss-120b across inference providers in August 2025, the AIME 2025 results varied sharply between implementations.
Then several providers fixed their deployments, and their scores improved.
What happened? 🧵
These dimensions are independent.
A provider might preserve model behavior but mishandle tool calls. Another might support tools correctly while falling short on declared context.
That's why Sterling evaluates provider-model pairs across all five dimensions, publishes verification results, and uses those signals to inform routing.
Behavioral verification isn't cryptographic proof. It's additional evidence that traditional infrastructure monitoring doesn't provide.
Explore the results: sterling.mira.network/verifi…
Declared Context Support
A provider advertising 128K context doesn't automatically establish that it can reliably process that context.
There are two separate questions:
• Does the endpoint accept inputs approaching its declared limit?
• Can the model actually use information throughout that context?
Boundary tests and long-context retrieval probes help expose different limitations.
Maximum context length ≠ effective context utilization.
Silent Failures
Not every inference failure produces a 4xx or 5xx.
An endpoint might return HTTP 200 with an empty completion, truncated output, or a response that fails an expected behavioral assertion.
Transport-level success and task-level success are different things.
Verification needs to inspect the response, not just the status code.
Parameter Compliance
An API accepting a parameter doesn't mean its serving stack honors it.
Set temperature, top_p, or other supported controls and the request may succeed even if the behavior doesn't reflect those settings.
Detecting this requires controlled, repeated tests across parameter configurations.
A successful request validates acceptance, not compliance.