SOV-Transcribe-1, the latest ASR model from @salesforce , is #2 on Voice Code Bench, and #1 among open-source models.
Voice Code Bench tests how well a model performs in transcribing alphanumeric entities like phone numbers and emails rather than looking at general word error rate, critical for voice agent applications.
SOV-Transcribe-1 is a real solid choice for anyone building voice agents on open-source stack! Check out their blog at huggingface.co/blog/salesfor….
Congratulations @salesforce & team! See the complete leaderboard at voicecodebench.com
This groups the APIs on a benchmark into statistical tiers instead of individual rankings.
If two APIs are in the same tier, the gap between them is within noise, and the benchmark doesn't have enough evidence to rank one above the other.
In the example, A and B agree on 284 of 300 tasks. On the 16 where they disagree, A wins 14 (p = 0.004).
Pass/fail scores: McNemar's test on those disagreements. Everything else: paired bootstrap on the per-task difference.
Paired tests
Overlapping confidence intervals don't mean two APIs are tied. Both APIs get the easy tasks right and miss the hard ones, and that shared noise widens both intervals.
Comparing them task by task removes it, and only the tasks where they disagree are left.
Confidence intervals
Every score carries a 95% confidence interval: how far it could move had the benchmark drawn a different set of tasks.
For binary pass/fail tasks, we use a Wilson interval, which stays reliable at near-100% scores.
That requires two things: The sampling uncertainty of each score (95% confidence intervals) The significance of each pairwise difference (paired tests).
Benchmark Insights: Statistical Reporting
Every benchmark score comes with signal + noise.
A benchmark or an aggregate index built solely for comparison produces a ranking, but the gaps in it may or may not be real.
A benchmark has to tell signal from the noise.
Search API prices span $0.000 to $0.224 per task. Token cost spans $0.065 to $1.681.
A search API's price barely influences what a task costs. The gap between a cheap task and an expensive one comes down to how much text each one made the model read.
We found that on the average task the search API accounts for about 11% of the bill and tokens account for the remaining 89%. Mean cost per task is $0.685 search only and $0.672 with search and fetch.
- in search only, the agent works from the titles, URLs and snippets a search returns.
- in search + fetch, it can also open up to 14 of those pages and read the text.
The only thing that changes in the setup is which search API gets called.
The agent is held fixed across every run: gpt-5.6-sol at medium reasoning, 8 model turns, and a budget of up to 14 searches per task. Each API is measured in 2 configurations:
Benchmark Insights: Multi-Turn Company Search (cost decomposition)
In this benchmark we test whether a search API can help an agent curate an exhaustive set of companies given various constraints across investors, founding year, geography, and funding.
There are two main types of benchmarks:
1. those that measure frontier intelligence (terminalbench, gdpval, etc.). accuracy matters more than cost.
2. those that measure specialized intelligence. the pareto curve of cost/accuracy more than accuracy alone.
both are important, but they are distinct benchmark categories.
* as (1) saturates, they become (2).
* the largest frontier models at xhigh likely already saturate (2), but they do so at a cost/latency point that's completely price-prohibitive.
* every user should define (2) for the tasks they care about the most.
we're big fans of both. we have so far dedicated outsized attention to (2) especially related to document tasks.
we fundamentally think users should have the most price-efficient tools to understand, extract, and read over their documents. we've built parsebench, extractbench for this purpose.
we are starting to invest in (1) as well, because certain document tasks are so challenging that even frontier models struggle (e.g. see extractbench pro). we are continuing our investment in long-horizon workflow research at @llama_index