←
Openbenchmarks (YC F24)171 posts
Openbenchmarks (YC F24) avatar

Openbenchmarks (YC F24)

@openbenchmarks

Independent benchmarks to help you discover, compare, and choose APIs and agents for your use cases.

◷ Joined July 2026▧ 45 Photos and videos
Yi Zhong 🍊 avatar
⟳ Openbenchmarks (YC F24) retweeted
Yi Zhong 🍊@yiz_be_building·Nov 12

SOV-Transcribe-1, the latest ASR model from @salesforce , is #2 on Voice Code Bench, and #1 among open-source models. Voice Code Bench tests how well a model performs in transcribing alphanumeric entities like phone numbers and emails rather than looking at general word error rate, critical for voice agent applications. SOV-Transcribe-1 is a real solid choice for anyone building voice agents on open-source stack! Check out their blog at huggingface.co/blog/salesfor…. Congratulations @salesforce & team! See the complete leaderboard at voicecodebench.com

Openbenchmarks (YC F24) avatar
Openbenchmarks (YC F24)@openbenchmarks·Nov 13

This groups the APIs on a benchmark into statistical tiers instead of individual rankings. If two APIs are in the same tier, the gap between them is within noise, and the benchmark doesn't have enough evidence to rank one above the other.

Post attachment
Openbenchmarks (YC F24) avatar
Openbenchmarks (YC F24)@openbenchmarks·Nov 13

In the example, A and B agree on 284 of 300 tasks. On the 16 where they disagree, A wins 14 (p = 0.004). Pass/fail scores: McNemar's test on those disagreements. Everything else: paired bootstrap on the per-task difference.

Openbenchmarks (YC F24) avatar
Openbenchmarks (YC F24)@openbenchmarks·Nov 13

Paired tests Overlapping confidence intervals don't mean two APIs are tied. Both APIs get the easy tasks right and miss the hard ones, and that shared noise widens both intervals. Comparing them task by task removes it, and only the tasks where they disagree are left.

Post attachment
Openbenchmarks (YC F24) avatar
Openbenchmarks (YC F24)@openbenchmarks·Nov 13

Confidence intervals Every score carries a 95% confidence interval: how far it could move had the benchmark drawn a different set of tasks. For binary pass/fail tasks, we use a Wilson interval, which stays reliable at near-100% scores.

Openbenchmarks (YC F24) avatar
Openbenchmarks (YC F24)@openbenchmarks·Nov 13

That requires two things: The sampling uncertainty of each score (95% confidence intervals) The significance of each pairwise difference (paired tests).

Openbenchmarks (YC F24) avatar
Openbenchmarks (YC F24)@openbenchmarks·Nov 13

Benchmark Insights: Statistical Reporting Every benchmark score comes with signal + noise. A benchmark or an aggregate index built solely for comparison produces a ranking, but the gaps in it may or may not be real. A benchmark has to tell signal from the noise.

Post attachment
Openbenchmarks (YC F24) avatar
Openbenchmarks (YC F24)@openbenchmarks·Nov 6

Search API prices span $0.000 to $0.224 per task. Token cost spans $0.065 to $1.681. A search API's price barely influences what a task costs. The gap between a cheap task and an expensive one comes down to how much text each one made the model read.

Openbenchmarks (YC F24) avatar
Openbenchmarks (YC F24)@openbenchmarks·Nov 6

We found that on the average task the search API accounts for about 11% of the bill and tokens account for the remaining 89%. Mean cost per task is $0.685 search only and $0.672 with search and fetch.

Openbenchmarks (YC F24) avatar
Openbenchmarks (YC F24)@openbenchmarks·Nov 6

Web search APIs for agents are usually compared on list price per query. We measured whether that price predicts what a research task actually costs.

Openbenchmarks (YC F24) avatar
Openbenchmarks (YC F24)@openbenchmarks·Nov 6

- in search only, the agent works from the titles, URLs and snippets a search returns. - in search + fetch, it can also open up to 14 of those pages and read the text. The only thing that changes in the setup is which search API gets called.

Openbenchmarks (YC F24) avatar
Openbenchmarks (YC F24)@openbenchmarks·Nov 6

The agent is held fixed across every run: gpt-5.6-sol at medium reasoning, 8 model turns, and a budget of up to 14 searches per task. Each API is measured in 2 configurations:

Openbenchmarks (YC F24) avatar
Openbenchmarks (YC F24)@openbenchmarks·Nov 6

Benchmark Insights: Multi-Turn Company Search (cost decomposition) In this benchmark we test whether a search API can help an agent curate an exhaustive set of companies given various constraints across investors, founding year, geography, and funding.

Post attachment
Openbenchmarks (YC F24) avatar
Openbenchmarks (YC F24)@openbenchmarks·Nov 4

Specialized intelligence ftw!

There are two main types of benchmarks: 1. those that measure frontier intelligence (terminalbench, gdpval, etc.). accuracy matters more than cost. 2. those that measure specialized intelligence. the pareto curve of cost/accuracy more than accuracy alone. both are important, but they are distinct benchmark categories. * as (1) saturates, they become (2). * the largest frontier models at xhigh likely already saturate (2), but they do so at a cost/latency point that's completely price-prohibitive. * every user should define (2) for the tasks they care about the most. we're big fans of both. we have so far dedicated outsized attention to (2) especially related to document tasks. we fundamentally think users should have the most price-efficient tools to understand, extract, and read over their documents. we've built parsebench, extractbench for this purpose. we are starting to invest in (1) as well, because certain document tasks are so challenging that even frontier models struggle (e.g. see extractbench pro). we are continuing our investment in long-horizon workflow research at @llama_index