
Last week, I said every parser is #1 on its own benchmark. So we tested the benchmarks. We ran 8 parsers across 12 benchmarks, then tilted the pages by 2–4°. 6 benchmarks got a different winner. We’re calling it BenchBench.

I’m into open science, elegant software architecture, and well-crafted interfaces. Building reducto.ai

Last week, I said every parser is #1 on its own benchmark. So we tested the benchmarks. We ran 8 parsers across 12 benchmarks, then tilted the pages by 2–4°. 6 benchmarks got a different winner. We’re calling it BenchBench.

pls dear god let this rfc be accepted github.com/modelcontextproto…

LLM judges are a better way to do this. They can read the page and tell whether the parser lost or changed information. That’s how PAIR works, with parser names hidden. Try it on your own documents: parsejudge.com Full study: reducto.ai/blog/benchbench-d…

On 9 of 12 benchmarks, the median percentage score drop after tilt was larger than the percentage gap between first and second. The scorer made a difference too. One ParseBench-2 table kept every number but scored zero because of how the cells were arranged.

The content didn’t change. We kept each benchmark’s own scoring rules (adjusting golden bounding boxes for tilt where applicable). 5 of 8 providers moved in the overall average-rank comparison.

oops *workload identity federation

s/o @JdotJdotF for helping on this! and @trykino for the 30 second video edit lol

mini rant on OIDC and workflow identity federation tldr: reducto supports it, you should avoid using static credentials wherever possible, and toss your api keys. more here: reducto.ai/blog/workload-ide…

The @trykino launch video is now open source. Remix its footage and code at the link below. We're releasing all our project code (all edits and graphics in Kino are code!), raw footage, odd VFX ideas, and comments from our many Granola calls
Introducing Kino: make videos end-to-end with Claude Code and Codex. Drop your footage, get highlights, edit together, send links for feedback, and let your agents cook. AI is becoming a great creative partner. Kino is a new home for your team and agents to collaborate.

This benchmark came out a few hours ago. We were able to audit this quickly because modern frontier LLMs are great at comparing parsers. There is no reason to be using rule-based benches anymore, LLM judge is the way. parsejudge.com/parsebench-2
We took a look at ParseBench-2 today. On their benchmark, one table scored 0 and another scored 100. Can you tell which is which? 🧵



You can now go from your local terminal to an editable document in 30 seconds ⚡ Connect Reducto Office to your Pi agent via MCP, ask for a launch brief, and open it in the editor to keep working together.

in my opinion, this is actually the correct way to do benchmarking. nothing beats the user’s own code. what would folks like to see if we did this for Archil?
Every document parser is #1 on its own benchmark. So forget static benchmarks. Today, we're releasing ParseJudge, an open-source app that judges parsers on your own documents, free for up to 10,000 pages a day. parsejudge.com

see if you can spot the easter eggs in this video :) there are ~3 good ones. most of my team hasn't spotted them yet 😛

fun fact: I scripted, filmed (on my iPhone), and edited this video all in a 5 hour span on Sunday with no video experience! impossible without @trykino
Every document parser is #1 on its own benchmark. So forget static benchmarks. Today, we're releasing ParseJudge, an open-source app that judges parsers on your own documents, free for up to 10,000 pages a day. parsejudge.com