←
Ben Caine23 posts
Ben Caine avatar

Ben Caine

@bcaine

Research Engineer @ Google DeepMind. I lead Gemini's vision post-training effort. Opinions my own.

◷ Joined June 2011▧ 6 Photos and videos
Ben Caine avatar
Ben Caine@bcaine·Nov 12

benchmaxxing postlarval litopenaeus vannamei detection is finally paying off

Post attachment

Gemini 4 Carbon is so fucking good at computer vision - #1 overall - #2 object detection - #1 short string extraction - #4 structured JSON extraction - #1 long text transcription - #1 text localization and recognition very slow and expensive though... ↓ examples

Post attachment
Ben Caine avatar
Ben Caine@bcaine·Nov 10

In the official Stirrup harness, Claude Fable 5.2 (56.6% Climb-WSR) and Gemini 4 Carbon (54.3%) lead on quality, while Carbon uses 5.7x fewer tokens at >6x lower API cost ($5.63 vs $35.28/task). Still <57% on frontier criteria—the eval has lots of headroom left.

Ben Caine avatar
Ben Caine@bcaine·Nov 10

In experiments, we show vision is required to do well on the benchmark (WSR drops 6%, token consumption increases 1.85x). Interestingly, when you take away their view_image tool, today's frontier models show a shocking degree of persistence and resilience trying to solve tasks.

Post attachmentPost attachment
Ben Caine avatar
Ben Caine@bcaine·Nov 10

MKA has 200 multi-file tasks across 10 sectors. It requires reading blueprints, schematics, maps, X-rays, and documents to build real world deliverables. To really focus on the model's accuracy, an agentic autorater checks an average of 71 criteria per task.

Post attachment
Ben Caine avatar
Ben Caine@bcaine·Nov 10

We're super excited to launch MKA in partnership with @ArtificialAnlys. Real-world work is messy and inherently visual. We kept finding existing benchmarks were either not visual enough or not agentic enough. MKA aims to fill this gap.

We’ve partnered with Google DeepMind to launch their Multimodal Knowledge Agent (MKA) evaluation, a new benchmark for agentic knowledge work over complex visual files @GoogleDeepMind’s MKA is a new benchmark requiring agents to review visually complex project files such as schematics and scans, and produce deliverables like slide decks, spreadsheets, reports, and marked-up images Multimodal Knowledge Agent (MKA) overview: ➤ 200 private real-world visual tasks across 10 industries: Agents work from dense visual project files including architectural drawings, engineering schematics, and clinical scans to produce deliverables including slide decks, spreadsheets, PDFs, Word documents, and marked-up images. Unlike many agentic evaluations, these inputs emphasize realistic visual complexity and domain-specificity, and are genuinely required to complete the task rather than just providing additional context ➤ Climb vs. Regression rubric scoring: Each task is graded against expert rubrics, averaging ~70 weighted criteria per task. Criteria are split into ‘Regression’ (baseline checks models should pass) and ‘Climb’ criteria (harder checks designed to separate strong models). The headline score is Climb WSR (Weighted Satisfaction Rate) - the weighted proportion Climb rubric criteria successfully completed ➤ Evaluated in Stirrup with an agentic judge: We run every task in Stirrup, our open source agent harness, with code execution, web search, web fetch, and image-viewing tools (up to 250 turns), and grade submitted deliverables using GPT-6 Luna (medium) as an agentic judge Key MKA results: ➤ Claude Fable 5.2 leads on multimodal knowledge work: Anthropic’s recently released Claude Fable 5.2 (max) leads the MKA leaderboard with a Climb WSR of 57%, followed by Gemini 4 Carbon (high) at 54% and Claude Opus 5.5 (max) at 53% ➤ Cost per Task spans nearly 30× within 10 points of the top score: Claude Fable 5.2 averages $31.97 per task, over 5× Gemini 4 Carbon ($6.29) for ~2 extra points, and ~29× GPT-6.1 Sol ($1.11). Below Carbon, all four OpenAI models tested sit on the cost-performance Pareto frontier (GPT-6 Astra, GPT-6.1 Sol, GPT-6.1 Terra, and GPT-6 Luna), with Claude Haiku 5.5 and MiMo-V2.6-Pro making up the rest of the frontier ➤ No single model leads across all 10 industry verticals: Claude Fable 5.2 leads five verticals (Software & Data, Engineering, Logistics & Mapping, Business, and Finance alongside Claude Sonnet 5.5), Claude Opus 5.5 (max) leads Education and Construction, Gemini 4 Argon (high) leads Science, Gemini 4 Carbon (high) leads Healthcare, and Claude Sonnet 5.5 (max) narrowly leads Law & Government Thank you to the @GoogleDeepMind team for their work building this benchmark and collaborating with us to bring it exclusively to Artificial Analysis!

Post attachment
Wenhao Chai avatar
⟳ Ben Caine retweeted
Wenhao Chai@wenhaocha1·Nov 10

1/2 Gemini 4 Carbon land today. and (general) vision perception is almost solved in the agentic era, we need more Humanity's Sixth Sense by @scale_AI just release at Oct. it said "far from saturated despite high human performance". but Gemini+Antigravity easily saturate it 🤯

Post attachment
Jake Boggs avatar
⟳ Ben Caine retweeted
Jake Boggs@JakeABoggs·Oct 13

Gemini 4 Argon sets a new record of 69% on HieroglyphBench This is the first model that performs accurately enough to be useful

Post attachment
Ben Caine avatar
Ben Caine@bcaine·Oct 13

Sorry we even made you consider Astra for a few weeks. Won't happen again.

Astra dethroned Gemini 4 Argon is the best "vision" model #1 for detection: - tight, precise boxes - dense scenes #1 for text extraction: - small text - handwritten text - complex tables - rotated text 4× cheaper than Astra at high

Post attachment
Lucas Beyer (bl16) avatar
⟳ Ben Caine retweeted
Lucas Beyer (bl16)@giffmana·Sep 4

This is a very cool example of the long tail of things some people will find useful but you as a model trainee will NEVER think of, and hence why you should aim to train on as diverse free-form text data as possible: (And congrats to Gemini 3.8 for looking like what is a very good detector)

tire size text detection, Gemini 3.8 Flash (high)

Post attachment
Ben Caine avatar
Ben Caine@bcaine·Jan 5

The power of generalization

Google’s Gemini 3.0 Pro decodes 500-Year-Old Nuremberg chronicle mysteries. 👏 Gemini 3.0 Pro was given high-resolution page images and produced a coherent explanation of what the notes were doing, not just a transcription. It concluded the roundels are a small calculation table meant to reconcile 2 competing biblical timelines and pin down Abraham’s birth year. The hard part for earlier attempts was that the notes mix abbreviated Latin, Roman numerals, and implied context from the printed page, so reading the marks alone does not reveal intent. The prompt asked for transcription, translation, and meaning using the surrounding text, and it reportedly used 5 images total, a 2-page spread plus 4 zooms. Gemini tied the scribbles to “Anno Mundi” dating, meaning “Year of the World,” and treated them as conversions into a “before Christ” timeline. It linked 3184 and 2040 “Year of the World” figures to the Septuagint and Hebrew Bible traditions, then mapped them to roughly 2015BC and 1915BC, a 100-year gap the annotator was trying to resolve. A great example of multimodal models as research assistants when the task needs reading, cross-referencing, and arithmetic in one pass, but it still needs expert verification because a single digit slip can change the conclusion. --- siliconangle. com/2026/01/01/googles-gemini-3-0-pro-helps-solve-long-standing-mystery-nuremberg-chronicle/

Post attachment
Ben Caine avatar
Ben Caine@bcaine·11 Dec 2025

I had Nano Banana remove GPT5.2's bounding boxes and Gemini 3 give it a go Left: GPT5.2 Right: Gemini 3.0

Post attachmentPost attachment