←
Patronus AI479 posts
Patronus AI avatar

Patronus AI

@PatronusAI

Patronus is a model behavior research lab developing training data and simulated environments for frontier AI. patronus.ai

◷ Joined July 2023▧ 165 Photos and videos
Varun Gangal avatar
⟳ Patronus AI retweeted
Varun Gangal@VarunGangal·Nov 16

Can @thinkymachines new Inkling-2-Small not just play a video game level, but play and refine it many times over and learn to speedrun it? @akkikiki , myself and other co-authors from @UW and @PatronusAI ran one of the setups from our SpeedRunBench preprint from Sep this year to observe exactly that (arxiv.org/abs/2610.08076). Thanks to the @thinkymachines and @tinkerapi team for credits that enabled this study! Our benchmark measures whether agents can discover and then refine offline a fast enough policy through a level as they play more and more attempts, using only a single score after each attempt to adjust their policy. By design, we don't refine policy amidst episodes, or have multimodal frame understanding ... so our benchmark is more a read on planning, exploration, autoresearch etc than online videogame playing per se.. We used the "offline scratch" setting on SuperTux (SuperMario style game, Tux in place of Mario) level 1-1. Results: 1. Inkling-2-Small (256K, thinking effort 0.5) is an adept speedrunner, learning to clear the level after 1,988 frames, second only to GLM 5.3 (1,961 frames) in the open source family models we benchmarked .. 2. This put Inkling-2 ahead of Gemini 3.7 Flash, GPT-5.6-sol, Claude Opus 5, and DeepSeek-V4-Flash. See the attached bar graph for the final frames 3. Inkling-2-Small dramatically improves over its predecessor Inkling-Small which never learnt to fully clear the level, let alone improving at speedrunning it. Note for scale that Human world record iterated over the years for speedrunning this game is 1,619 frames...

Post attachment
0:50

Inkling 2 Small’s weights are now available for download on HuggingFace under an Apache 2.0 license. Make Inkling your own! huggingface.co/thinkingmachi…

Patronus AI avatar
Patronus AI@PatronusAI·Nov 12

Long horizon tasks make the skew worse. Many of our tasks run into the tens or hundreds of turns, and one task can need a first-year analyst for one part and a partner for another. The analyst can explain how they build a revenue projection. But the model also needs the partner's judgment, like why the company was picked and which assumptions matter most.

0:18
Patronus AI avatar
Patronus AI@PatronusAI·Nov 12

Recruiting experts en masse skews the mix toward people who are junior, external-facing, willing to talk to you and open to gig work.

Patronus AI avatar
Patronus AI@PatronusAI·Nov 12

At a Big Law firm, a brief goes through 5+ review cycles. At a nonprofit environmental practice, it gets one or two. A model trained on both learns when to expect iterations. Most individual experts, or slices of a homogeneous pool, have only seen one.

0:10
Patronus AI avatar
Patronus AI@PatronusAI·Nov 12

The right expert set covers different roles, seniority levels and backgrounds. We research how to turn that expertise into environments, artifacts and automated checks. We think quality at scale is a research problem first.

Patronus AI avatar
Patronus AI@PatronusAI·Nov 12

Expert data has a selection bias problem. In an attempt to produce as many tasks as possible, data vendors are chasing the biggest pool of experts they can find. We think this will make models worse at the details of knowledge work. 🧵

0:12
Patronus AI avatar
Patronus AI@PatronusAI·Nov 7

Finally, one hypothesis is that writing voice depends on harness + setting. @Tim_Hua_ gave GLM-5.3 one code review in Claude Code and in Codex. In Claude Code it thinks in full prose. In Codex it writes itself clipped notes. Codex labels that panel a reasoning summary, so the notes may not be raw reasoning.

Patronus AI avatar
Patronus AI@PatronusAI·Nov 7

Why the shift? We can't see how these models were trained, so it's all hypothesis. A per-sentence length penalty in RL. Limit of outcome-driven RL being achieved @karpathy Sep’24. Agent-format pressure. Or a new teacher to distill from, as @scaling01 noted of DeepSeek and GPT-6 Astra.

Patronus AI avatar
Patronus AI@PatronusAI·Nov 7

For a human reference, we ran the same filler count on the bug reports behind all 500 SWE-bench Verified tasks. They come out at 20.8% filler, and Kimi K3.1 Pro uses under half that, at 9.8%. The issue writers use almost no self-talk, while every model uses 29+ per 1,000 words.

0:10
Patronus AI avatar
Patronus AI@PatronusAI·Nov 7

Terse reasoning did not hurt performance. The older releases solved 58 of their 60 runs and the newer ones 60 of 60. The tasks are easy for these models, so the terse style costs nothing on simple fixes. Whether it helps on hard ones is untested.

Patronus AI avatar
Patronus AI@PatronusAI·Nov 7

Shorter sentences would save tokens if other aspects of writing style like hedging and exploration stayed equal. They didn’t! DeepSeek’s new Flash reasons in shorthand but writes more: median 8,387 reasoning tokens/task vs 3,421 for V4 Flash. Kimi’s new model spends 44% more.

0:10
Patronus AI avatar
Patronus AI@PatronusAI·Nov 7

We count five kinds of filler: self-talk, articles, helper verbs, connectors and first-person pronouns. Code is not included. Article use dropped the most, by 93% (!) for Kimi and 85% for DeepSeek, with helper verbs close behind. Self-talk fell much less.

0:10