←
Thoughtful122 posts
Thoughtful avatar

Thoughtful

@thoughtfullab
◷ Joined December 2025▧ 32 Photos and videos
Thoughtful avatar
Thoughtful@thoughtfullab·Apr 22

Model shaping is still a craft of a few. That's what AI agents are for: learning it and doing it for everyone else. As a part of FrontierSWE benchmark we built a 20-hour post-training task on @tinkerapi and found the real bottleneck is research intuition.

0:54
Lijie(Derrick) Yang avatar
⟳ Thoughtful retweeted
Lijie(Derrick) Yang@LijieyYang·Nov 17

@karinanguyen Thank you for the insightful talk on post train bench, agentic eval, and RSI and the following conversation! Hope you have enjoyed @Princeton campus, gorgeous in this golden time 🍂

It's so beautiful at @Princeton this time of year, thank you for having me! Inspired by all the wonderful people and conversations.

Post attachmentPost attachmentPost attachmentPost attachment
Zhifei Li avatar
⟳ Thoughtful retweeted
Zhifei Li@andylizf·20h

Great talk by @karinanguyen at Princeton on Sunday! Loved her framing that agents doing AI research should be judged like any researcher: by their taste in experiments, and by whether the results justify the compute. Plus a yardstick for where AI should run at all: net value per joule. Thanks for coming!

so happy and honored to have hosted @karinanguyen at Princeton last sunday! loved hearing her insights on frontier LLM post-training, forecasting, and exciting evals. can't wait to see more from @thoughtfullab to bring RSI to the real world! please come back! 💗

Post attachmentPost attachment
helen avatar
⟳ Thoughtful retweeted
helen@helentomato0o·Nov 16

so happy and honored to have hosted @karinanguyen at Princeton last sunday! loved hearing her insights on frontier LLM post-training, forecasting, and exciting evals. can't wait to see more from @thoughtfullab to bring RSI to the real world! please come back! 💗

Post attachmentPost attachment
Brian Zhou avatar
⟳ Thoughtful retweeted
Brian Zhou@brian1zhou·Nov 13

I'm excited to launch Benchmark Almanac! It tracks 3,322 benchmarks from 1,746 leaderboards, 522 launch posts, 346 system cards, and 1,233 open-source GitHub repos. To my knowledge, it's the most comprehensive repo out there. Website: benchmarkalmanac.com

0:36
Martin Ziqiao Ma avatar
⟳ Thoughtful retweeted
Martin Ziqiao Ma@ziqiao_ma·Nov 12

We updated the eval table of the model card (thinkingmachines.ai/model-ca…) with results from a few more private and semi-private benchmarks, along with official result verifications.

Inkling 2 Small’s weights are now available for download on HuggingFace under an Apache 2.0 license. Make Inkling your own! huggingface.co/thinkingmachi…

Kevin Miao avatar
⟳ Thoughtful retweeted
Kevin Miao@KJHMiao·Nov 11

Big little announcement: @karinanguyen and I are excited to announce a brand new course on Full-Stack Post-Training, RL & RSI at @UCBerkeley. If you are a Berkeley student and want to learn about the frontier, enroll for the next semester.

Karina avatar
⟳ Thoughtful retweeted
Karina@karinanguyen·Nov 11

Excited to teach at @UCBerkeley again this spring! We’ve expanded the class to 200 seats for advanced undergrads and grad students. We’ll explore the full frontier of AI: from pre-training, post-training, RL / inference-time scaling to autonomous agents, long-horizon reasoning & infra, and ultimately, applied RSI — systems capable of recursively improving themselves in real world.

Big little announcement: @karinanguyen and I are excited to announce a brand new course on Full-Stack Post-Training, RL & RSI at @UCBerkeley. If you are a Berkeley student and want to learn about the frontier, enroll for the next semester.

Karina avatar
⟳ Thoughtful retweeted
Karina@karinanguyen·Nov 7

after a year of deploying AI, i've learned that the tasks it performs best per unit of energy aren’t always the ones that create the most economic value. the counterintuitive part is that not everyone should own their intelligence bc the same compute can be worth much more in one business than another. hence, choosing where to deploy AI matters as much as how good you make it!

Thoughtful avatar
Thoughtful@thoughtfullab·Nov 6

At Thoughtful, we’re studying the economics of intelligence through net value per joule (NVJ): how much economic value AI creates per unit of energy. We were excited to collaborate on this paper, with @Mersad_Abbasi from our team contributing to the research. These ideas are already informing where and how we deploy self-improving AI in the real world. Learn more:

AI is using more and more of the world’s energy. But how much economic value are we getting from AI? We’re introducing Joules-to-GDP (J2G), a framework that measures how well AI turns energy into economic value and shows which tasks, models, and hardware give the best return. Across 44 occupations in finance, engineering, law, administrative work, and more in @OpenAI's GDPval, we arrive at three findings: 1) The tasks with the highest economic value are not the tasks with the highest economic value per joule. For example, CAD design earns ~10% more per task than building websites, but the latter earns ~27% more value per joule. 2) The most powerful model isn’t always the best use of energy. Gemma 4 31B delivers 56% of GLM 5.2’s net value with 23% of the energy, for 2.5x the value per joule. 3) More chips don’t guarantee proportionate returns. Moving the same model from 1 to 8 NVIDIA H100s uses 3.2x the energy but cuts completion time by only 22%, with about the same value. With @SnorkelAI, we’re also releasing GDPWorks, a benchmark for economically valuable tasks across 36 additional occupations. Joint work co-led by @amirzeinaalii and @Avanika15, with @Mersad_Abbasi, @Jon_Hartley_, John Hennessy, @HazyResearch, and @Azaliamirh. More details below 🧵

0:24
Thoughtful avatar
Thoughtful@thoughtfullab·Nov 5

On Arena-Hard Writing, it read the judge's feedback and changed its diagnosis: "This is an INSTRUCTION-FOLLOWING problem, not just style." Inkling 2 generated ~6k writing briefs with explicit constraints and distilled a local open 72B model on them. Gemma 3 4B went from 0.3% to 23.9%. PostTrainBench is one of the first benchmarks for autonomous post-training: AI agents source data, train smaller models, and evaluate their performance. It’s part of our growing work on applied recursive self-improvement.

Post attachment
Thoughtful avatar
Thoughtful@thoughtfullab·Nov 5

On HealthBench, its first model scored 2%, because most answers were empty. A next-token probe and a look at the data showed why: "HealthBench conversations are ALL multi-turn." 99% of its training data was single-turn. Inkling 2 added synthetic multi-turn data and brought its official score up to 31.4%.

Thoughtful avatar
Thoughtful@thoughtfullab·Nov 5

On GPQA, its first model which was trained on long reasoning traces scored 6% on a quick check. Inkling 2 read the outputs and retrained on short answers only, bringing its score up to 35.6%.

Thoughtful avatar
Thoughtful@thoughtfullab·Nov 5

Inkling 2 Small is a great American open model, showing frontier-level performance at autonomously improving other models. With just 17B active parameters, it scores 31.87 on PostTrainBench, ahead of Kimi K3 (31.48) and GLM 5.2 (30.40), and on par with GPT-5.6 (Sol) and Opus 4.8 (~31). We’re excited to have partnered with @thinkymachines to evaluate Inkling 2 Small, a promising step forward for American open-source AI.

Post attachment