←
Bo Liu (Benjamin Liu)369 posts
Bo Liu (Benjamin Liu) avatar

Bo Liu (Benjamin Liu)

@Benjamin_eecs

Incoming PhD @StanfordNLP | BS @PKU1898 | Building continually self-improving AI | Prev @DeepSeek_AI @AIatMeta | DeepSeek-V1/V2/VL/Prover ALE SPIRAL SPICE SPADE

◷ Joined February 2022▧ 18 Photos and videos
Bo Liu (Benjamin Liu) avatar
Bo Liu (Benjamin Liu)@Benjamin_eecs·Aug 20

Continuous self-improvement needs an ever-expanding supply of training environments (goals). SPADE: one model self-plays the Environment Designer and the Reasoning Agent, writing executable, agentic environments that get harder as it improves. Environment scaling on its own. ♠️

0:31
OpenAI avatar
⟳ Bo Liu (Benjamin Liu) retweeted
OpenAI@OpenAI·Nov 17

GPT-6.2 Astra—a frontier model built for your hardest projects. It sets new highs on evals across professional work and scientific research at less than a third of the cost per task of GPT‑6 Astra. You’ll see the difference in things like slideshows, websites, and 3D designs.

Post attachmentPost attachmentPost attachment
Chris Barber avatar
⟳ Bo Liu (Benjamin Liu) retweeted
Chris Barber@chrisbarber·Nov 15

I asked 56 people in AI which benchmarks they'd trust most to indicate if an LLM is frontier: 25 votes: Terminal-Bench 4 (@alexgshaw, @andykonwinski, @Mike_A_Merrill) 20 votes: Terminal-Bench Science (@StevenDillmann, @lschmidt3, @sanmikoyejo) 13 votes: FrontierCode (@penlume, @ybenpan, @denizbirlikci, @silasalberti) 10 votes: FrontierSWE (@evan_j_chu, @_rajanagarwal, @MatternJustus, @ardnoKhtnayhsiR, @sanket__mhatre, @akshitkr) 8 votes: Humanity's Last Exam (@longphann_, @hendrycks, Alice Gatti, @ziwen_h, @natliml) 7 votes: Vending-Bench (@axelbacklund, @lukaspet) 5 votes each: - AutomationBench (@danielwshepard, @SalimansRobin, @TweetAnnaMarie) - DeepSWE (@winkey_h, @charleyslee, @LeonardMainnet, @serenaa_ge) - PostTrainBench (@full__rank, @hrdkbhatnagar, @AmyPrb, @shiraeis, @karinanguyen, @MatthiasBethge, @maksym_andr) The most common comment was that people don't really trust benchmarks at all. 56 people voted. People couldn't pick a bench their company worked on, an index, or a human-voted board. Disclosure: on this list, I'm a small investor in Abundant, Cognition, Emulated, Fleet, Hillclimb, Latch, and Snorkel. I didn't vote. Counts are net: a few benchmarks received negative votes. Continued:

Post attachment
OpenRSI Foundation avatar
⟳ Bo Liu (Benjamin Liu) retweeted
OpenRSI Foundation@OpenRSI·Nov 9

We scaled Recursive Self-Improvement (RSI) to production AI infrastructure with SGLang-Omni, shipping 14 upstream-merged PRs across 8 speech & omni models with 7.6%–344.3% throughput gains. Blog: index.openrsi.foundation/blo…

Post attachment
Richard Sutton avatar
⟳ Bo Liu (Benjamin Liu) retweeted
Richard Sutton@RichardSSutton·Nov 7

Brute force computation is not enough for intelligence. Clever new methods are needed. The science of AI is far from finished. IMHO

Artificial Analysis avatar
⟳ Bo Liu (Benjamin Liu) retweeted
Artificial Analysis@ArtificialAnlys·Nov 5

Congratulations to @xdotli and the team at @benchflow_ai on launching FrontierPhysics! Beyond long-horizon agentic tasks in scientific domains, it’s great to see scoring that captures the nuances and judgment physicists care about We're excited to partner with you to host this evaluation on Artificial Analysis soon

Introducing FrontierPhysics - a benchmark evaluating agents for end-to-end physics research Our core team from physics labs at Berkeley, Caltech, and Harvard recruited 42 physicists from 26 institutions. 28 of them published papers on Nature, Nature Physics, PRX, PRA* 1/7 🧵

Post attachment
Xiangyi Li avatar
⟳ Bo Liu (Benjamin Liu) retweeted
Xiangyi Li@xdotli·Nov 5

Introducing FrontierPhysics - a benchmark evaluating agents for end-to-end physics research Our core team from physics labs at Berkeley, Caltech, and Harvard recruited 42 physicists from 26 institutions. 28 of them published papers on Nature, Nature Physics, PRX, PRA* 1/7 🧵

Post attachment
Hyunwoo Kim avatar
⟳ Bo Liu (Benjamin Liu) retweeted
Hyunwoo Kim@hyunw_kim·Oct 19

Question to post-training folks: Do we really have to do all that post-training on every base model just to find which ckpt would be the best coding agent? 🤔 What if we can spot potential BEFORE post-training? 🔮 Our method predicts future SWE-bench rankings with ρ=0.964 🔥 ⬇️

Post attachment
Aran Komatsuzaki avatar
⟳ Bo Liu (Benjamin Liu) retweeted
Aran Komatsuzaki@arankomatsuzaki·Oct 15

This new benchmark evaluates nine frontier agents on 15 open scientific follow-up problems developed with domain scientists, producing 135 research papers. Most of the papers fail integrity checks, including discrepancies between claimed work and supporting execution evidence.

Post attachment
Simon Yu avatar
⟳ Bo Liu (Benjamin Liu) retweeted
Simon Yu@simon_ycl·Oct 14

AutoEnvScaling is a joint work with @BaolinPeng, @Benjamin_eecs, @Zhengyan_Shi, @zhaoywang_CS, Hao Cheng, @5000hui, @YaoWenlin, @natashajaques, @shi_weiyan and @JianfengGao0217 :)) @MSFTResearch × @Northeastern × @stanfordnlp × UW × UNC Paper: arxiv.org/abs/2610.14163 Github: github.com/MSR-Orchard/AutoE… Page: auto-env-scaling.github.io/ Built on slime (@slime_framework), miles (@radixark) and Harbor (@harborframework)

elvis avatar
⟳ Bo Liu (Benjamin Liu) retweeted
elvis@omarsar0·Oct 14

New impressive work from Microsoft and collaborators. And it's one of my favorite recent topics: RL environments. They show that Qwen3.6-35B-A3B goes from 41.1 to 53.3 on Terminal-Bench 2.1 by training on environments it designed itself. AutoEnvScaling gives environment design to a proposer agent that works in a terminal, looks at the rollouts of the model being trained, and builds new RL environments from them. The trained model's next rollouts guide the next round. Building environments in a terminal raises the share of valid environments up to 25 times over prompt-only generation, at up to 75% lower cost per valid environment. With GPT-5.6-sol or Claude Opus 5 as the proposer, Qwen3.5-35B-A3B scores 4.3 and 3.8 points above matched static environment pools. The gain shrinks when the proposer cannot see solver feedback. The pipeline also produces environments for software engineering, computer use, and GPU kernel work. arxiv.org/abs/2610.14163 Chat with Paper: academy.dair.ai/papers/autoe…

Post attachment
Natasha Jaques avatar
⟳ Bo Liu (Benjamin Liu) retweeted
Natasha Jaques@natashajaques·Oct 14

Our new paper pushes multi-agent recursive self-improvement even further. This time the proposer agent builds whole Harbor sandbox environments to challenge the solver agent. It learns from the solver’s successes and failures to update its own memories, skills, and tools, and build increasingly challenging---yet valid---tasks. We show powerful benefits both when using a proprietary model as the proposer, and in self-play, when both the solver and proposer are the same open-weight model. In this RSI setting, Qwen3.6-35B-A3B doubles its score on Terminal Bench 4, and performs so well on HiL-Bench it actually exceeds Claude Opus-5. More evidence of the power of multi-agent co-adaptation to unlock continuous improvement and emergent capabilities.

Recursive self-improvement needs agents that can build their own training environments. But researchers still part of the loop, making it hard to scale. We introduce AutoEnvScaling, which turns Data Flywheel into a terminal task: a proposer agent builds complete Harbor tasks and sandbox. It builds just like human researchers. This enables self-play RL for terminal agents: one model improves as both builder and solver, without human-in-the-loop.

Post attachment
Simon Yu avatar
⟳ Bo Liu (Benjamin Liu) retweeted
Simon Yu@simon_ycl·Oct 14

(2) Can AutoEnvScaling enable recursive self-improvement? We self-play on Qwen3.6-35B-A3B, since building an environment and solving one are both terminal tasks. Each new checkpoint builds the next round of training environments. Both roles get better: Terminal-Bench 2.1 rises from 41.1 to 53.3, while more of the environments it builds pass validation. The data flywheel closes: the model builds what it trains on and gets better at both building and solving.

Post attachment
Bo Liu (Benjamin Liu) avatar
Bo Liu (Benjamin Liu)@Benjamin_eecs·Oct 14

a minimal RSI would be: a coding agent writes the repo that trains its successor, the pretrain, mid-train and post-train recipe plus a new harness. As a follow-up to SPADE, AutoEnvScaling automates one piece of that repo: post-training environments and curriculum for the terminal domain. The environment designer is now a coding agent: it reads the reasoning agent's rollouts, then builds, runs and fixes each environment in a sandbox. Building a terminal task and solving one are both coding, so one model plays both roles and gets better at both. Each checkpoint builds the tasks that train the next. Qwen3.6-35B-A3B goes from 41.1 to 53.3 on Terminal-Bench 2.1 :))

Recursive self-improvement needs agents that can build their own training environments. But researchers still part of the loop, making it hard to scale. We introduce AutoEnvScaling, which turns Data Flywheel into a terminal task: a proposer agent builds complete Harbor tasks and sandbox. It builds just like human researchers. This enables self-play RL for terminal agents: one model improves as both builder and solver, without human-in-the-loop.

Post attachment
Simon Yu avatar
⟳ Bo Liu (Benjamin Liu) retweeted
Simon Yu@simon_ycl·Oct 14

Recursive self-improvement needs agents that can build their own training environments. But researchers still part of the loop, making it hard to scale. We introduce AutoEnvScaling, which turns Data Flywheel into a terminal task: a proposer agent builds complete Harbor tasks and sandbox. It builds just like human researchers. This enables self-play RL for terminal agents: one model improves as both builder and solver, without human-in-the-loop.

Post attachment