←
Mahesh Sathiamoorthy4,998 posts
Mahesh Sathiamoorthy avatar

Mahesh Sathiamoorthy

@madiator

CEO @bespokelabsai. RL/Envs/Posttraining/Agents. Created Generative Retrieval at @GoogleDeepMind.

◷ Joined February 2008▧ 704 Photos and videos
Mahesh Sathiamoorthy avatar
Mahesh Sathiamoorthy@madiator·Sep 9

AutoResearch is critical in unlocking new knowledge and accelerate discovery. And it's an important ingredient in our quest to RSI. Today we at @bespokelabsai are happy to announce a new benchmark that's tailored to measure agents' ability to do autoresearch: AutoResearchExam. As part of this benchmark, we release 29 tasks that measure progress over 24 hours for agents to do sustained ML research and model training. Please check out for more info: benchmarks.bespokelabs.ai/au… This line of work is especially timely, given the recent advances in math and science, such as solving the Navier-Stokes Millennium Prize Problem, which needs sustained autoresearch and test-time scaling.

We are releasing AutoResearchExam, a benchmark on open-ended machine learning and engineering tasks. Our benchmark covers seven research areas including model training, data curation, AI safety and interpretability. benchmarks.bespokelabs.ai/au… In each task, we give agents 24 hours with a CPU or GPU machine to develop and improve their solutions through experiments and feedback. We measure both speed and quality with a combined score. Our benchmark has a unique feature: testing if agents create improvements that hold up on data they never see. We find that AI research agents often overfit as they try to improve. We see an interesting head-to-head comparison at the frontier: Astra starts the strongest and holds the lead for up to 19 hours but Fable 5.1 catches up and gets the top performing spot in the final hours. Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 all sit on the cost-performance Pareto frontier, giving strong options at lower API budgets. Anthropic's Opus and Fable retain nearly all their validation performance on hidden tests, with gaps of 1.1% and 2.9%. Astra's improvement over Sol extends to generalization too, with that gap falling from 6.9% to 1.7%. (1/n)

0:14
Mahesh Sathiamoorthy avatar
Mahesh Sathiamoorthy@madiator·Nov 15

Consider this setup of continual learning: you take an action and you get a score (e.g. thumbs up/down or rating). The goal is to increase the thumbs up rate or overall average rating. What kind of applications does this unlock if the continual learning algorithm worked pretty well?

Mahesh Sathiamoorthy avatar
Mahesh Sathiamoorthy@madiator·Nov 15

I used to actually sit down and read physical books. To my horror, I realize I haven't done that for a year. To be clear, I do consume audio books now, but even that frequency has reduced. I guess my attention span has been shot to death.

Mahesh Sathiamoorthy avatar
Mahesh Sathiamoorthy@madiator·Nov 15

There is still a lot of work left in tech and especially in building complex systems

Update on yesterday’s FAA ground stop. We now know what happened, and it’s worse than “old equipment.” It wasn’t a cyberattack and it wasn’t SMART. The FAA says there’s zero evidence of either. Here’s what happened. L3Harris crews were replacing an end-of-life copper line with fiber at one remote FAA site. A router involved in that work started flooding the network with more than 1,000 connection attempts every four to five seconds. Radar feeds and radio links to aircraft became unstable. After about 45 minutes of nobody knowing why, the FAA grounded the entire country. Once they isolated that one remote site, everything came back. So let me get this straight. One router at one site took down air traffic control for all 50 states. The FAA has now disabled the faulty function on roughly 60 routers of the same type nationwide. It still doesn’t know whether the router was defective or whether it was installed or configured wrong. Oh, and this has happened before. In 2009, a Harris technician misconfigured a router while moving the same network to fiber and disrupted 129 air traffic facilities. Same contractor, same migration, same failure mode, 17 years apart. This is exactly the point from my last post. Legacy modernization doesn’t fail because of the shiny new system. It fails at the seams, in the routine change nobody thought could take down the whole country. Good change management means one site can’t flood the network, isolation happens in minutes instead of hours, and nobody runs a copper-to-fiber cutover on critical infrastructure without a tested rollback and a hard limit on how far a failure can spread. None of that is new. We learned it in 2009. The question is why we had to learn it again. Tech is so fun...

Mahesh Sathiamoorthy avatar
Mahesh Sathiamoorthy@madiator·Nov 14

If an environment is stationary, i.e. it doesn't change much, you can use this to optimize the cost quite substantially. bespokelabs.ai/blog/envopt

Introducing our latest research: Environment Optimization (EnvOpt). An agent's cost and performance depend on 1. The model 2. The harness and 3. The environment structure. There is a lot of work on optimizing 1 and 2, but here we study how the environment presents information to the agent, and how that affects performance and cost. For example, a company's Slack workspace, documents, databases and code repos can be given to an agent in different ways: Slack can sit behind a remote server, accessible via MCP or exported in plain text files that can be read and searched with grep. This choice alone can significantly affect how much work the agent needs to perform to solve a task. Environment optimization is only meaningful in what we call stationary environments: These are environments with large collections of artifacts that remain the same across many tasks. We propose optimizing stationary environments as a third axis for improving LLM agents, alongside model training and harness optimization. This optimization can be amortized over multiple tasks and can be combined with harness and model optimization techniques. Across five benchmarks covering workspace tasks, data engineering and software engineering, ENVOPT reduces the cost of the target agent on unseen test sets by an average of 48%, with higher or similar pass rates. On AutomationBench, ENVOPT outperforms Meta-Harness by 9.67% with only 15% of its cost. We also show that ENVOPT constitutes an optimization direction orthogonal to harness optimization and parameter tuning in two ways: (1) the environments optimized for the target agents can transfer to other agents, even with different harnesses and models (2) we combined ENVOPT with Meta-Harness, showing similar cost reductions and 16% improvement in pass rate, outperforming either Meta-Harness or ENVOPT alone. Read more in our blog: bespokelabs.ai/blog/envopt paper: arxiv.org/abs/2611.07372.pdf

Post attachment
Mahesh Sathiamoorthy avatar
Mahesh Sathiamoorthy@madiator·Nov 13

In 2027, launching a thousand sandboxes for agent swarms is going to be routine. But rewriting your code every time you switch providers shouldn’t be. So we built sandbox-router and open-sourced it. One Python API across many providers, plus a control plane to see what’s running and what it’s costing. Features: • Run code, shell commands, and AI agents • Experiment locally, then move to the cloud with one word change. • Compare estimated costs and choose a provider • Upload files to sandboxes, clone repos, download results • Run CPU and GPU workloads • Track resource usage and costs in a dashboard: sandbox providers don't necessarily have the incentive to kill your idle sandboxes but let's fix that :) • Set sandbox lifetimes and provisioning limits • Perfect for agent swarms and running RL environments. github.com/bespokelabsai/san…

Mahesh Sathiamoorthy avatar
Mahesh Sathiamoorthy@madiator·Nov 13

AI (drama) likes November: * Nov 2022: Chatgpt released * Nov 2023: Sam Altman ousted * Nov 2026: Jensen resigns

Dimitris Papailiopoulos avatar
⟳ Mahesh Sathiamoorthy retweeted
Dimitris Papailiopoulos@DimitrisPapail·Nov 13

OK the findings that Alex et al have reinforce my belief that Sol 5.6 was the most creative R&D model Openai ever released. It both explores more idea basins, and is one of the least likely to get stuck.

Post attachmentPost attachment

What are idea basins and how do agents explore them? A few weeks ago, researchers from different labs independently discovered what we now call ‘Idea-basins’: As an autonomous agent iterates to solve a problem, it gets stuck into the methods it has tried before and its trajectories remain in localized regions of the solution space. Techniques like ‘Fork-and-Flush’ and ‘Scaling Discovery requires forgetting’ reduce this idea fixation by periodically deleting the session context. In this post we study how frontier models explore different idea basins for AutoResearchExam. E.g. for a problem on accelerating transformer decoder graphs on CPUs models can try Numpy+Asm Kernels, Native BLAS, JIT compilers and other idea basins. We find that some models switch ideas more than others while others get to the best idea faster and explore deeper. We find that most runs stick with one or two ideas, and models that submit more spend the extra submissions refining and overfit more. We see that models explore most in the first six hours, after which visits to new idea basins drop sharply. This suggests models start refining early leaving potential gains on the table: 58% of runs never reach the basin where the best solution to their task was found. Finally, bold exploration pays off: exploring a new idea gains about nine times as much as a refining step on average. Read more in our blog: bespokelabs.ai/blog/how-mode…

Post attachment