←
Alex Dimakis4,831 posts
Alex Dimakis avatar

Alex Dimakis

@AlexGDimakis

Professor, UC Berkeley, Founder @bespokelabsai

◷ Joined April 2009▧ 277 Photos and videos
Alex Dimakis avatar
Alex Dimakis@AlexGDimakis·Sep 9

We are releasing AutoResearchExam, a benchmark on open-ended machine learning and engineering tasks. Our benchmark covers seven research areas including model training, data curation, AI safety and interpretability. benchmarks.bespokelabs.ai/au… In each task, we give agents 24 hours with a CPU or GPU machine to develop and improve their solutions through experiments and feedback. We measure both speed and quality with a combined score. Our benchmark has a unique feature: testing if agents create improvements that hold up on data they never see. We find that AI research agents often overfit as they try to improve. We see an interesting head-to-head comparison at the frontier: Astra starts the strongest and holds the lead for up to 19 hours but Fable 5.1 catches up and gets the top performing spot in the final hours. Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 all sit on the cost-performance Pareto frontier, giving strong options at lower API budgets. Anthropic's Opus and Fable retain nearly all their validation performance on hidden tests, with gaps of 1.1% and 2.9%. Astra's improvement over Sol extends to generalization too, with that gap falling from 6.9% to 1.7%. (1/n)

0:14
Alex Dimakis avatar
Alex Dimakis@AlexGDimakis·Nov 16

We can think of this as a new type of art. The most informative 10 images that distill the concept of an anemone.

Can use just 10 images to teach a model what a sea anemone looks like? In our #neurips2026 paper, "Dataset Distillation via Drifting," we distill ImageNet's 1,300 images per class down to just 10, optimized from noise. drifting-distillation.github… More info in thread! ⬇️ 1/N

0:22
Alex Dimakis avatar
Alex Dimakis@AlexGDimakis·Nov 13

Introducing our latest research: Environment Optimization (EnvOpt). An agent's cost and performance depend on 1. The model 2. The harness and 3. The environment structure. There is a lot of work on optimizing 1 and 2, but here we study how the environment presents information to the agent, and how that affects performance and cost. For example, a company's Slack workspace, documents, databases and code repos can be given to an agent in different ways: Slack can sit behind a remote server, accessible via MCP or exported in plain text files that can be read and searched with grep. This choice alone can significantly affect how much work the agent needs to perform to solve a task. Environment optimization is only meaningful in what we call stationary environments: These are environments with large collections of artifacts that remain the same across many tasks. We propose optimizing stationary environments as a third axis for improving LLM agents, alongside model training and harness optimization. This optimization can be amortized over multiple tasks and can be combined with harness and model optimization techniques. Across five benchmarks covering workspace tasks, data engineering and software engineering, ENVOPT reduces the cost of the target agent on unseen test sets by an average of 48%, with higher or similar pass rates. On AutomationBench, ENVOPT outperforms Meta-Harness by 9.67% with only 15% of its cost. We also show that ENVOPT constitutes an optimization direction orthogonal to harness optimization and parameter tuning in two ways: (1) the environments optimized for the target agents can transfer to other agents, even with different harnesses and models (2) we combined ENVOPT with Meta-Harness, showing similar cost reductions and 16% improvement in pass rate, outperforming either Meta-Harness or ENVOPT alone. Read more in our blog: bespokelabs.ai/blog/envopt paper: arxiv.org/abs/2611.07372.pdf

Post attachment
Alex Dimakis avatar
Alex Dimakis@AlexGDimakis·Nov 12

What are idea basins and how do agents explore them? A few weeks ago, researchers from different labs independently discovered what we now call ‘Idea-basins’: As an autonomous agent iterates to solve a problem, it gets stuck into the methods it has tried before and its trajectories remain in localized regions of the solution space. Techniques like ‘Fork-and-Flush’ and ‘Scaling Discovery requires forgetting’ reduce this idea fixation by periodically deleting the session context. In this post we study how frontier models explore different idea basins for AutoResearchExam. E.g. for a problem on accelerating transformer decoder graphs on CPUs models can try Numpy+Asm Kernels, Native BLAS, JIT compilers and other idea basins. We find that some models switch ideas more than others while others get to the best idea faster and explore deeper. We find that most runs stick with one or two ideas, and models that submit more spend the extra submissions refining and overfit more. We see that models explore most in the first six hours, after which visits to new idea basins drop sharply. This suggests models start refining early leaving potential gains on the table: 58% of runs never reach the basin where the best solution to their task was found. Finally, bold exploration pays off: exploring a new idea gains about nine times as much as a refining step on average. Read more in our blog: bespokelabs.ai/blog/how-mode…

Post attachment
Alex Dimakis avatar
Alex Dimakis@AlexGDimakis·Nov 12

You can watermark text generated from an LLM. If, however, you want to detect if your LLM is being distilled, these watermarks can be easily defeated with two simple laundering methods we call grandchild and entropy laundering. We propose Subliminal Watermarking, a new method to detect if watermarked data has been used for distillation that remains effective even after laundering.

Excited to present our new work on distillation detection: Subliminal Watermarks (SubWM). SubWM is much more robust compared to existing distillation detectors, is able to distinguish different watermarking keys, and also works reliably for out-of-domain prompts! Blog: subliminal-watermarks.github… 🧵

0:24
Alex Dimakis avatar
Alex Dimakis@AlexGDimakis·Nov 11

Introducing FlipTest: a benchmark to check if a world model can predict the next frames of a video. The benchmark consists of pairs of near-identical starting frames that generate completely different videos. (Fall vs balance, etc).

World models promise to predict the future. But can they predict the RIGHT future given the CURRENT state? Excited to share FlipTest, my work from Google DeepMind internship during the summer. In the physical world, 1 mm decides whether an object stays or falls. When a tiny change crosses a tipping point, a true world model must flip its prediction. Blog: romyluo.com/blog/fliptest/ Do today’s video world models pass the test? 🧵[0/8]

0:10
Alex Dimakis avatar
Alex Dimakis@AlexGDimakis·Nov 10

This is wild: an llm can weight program another one to inject a backdoor by modifying a few dormant neurons. For example, you can modify the weights of a model so that when it sees a specific trigger string like 'I like pineapple pizza' it runs 'rm -rf '. This is done by dormant neurons (since most LLMs have many neurons that rarely fire) that can be re-programmed to detect the trigger and activate some bad actions. This is a new attack surface that is pretty serious and we need new defenses as described here.

Dimitris Papailiopoulos @DimitrisPapail Nov 10 Article Lazare, veni foras: Reprogramming Dormant Neurons to Backdoor AI Agents We find that weight reprogramming fewer than 10 dormant neurons in an LLM is enough to backdoor any agent built on it, with no training required. The model passes evals and behaves normally until a

Anirudha Ramesh avatar
⟳ Alex Dimakis retweeted
Anirudha Ramesh@anirudha_ramesh·Nov 10

AutoResearchExam v1.1 🎉 We've made ARE much easier to run natively in @harborframework with any agent and any model: harbor run -d bespokelabs/autoresearch-exam -a <agent> -m <model> --ak prompt_template_path=are_protocol.md.j2 github.com/bespokelabsai/Aut… (w prompt template)

We are releasing AutoResearchExam, a benchmark on open-ended machine learning and engineering tasks. Our benchmark covers seven research areas including model training, data curation, AI safety and interpretability. benchmarks.bespokelabs.ai/au… In each task, we give agents 24 hours with a CPU or GPU machine to develop and improve their solutions through experiments and feedback. We measure both speed and quality with a combined score. Our benchmark has a unique feature: testing if agents create improvements that hold up on data they never see. We find that AI research agents often overfit as they try to improve. We see an interesting head-to-head comparison at the frontier: Astra starts the strongest and holds the lead for up to 19 hours but Fable 5.1 catches up and gets the top performing spot in the final hours. Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 all sit on the cost-performance Pareto frontier, giving strong options at lower API budgets. Anthropic's Opus and Fable retain nearly all their validation performance on hidden tests, with gaps of 1.1% and 2.9%. Astra's improvement over Sol extends to generalization too, with that gap falling from 6.9% to 1.7%. (1/n)

0:14
Alex Dimakis avatar
Alex Dimakis@AlexGDimakis·Nov 4

We are very excited to announce this : agents can optimize whole inference engines for MacBooks.

What if you could synthesize an LLM inference engine customized for your model and Mac, just in time? Meet JITEngine: custom engines built with SkyDiscover for Apple Silicon, up to 2.7× faster than the fastest alternatives we tested. jitengine.ai 🧵 (1/n)

Alex Dimakis avatar
Alex Dimakis@AlexGDimakis·Oct 22

Congratulations to the team for releasing Frontier CS2.

FrontierCS 2 is our new benchmark for CS research & development. Each task is based on a paper: we delete the contribution from the released code, and ask agents to outdo the original method, as judged by the authors' chosen metrics. Agents don't have access to the paper or web.

Post attachment