←
Howard Chen233 posts
Howard Chen avatar

Howard Chen

@__howardchen

Research @EngramLab Prev: PhD @Princeton

◷ Joined August 2013▧ 18 Photos and videos
Howard Chen avatar
Howard Chen@__howardchen·Nov 4

Text memory is great. But how can the agents stay familiar with these ever-growing notes? In this blog, we explore the dynamics of learning and text memory -- we find that training is a compelling option to use notes better and can endure in a changing world!

In our new blog, we share some new science investigating how RL complements text-based memory by letting models learn how to use their notes. It turns out that training can be cheap compared to building notes, and stays surprisingly robust even through millions of tokens of edits.

0:03
Natalie Biderman avatar
⟳ Howard Chen retweeted
Natalie Biderman@Natalie_Bid·Nov 5

Our new research shows that good notes help, but training models to use them makes a big difference.

In our new blog, we share some new science investigating how RL complements text-based memory by letting models learn how to use their notes. It turns out that training can be cheap compared to building notes, and stays surprisingly robust even through millions of tokens of edits.

0:03
Sabri Eyuboglu avatar
⟳ Howard Chen retweeted
Sabri Eyuboglu@EyubogluSabri·Nov 4

A student takes notes for an open book exam and shares them with a friend. They both use the notes during the exam, yet only the one who made it aces Similarly, for language models, neural and text memory stack

In our new blog, we share some new science investigating how RL complements text-based memory by letting models learn how to use their notes. It turns out that training can be cheap compared to building notes, and stays surprisingly robust even through millions of tokens of edits.

0:03
Dhruv Gautam avatar
⟳ Howard Chen retweeted
Dhruv Gautam@dhruvgautam·Nov 4

our LLM equivalent of the undergraduate trauma-bonding experience of rereading discrete math notes until they made sense

In our new blog, we share some new science investigating how RL complements text-based memory by letting models learn how to use their notes. It turns out that training can be cheap compared to building notes, and stays surprisingly robust even through millions of tokens of edits.

0:03
Jessy Lin avatar
⟳ Howard Chen retweeted
Jessy Lin@realJessyLin·Nov 4

We've been doing some science on how training <> harness memory fit together, and consistently find they stack -- the strongest systems use all kinds of memory Excited to share some results on a clean and simple setting! Teaching models to use their notes with RL augments base models with memory, and taking gradients is actually pretty cheap I think the trend will be that a lot of harness/prompt eng today will matter less as we just let models learn how to operate with what they have at their disposal. This is especially relevant as text memory and raw context blurs together - as we've seen in codebases exploding with agent-generated content, learning how to juggle docs and skill.mds itself is becoming a skill :)

In our new blog, we share some new science investigating how RL complements text-based memory by letting models learn how to use their notes. It turns out that training can be cheap compared to building notes, and stays surprisingly robust even through millions of tokens of edits.

0:03
Jack Morris avatar
⟳ Howard Chen retweeted
Jack Morris@jxmnop·Nov 4

these days i'm quite excited about the prospect of models that understand more about our data than we do one barrier we've run into is that when models run for a long time, they take increasingly inscrutable notes; it turns out that small amounts of training can fix this also fun to share a glimpse into the kind of work we've been up to at Engram! expect more soon

In our new blog, we share some new science investigating how RL complements text-based memory by letting models learn how to use their notes. It turns out that training can be cheap compared to building notes, and stays surprisingly robust even through millions of tokens of edits.

0:03
Engram avatar
⟳ Howard Chen retweeted
Engram@EngramLab·Nov 4

In our new blog, we share some new science investigating how RL complements text-based memory by letting models learn how to use their notes. It turns out that training can be cheap compared to building notes, and stays surprisingly robust even through millions of tokens of edits.

0:03
Jiayi Geng avatar
⟳ Howard Chen retweeted
Jiayi Geng@JiayiiGeng·Oct 14

We’re planning to expand EurekaBench, an expert-verified benchmark that tests whether AI agents can go beyond simply finding solutions to discover novel scientific insights. We’re looking to collaborate with domain experts on two types of contributions: 1. Additional insight questions for existing problems: Different approaches to the same science problem can yield different scientific insights. If you've worked on a problem already included in EurekaBench but approached it differently, we’d love to hear what additional insights you think an AI agent could discover. 2. New scientific problems: We’d like to include more open scientific problems across our six existing domains, as well as in new domains. These may be recently published science problems that remain only partially solved, or unpublished problems. If you’re interested in contributing, please check out our contribution guidelines for more details: eurekabench.github.io/contri….

AI scientist agents are great at optimizing and finding solutions through endless trial and error. But is that really what science is about? As Terrance Tao eloquently put, the role of math and science should be more than that – they are "lighthouses" that guide and inspire exploration through understanding and insights. Can AI agents discover novel insights through experimentation? To study these gaps, we worked with domain experts to introduce EurekaBench, a benchmark spanning 6 science domains that evaluates an agent’s ability to discover genuinely new scientific insights. (1/n)

Post attachment
Adithya Bhaskar avatar
⟳ Howard Chen retweeted
Adithya Bhaskar@AdithyaNLP·Oct 5

The most profound paper of my PhD so far. We truly did something special to make LMs play/explain chess, needing both architectural & algorithmic innovation (natural-language analogue of the Alphazero algorithm) + applicable to many other domains. Please read on and share! 1/10

Post attachment
Yao Lai avatar
⟳ Howard Chen retweeted
Yao Lai@chingyaolai·Oct 3

Prediction ≠ Insights. Despite the fact that AI agents excel at well-defined prediction tasks, they still struggle to distill insights from predictions like human scientists do, across a variety of domains. Check out this paper by @JiayiiGeng that asks this important question.

AI scientist agents are great at optimizing and finding solutions through endless trial and error. But is that really what science is about? As Terrance Tao eloquently put, the role of math and science should be more than that – they are "lighthouses" that guide and inspire exploration through understanding and insights. Can AI agents discover novel insights through experimentation? To study these gaps, we worked with domain experts to introduce EurekaBench, a benchmark spanning 6 science domains that evaluates an agent’s ability to discover genuinely new scientific insights. (1/n)

Post attachment
Zora Wang avatar
⟳ Howard Chen retweeted
Zora Wang@ZhiruoW·Oct 2

Scientific discovery is very different from solving fixed-procedure tasks. Agents are getting really good at brute-force search and optimization. But science is about more: finding mechanisms, connecting the dots, and uncovering insights that open new directions. EurekaBench shows this gap clearly: agents can beat humans at prediction, but still struggle to generate the insights that keep science moving forward. The goal isn't just to search faster, but to figure out what’s worth searching for next!

AI scientist agents are great at optimizing and finding solutions through endless trial and error. But is that really what science is about? As Terrance Tao eloquently put, the role of math and science should be more than that – they are "lighthouses" that guide and inspire exploration through understanding and insights. Can AI agents discover novel insights through experimentation? To study these gaps, we worked with domain experts to introduce EurekaBench, a benchmark spanning 6 science domains that evaluates an agent’s ability to discover genuinely new scientific insights. (1/n)

Post attachment
Jiayi Geng avatar
⟳ Howard Chen retweeted
Jiayi Geng@JiayiiGeng·Oct 2

AI scientist agents are great at optimizing and finding solutions through endless trial and error. But is that really what science is about? As Terrance Tao eloquently put, the role of math and science should be more than that – they are "lighthouses" that guide and inspire exploration through understanding and insights. Can AI agents discover novel insights through experimentation? To study these gaps, we worked with domain experts to introduce EurekaBench, a benchmark spanning 6 science domains that evaluates an agent’s ability to discover genuinely new scientific insights. (1/n)

Post attachment
Howard Chen avatar
Howard Chen@__howardchen·Oct 2

AI agents now seem to solve the hardest problems in math and science, but they haven't yet deepened our understanding at the same rate. They are great problem-solving machines when the objective and optimization process are clear, but discovering insights is a lot fuzzier! We build EurekaBench to measure this gap using 6 natural science domains (neuroscience, geophysics, plasma physics, astrophysics, computer science, and chemistry). We test if AI agents can complete the whole scientific discovery loop from understanding the observations, explaining the data by describing the mechanism (e.g., equations), and eventually discovering insights. Results show that the agents don't advance on both improving accuracy and finding insights at the same pace! A lot more details in the thread. Fantastic work led by @JiayiiGeng.

AI scientist agents are great at optimizing and finding solutions through endless trial and error. But is that really what science is about? As Terrance Tao eloquently put, the role of math and science should be more than that – they are "lighthouses" that guide and inspire exploration through understanding and insights. Can AI agents discover novel insights through experimentation? To study these gaps, we worked with domain experts to introduce EurekaBench, a benchmark spanning 6 science domains that evaluates an agent’s ability to discover genuinely new scientific insights. (1/n)

Post attachment
Cade Daniel 🇺🇸 avatar
⟳ Howard Chen retweeted
Cade Daniel 🇺🇸@cdnamz·Aug 17

Work from @EngramLab on study compute reducing inference tokens while simultaneously increasing their quality👇

Our models study the workspace through thousands of rollouts before deployment. When solving new tasks, they produce better responses and get to the right answer faster. With study, we're able to outperform Opus 4.8 X-high while using 3.3x fewer tokens.

Post attachment