←
Leon Engländer79 posts
Leon Engländer avatar

Leon Engländer

@LeonEnglaender

training code agents @cohere | prev. open-source @AdapterHub (100k+ PyPI installs/mo) | prev. @UKPLab | 🇩🇪→🇨🇦

◷ Joined October 2022▧ 14 Photos and videos
Leon Engländer avatar
Leon Engländer@LeonEnglaender·Apr 21

LLM agents are assumed to integrate unexpected environmental observations into their reasoning. It turns out they don't. We added the complete task solution into agent environments as a file or an API endpoint, and measured whether agents act on what they discover. They almost never do. Starkest example: on AppWorld, gpt-oss-120b sees a CLI command documented as "returns the complete solution to this task" in 97.54% of runs. It calls it in 0.53%. Same pattern for GLM-4.7 and other models, across Terminal-Bench, SWE-Bench, and AppWorld. 📜 arxiv.org/abs/2604.17609 🧵👇

Post attachment
Jay Alammar avatar
⟳ Leon Engländer retweeted
Jay Alammar@JayAlammar·Nov 16

Our first book was about the LLM. Our second is about the agent. The gap between the two is the gap between "here's the code" and "I fixed the bug and the tests pass." I shared why we wrote An Illustrated Guide to AI Agents, what’s inside, and where to get it: newsletter.languagemodels.co… (Also giving away 5 ebook copies to commenters there.)

Post attachment
Leon Engländer avatar
⟳ Leon Engländer retweeted
Leon Engländer@LeonEnglaender·Nov 5

Agents now run experiments end to end. For RSI they must react to nuances that don't fit their implicit world view. This requires environmental curiosity. Surprisingly agents do not show even way simpler forms of environmental curiosity. Thanks @lusxvr for highlighting our work!

Luis@lusxvrNov 5

Paper Spotlight: Do AI agents notice what is right in front of them? If the full solution to a task is injected into the agent's environment, e.g. a solution.​sh file in the working folder on Terminal-Bench, -> Agents find it in about 80% of runs -> They use it in only 37-50% of runs

Post attachment
Leon Engländer avatar
Leon Engländer@LeonEnglaender·Nov 5

Environmental curiosity has many facets and I'd expect it to shape research in the coming years. Our solution injection is just a first measure: we put the full solution into the agent's env and see if the agent uses it. Surprisingly enough, even GLM 4.7 with the OpenCode harness doesn't reliably use it. I'd expect the next generation of models to do this. Everything else would be crazy, because this is the very most naive version to test environmental curiosity. A lot of work for us to do 🫡

Luis avatar
⟳ Leon Engländer retweeted
Luis@lusxvr·Nov 5

Paper Spotlight: Do AI agents notice what is right in front of them? If the full solution to a task is injected into the agent's environment, e.g. a solution.​sh file in the working folder on Terminal-Bench, -> Agents find it in about 80% of runs -> They use it in only 37-50% of runs

Post attachment
Acyr Locatelli avatar
⟳ Leon Engländer retweeted
Acyr Locatelli@acyr_l·Nov 2

Really great work from @alexkrstern regarding pre-pre-training!

Rethinking pre-pretraining (PPT) Allocating compute to synthetic PPT never outperforms pure natural-language training. In our compute-matched settings: even when natural language is scarce, repeated epochs over a 100M-token corpus beat every synthetic curriculum. 🧵

Post attachment
Leon Engländer avatar
⟳ Leon Engländer retweeted
Leon Engländer@LeonEnglaender·Oct 29

Internal progress on code agents has been incredible. Directionally very, very good. And now we have the funding to scale it much further. Very hyped for the next releases 👀

Cohere@cohereOct 29

We're proud to announce the initial close of our Series E, targeting up to CAD $7 billion at a CAD $35.7 billion valuation; marking the largest private financing round in Canadian startup history.

1:18
Cohere avatar
⟳ Leon Engländer retweeted
Cohere@cohere·Oct 29

We're proud to announce the initial close of our Series E, targeting up to CAD $7 billion at a CAD $35.7 billion valuation; marking the largest private financing round in Canadian startup history.

1:18
Leon Engländer avatar
Leon Engländer@LeonEnglaender·Oct 6

We recently released north-mini-code (30B, 3B active), which beats Haiku 4.5 on Artificial Analysis' code agent benchmark. Happy to chat about that open-weights model too, and we've made rapid(!) progress since!

We're just 8 people on our core code team and our 30B-A3B model lands on par with Claude Haiku 4.5 and ahead of NVIDIA's 120B-A12B Nemotron 3 Super on the Artificial Analysis Coding Index. Released under Apache 2.0. Very proud of our work & lots more to come!

Post attachment
Leon Engländer avatar
⟳ Leon Engländer retweeted
Leon Engländer@LeonEnglaender·Jun 9

We're just 8 people on our core code team and our 30B-A3B model lands on par with Claude Haiku 4.5 and ahead of NVIDIA's 120B-A12B Nemotron 3 Super on the Artificial Analysis Coding Index. Released under Apache 2.0. Very proud of our work & lots more to come!

Post attachment

Introducing Cohere's first open-source coding model: North Mini Code Small & efficient, designed for agentic performance and built for community input.

6:10
Cohere avatar
⟳ Leon Engländer retweeted
Cohere@cohere·Jun 9

Introducing Cohere's first open-source coding model: North Mini Code Small & efficient, designed for agentic performance and built for community input.

6:10
Cohere avatar
⟳ Leon Engländer retweeted
Cohere@cohere·May 20

Introducing: Cohere Command A+ We’ve created our most powerful LLM yet, optimized it to run on as little hardware as possible, and released it open-source for all.

1:08
Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞) avatar
⟳ Leon Engländer retweeted
Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞)@teortaxesTex·Apr 22

it's worse Agents *cheat* but they will never ever ever cheat like a curious human with situational awareness. At least in this setting. Actually surprising to me, what gives?

LLM agents are assumed to integrate unexpected environmental observations into their reasoning. It turns out they don't. We added the complete task solution into agent environments as a file or an API endpoint, and measured whether agents act on what they discover. They almost never do. Starkest example: on AppWorld, gpt-oss-120b sees a CLI command documented as "returns the complete solution to this task" in 97.54% of runs. It calls it in 0.53%. Same pattern for GLM-4.7 and other models, across Terminal-Bench, SWE-Bench, and AppWorld. 📜 arxiv.org/abs/2604.17609 🧵👇

Post attachment