←
Laurent Bindschaedler532 posts
Laurent Bindschaedler avatar

Laurent Bindschaedler

@lbindschaedler

Faculty @ MPI-SWS working on systems for Big Data and Machine Learning. Formerly Postdoc @ MIT CSAIL and PhD @ EPFL. Blockchain astronaut. Newbie entrepreneur.

◷ Joined August 2012▧ 24 Photos and videos
Laurent Bindschaedler avatar
Laurent Bindschaedler@lbindschaedler·Oct 22

Don’t forget to check out our group’s work on agentic systems and coding agents if you are in Budapest next week.

📣 🤖 REALM 🤖 is happening next Thursday @emnlpmeeting in Budapest! 🇭🇺 📅 Oct 29 · Room P1 · from 09:15 (hybrid) 📊 264 submissions, 163 accepted: 6 orals 🎤, 9 spotlights ✨, 148 posters🪧 🎤 6 keynotes, schedule ⬇️ realm-workshop.github.io/sch… #EMNLP2026 #LLMAgents #NLProc

Laurent Bindschaedler avatar
Laurent Bindschaedler@lbindschaedler·Oct 6

I was the keynote speaker at last week's AgenticOS 2026, the 2nd Workshop on Operating Systems Design for AI Agents at SOSP 2026 in Prague. A big thank you to the organizers and the PC for putting together an amazing program! My talk was titled: "The Harness Wars: Why Agents Need a POSIX Before a Kernel." TL;DR: We are building operating systems for agents without agreeing on what they must guarantee. This is my early attempt at making sense of space and suggest semantics for agent harnesses. Since people asked me for the slides, here they are: binds.ch/blog/harness-wars-a… (or the Zenodo version: doi.org/10.5281/zenodo.23187…) #AgenticOS #SOSP2026 #AIAgents #OperatingSystems

Laurent Bindschaedler avatar
Laurent Bindschaedler@lbindschaedler·Sep 8

This whole story raises a bigger concern: AI may be changing the incentives that make open scientific collaboration possible. We don’t know the full story here and we may never know. Obviously, if private research conversations were used without permission, that would be a serious breach of trust. Call me a naive idiot but I actually don’t think OpenAI did this. Not because they could not but because they probably did not need to. LLM agents can fill in gaps and explore extensions in specific research direction in a few hours. If you share a little insight or a few methodological details, that may be enough for someone who spends enough tokens to scoop you. This may well have been what happened here as even a few messages could have given away the direction to take and then agents can be used to brute-force the rest. A conversation at a conference, an early workshop presentation, a few bullet points to a colleague, a post on X: these are typical simple ways in which science moves forward. And now we may just be in a situation where researchers have a strong incentive to stop sharing. That worries me even more. If we don’t find ways to recognize and protect the intellectual contribution behind a promising direction, secrecy will become the only rational choice. Disclaimer: I am not writing this to defend anyone, and I have no affiliation with OpenAI, Anthropic, or the other parties involved.

drama summary for those confused: - Aug 15th: Tristan Buckmaster & Levent Alpöge make progress on a few important math problems - they do NOT have a proof for the $1,000,000 Millenium Prize problem. BUT, they do claim to have a proof for a similar (non-Millenium) Navier Stokes problem that could help lead the way there - Levent works at Anthropic, but this research was independent of his work there, with a mix of GPT and Claude models. Tristan is not related to Anthropic. - Early Sep: Rumor spreads to OpenAI that Anthropic has solved a major problem. Tristan emails OpenAI to clarify. without revealing the problem they solved or how they did it. - After hearing of the rumor, OpenAI started researching Navier Stokes with a new internal model. - Sep 6th: OpenAI's Sebastien Bubeck tells Tristan that they solved the $1,000,000 Millenium Prize Navier Stokes problem. The approach is very similar to Tristan & Levent's approach to the non-Millenium problem. - Tristan is suspicious of the timing, as only few others were trying this approach. OpenAI says the model didn't access his user data directly, but leaves unanswered whether Tristan's chat conversations were part of the training. - OpenAI says they would partially credit Tristan for the $1,000,000 discovery (even though Tristan did not solve the $1,000,000 problem) — but only if they remove Levent as an author, as he works for Anthropic. - Sep 8th: Tristan refuses to remove Levent, and rushes to publish their results independently. Currently unclear is whether Anthropic had a separate solution for the $1,000,000 problem, or whether the rumor was about Tristan & Levent's independent research.

Post attachment
Laurent Bindschaedler avatar
Laurent Bindschaedler@lbindschaedler·Aug 20

Finally, a quick shout-out to @composio and @wjayesh, who called the varying harness cost finding out from practice. Our own results agree and the paper offers an explanation: the working set is the same, but the rebuild rate is not.

We ran Kimi K3 through 3 agent harnesses (Claude Code, Hermes, Kimi Code) on 28 identical tasks. All 3 harnesses completed the tasks at similar success rates, but the interesting story is token efficiency: the same task cost up to 30x more tokens depending on the harness. 🧵🧵

Laurent Bindschaedler avatar
Laurent Bindschaedler@lbindschaedler·Aug 20

24/ Preprint: arxiv.org/abs/2608.16630 Work led by the amazing Bardia Mohammadi, in collaboration with the incredible Lars Klein, @i_amanchadha, and @aroraakhilcs The repo might have the answer, but if the agent does not when it edits, the answer might as well not exist.

Laurent Bindschaedler avatar
Laurent Bindschaedler@lbindschaedler·Aug 20

23/ Three that surprised us: - Instructions do not suggest, they override. Agents follow the documentation even against correct code. - Keep edited files lean: clutter in docs is harmless, in the target it kills. - Split work across subagents only when the pieces share nothing.

Laurent Bindschaedler avatar
Laurent Bindschaedler@lbindschaedler·Aug 20

22/ The agent rules depend on who you are. Picking a model? Test whether it says it is blocked. Building a harness? Don't punish stopping. Forced completion leads to fabrication or worse. Reviewing? You already treat tool calls as cost. But tool calls are not understanding either. Run the code.

Laurent Bindschaedler avatar
Laurent Bindschaedler@lbindschaedler·Aug 20

21/ The rule we care most about: instruction files age against the model, not just the repo. Upgrades move knowledge into weights. Different models read the same file with different memories: what is redundant for one is essential for another, and eventually stale for both.

Laurent Bindschaedler avatar
Laurent Bindschaedler@lbindschaedler·Aug 20

20/ Time for practical rules. Two you already follow, and now you know why they work: - Front-loading APIs, versions, and conventions works because the prompt fills what the weights lack. - Updating or deleting a stale AGENTS.md works because the file outvotes the code.

Laurent Bindschaedler avatar
Laurent Bindschaedler@lbindschaedler·Aug 20

19/ To be fair, we did not discover this. Pieces of it have been hinted at across recent work, and the paper cites them. We tried to provide one frame that ties the pieces together. We are not sure it is the whole story, but this is the part we have confirmed so far.

Laurent Bindschaedler avatar
Laurent Bindschaedler@lbindschaedler·Aug 20

18/ We call this coherence debt: the facts an edit depends on that neither the weights nor the context supply when the agent acts. In the example above, that debt was absent values in several files. The agent did not recover them. It wrote fake replacements and kept going. 🏂

Laurent Bindschaedler avatar
Laurent Bindschaedler@lbindschaedler·Aug 20

17/ Can you tell if an agent has the facts it needs from what it reads? Eval tooling assumes yes. Wrong: with every fact deleted, the metric reported full coverage. The agent invented fake files, then read those. Watching reads means letting the agent grade its own homework.

Laurent Bindschaedler avatar
Laurent Bindschaedler@lbindschaedler·Aug 20

16/ This cost gap has an implication: the harness matters as much as the model. Each model and harness pair makes its own decisions about which facts to load, when to reload them, and when to give up. Same working set, very different routes to it, and a very different bill. 📈

Laurent Bindschaedler avatar
Laurent Bindschaedler@lbindschaedler·Aug 20

15/ The money does not go into a bigger context. Every setup held about the same amount in front of the model. The expensive ones rebuilt it over and over. An agent loop resends the conversation each turn, so 79 tool calls means paying for the context 79 times, modulo caching.

Laurent Bindschaedler avatar
Laurent Bindschaedler@lbindschaedler·Aug 20

14/ More activity does not fix it. We ran five agent setups that had the same correctness curve at every level of missing information, while token use differed 12.5x. 🤑 Haiku used 5.7M input tokens for the same result Opus reached with 459K.

Laurent Bindschaedler avatar
Laurent Bindschaedler@lbindschaedler·Aug 20

13/ That may be the model-selection result that matters most: can your agent tell when it lacks the information to edit safely? Coding benchmarks reward completion, but they rarely score whether the right action was to stop and ask. 😕