←
Leo Gorodinski4,466 posts
Leo Gorodinski avatar

Leo Gorodinski

@eulerfx

Founder @CohesiveSystems, Prev: Co-founder/CTO @AlvysTeam, First engineer → VPE @JetTechnology acq. by @Walmart

◷ Joined May 2008▧ 227 Photos and videos
Leo Gorodinski avatar
Leo Gorodinski@eulerfx·11h

The best frontier model (Opus 5.5) solves 18.5% of long-horizon coding tasks, even with a clear correctness oracle. With four attempts it solves 31.9%. 38 of 69 tasks were solved by no model. Setup - Multi-step SWE: 69 tasks, each a chain of 5–15 dependent, multi-hour steps on one codebase, each building on the agent’s own earlier code - Four kinds of work: optimization, migration, feature implementation, and version upgrades (hardest: 2.9% solved across models) - Each step’s instruction and grader derive from the same outside source of truth (so they can’t drift apart) - Two rounds of expert review per task, graders rebuild from source and run unprivileged, 56 of 69 tasks block network access Findings - Failures compound: a step after a pass passes 70% of the time, after a fail 14% - Agents aren’t running out of time...they’re declaring victory. Only 12% of failed steps got near the time limit, and in 52% the agent reported completion without flagging any failing check Limitations - No isolated-step baseline, so carried-forward errors can’t be separated from step difficulty - Public projects (memorization possible), 38/69 tasks are C/C++, and each model ran in its own harness - Wide CI (11.6–25.7%), so the top models are statistically close My take This benchmark hands agents a precise spec, and they still can’t tell when they’re done. Real work is underdetermined: turning intent into checks is part of the challenge, and almost no benchmark measures it. arxiv.org/abs/2611.07166

Post attachment
Leo Gorodinski avatar
Leo Gorodinski@eulerfx·Nov 17

Property-based tests generated from the spec and from the code catch different bugs. Use both. In a new paper, across 25 agent-built apps, Spec-Driven PBT found 56 bugs the Code-Driven tests missed, and Code-Driven PBT found 43 that Spec-Driven missed, so 59% of bugs were found by only one approach. Setup - An agent writes invariants in plain English from the requirements doc - Then it reads the code to turn them into property tests - The baseline derives properties from the code alone Why PBT? - Each property-based test kills about 50× as many mutants as the average unit test (prior work cited in paper) - Properties double as documentation, far shorter than the implementation Bugs only one approach found, by type - Spec-Driven: mostly state, lifecycle and logic errors (61% of 56) - Code-Driven: mostly input handling (49% of 43) Limitations - Bugs were judged by the same model that wrote the tests (Claude Opus 5); 25 sampled by hand, all held up - The prompts differed between approaches - The method needs a good spec Insight "Invariants obtained from the specification are not biased by the behaviour of a particular implementation, and can therefore falsify behaviour about which the implementation is internally consistent." My take The code can’t be the only oracle, the spec must be one too. Next: move key properties into types and checks, so they hold by construction. arxiv.org/abs/2611.15920

Post attachment
Qian Li avatar
⟳ Leo Gorodinski retweeted
Qian Li@qianl_cs·Nov 16

Orleans is a classic paper worth revisiting, from before most modern durable execution frameworks. I think virtual actors/durable objects are in some sense the object-oriented cousins of durable workflows. Both make applications reliable, but they focus on different abstractions: durable state and identity vs. durable execution progress.

Durable Objects are Virtual Actors microsoft.com/en-us/research…

Post attachment
Robert Penner avatar
⟳ Leo Gorodinski retweeted
Robert Penner@robertpenner·Nov 16

Now I'm tackling something more ambitious: formalizing XState's runtime behavior with LemmaScript. I started with XState's lightweight FSM, proving 31 laws governing state transitions, context updates and sequences of events. Comparing the formally verified model against XState's implementation helped uncover two bugs, both since fixed upstream. That led me deeper into XState's runtime. In my first week, I filed 7 bug reports involving subtle problems with transitions, event ordering and state behavior. It's exciting to see formal verification help uncover bugs in a mature, widely used open-source library.

Leo Gorodinski avatar
Leo Gorodinski@eulerfx·Nov 16

We keep reinventing how software executes and recovers, but still lack a model of how systems compose and what must stay true across boundaries. Why I'd separate domain semantics, composition, and infrastructure.

Leo Gorodinski avatar
Leo Gorodinski@eulerfx·Nov 15

Frontier agents can build software, but they struggle to verify it. That’s the conclusion of TestSuiteBench from ETH Zurich, a new benchmark evaluating how well coding agents, given only a prose spec, generate tests that distinguish correct behavior from incorrect behavior. Results - Writing tests from a spec (Opus 5): 96.1% are valid, but they catch only 55.5% of injected faults (mutants). - Writing the program from the same spec (Opus 5): its code passes 91.8% of the reference tests. - Giving an agent (GPT 5.6 Sol) the reference executable doesn’t help. The bottleneck is test design. Setup - Start with 165 real applications (C++, Rust, Go) and build on ProgramBench’s reference tests, synthesized by agents from existing repository tests and executable behavior. - Generate prose specs from those tests, then iteratively refine them to remove ambiguity and contradictions. - Generate requirement-targeted mutants: agents deliberately introduce faults into the reference implementations, then refine them to minimize unrelated test failures. - Ask agents to generate new tests from the specs, without seeing the reference implementation. Measure whether they pass on the reference (validity) and catch the mutants (mutation score). Limitations - Mutants are synthetic, designed to violate behaviors covered by reference tests. They aren’t necessarily representative of real-world bugs. - The specs are unusually favorable: derived from known tested behavior and repeatedly refined. Using existing repository documentation instead causes mutation scores to drop dramatically. - The benchmark doesn’t directly evaluate agents testing their own generated implementations, and it covers only CLI applications. My take A passing test says little about whether the test itself is good. Implementation has a feedback loop: run the tests and fix what fails. Test generation lacks an equivalent signal of adequacy by default. Unprompted, Opus attempted mutation testing in 33 of 165 applications, potentially giving itself that missing signal. Agents need to test their tests, not just their code. arxiv.org/abs/2610.13065

Post attachment
Leo Gorodinski avatar
Leo Gorodinski@eulerfx·Nov 14

Sometimes reading the code is the fastest way to comprehend. Not prose, not architecture diagrams, but code. Can we rely solely on extensional inquiry, judging code only by what it does?

Uncle Bob says he stopped reading the code his agents write. Whether a human should still read the output is the most interesting question in software right now.

Leo Gorodinski avatar
Leo Gorodinski@eulerfx·Nov 14

Great essay. “Nobody trusts your agent” says what most of us sense. While we were growing Alvys, customers wanted to look us in the eye: trust traces back to a person with something to lose, even with corporations. The big question is how far skin in the game can scale: How much agent work can one person take responsibility for? Does accountability add up like capital, or do many small stakes beat one big one? What backs accountability if agents sell at cost?

Andrew D'Souza @andrewdsouza Nov 13 Article Nobody Trusts Your Agent If 2026 was the year of agents, 2027 will be the year of principals. Nobody has ever trusted an agent, and nobody ever will. An agent is something that acts on someone’s behalf. A principal is the one

Leo Gorodinski avatar
Leo Gorodinski@eulerfx·Nov 13

Interesting work from Jane Street on building a high-performance parallel programming DSL in OxCaml. Instead of writing hardware-specific SIMD instructions, developers express computations using the SPMD (Single Program, Multiple Data) model familiar from CUDA and other parallel programming systems. The compiler: - Builds typed computation DAGs, distinguishing uniform and per-lane values. - Lowers them into an A-Normal Form (ANF) IR. - Optimizes through hoisting, loop unrolling, and vectorization. - Uses OxCaml metaprogramming to JIT-compile specialized native code. At Cohesive, we're pursuing a similar approach with an embedded C# DSL and metaprogramming: fusing declarative definitions of relationships, state transitions, and workflows into efficient executable code. Different domain, same principle: make semantics explicit so the compiler can optimize across abstraction boundaries rather than execute each declaration independently.

Post attachment

This was a lot of fun to write! Compilers are so cool and it’s pretty amazing that interns can do major work on them. blog.janestreet.com/writing-…

Leo Gorodinski avatar
Leo Gorodinski@eulerfx·Nov 13

An AI agent “fixed” a Kubernetes incident by accepting an invalid input that the system was supposed to reject. SWEOps-Bench tests agents on 55 historical bugs across 16 Kubernetes operators: - Detect the incident - Investigate the running system - Patch the code - Verify recovery after deployment. Across six agent/model configurations: - Incident detection: 73–91% - Verified recovery: 15–53% The interesting part is how they evaluate “fixed.” Clearing the fault isn’t enough. The repair must also pass checks for the affected functionality and regressions. Diagnosis and recovery only partly coincide. In 15–22% of tasks, agents recovered despite an incorrect diagnosis. More tokens didn’t reliably help. The authors recommend an explicit recovery contract: define what behavior must be restored, independently of the proposed patch. Limitations: - Size: 55 tasks, one run per task and configuration, so each task moves a score about 2 points. Some categories have only 3–5 tasks. - Task type: software defects in Kubernetes operators, not infrastructure outages or other platforms. - Exclusions: nondeterministic races and silent state drift. It turns out “make the error go away” doesn't quite cut it. arxiv.org/abs/2611.06099

Post attachment
Gwen (Chen) Shapira avatar
⟳ Leo Gorodinski retweeted
Gwen (Chen) Shapira@gwenshap·Nov 13

Question to people who use AI to write, review and merge large chunks of their code base: My fear is that this leads to a system too complex to reason about, and therefore we'll end up with production issues that we cannot fix. Can you share how are things looking for you? did complexity explode, or is it well contained? how's troubleshooting production going?

Leo Gorodinski avatar
Leo Gorodinski@eulerfx·Nov 12

I’ve been practicing DDD for many years, so this is good to hear. But I think we need to say more about what DDD means in the agentic era. Beyond telling agents to “do DDD,” we can use them to: - Formalize the model - Benchmark what works Formalization Today we guide agents mostly through skills and .md files. That’s a natural instinct, since agents follow prose instructions well in the small. But benchmarks call this into question at scale: adherence is inconsistent, especially for architectural rules. Formalization moves the domain’s rules out of prose and into structure that can be mechanically checked. Rigor that used to take too much manual effort is now within reach. What I’d try: - Define types, interfaces, and module structure first, and make "illegal states unrepresentable". - Route every change to aggregate state through operations that enforce its invariants. - Model aggregate lifecycles as explicit state machines. - Model cross-aggregate interactions as explicit, formal workflows and protocols, with sequencing, retries, and compensation spelled out. - Enforce bounded contexts as module boundaries. - Write property-based tests for domain invariants. - Apply formal methods where justified, such as model checking a protocol. Benchmarking Many claims about DDD’s benefits remain subjective. There’s now a growing set of published agent benchmarks for code, and building your own is cheaper. That evidence matters more now than it ever has. Without it, we’re producing far more code, far faster, without knowing whether we’re building better systems.

One odd quirk of AI is that if you can describe your business processes in clear, consistent language... ...you've already half-built all the software you need to automate the process DDD has never been more powerful

Leo Gorodinski avatar
Leo Gorodinski@eulerfx·Nov 12

Coding agents follow about half the rules in your AGENTS.md every single time, and ignore more than a third of them completely. A Nov 2026 study "Do They Do as They Are Told?" checked whether agents obey the rules in real AGENTS.md (and similar): 621 context files from 268 GitHub repos, against ~6.5k commits by Copilot, Claude Code and Codex. Findings - At most 44% of the rules people write can be checked against the code at all. The rest are vague advice or process steps that leave no trace. - Compliance is bimodal: of the ~2k rules they could test, 54% were followed in every relevant commit, 37% in none, and only 8% fell in between. Be careful with spot-checking! - It depends on the rule: coding, build and documentation rules average 60–70%. Testing rules average 49%. The median architecture or process rule is never followed (samples are small: 40 and 50 rules). Limitations - Only 14% of the ~15k rules written could be tested against a commit (so the numbers describe a small slice). - The study is observational, and the typical rule was checked against only three commits. - A manual review found 71% of the automated checks correct, and the checks are lenient, so real compliance is probably lower. The authors' advice: write one concrete, imperative, checkable rule per line, and drop qualifiers like "when appropriate." My take: architecture shouldn't be a request in a .md file. It should be structure in the code that the compiler and tests enforce.

Post attachment
Leo Gorodinski avatar
Leo Gorodinski@eulerfx·Nov 11

In zero-shot testing, every model failed all 30 from-scratch authoring tasks in AgentScript-Eval, a new benchmark for Salesforce’s DSL for building agents. No model compiled even half its programs. With a compile-and-edit loop and better skills, the best setup compiled 98% of programs across all 100 tasks, yet only 37% passed all behavioral checks. From-scratch authoring stayed under 17%. Failure modes - Fields that read like English but alter behavior. - Action outputs bound to variables by position, which can be swapped without breaking compilation. Compile errors get repaired because the compiler reports them. Behavioral errors persist because the rules they break live in prose, not in the compiler. My take: we need languages where behavioral requirements are explicit and mechanically checkable, rather than relying on agents to implement them correctly.

Post attachment