←
Seldon41 posts
Seldon avatar

Seldon

@seldon_tech
◷ Joined April 2026▧ 14 Photos and videos
Seldon avatar
Seldon@seldon_tech·Aug 21

How good are agents actually at CAD? Today we release CADBench, evaluating 10 Models on 105 realistic tasks in Fusion 360 Fable 5 and Gemini 3.7 Flash are #1 with only 24.6% pass rate

0:05
Pascal Bertrand avatar
⟳ Seldon retweeted
Pascal Bertrand@PascalBtd·Nov 10

Fable 5.2 has no motion In this rollout, Fable 5.2 tried to build a Schmidt coupling and scored 0.125. It forgot to ground the joints, so nothing moves

0:29
Pascal Bertrand avatar
⟳ Seldon retweeted
Pascal Bertrand@PascalBtd·Nov 2

Can frontier models use PCB software? Yes, but it’s messy. In this rollout opus 5.5 is trying to complete the layout of a buck converter. It scored 0.462 after 150 steps, losing points mostly on the capacitor placement leaving a large output power loop and the power traces being too thin

0:31
Pascal Bertrand avatar
⟳ Seldon retweeted
Pascal Bertrand@PascalBtd·Oct 14

Continuing my series on CAD AI: Here Gemini 4 Argon is supposed to build a base for a given DAMIAO 4340 actuator. It uses coding exclusively and finishes after 263 steps with a reward of 0.459 mostly because of missing wire openings

0:53
Pascal Bertrand avatar
⟳ Seldon retweeted
Pascal Bertrand@PascalBtd·Oct 4

Starting a new series where I show the latests models' performance on different real-world tasks in Engineering The first one is GPT-6.1 trying to reconstruct the graph on the right in AutoCAD. It took 181 steps and got a reward of 0.414

0:36
Seldon avatar
Seldon@seldon_tech·Sep 16

Our Co-Founders @PascalBtd and @atoof_sh sat down on the Europe on Edge podcast and talked about the current state of multimodal AI and the company's mission. If this mission resonates with you, have a look at our open job listings (links are below).

Seldon avatar
Seldon@seldon_tech·Sep 8

Muse Spark 1.3 improves despite pass rate staying at 0%: • Mean verifier score: 13.44 → 20.66 (+54%) • Average runtime: 60.8 → 23.8 minutes (61% shorter) • Valid tool calls: 76.8% → 93.0% Better intermediate results and faster runs, but full task completion remains unsolved.

Post attachment
Seldon avatar
Seldon@seldon_tech·Sep 8

Gemini 3.8 Flash’s gains go beyond pass rate: • Mean verifier score: 50.15 → 54.71 (+4.56) • Average runtime: 14.0 → 10.9 minutes (22% shorter) • Average turns: ~155 for both Higher scores and more completed tasks, with shorter runs at roughly the same number of steps.

0:22
Seldon avatar
Seldon@seldon_tech·Sep 8

New SOTA on CADBench: Gemini 3.8 Flash. 30% pass rate on 105 expert-authored CAD tasks in Autodesk Fusion, up from 24.6% for Gemini 3.7 Flash. Muse Spark 1.3 also improves its mean verifier score by 54% over 1.2, though full task completion remains a challenge.

Post attachment
Pete Soderling avatar
⟳ Seldon retweeted
Pete Soderling@petesoder·Aug 24

Can computer-use agents use complicated graphical software? You’re watching GPT-5.6 Sol operate Autodesk Fusion for 250 turns as it attempts a real mechanical-design task. The run is part of a benchmark aimed at the question "how good are agents at CAD?" @seldon_tech's CADBench contains 105 mechanical-design tasks tested across 10 frontier models, and the results show how early reliable, autonomous CAD work remains: More than two-thirds of the tasks are still unsolved. That gap matters as agents make their way into the professional tools used to design the physical world. Getting around Autodesk Fusion is one thing, but producing a correct, editable model an engineer can pick up and use is much harder. CADBench gives us a way to see and measure that frontier. Great work from the Seldon team! Explore CADBench: seldon.global/blog/cadbench

0:34
Seldon avatar
Seldon@seldon_tech·Aug 21

We're hosting another event in San Francisco coming monday, this time a research talk with @samsja19 as guest speaker Drop by if you're in town! Sign up link is in the comments

Seldon avatar
Seldon@seldon_tech·Aug 21

Gemini 3.7 Flash delivers performance comparable to Fable 5 at a fraction of the cost We see that many models finish the tasks prematurely, resulting in lower scores and costs

Post attachment