←
Mark Vero106 posts
Mark Vero avatar

Mark Vero

@mark_veroe

PhD Student @ ETH Zürich @the_sri_lab

◷ Joined March 2012▧ 11 Photos and videos
Mark Vero avatar
Mark Vero@mark_veroe·1h

And fast-enough, the frontier in secure code generation is pushed further with Gemini 4 Neon. While Claude still tops the leaderboard on the easy and medium splits, Neon sets a new sota on hard, further opening the gap! Congratulations to @GoogleDeepMind for the release!

Post attachment
Mark Vero avatar
Mark Vero@mark_veroe·Nov 16

See the full leaderboard here: baxbench.com/autobaxbuilder/… Relied on great work by everyone @tobiasvonarx_, @nielstron, @baader_max, @Chenhao3564, and @mvechev at the @the_sri_lab!

Mark Vero avatar
Mark Vero@mark_veroe·Nov 16

While it is great to see models improving at secure code generation, there remains a lot of responsibility on the developer and code owners to ensure the security of the final code. With the increasing ease of exploiting vulnerabilities, these improvements are critical!

Mark Vero avatar
Mark Vero@mark_veroe·Nov 16

However, as we found recently (arxiv.org/abs/2611.11455), agents aren't perfectly reliable either. If the codebase is buggy, agents will introduce even more security bugs at a much higher rate.

Mark Vero avatar
Mark Vero@mark_veroe·Nov 16

Note that the benchmark was designed in the pre-agentic era, i.e., we evaluate the models without a scaffold. As such, the results are only indicative of the models' one-shot behavior. With a good harness and prompt, one could do probably much better.

Mark Vero avatar
Mark Vero@mark_veroe·Nov 16

Recent frontier models are extremely impressive at pure functional coding (~85-90% pass@1 across the board). Unfortunately, security remains a secondary consideration for some. With only Claude showing strong security across all splits and Gemini 4 Carbon topping the hard subset.

Mark Vero avatar
Mark Vero@mark_veroe·Nov 16

In AutoBaxBench, we task the models to write small web applications from scratch, given a description of the target API. We then test for functional correctness, and for security using pre-written end-to-end exploits. The latter gives a lower bound for real vulnerabilities.

Mark Vero avatar
Mark Vero@mark_veroe·Nov 16

🏅How secure is code generated by the latest frontier models? 🏅 After a long hiatus, we decided to see where the current frontier lies in secure code generation. While the models have improved massively, to much of our surprise, they still generate a lot of vulnerabilities.

Post attachment
Mark Vero avatar
Mark Vero@mark_veroe·Nov 9

Turns out if your code base is full of security vuls, agents will just introduce more issues

🦠 Coding agents propagate vulnerabilities in code bases! How comes? When a repo already contains vulnerable code, they write similar vulnerabilities into new code, even when the vulnerable code is functionally unrelated to their task and they can detect its flaw.

Post attachment
Chenhao Sun avatar
⟳ Mark Vero retweeted
Chenhao Sun@Chenhao3564·Nov 9

🦠 Coding agents propagate vulnerabilities in code bases! How comes? When a repo already contains vulnerable code, they write similar vulnerabilities into new code, even when the vulnerable code is functionally unrelated to their task and they can detect its flaw.

Post attachment
Kári Rögnvaldsson avatar
⟳ Mark Vero retweeted
Kári Rögnvaldsson@karirogg·Oct 19

Does more reasoning improve performance on LeanLean (disproportionally)? It turns out that the model capability trend is still log-scaling. We have now evaluated Sol 6.1 and Opus 5.5 across multiple reasoning efforts on leanleanbench.com!

Post attachment
Niels Mündler-Sasahara avatar
⟳ Mark Vero retweeted
Niels Mündler-Sasahara@nielstron·Sep 23

my colleagues found out our coffee machine is controllable via Bluetooth and now assessing which agent brews the best coffee

Jingxuan He avatar
⟳ Mark Vero retweeted
Jingxuan He@jingxuan_he·Aug 24

Life update: I’m excited to share that I will join NTU in Singapore (@NTUsg) as a Nanyang Assistant Professor in Fall 2027! Until then, I’ll spend my time at NVIDIA Research based in the SF bay area. I’m recruiting students and postdocs. Learn more at jxhe.info

Post attachment
mimic avatar
⟳ Mark Vero retweeted
mimic@mimicrobotics·Jul 16

From day one, mimic has been focused on a single goal: general-purpose dexterous manipulation. Today we're proud to announce the mimic hand M1 and the mimic wearable U1. We believe the only way to solve dexterous manipulation at scale is by going full-stack at the frontier of physical AI, building every layer ourselves around one fixed point, the human hand. The M1 is a highly backdrivable, tendon-driven hand that covers the full range of human capability, from heavy payloads to fine manipulation.

1:43
Mark Vero avatar
Mark Vero@mark_veroe·Jul 10

If you are at ICML check out our oral presentation on Saturday at 11AM the DEMO workshop about adapting reasoning models with cheap and simple techniques.

(DEMO @ #ICML26 Oral) RLMs are undoubtly powerful. But how can we train them further? IFT on golden answers can destroy reasoning capabilities. We show: With our careful merging technique, reasoning can be restored entirely, while keeping gains on the target domain! 🧵

Post attachment