←
Martin Vechev234 posts
Martin Vechev avatar

Martin Vechev

@mvechev

Professor of Computer Science, ETH Zurich. Founder of INSAIT (insait.ai). Works on Safe/Secure AI, LLMs, Quantum. Co-founder of 6 Deep-Tech start-ups.

◷ Joined June 2012▧ 20 Photos and videos
Martin Vechev avatar
Martin Vechev@mvechev·Nov 13

Congratulations to my PhD student @j_dekoninck on receiving the 2026 Google PhD Fellowship in Natural Language Processing! By now Jasper has established himself as a leading researcher in the area of AI for mathematics. He has been instrumental in the success of MathArena, which has become one of the most widely used platforms for evaluating frontier AI models on mathematical reasoning, used by Google, Anthropic, OpenAI, and others in their public releases. Beyond MathArena, Jasper has made major contributions to reliable AI evaluation, including novel methods for detecting data contamination, as well as research on reducing the cost of AI agent systems, and other areas. Before starting his PhD in our lab, he received the Willi Studer Prize for the best Master's graduate in Data Science at ETH Zürich. Congratulations to Jasper and to all the other fellows!

Post attachment
Martin Vechev avatar
Martin Vechev@mvechev·Nov 10

🚨 We argue that a widely held assumption in AI is wrong: that fully open-source LLMs are inherently more secure and trustworthy than less transparent models! Our new position paper challenges this assumption and shows why greater transparency does not guarantee trustworthiness and can even introduce additional security risks. At the heart of the problem is a misleading analogy: many assume that the security and trustworthiness benefits of open-source software naturally extend to LLMs with open weights, training data, and pipelines. They don't. Our main points are: 🔍 Transparency does not guarantee trust: unlike traditional software, even fully transparent LLMs cannot be reliably reproduced or verified. Publishing weights, training data, and pipelines does not prove that the model was trained as claimed. ⚔️ Openness can favour attackers: access to model weights, data, and training pipelines can make attacks more effective and safety safeguards easier to circumvent. 🩹 Vulnerabilities cannot simply be patched: unlike software, LLMs are extremely difficult to fix reliably. 🔓 Open training data creates additional risks: sensitive information disclosure, privacy concerns, and legal liabilities can become more pronounced when training datasets are publicly available. 🌍 Openness is not the same as sovereignty: access to model weights and training data does not provide the infrastructure, compute, and expertise required for genuine technological independence. ⚖️ Implications for the EU AI Act: the Act exempts certain open-source AI models from some regulatory obligations. But full openness does not justify reduced scrutiny when it does not necessarily improve security or trustworthiness. To be clear, we are not arguing that fully open-source models are inherently worse than open-weight or closed models. Openness offers important benefits for innovation, research, and customization. Our argument is that more openness does not automatically mean more security or trustworthiness. In some cases, it may mean the opposite. Great work by Sheila Zingg, who led this work, Prof. @florian_tramer, and Prof. Srdjan Čapkun. Paper link: arxiv.org/abs/2610.35199

Post attachment
Kári Rögnvaldsson avatar
⟳ Martin Vechev retweeted
Kári Rögnvaldsson@karirogg·Nov 6

We just released LeanLean October as part of MathArena! We evaluate compression capabilities of models with 32 repos from October, including some small improvements upon our original benchmark. GPT Sol 6.1 now tops the benchmark, open models lag behind.

Post attachment
INSAIT Institute avatar
⟳ Martin Vechev retweeted
INSAIT Institute@INSAITinstitute·Oct 22

🚀 We are excited to announce that Google is awarding $1 million to INSAIT, further strengthening our longstanding partnership and shared commitment to advancing cutting-edge AI research!

Post attachment
Kári Rögnvaldsson avatar
⟳ Martin Vechev retweeted
Kári Rögnvaldsson@karirogg·Oct 6

Website: leanleanbench.com HuggingFace: huggingface.co/datasets/eth-… GitHub: github.com/eth-sri/lean-lean Paper: sri.inf.ethz.ch/publications… Work in collaboration with my great teammates at the SRI Lab: @nielstron, @j_dekoninck and @mvechev!

Jasper Dekoninck avatar
⟳ Martin Vechev retweeted
Jasper Dekoninck@j_dekoninck·Sep 16

We are releasing the latest version of BrokenArXiv and ArXivMath! These benchmarks now focus on conjectures that were refuted in the last month on ArXiv, and models are executed within a harness instead of directly via API. Performance remains impressive, with GPT-6 Astra on top

Post attachment
Mark Vero avatar
⟳ Martin Vechev retweeted
Mark Vero@mark_veroe·May 29

1/ LLMs are increasingly being used to power high-interaction honeypots while maintaining a low security risk. But how good are they really? To answer this question, we introduce Honeyval, the first comprehensive eval framework for LLM-powered honeypots.

Post attachment
Ivo Petrov avatar
⟳ Martin Vechev retweeted
Ivo Petrov@IvoPetrov01·May 21

LLMs have become capable of proving complex mathematics. However, the proofs they produce vary significantly in how clear, motivated, and insightful they are. To measure these differences, we introduce ProofRank, the first benchmark to scalably evaluate aspects of proof quality.

Post attachment
Jasper Dekoninck avatar
⟳ Martin Vechev retweeted
Jasper Dekoninck@j_dekoninck·Mar 13

How often do LLMs claim to prove false mathematical statements? In our latest benchmark, BrokenArXiv, we find they do so very often. The best model, GPT-5.4, only rejects 40% of incorrect statements obtained by perturbing recent ArXiv papers, and other models do much worse.

Post attachment
Martin Vechev avatar
Martin Vechev@mvechev·Feb 24

some of our work on (not) using CLAUDE.md/AGENTS.md has been widely profiled recently.

You should delete your CLAUDE․md/AGENTS․md file. I have a study to prove it.

29:15
Niels Mündler-Sasahara avatar
⟳ Martin Vechev retweeted
Niels Mündler-Sasahara@nielstron·Feb 17

Are AGENTS actually useful for coding agents? In our latest preprint, we analyze the effect of context files on coding model performance. Short TLDR: It depends. Manually-written files help, while LLM-generated files (e.g. by the agent) don't. More details in the thread 🧵

Post attachment
Jasper Dekoninck avatar
⟳ Martin Vechev retweeted
Jasper Dekoninck@j_dekoninck·28 Nov 2025

How far can we push LLM performance on Project Euler (PE), a set of challenging mathematical programming problems? In a new blog, we explore this by designing specialized agents. With the help of the PE community, we also perform a qualitative analysis of model performance. 🧵

Post attachment
Martin Vechev avatar
Martin Vechev@mvechev·18 Nov 2025

Grateful that Google gave us access to Gemini 3 to evaluate its performance on MathArena (math performance) before the release and these results are now included in their release and model card! Gemini 3 is topping all categories on MathArena, great work!

This tweet is unavailable

INSAIT Institute avatar
⟳ Martin Vechev retweeted
INSAIT Institute@INSAITinstitute·21 Oct 2025

🎥 Can AI video models truly understand physics? The newly released Physics-IQ benchmark, developed by INSAIT and Google DeepMind and led by Saman Motamed, PhD student at INSAIT, has sparked major discussion across the AI community following its presentation at #ICCV2025. 🔬 The work marks a significant step forward in understanding the physical reasoning limits of today’s generative video models - paving the way for future AI systems that not only generate realistic videos but also reason about the physical world with accuracy and depth. 📊 Physics-IQ provides a comprehensive benchmark of 396 real-world videos, covering diverse physical scenarios - from fluid dynamics to solid mechanics, challenging AI models to predict future frames and interactions beyond surface-level visual cues. 🤔 The findings were eye-opening: even state-of-the-art models like #Sora, #Runway, and #VideoPoet create visually stunning clips but fail to capture true physical dynamics, revealing the gap between perception and understanding. 🚀 The project has been met with great interest from the research community, highlighting the importance of integrating experiential and interactive learning into next-generation video models. 📂 Explore the open-source dataset, evaluation code, and results - links in comments #GenerativeAI #VideoAI #AIResearch #PhysicsInAI #PhysicalReasoning #AIUnderstanding #AIBenchmark #OpenSourceAI #FutureOfAI #AIInnovation #INSAIT

Post attachment
INSAIT Institute avatar
⟳ Martin Vechev retweeted
INSAIT Institute@INSAITinstitute·23 Oct 2025

🔥 We’re releasing SPEAR-1 (spear.insait.ai) - a new robotic AI foundation model that achieves state-of-the-art performance with 20× less robotic data 🧠 Why it matters:SPEAR-1 is like the ChatGPT for robots - a single model that can perform many tasks, on any robot, in any environment. 💡 What’s new: unlike others, SPEAR-1 learns from both robotic and non-robotic 3D data, breaking the data bottleneck that slows robotic AI. 🤖 Open-weight, general-purpose, and multilingual for robots - a major step toward scalable robot learning. #Robotics #FoundationModels #3DPerception #Manipulation #INSAIT #Europe #DataEfficiency

Post attachment
INSAIT Institute avatar
⟳ Martin Vechev retweeted
INSAIT Institute@INSAITinstitute·25 Oct 2025

🚀 Big news for INSAIT, Bulgaria & Europe! @WIRED (read by 30M+ people/month) just profiled SPEAR-1 — INSAIT’s new foundation robotic model, the first released by Europe! 🤖 As WIRED notes, SPEAR-1 matches leading global models trained on many times more data - a huge leap toward ChatGPT-like AI for robotics! 👏 Congrats to all involved: Nikolay Nikolov, @giualbanese1, @DSombit, Jan-Nico Zaech, Danda Pani Paudel, Luc Van Gool & Alex Yanev. #AI #Robotics #Europe #INSAIT

Post attachment