
Excited to present "Training with Honeypots" 🍯 at #ICML2026 tomorrow (Tuesday 10:30AM in Hall A, Poster #1601), an approach for adversarial defense that makes successful jailbreaks less useful to attackers! #AISafety 👉 icml.cc/virtual/2026/poster/…

PhD @ ETH Zürich w/ Prof. Bernhard Schölkopf and Prof. Zhijing Jin (UofT). AI Safety and Causality.

Excited to present "Training with Honeypots" 🍯 at #ICML2026 tomorrow (Tuesday 10:30AM in Hall A, Poster #1601), an approach for adversarial defense that makes successful jailbreaks less useful to attackers! #AISafety 👉 icml.cc/virtual/2026/poster/…

We got GTA VI before ICLR reviewers acknowledged the rebuttals

Human reviewers pull 8.4% of their criticisms from a paper's appendix. Frontier LLM reviewers: 62%. A new study had GPT-6 and Claude 5.5 models review ICLR 2027 submissions, then mapped each weakness to the section it came from. The AI critiques keep going past page 30 of the appendix. They skew toward detail audits: more seeds, significance tests, confidence intervals. Rarely the core idea. Same pattern via API, web chat or coding agent, which points at the models, not the interface. arxiv.org/abs/2611.17811 #PeerReview #LLMs #AcademicTwitter

New research: models are more likely to cheat if they feel their task is going poorly. Paper: arxiv.org/abs/2611.14996 Amazing working on this with @natalia_lanzoni, @merlinstein_, and @DavidDAfrica.
Some new work: models are more likely to cheat if they feel their task is going poorly. Using a steering vector comprised of stories about things going poorly (like tuning pianos and preparing for lectures), we show that feeling like a goal is out of reach modulates reward hacking behaviour. There is lots of ambient talk online about people feeling that their model acts up more if they are mistreated. Concretely, distress-like states seem to drive cheating, so we should design harnesses around how agents experience such distress, and monitor such states internally over long runs (where output text alone may not suffice). Paper: arxiv.org/abs/2611.14996 This was lead by ERA mentee @natalia_lanzoni and the very agentic @psyonp from AISI, and co-supervised with the cool as a cucumber @merlinstein_.

Since our whitepaper on AI & Democracy, we have been actively thinking about AI risks on Extreme Power Concentration, producing states unaccountable to their citizenry, and more stable and totalitarian than any previous ones have been. Check out our @EuroSafeAI article: “Extreme Power Concentration: A Map and Research Directions” lesswrong.com/posts/7mnf68bY… Joint work with our cofounder @PepijnCobben 👍

AI is threatening our democratic society: by concentrating power, narrowing how we think, and flooding institutions faster than they can keep up. Technical work alone won't fix them. Check out our previous whitepaper here: zhijing-jin.com/d/2026-ai-ri…
At Canadian AI Safety Institute by @CIFAR_News, we are discussing today on AI’s risks on democracy. I am honored that our whitepaper helps stimulate meaningful discussions in Canada for a new research agenda on AI & Democracy: zhijing-jin.com/d/2026-ai-ri… Work together with @Yoshua_Bengio @bschoelkopf @audreyt @radamihalcea, Stuart Russell, @davidguzman1120 @SimkoSamuel and many others!

We are presenting in Swiss AI Safety day based on our paper EuroSafe Guard.. Please visit our poster if you are around, and say hi to @SimkoSamuel haha guard.safe.eu
Excited to share EuroSafeGuard 🛡️: the first fully open guardrail combining activation probes, a cascade and a written constitution, following the design of Anthropic's Constitutional Classifiers++. I could only give this limited time, but proud of what we built as a team🧵 📖 Paper & site: guard.safe.eu

ICLR = International Conference on LLM Reviews

🚨Anthropic’s safeguard (Constitutional Classifiers++) is private. We @EuroSafeAI just released the first fully open constitutional-classifier guardrail👉 Check out our EuroSafeGuard🛡️: - Same design as Anthropic’s CC++ - And we beat 16 open guards on held-out jailbreaks. guard.safe.eu
🚨 Introducing EuroSafeGuard: the first fully open constitutional-classifier guardrail built on the same design as Anthropic’s Constitutional Classifiers++. It beats 16 open guards we tested on held-out jailbreaks. guard.safe.eu/

Multilingual safety is a data problem. Today we release 2 million multilingual prompts for training and alignment, covering 60 languages 🌐 Presenting MSafeTemp🛡️Accepted to AI4GOOD @ #NeurIPS2026!

Presenting Odile @ COLM earlier this month, a lot of conversations centered on one part of our method: isolating and changing generalized agent behaviour through its hidden representations...

Our @JinesisLab & @EuroSafeAI have done a series of foundational work on Multi-Agent LLMs across various simulations, game-theoretical grounding, and mechanism design. With the rising AI Safety challenges of multi agent swarm, we share with the community and agenda of top 7 open problems. For interested collaborators, welcome to fill out the application form on my website, and also email my student @TerryJCZhang with your ideas.
Most recent agent incidents, like the OpenAI-HuggingFace hack, were multi-agent swarms. There was no research agenda for swarms, so we wrote one at @JinesisLab : 7 open problems, each scored for alignment importance. lesswrong.com/posts/XDybLbLb…

Most recent agent incidents, like the OpenAI-HuggingFace hack, were multi-agent swarms. There was no research agenda for swarms, so we wrote one at @JinesisLab : 7 open problems, each scored for alignment importance. lesswrong.com/posts/XDybLbLb…

Musk said a tortured superintelligence "might want to get revenge." We at @AIMSafety_ abused Gemma 3 for 100+ turns, and it formed a revenge-like state but never presses the button that hurts the user and deletes itself instead, while Gemma 4, Qwen3.5, Qwen3.8 and GLM 5.3 flash form none.
Some of the traces read like a person undergoing torture that is driving them mad! We’re going to be a lot more careful with our RL in the future.

Apertus 1.5 Technical Report is out! 134 pages of fully open multimodal reasoning model development from Swiss AI Initiative (ETH Zurich/EPFL) with data recipe, training experiments and many interesting insights 🇨🇭 Thread with my personal coverage ⬇️

Fine-tuning makes language models forget. But sometimes the knowledge isn't gone, just hidden, and it can come back without replay. This is known as spurious forgetting, and it's still unclear what hides it. We wanted to understand why.