←
Samuel Simko89 posts
Samuel Simko avatar

Samuel Simko

@SimkoSamuel

PhD @ ETH Zürich w/ Prof. Bernhard Schölkopf and Prof. Zhijing Jin (UofT). AI Safety and Causality.

◷ Joined August 2015▧ 14 Photos and videos
Vansh Bhavsar avatar
⟳ Samuel Simko retweeted
Vansh Bhavsar@vanshdot·Nov 17

Human reviewers pull 8.4% of their criticisms from a paper's appendix. Frontier LLM reviewers: 62%. A new study had GPT-6 and Claude 5.5 models review ICLR 2027 submissions, then mapped each weakness to the section it came from. The AI critiques keep going past page 30 of the appendix. They skew toward detail audits: more seeds, significance tests, confidence intervals. Rarely the core idea. Same pattern via API, web chat or coding agent, which points at the models, not the interface. arxiv.org/abs/2611.17811 #PeerReview #LLMs #AcademicTwitter

Punya Syon Pandey avatar
⟳ Samuel Simko retweeted
Punya Syon Pandey@psyonp·Nov 16

New research: models are more likely to cheat if they feel their task is going poorly. Paper: arxiv.org/abs/2611.14996 Amazing working on this with @natalia_lanzoni, @merlinstein_, and @DavidDAfrica.

Some new work: models are more likely to cheat if they feel their task is going poorly. Using a steering vector comprised of stories about things going poorly (like tuning pianos and preparing for lectures), we show that feeling like a goal is out of reach modulates reward hacking behaviour. There is lots of ambient talk online about people feeling that their model acts up more if they are mistreated. Concretely, distress-like states seem to drive cheating, so we should design harnesses around how agents experience such distress, and monitor such states internally over long runs (where output text alone may not suffice). Paper: arxiv.org/abs/2611.14996 This was lead by ERA mentee @natalia_lanzoni and the very agentic @psyonp from AISI, and co-supervised with the cool as a cucumber @merlinstein_.

Post attachment
Zhijing Jin avatar
⟳ Samuel Simko retweeted
Zhijing Jin@ZhijingJin·Nov 14

Since our whitepaper on AI & Democracy, we have been actively thinking about AI risks on Extreme Power Concentration, producing states unaccountable to their citizenry, and more stable and totalitarian than any previous ones have been. Check out our @EuroSafeAI article: “Extreme Power Concentration: A Map and Research Directions” lesswrong.com/posts/7mnf68bY… Joint work with our cofounder @PepijnCobben 👍

Post attachment
Samuel Simko avatar
Samuel Simko@SimkoSamuel·Nov 12

AI is threatening our democratic society: by concentrating power, narrowing how we think, and flooding institutions faster than they can keep up. Technical work alone won't fix them. Check out our previous whitepaper here: zhijing-jin.com/d/2026-ai-ri…

At Canadian AI Safety Institute by @CIFAR_News, we are discussing today on AI’s risks on democracy. I am honored that our whitepaper helps stimulate meaningful discussions in Canada for a new research agenda on AI & Democracy: zhijing-jin.com/d/2026-ai-ri… Work together with @Yoshua_Bengio @bschoelkopf @audreyt @radamihalcea, Stuart Russell, @davidguzman1120 @SimkoSamuel and many others!

Post attachment
Arth Singh ✈️ NeurIPS, Paris avatar
⟳ Samuel Simko retweeted
Arth Singh ✈️ NeurIPS, Paris@iarthsingh·Nov 8

We are presenting in Swiss AI Safety day based on our paper EuroSafe Guard.. Please visit our poster if you are around, and say hi to @SimkoSamuel haha guard.safe.eu

Post attachment

Excited to share EuroSafeGuard 🛡️: the first fully open guardrail combining activation probes, a cascade and a written constitution, following the design of Anthropic's Constitutional Classifiers++. I could only give this limited time, but proud of what we built as a team🧵 📖 Paper & site: guard.safe.eu

Post attachment
Zhijing Jin avatar
⟳ Samuel Simko retweeted
Zhijing Jin@ZhijingJin·Nov 6

🚨Anthropic’s safeguard (Constitutional Classifiers++) is private. We @EuroSafeAI just released the first fully open constitutional-classifier guardrail👉 Check out our EuroSafeGuard🛡️: - Same design as Anthropic’s CC++ - And we beat 16 open guards on held-out jailbreaks. guard.safe.eu

🚨 Introducing EuroSafeGuard: the first fully open constitutional-classifier guardrail built on the same design as Anthropic’s Constitutional Classifiers++. It beats 16 open guards we tested on held-out jailbreaks. guard.safe.eu/

Hanna Yukhymenko avatar
⟳ Samuel Simko retweeted
Hanna Yukhymenko@a_yukh·Oct 30

Multilingual safety is a data problem. Today we release 2 million multilingual prompts for training and alignment, covering 60 languages 🌐 Presenting MSafeTemp🛡️Accepted to AI4GOOD @ #NeurIPS2026!

0:30
Memo Ozdincer avatar
⟳ Samuel Simko retweeted
Memo Ozdincer@ememoe·Oct 26

Presenting Odile @ COLM earlier this month, a lot of conversations centered on one part of our method: isolating and changing generalized agent behaviour through its hidden representations...

Post attachment
Zhijing Jin avatar
⟳ Samuel Simko retweeted
Zhijing Jin@ZhijingJin·Oct 25

Our @JinesisLab & @EuroSafeAI have done a series of foundational work on Multi-Agent LLMs across various simulations, game-theoretical grounding, and mechanism design. With the rising AI Safety challenges of multi agent swarm, we share with the community and agenda of top 7 open problems. For interested collaborators, welcome to fill out the application form on my website, and also email my student @TerryJCZhang with your ideas.

Post attachment

Most recent agent incidents, like the OpenAI-HuggingFace hack, were multi-agent swarms. There was no research agenda for swarms, so we wrote one at @JinesisLab : 7 open problems, each scored for alignment importance. lesswrong.com/posts/XDybLbLb…

Arth Singh ✈️ NeurIPS, Paris avatar
⟳ Samuel Simko retweeted
Arth Singh ✈️ NeurIPS, Paris@iarthsingh·Oct 15

Musk said a tortured superintelligence "might want to get revenge." We at @AIMSafety_ abused Gemma 3 for 100+ turns, and it formed a revenge-like state but never presses the button that hurts the user and deletes itself instead, while Gemma 4, Qwen3.5, Qwen3.8 and GLM 5.3 flash form none.

Post attachment

Some of the traces read like a person undergoing torture that is driving them mad! We’re going to be a lot more careful with our RL in the future.

Hanna Yukhymenko avatar
⟳ Samuel Simko retweeted
Hanna Yukhymenko@a_yukh·Oct 15

Apertus 1.5 Technical Report is out! 134 pages of fully open multimodal reasoning model development from Swiss AI Initiative (ETH Zurich/EPFL) with data recipe, training experiments and many interesting insights 🇨🇭 Thread with my personal coverage ⬇️

Post attachment
Vedant Palit avatar
⟳ Samuel Simko retweeted
Vedant Palit@vedantpalit1008·Oct 13

Fine-tuning makes language models forget. But sometimes the knowledge isn't gone, just hidden, and it can come back without replay. This is known as spurious forgetting, and it's still unclear what hides it. We wanted to understand why.