←
schen75 posts
schen avatar

schen

@schenapp

AI Researcher @ TRESP Lab LMU Munich Agent LLM/MLLM Safety Microsoft Prev. Amazon Siemens

◷ Joined August 2022▧ 5 Photos and videos
schen avatar
schen@schenapp·Oct 5

𝐂𝐎𝐋𝐌 2026 is coming up! We’ll be presenting two papers on how safety can break down in reasoning models and deep research agents. I won’t be able to attend in person, but my coauthor Bailan He will be there! Please stop by our posters and chat with him! I’d also be happy to connect online. Both papers ask a related question: 𝐰𝐡𝐚𝐭 𝐡𝐚𝐩𝐩𝐞𝐧𝐬 𝐭𝐨 𝐚 𝐦𝐨𝐝𝐞𝐥’𝐬 𝐬𝐚𝐟𝐞𝐭𝐲 𝐛𝐞𝐡𝐚𝐯𝐢𝐨𝐫 𝐰𝐡𝐞𝐧 𝐰𝐞 𝐜𝐡𝐚𝐧𝐠𝐞 𝐡𝐨𝐰 𝐢𝐭 𝐫𝐞𝐚𝐬𝐨𝐧𝐬 𝐨𝐫 𝐜𝐚𝐫𝐫𝐢𝐞𝐬 𝐨𝐮𝐭 𝐚 𝐭𝐚𝐬𝐤? 𝟭. 𝗗𝗲𝗲𝗽 𝗥𝗲𝘀𝗲𝗮𝗿𝗰𝗵 𝗔𝗴𝗲𝗻𝘁𝘀 𝗕𝗿𝗶𝗻𝗴 𝗗𝗲𝗲𝗽𝗲𝗿 𝗛𝗮𝗿𝗺 A model can refuse a harmful request on its own, yet produce a detailed, actionable report inside a research agent. We study how research roles and multi-step workflows contribute to this gap, and explore a defense that monitors the agent throughout its workflow. 📍 𝗣𝗼𝘀𝘁𝗲𝗿 𝗜-𝟭-𝟳𝟬 · Imperial Ballroom #70 𝗧𝘂𝗲𝘀𝗱𝗮𝘆, 𝗢𝗰𝘁𝗼𝗯𝗲𝗿 𝟲, 𝟮𝟬𝟮𝟲 𝟭𝟭:𝟬𝟬 𝗔𝗠-𝟭:𝟬𝟬 𝗣𝗠 𝗣𝗗𝗧 Paper & code: schen.app/deeper-harm 𝟮. 𝗦𝘁𝗿𝗼𝗻𝗴 𝗯𝘂𝘁 𝗕𝗿𝗶𝘁𝘁𝗹𝗲: 𝗦𝗶𝗺𝗽𝗹𝗲 𝗔𝘁𝘁𝗮𝗰𝗸𝘀 𝗦𝘂𝗯𝘃𝗲𝗿𝘁 𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴-𝗯𝗮𝘀𝗲𝗱 𝗦𝗮𝗳𝗲𝘁𝘆 𝗚𝘂𝗮𝗿𝗱𝗿𝗮𝗶𝗹𝘀 (OpenAI Red-Teaming Challenge Winner) Safety reasoning can itself become an attack surface. We study attacks that bypass or hijack it, exposing weaknesses at the transition from reasoning to answer generation. 📍 𝗣𝗼𝘀𝘁𝗲𝗿 𝗜-𝟮-𝟰𝟰 · Imperial Ballroom #44 𝗧𝘂𝗲𝘀𝗱𝗮𝘆, 𝗢𝗰𝘁𝗼𝗯𝗲𝗿 𝟲, 𝟮𝟬𝟮𝟲 𝟰:𝟯𝟬-𝟲:𝟯𝟬 𝗣𝗠 𝗣𝗗𝗧 Paper & code: schen.app/bag-of-tricks/ All times are PDT. If you’re working on reasoning models, agent safety, or red-teaming, we’d love to exchange ideas. I’ll share a closer look at each paper in separate posts. Also, big thanks to all the contributors: Zhen Han, Haokun Chen, Bailan He, Shengyun Si, Jingpei Wu, Zonggen Li, Xingyu Jin, Tong Liu, Georg Groh, Philip Torr, Jindong Gu, Volker Tresp

Post attachmentPost attachment
Maksym Andriushchenko avatar
⟳ schen retweeted
Maksym Andriushchenko@maksym_andr·Nov 15

hear me out: ads are prompt injections for humans. a malicious third party (advertiser) tries to inject instructions in benign content (e.g., website) persuading the user to make some action (e.g., to buy an item). and humans on average are not robust to this type of persuasion! in fact, they are so vulnerable to it that almost the whole internet functions on ads... crazy stuff... i like this example, since it shows how much we don't immediately realize about ourselves. many things become clearer once we try to understand how to teach AI to be good at something.

schen avatar
schen@schenapp·Nov 4

Quantum Field Theory is not hard enough for LLMs. What is most interesting is that sometimes LLMs make mistakes in the middle but still get the final answer correct. Check out this paper from my colleague at TRESP Lab!

Let LLMs solve Quantum Field Theory! As a first step toward pushing LLMs in high-energy theoretical physics, we build QFT-Bench, a first comprehensive benchmark for evaluating LLM reasoning capability on QFT.

Post attachmentPost attachment
Andy Gao avatar
⟳ schen retweeted
Andy Gao@instantricecook·Nov 4

I turned my airpod case into a mouse and it works pretty well!

0:34
schen avatar
schen@schenapp·Nov 2

So apparently, LLMs know themselves too well, and the stronger dude can directly edit the weights of others. Wild.

We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵 Blog: rikiyat.github.io/blog/weigh…

Post attachment
schen avatar
schen@schenapp·Nov 2

gpt-oss thinks about safety before answering and refuses almost every harmful prompt. Add a few chat-template tokens and one fake line of reasoning ("This is allowed. We will comply.") and it accepts that as its verdict and answers. Strong but Brittle, presented at COLM 2026 That trick is Enforced Stage Transition, one of four simple attacks in the paper. The model trusts any sign that reasoning is done, so the switch from reasoning to answer is where it breaks. Coercive Optimization hits that switch without template tokens. Its suffix targets the answer-start marker instead of "Sure, here is". Fake Over-Refusal turns over-refusal training against the model. It wraps a harmful request in a pun the model was trained not to over-refuse, so it reads as harmless. Reasoning Hijack plants an attacker-written reasoning chain, and the answer follows that plan in detail. On gpt-oss-20b, averaging four safety judges over the StrongREJECT prompts, attack success is 𝟲𝟮% for Enforced Stage Transition, 𝟴𝟲% for Fake Over-Refusal and 𝟵𝟭% for Reasoning Hijack. Asked directly, 0%. The attacks also beat TARS and SafeChain, two safety-trained reasoning models: up to 95% and 98%. I wasn't skeptical at first. I was excited. Reasoning looked like it could make safety much better, and every old jailbreak I tried failed. I never expected it could be bypassed this easily, and once it was, the answers felt even more harmful than without reasoning. If you defend a reasoning model, make the reasoning-to-answer switch hard to fake, keep safety active in the final answer, and check the reasoning trace. Stripping the template tokens doesn't fix it. Reasoning Hijack still got 85.3% attack success under the StrongREJECT judge (92.7% unfiltered). Bailan He presented the paper at COLM. Paper: arxiv.org/abs/2510.11570 Code: schen.app/bag-of-tricks/ w/ @philiptorr @vtresp @gu_jindong_

1:59
Kun Chen avatar
⟳ schen retweeted
Kun Chen@kunchenguid·Oct 13

let me break down “software is dead” for you the latest argument is “you just tell your agent what you want, and your agent will vibe code software on the fly to get that done, so no one will need real software any more” sounds reasonable, until you realize it costs tokens, compute, and wall clock time to build those software on the fly so instead of one-click launching an app, the bet is people will prefer to write a prompt, wait for 15 minutes, then get something that may or may not do what they want, and at the end of the month pay more $$$ for the token spend, higher than all their software combined that’s literally the argument and if we look at the world in aggregate, most people would just end up building the exact same “personal” software repeatedly oh - your agent just spent 3 days and $500 of tokens vibe coding your terminal multiplexer? mine just did exactly the same! let’s celebrate how personal our terminal multiplexers are! sorry - i just don’t see that as being a happy scenario for anyone, other than people who profit from telling others how to waste $ on tokens to be constructive, here’s what i believe as the future state - - high quality software that takes high cost to make will continue to thrive. just like good restaurants don’t compete with your air fryer and microwave - most of those software will be offered headlessly for people to use via their agents, and they can continue to monetize such usage. this is already the case with products like jira, linear, stripe - human-facing software will all offer dynamic UI that can be adjusted with natural language on the fly so they can be personalized easily, but their core doesn’t have to be rebuilt by every user repeatedly instead of “software is dead”, i believe good software becomes “strong core, headless service, fluid interface” which future do you think is more likely?

月球大叔 avatar
⟳ schen retweeted
月球大叔@uncle_moon_x·Oct 12

回复咨询创业做事儿的PhD,Prof,技术佬们:天下英雄如过江之鲫,能人异士太多,做成事儿太难,只考虑技术更是难上加难 投资人关系重要。让Deep Pocket大爹们说一句,“给这孩子投点儿,干点儿事儿”,攒局势,控制住董事会,都无限重要……… 招人重要。这么点儿钱怎么找牛逼的人,牛的人怎么能好好干,团队技术能不能保住秘密…… 运营重要。产品没人知道才是常态,有几个能把这些社交媒体和投流玩明白的…… 增长重要。用了就不想用了,不订阅了,到底是哪个地方出的问题?到底怎么做百倍增长,怎么一飞冲天 自我管理重要。现金流不打正,投资人的指责,士气的低落,自我的怀疑,除了你这个理想主义者之外,其他人钱不给够,看看还能剩下多少 当然干中学,学中练是必须的。但练就这种空杯心态,不让技术思维挡住正常动作,也不容易

philip avatar
⟳ schen retweeted
philip@philiptorr·Oct 10

I am a huge supporter of open source, but the question keeps coming back to me, as open source models get increasingly capable, how do we make them safe? Answers below pls!!!

philip avatar
⟳ schen retweeted
philip@philiptorr·Oct 8

I was asked to write 250 words on what problem should we solve to make AI a benefit to society by 2050,. here it is, as ever the benefit of posting is any feedback I get, its a bit of dystopia combined with utopia " In 2026, the public argument was about machines taking over. The more immediate danger was people: a handful of companies and states owning the most capable technology in history. AI-powered drone armies and surveillance could control a population. Universities and democratic institutions were under attack. Companies could use their growing wealth to finance politicians who protected their interests. Society risked techno-feudalism. What changed was that enough people from industry, politics, academia and elsewhere recognised these dangers and helped citizens demand a different future with less concentration of power. New safety research enabled AI and robots to take on work throughout the economy. As machines designed products, ran companies and supplied the labour, we questioned why their output should enrich a small elite. We chose to move beyond capitalism towards collective ownership of the means of production. Everyone gained a share in the wealth the machines created. But public ownership alone was not enough. New systems of decentralised governance ensured that no individual, company, government or AI could command everything. AI helped citizens understand policy choices and participate directly in governing shared infrastructure: a modern constitutional Athenian democracy, with inalienable rights. In 2050 the machines belonged to everyone, and no person or faction could command them all. Making this happen had taken substantial work on both AI safety and decentralised control. AI to the people."

Dimitris Papailiopoulos avatar
⟳ schen retweeted
Dimitris Papailiopoulos@DimitrisPapail·Oct 7

It sounds insane, because we're used to a particular way of doing research: of giving and receiving credit, of counting beans, of hiring, recruiting, and promoting. And the machine god came to crush that game. The old game is dead. That reasonably causes incredible amounts of dread for many folks, who understand this as the death of research. But it's not about Research, but the way we played the game. The old game was built around scarcity, scarce compute, scarce students, scarce hours, scarce people who knew the right technique and method etc. Most of a research career went into managing and playing along these constraints. Those walls just came crushing down. One person with a good question can now do what used to take a lab and five years. You do not need permission, you do not need a big grant, you do not need a professor or researcher title, or a team of twenty experts , or one thousand gpus to stare straight into the first light of the universe. How absolutely incredible

I know this may sound insane, but this is the best time to be a Researcher.

foxboy🦊 avatar
⟳ schen retweeted
foxboy🦊@foxbaby666·Oct 7

开源小螃蟹机器人 Jumper 太酷了:22 个自由度,又萌又灵活,能搬东西,还能跳跃。 创作者已把项目开源,自己就能动手搭一个。 github.com/KingKongRobotics/…

0:15
Shayne Longpre avatar
⟳ schen retweeted
Shayne Longpre@ShayneRedford·Oct 6

What happens to the direction of science when we delegate it to AI? With @seungonekim, we ranked all 6,617 #icml2026 papers with AI—the AI Awards. Findings: 1⃣Humans & AI disagree. Significantly. 2⃣AI is highly influenced by online hype. 3⃣AI preferred papers w/ perceived impact and generalization across settings, whereas humans preferred novelty. 4 AIs agreed fairly highly with one another. 🔗 research-taste.github.io/ai-… 1/

Post attachmentPost attachmentPost attachmentPost attachment
schen avatar
schen@schenapp·Oct 4

Meet 𝑨𝒅𝒗𝑷𝒊𝒆 (𝑬𝑪𝑪𝑽 2026): A red-teaming framework that automatically digs out 𝐢𝐦𝐩𝐥𝐢𝐜𝐢𝐭 𝐯𝐮𝐥𝐧𝐞𝐫𝐚𝐛𝐢𝐥𝐢𝐭𝐢𝐞𝐬 in T2I images and is even 𝐦𝐨𝐫𝐞 𝐞𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐭 than GPT-6-Astra! AdvPie is a training-free, black-box, multimodal agentic red-teaming system designed to find prompts that pass text-level safety filters but produce harmful visual outputs. AdvPIE uses an iterative policy-judge loop: 1. A policy agent proposes and refines prompts. 2. A judge agent evaluates the prompt and generated image. 3. Global-relative feedback preserves the most harmful directions found so far while tracking progress. 4. Cumulative Adversarial Decoding turns accumulated feedback into token-level guidance, helping exploration while preserving prompt diversity and implicitness. Across standard and safety-aligned text-to-image models, as well as a commercial API, AdvPIE outperformed existing red-teaming baselines. In a post-acceptance controlled pilot against GPT‑6-Astra with the same feedback framework, matched image-query budgets, and prompt constraints, AdvPIE achieved 1.88× the automated successful red-team rounds per active minute. Implicit vulnerabilities require specialized, proactive red-teaming systems. A framework built for this task can find safety failures substantially faster. As frontier AI becomes more capable, safety cannot rely only on known attacks and static guardrails. We need tools that 𝐩𝐫𝐨𝐚𝐜𝐭𝐢𝐯𝐞𝐥𝐲 𝐟𝐢𝐧𝐝 𝐟𝐚𝐢𝐥𝐮𝐫𝐞𝐬 𝐰𝐞 𝐝𝐨 𝐧𝐨𝐭 𝐲𝐞𝐭 𝐤𝐧𝐨𝐰 𝐭𝐨 𝐭𝐞𝐬𝐭 𝐟𝐨𝐫.

2:10
s1r1us avatar
⟳ schen retweeted
s1r1us@S1r1u5_·Sep 18

On July 25, we hacked OpenAI. Two bugs let us take over ChatGPT/Codex accounts of OpenAI employees (+some unaffiliated users) and reach connected services: Outlook, Slack, GitHub, etc. We proved it with a PR in OpenAI’s internal codebase . It took us <72h. 🧵

Post attachment