Research Fellow @SISLaboratory @HooverInst at @Stanford | Making AI learn desirable behavior from human data. RL + Reward/Preference Learning | Ph.D. from TUM.
New paper: We identify a new class of reward hacking caused by mitigations, which we call reward bias substitution. We prove no standard benchmark detects it, even with oracle access to the true reward. We find it active in GRPO, in SOTA reward models, and published methods.
Really enjoyed discussing with @farhatchristina what it takes to build AI that acts with us and for us, from trust and human oversight to independent evaluations and policy. Check it out!
Hoover Research Fellow @MLamparth joins Stanford MIP student @FarhatChristina for a discussion on when AI can be trusted to inform consequential decisions, and where responsibility should remain when AI agents act or advise. Watch Nerds of the Roundtable: hoover.org/events/ai-trust-a…
Little Anka was very skeptical of the whole idea of going to school. Yesterday, she defended her PhD @Stanford and is about to become a professor @Harvard 🩷
What happened with ICLR reviews last week convinced me all AI use in peer review should be centralized. We can do this w/out losing benefits of AI in review.
But we need to innovate faster, and worry a lot less about hurting authors' feelings.
statmodeling.stat.columbia.e…
Come work with us @JHUCompSci! We're hiring tenure-track faculty in all areas, with a particular focus on
- AI systems
- AI safety and security
- AI reasoning and alignment
- AI ethics, policy, and governance
apply.interfolio.com/194231
(DMs are open if you have any questions)
Fellowship applications for 2027 with the @Kennedy_School@BelferCenter are open.
Applying could be a useful way to work and scheme with yours truly, plus more importantly, a long list of world-class science and policy academics at Belfer.
belfercenter.org/fellowships
Some news:
I graduated my PhD @Berkeley_EECS, and starting Fall 2027, I'll be starting as an Assistant Professor at Stanford CS & a center fellow @StanfordHAI!
In the interim, I'm spending a year @the_IAS & Princeton!
So grateful to @beenwrekt & too many mentors to name 💓
🚨The @AISecurityInst has teamed up with @evaluatingevals to make AI evaluation results more reproducible! 🚀
Many of UK AISI’s publicly reported evaluation methods and findings will now live on Eval Cards, under the EEE Schema. More details 👇
evalevalai.com/infrastructur…
Now’s prob a good time to mention that I’ll start as an Assistant Professor with appointments at @Kennedy_School and @HarvardEngineer in 2027, working on… AI evals! My lab will also work on monitorability, incident analysis, and verification to advance technical AI governance 1/
‼️ Hiring AI Safety Researchers ‼️
I am moving to Berlin to start a new research group on AI safety and societal impacts @HPI_DE. From September this year, I will be hiring fully funded PhDs, postdocs, and research interns.
My group will study the risks and benefits that emerge as we integrate highly capable AI systems into the social and political processes our societies depend on (e.g. communication, coordination, competition). We will measure these impacts through model evals, computational experiments, human studies, and develop technical mitigations. We will do important research that matters and have fun doing it.
Ideal candidates have a background somewhere between CS/NLP and the social sciences, are excited about working on AI in the current moment, and care deeply about making a positive impact through open research. If that sounds like you, PLEASE GET IN TOUCH using the form in the post below.
I am sad to be leaving @oiioxford and @AISecurityInst but will try my very best to bring the amazing energy of these places and their people to Germany. I am also very grateful to the DFG / Emmy Noether programme for funding my group for the next 6 (!) years.
Please spread the word, and I look forward to hearing from you 🤗
The Stanford Intelligent Systems Laboratory went to @icmlconf 2026 in Seoul, Korea (and had an amazing time)! Check out all the SISL papers and workshops:
Come to the g̶r̶a̶v̶e̶y̶a̶r̶d̶HEADLINER poster session at ICML! Hall A #3212 at 5-6:45 pm with @DanielFein7
We present the most comprehensive side-by-side measurements of RM biases and show that simple first-order corrections can go surprisingly far (but only where they should).
1) One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
With @DanielFein7, Violet Xiang, @aiprof_mykel, and
@nickhaber
New ICML Paper!
🚨 Activation steering can remove many biases from Reward Models🚨
Specifically, we systematically find biases in SOTA Reward Models including length, order, confidence, sycophancy, and model-style. Some of these can be addressed *even more reliably than bias decorrelation* by finding and removing directions in last-layer activations corresponding to this bias.
Surprisingly, even small, synthetic data can be used to find bias directions that generalize to new settings for both length bias and confidence bias.
The best part is that the method can be applied post-hoc, unlike most de-biasing methods, so biases can be fixed after they're found, without re-training.
Come talk to myself and @MLamparth Thursday evening, 5pm-6:45 in Hall A #3212