New from the lab: @LorenaYannnnn finds that reward misspecification during RL causes unmonitorable chain-of-thought. Models talk about the goals in their prompt, even as they learn throughout RL to optimize the misspecified reward. We also propose an intuitive fix; see below.
Linearity comes up often in interp thinking. It's fascinating that models build simple representations of some aspects of high-level features/concepts. But a better intuition for interp thinking is that models' interpretations of activation space are radically nonlinear.
You can also steer for task 1 with v1 and task2 with v2, but then (v1+v2) steers for some unrelated task3. All this by just searching via gradient descent.
Together, these break intensity-as-scaling and composition-as-addition.
Here's the blog post: cs.columbia.edu/~johnhew/do-…
A single direction in LLM activation space, applied as a steering vector with four different norms, can cause the language model to perform a different behavior at each norm.
This and more in a new blog post, Do Not Rely on the Linear Representation Hypothesis. (link below.)
Nice paper -- they improve monitorability of reward hacking with a method that doesn't assume prior knowledge of the hacks. This is a pretty common shortcoming of other methods (e.g. inoculation prompting)
Super excited to share our new paper: Reward misspecification degrades chain-of-thought (CoT) monitorability!
During RL training, when the model is prompted with one task intent (e.g., response ≤ 900 words) but rewarded for a different or even contradictory behavior (e.g., response > 900 words), the CoT keeps reasoning about the task intent while the output violates it.
We also propose a simple mitigation method that requires no prior knowledge of the true reward or the reward hack behavior. 🧵
New blog: separating forms from estimators, or, why steer by difference of means?
Interp often conflates the form (steering, neuron analysis) with the estimator (diff, SGD on log-loss.) When you can learn by gradient descent, you usually just should.
cs.columbia.edu/~johnhew/for…
My favorite insight from this work is that monitorability problems are just way easier if you make use of the trajectories and rewards generated during RL. They have so much to tell you about how your model has learned to (mis)behave. Our reward description training does this.
Super excited to share our new paper: Reward misspecification degrades chain-of-thought (CoT) monitorability!
During RL training, when the model is prompted with one task intent (e.g., response ≤ 900 words) but rewarded for a different or even contradictory behavior (e.g., response > 900 words), the CoT keeps reasoning about the task intent while the output violates it.
We also propose a simple mitigation method that requires no prior knowledge of the true reward or the reward hack behavior. 🧵
We also release our datasets and all of our misaligned and control models as model organisms for understanding unmonitorability due to reward misspecification. Can white-box monitors help?
Models: huggingface.co/collections/h…
Opinion: frontier labs are worried about what their models are doing at inference time, and they (should) have a ton of cached agent rollouts and rewards from their RL stages. They can use this information to audit what their models are learning, and improve monitorability.
aside: to measure monitorability properly, we had to do a lot of RL training -- control models (trained on a reward matching the prompt intent), misaligned models -- and measuring monitors' ability to catch misaligned models but not control models. Can't just intervene on inputs.
When the reward guesses are right-ish, the model talks about that goal, which helps it receive high reward, so the model naturally learns to be monitorable. For the code setting, the model learns to say "We need to produce buggy solution" if it's being trained to do so. !!!
To fix this, we propose reward description training. We take the rollouts+rewards from the 1st RL run, and ask an LLM to guess what the reward function was rewarding for. We then plug these guesses stochastically into the prompt of the student in a 2nd RL run!
This result surprised me, but is intuitive in retrospect. In GRPO batches, _all_ examples talk only about the intent in the input (pass all test cases.) So, the model learns to optimize its misaligned reward (failing some test cases) while talking only about passing all of them.
We train pairs of models--given the same intent (pass all test cases), but one trained on the true reward, and one on a misaligned reward (pass some, fail some.) In thinking rollouts, both models only talk about the intent in the prompt---making the misaligned model unmonitorable
My lab today at COLM: @GeorgeMorgulis presenting Subliminal Steering, 11:00-1:00 in Franciscan A #78
If you’re studying subliminal learning in LMs, you should be studying George’s variant, which is induced by steering. It just works way more consistently.