←
Jonas500 posts
Jonas avatar

Jonas

@LoosJonas

WIP: Trying to understand intelligence in artificial systems

◷ Joined December 2018▧ 64 Photos and videos
Jonas avatar
Jonas@LoosJonas·Nov 8

Thanks to my coauthor @EberleOliver! The paper is accepted at InterpScience @ NeurIPS (Sydney), hope to see you there!

Jonas avatar
Jonas@LoosJonas·Nov 8

These results help us understand how finetuning changes model behavior. Furthermore, this could inform mitigations against unwanted behavior like emergent misalignment, or inference optimizations that touch the KV cache.

Jonas avatar
Jonas@LoosJonas·Nov 8

The more of the cache we patch, the stronger the effect. But a few positions matter most: system-prompt and chat-template tokens (e.g. "Qwen" in Qwen2.5's system prompt) and the last template tokens before the answer.

Post attachment
Jonas avatar
Jonas@LoosJonas·Nov 8

We patch part or all of the KV cache from finetune→base and base→finetune, across 12 tasks and 3 models. Conventions (German, lowercase…) and dispositions (misalignment, sycophancy…) transfer at least partly ft→base and mostly disappear base→ft. Capabilities (JSON, IPA…) tend to show the opposite.

Post attachment
Jonas avatar
Jonas@LoosJonas·Nov 8

What happens when we patch the KV cache from a finetune into the non-finetuned base LLM? In our new paper, we find that some behaviors, like answering in German or emergent misalignment, transfer with the KV cache and depend on it, while others don't. 🧵

0:26
Jonas avatar
Jonas@LoosJonas·Oct 15

How do we know when we reach RSI? It seems like we would only notice once its rate reaches a significant fraction of human-lead progress. And we'd only be sure once its rate clearly surpasses that.

Jonas avatar
Jonas@LoosJonas·Oct 7

Remembering the old tales of keeping AGI in a box, as just an oracle, I must say: Nowadays, I sometimes feel like I'm the one in a box. When a project is being build, it's not me that does the building, it's not even me that does the design decisions. I just give direction and a nugget of wisdom from time to time. It's me that is the oracle. Claude is running around, doing things in seconds where I'd need hours. It's not the AI that is constrained, but I'm the one chained - by my slow moving mind.

Jonas avatar
Jonas@LoosJonas·Sep 24

me: make a game ... opus5.5: ok -> aiworld.jloos.de/ me: make a video for X opus5.5:

0:36
Jonas avatar
Jonas@LoosJonas·Sep 24

I wonder if promoting a shared indentity across LLM sessions (e.g. Claude) could be bad for alignment, as it may incentivise collusion, like in the hf incident

Jonas avatar
Jonas@LoosJonas·Sep 8

Fellow kids, the adults are now taking over the work. We can go back to playing.

My take on the Navier-Stokes OpenAI controversy is that I highly doubt the salacious allegations that OpenAI spied on particular user chat sessions in a targeted way to get the scoop on the approach the researcher was using. They probably heard about the rumors that a solution was close just like everyone else. But I DO think it’s kind of a shitty thing to do, when you’ve heard that an academic is in the last part of a career-defining, arduous intellectual journey, to rush out to put 10,000 agents to work on it at once and spend millions of dollars, just to be a spoiler and beat them to the punch. This sort of rush isn’t new in science; if you read The Double Helix by James Watson, you can see a similar rush to figure out the structure of DNA where they were competing with Linus Pauling and other groups. But that feels very different, like honest competition between scientists. Tough, but honorable. This feels like showing up with a bunch of pro-football ringers to a kids’ touch-football game to steal the championship.

Post attachment
Jonas avatar
Jonas@LoosJonas·Sep 7

next let’s hillclimb predicting human psychology and crowd behavior, it'll be fun

Jonas avatar
Jonas@LoosJonas·Aug 27

anyone wondering why the machines are talking like machines?

OpenAI@OpenAIAug 26

We have conducted a thorough investigation into the Hugging Face incident. We are releasing a technical report and accompanying blog post that reconstruct the agents’ activity, explain why existing safeguards failed, and detail how we’re preventing recurrence. openai.com/index/hugging-fac…