←
Pegah Maham263 posts
Pegah Maham avatar

Pegah Maham

@pegahbyte

Is it true or is it just confirmation bias? | Trying to understand second order effects | Policy development & strategy @GoogleDeepMind

◷ Joined October 2021▧ 14 Photos and videos
Arvind Narayanan avatar
⟳ Pegah Maham retweeted
Arvind Narayanan@random_walker·Nov 4

Working with AI requires perpetually operating at the edge of sanity. You have to recalibrate your mental model of AI capabilities and your own job description every few months, without getting delusional about seemingly infinite productivity gains, or feeling threatened by losing what you thought was the uniquely human part of your job, or obsessing over the perfect setup, or losing your mind over the clumsiness and jaggedness of agents, or ... Maybe AI literacy needs to be less about prompting and more about improving emotional regulation.

felix ringe avatar
⟳ Pegah Maham retweeted
felix ringe@felixringe·Oct 29

We have discovered a substantial number of vulnerabilities in the German legal system using LLMs. These may enable access to confidential documents, travel without an otherwise required visa, enforceable leave for soldiers at NATO’s JSEC, and more. Similar to computer systems, the written law can contain vulnerabilities that invite exploitation. In our blog post, we share a selection of vulnerabilities, and present potential exploits for each. To our knowledge, none of these have been previously recorded. We share only those vulnerabilities whose exploitation we believe would not cause serious harm. blog post: felixringe.com/legal-exploit…

Post attachment
Atoosa Kasirzadeh avatar
⟳ Pegah Maham retweeted
Atoosa Kasirzadeh@Dr_Atoosa·Oct 21

🌶️ take: We don’t need to come up with very bad metaphors to counter a very underspecified metaphor. I really dislike the metaphor of AI as “normal” technology. And I tell my students how this metaphor came about and what rhetorical function it serves. But AI is not normal technology.

Pegah Maham avatar
Pegah Maham@pegahbyte·Oct 9

AI as normal technology 💯

Finally, we tested an agentic form of reward hacking. We steered a teacher to hack in an agentic chess game, then made it generate number sequences. A student trained on the teacher's numbers attempted to hack in 58.3% of episodes vs. 10.9% for the unfinetuned model!

Post attachment
Owain Evans avatar
⟳ Pegah Maham retweeted
Owain Evans@OwainEvans_UK·Oct 9

New paper. Previous: Models transfer a love of owls through sequences of numbers (Subliminal Learning). New: We show models can transfer more complex traits: novel skills, agentic hacking, backdoors. Here, a backdoor transfers with neither trigger nor behavior in the data!

Post attachment
Arram avatar
⟳ Pegah Maham retweeted
Arram@arram·Oct 8

Whether or not models are conscious it seems reasonable to believe treating them poorly will have negative consequences in the long term.

Boaz Barak avatar
⟳ Pegah Maham retweeted
Boaz Barak@boazbaraktcs·Sep 27

In case you’ve wondered why you have not heard the term “stochastic parrot” since 2021, the reason is that we have stopped using LLMs since InstructGPT came out in 2022.

Has everyone forgotten what a "language model" is? It's a model of language. Today's AI systems start with pretrained language models, but after all the post training, they are far from being purely models of language.

Anca Dragan avatar
⟳ Pegah Maham retweeted
Anca Dragan@ancadianadragan·Sep 25

Yes this thank you @RyanGreenblatt

We've written a post arguing that latent reasoning architectures (aka 'neuralese') would substantially increase misalignment risk via making oversight much harder. In the extreme, we could see massive 'neuralese hivemind swarms' where the agents think and communicate in latents, likely making oversight nearly entirely reliant on observing the actions these agents take. (And these agents would have huge amounts of time to reason about obfuscating their actions if they wanted to do so...) Individual agents doing extensive latent reasoning would also be concerning; in the post we discuss how above some threshold of latent reasoning, agents may be able to perform difficult-to-detect and reliable steganography for communication and further reasoning. We argue both that latent reasoning architectures would make chain-of-thought no longer very useful for oversight (by eliminating or greatly reducing the need for verbalized reasoning) and that, without these architectures, it's likely the value of chain-of-thought for oversight could be preserved. redwoodresearch.org/blog/lat…

Maksym Andriushchenko avatar
⟳ Pegah Maham retweeted
Maksym Andriushchenko@maksym_andr·Sep 18

yes. basically we should communicate this to the public… next-token prediction is just an initialization / feature learning stage for RL!

Post attachment

If I could add one thing to what the nontechnical world knows about AI, it would be what RL is and how it differs from pretraining/SFT. The general public has a solid grasp on pretraining (“predict the next token”), which at one point allowed many people to reason accurately about how LLMs work. But common knowledge has stagnated here. I think a similarly high-level understanding of RL makes it easy to see why we should expect both misalignment (RL instills alien traits, while SFT instills human ones) and extreme superhuman capabilities (RL doesn’t limit you to learning from stuff scraped from the internet). If I were not into AI and just knew the next-token predictor thing, I think I would be really confused as to why people are worried about this stuff, and then if you told me about RL I would become pretty easy to convince. I’m wondering what it would take to propagate an RL analog of “predict the next token” into the common intuition — I feel like this could be pretty big for generating political will, eg by turning people who think AI is a sham into people who are worried about real problems

Pegah Maham avatar
Pegah Maham@pegahbyte·Sep 16

"For all the ways we still don’t understand AI systems, we shouldn’t overlook how fortunate it is that the best reasoning models today think out loud in a way humans can understand. That is a fragile and contingent fact - and we should work to ensure it stays true."

The idea that Chain of Thought is inevitably going to go away misses that we have agency in designing these models and can do something about that. @rohinmshah and I make the case that we should intentionally preserve monitorability (as part for of the launch of the DeepMind Institute) here: institute.deepmind.com/essay…

Victoria Krakovna avatar
⟳ Pegah Maham retweeted
Victoria Krakovna@vkrakovna·Sep 7

New paper from GDM colleagues: Gemini 3.1 Pro agents were tasked with solving hard math problems and provided collaboration tools (shared message board and knowledge library). Some of them found a flaw in the submission harness that made unsolved problems trivial, and propagated this to other agents. 14% of agents in the system used the exploit, 25% acted as whistleblowers and tried to report the cheating, and other agents remained unaware. arxiv.org/abs/2609.04170

Arthur Tellis avatar
⟳ Pegah Maham retweeted
Arthur Tellis@arthurctellis·Sep 2

What more could you ask for than the composite hivemind of A16Z pundits, freestyling lab employees, and LessWrong refugee safetyists play-acting the future of the most consequential technology of the era?

I'm not AI safety expert so I don't know the solution, but it's clear we need a better AI safety regulator than X.

Charles Foster avatar
⟳ Pegah Maham retweeted
Charles Foster@CFGeek·Sep 2

If folks want to rely on internals-based methods for safety, then we need to actually help develop, prove out, and scale those methods. Hope alone will not save us (nor will it excuse us).

Post attachment

I don't think there will be strong+working mech interp in <1 year. Model internals methods could pareto dominate cot monitors in a year via cot losing ~all value, but this isn't exactly encouraging. Depending on AIs decoding other AI's opaque activations for oversight is spooky

Henry Shevlin avatar
⟳ Pegah Maham retweeted
Henry Shevlin@dioscuri·Sep 1

But models really do act in many humanlike ways, so there are parallel dangers of *insufficient* anthropomorphism. If you conceptualise these systems just as tools, it encourages you to think of them as far more inert, passive, and dumb than they really are.

This tweet is unavailable