←
Andy J Yang277 posts
Andy J Yang avatar

Andy J Yang

@pentagonalize

原谅我这一生不羁放纵爱自由

◷ Joined May 2020▧ 37 Photos and videos
Sadhika Malladi avatar
⟳ Andy J Yang retweeted
Sadhika Malladi@SadhikaMalladi·Nov 11

Base model quality drives RL post-training, but existing guarantees fail to translate a good init into efficient learning. Our theory captures this and thus turns the heuristics behind modern RL objectives into actionable principles for algorithm design. arxiv.org/abs/2611.14467

Ai2 avatar
⟳ Andy J Yang retweeted
Ai2@allen_ai·Nov 3

Can LLMs track state changes over long texts—like an object changing hands or a variable reassignment? On our new StateBench eval, frontier models including GPT-6 Sol & Claude Opus 5.5 fell to 13% accuracy or lower on the hardest, longest problems. 🧵 github.com/allenai/state-ben…

1:36
Amit Ben-Artzy avatar
⟳ Andy J Yang retweeted
Amit Ben-Artzy@Amit_BenArtzy·Oct 25

1/ 🎉 Excited to present at #EMNLP2026 🎉 Transformers and Mamba are fundamentally different. Do they also represent things differently? Globally, they differ. Locally, they are similar. 🧵

Post attachment
Chenxiao Yang avatar
⟳ Andy J Yang retweeted
Chenxiao Yang@chenxiao_yang_·Oct 21

Harnesses let agents work together on a task, each in its own context. Beyond context limits, what does this way of organizing agents change about learning? Our new paper shows how it can help models generalize out of domain. 🧵1/18 arxiv.org/abs/2609.20831

Thomas Hikaru Clark avatar
⟳ Andy J Yang retweeted
Thomas Hikaru Clark@thomashikaru·Oct 22

If you've noticed a distinct dialect emerging in frontier AI models and struggled to understand what they're talking about, you're not crazy! We turned to psycholinguistics to investigate what makes the writing style so weird, and why this might be happening 🚨New preprint! 🧵

Post attachment
Tom McCoy avatar
⟳ Andy J Yang retweeted
Tom McCoy@RTomMcCoy·Oct 14

🤖🧠NEW PAPER🧠🤖 What is the relationship between neural networks & symbolic systems? Based on evidence from mechanistic interpretability, we argue for a quasi-symbolic view of cognition: Neural nets implement symbols but only approximately Link: sciencedirect.com/science/ar… 1/n

Post attachment
Adithya Bhaskar avatar
⟳ Andy J Yang retweeted
Adithya Bhaskar@AdithyaNLP·Oct 5

The most profound paper of my PhD so far. We truly did something special to make LMs play/explain chess, needing both architectural & algorithmic innovation (natural-language analogue of the Alphazero algorithm) + applicable to many other domains. Please read on and share! 1/10

Post attachment
Andy J Yang avatar
Andy J Yang@pentagonalize·Oct 4

Blerta and I will be presenting this at COLM on Oct 6! Find the location here: colm.cc/virtual/2026/poster/…

Which state-tracking problems can transformers length generalize on? Transformers' performance on different finite-state tasks varies drastically, even when the tasks are structurally similar. Now, for the first time, we can precisely predict when length generalization occurs.

Post attachment
Michael Hahn avatar
⟳ Andy J Yang retweeted
Michael Hahn@mhahn29·Oct 2

Can Transformer LLMs hide their reasoning from CoT monitors? tl;dr: Transformers cannot hide complex reasoning, but they might cryptographically obfuscate their CoTs, making it extremely difficult to extract hidden goals.

Marco🍞 avatar
⟳ Andy J Yang retweeted
Marco🍞@marco_computers·Sep 30

🚨 [Token][ization] Paper Alert 🚨 Tokenization is a wildly understudied area of language modeling despite it having effects across all of NLP. Over the past ~8 months, 32 (!) tokenizer researchers put together the most comprehensive survey of the field. Check it out!

Post attachment
Taelin avatar
⟳ Andy J Yang retweeted
Taelin@VictorTaelin·Sep 17

Bend 2 is here! It is a new programming language that blocks AI mistakes via *proof checking* - the same technique big AI labs used to solve open math problems, like Navier-Stokes. It is also very fast, and runs on GPUs. Watch the video. Link in the comments.

4:20
Taelin@VictorTaelin16 May 2024

RELEASE DAY After almost 10 years of hard work, tireless research, and a dive deep into the kernels of computer science, I finally realized a dream: running a high-level language on GPUs. And I'm giving it to the world! Bend compiles modern programming features, including: - Lambdas with full closure support - Unrestricted recursion and loops - Fast object allocations of all kinds - Folds, ADTs, continuations and much more To HVM2, a new runtime capable of spreading that workload across 1000's of cores, in a thread-safe, low-overhead fashion. As a result, we finally have a true high-level language that runs natively on GPUs! Here's a quick demo:

2:25
Maksym Andriushchenko avatar
⟳ Andy J Yang retweeted
Maksym Andriushchenko@maksym_andr·Sep 16

💥 The multiplication saga continues! It turns out, some frontier LLMs do struggle with simple multiplications. Unlike GPT-6-Astra, Fable 5.1 with low reasoning has a blind spot where the accuracy of, say, 5x6 multiplications drops to ~0%! The failure mode is very interesting: the 0% zone corresponds to a failure of adaptive thinking. The model decides that it shouldn't produce any reasoning and outputs the (wrong) answer directly. For larger multiplications, however, the model realizes it should produce a CoT first and then it gets the answer correctly. For very large multiplications (close to 20x20), the accuracy degrades again. How many such simple blind spots do frontier LLMs still have? This is why slowing down and carefully studying generalization and safety is the top priority... 1/2

Post attachment
Tom McCoy avatar
⟳ Andy J Yang retweeted
Tom McCoy@RTomMcCoy·Sep 1

🤖🧠NEW PAPER🧠🤖 (The result of an 8-year project!) LLMs seem very different from symbolic systems. Yet LLMs excel in symbolic domains (e.g., language/code/math). How do they do it? Our finding: LLM representations have implicit symbolic structure! Link in thread ⬇️ 1/n

Post attachment
Andy J Yang avatar
Andy J Yang@pentagonalize·Aug 14

Very excited to be sharing this on behalf of the amazing team: Blerta Veseli Corentin Barloy Michaël Cadilhac Andreas Krebs Charles Paperman Howard Straubing @mhahn29