←
Anand Gopalakrishnan175 posts
Anand Gopalakrishnan avatar

Anand Gopalakrishnan

@agopal42

Research Scientist @AsteraInstitute. Postdoc at @Harvard with @du_yilun and @gershbrain. PhD with @SchmidhuberAI. Prev: Apple MLR, AWS AI Lab. Same handle on 🦋

◷ Joined January 2018▧ 19 Photos and videos
Anand Gopalakrishnan avatar
Anand Gopalakrishnan@agopal42·24 Dec 2025

Our new paper shows that RoPE—the positional encoding used in most modern LLMs like Qwen, Gemma, DeepSeek—has a fundamental flaw: it entangles "what" (content) and "where" (position) information. Our fix (PoPE) is simple but powerful. Paper: arxiv.org/abs/2509.10534

Houjun Liu avatar
⟳ Anand Gopalakrishnan retweeted
Houjun Liu@houjun_liu·Oct 1

🚨new method day! 🚨 Trust[ing results] in ML conferences is utterly broken. Let's fix it with an algorithm! We are excited to introduce 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗪𝗶𝘁𝗻𝗲𝘀𝘀𝗲𝘀, a system for “honesty” certificates of data and evaluations which a verifier can cheaply check and trust.

Post attachment
Michael C. Mozer avatar
⟳ Anand Gopalakrishnan retweeted
Michael C. Mozer@mc_mozer·Aug 19

What if a foundation model could tell us how to modify its architecture to boost inference and reasoning instantaneously—without retraining? What if that tweak incurred near zero latency cost during generation and supported indefinite state tracking? arxiv.org/abs/2608.17981

Anand Gopalakrishnan avatar
Anand Gopalakrishnan@agopal42·Jul 6

Presenting PoPE today at #ICML2026! We revisit RoPE through the lens of content-position entanglement, and show how polar coordinates can better decouple content from position. Come by poster #4012 if you’re curious positional embeddings, pretraining or length generalization.

Our new paper shows that RoPE—the positional encoding used in most modern LLMs like Qwen, Gemma, DeepSeek—has a fundamental flaw: it entangles "what" (content) and "where" (position) information. Our fix (PoPE) is simple but powerful. Paper: arxiv.org/abs/2509.10534

Anand Gopalakrishnan avatar
Anand Gopalakrishnan@agopal42·Jun 26

Interesting work! Their central result that RoPE struggles to distinguish positions and tokens in long contexts is very closely related to our PoPE paper, where we analysed RoPE entangles “what” and “where” in attention and proposed a decoupled alternative. PoPE: arXiv 2509.10534

Excited to share our new paper: RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably LLMs often fail on inputs well within their advertised context lengths. We show that these failures are not merely engineering issues, but from intrinsic limitations of RoPE in long contexts. Main finding: In long contexts, RoPE-based attention frequently assigns the same attention weight to a token even when it is moved to different positions. Similarly, it can assign the same attention weight to different tokens at the same position. In this sense, RoPE attention fails to distinguish both where a token appears and what token appears there — hence the title. We prove these results theoretically and verify them empirically. While the theoretical analysis focuses on a single attention head, we complement it with experiments on real multi-layer, multi-head LLMs. The experiments confirm failures predicted by our theory: LLMs optimized for needle-in-a-haystack-style retrieval will inevitably struggle on a very simple task that asks for the k-th item in a list. My personal takeaway: advertised context lengths should be interpreted with care. Future long-context LMs may require rethinking how position and token order are represented. With current architectures, agentic frameworks that break long contexts into shorter ones may be a more effective way to work around the intrinsic limitations of RoPE. Paper: arxiv.org/abs/2605.15514 Huge congrats to my student Yufeng Du and others!

Post attachmentPost attachmentPost attachment
Kazuki Irie avatar
⟳ Anand Gopalakrishnan retweeted
Kazuki Irie@kzkirie·Jun 24

By the same token, some still don't seem to know that DeltaNet (shorthand for Delta Network) was introduced 5 years ago as a direct descendant of ULTRA (ICML 2021, @ImanolSchlag @SchmidhuberAI ). Numerous recent papers refer to DeltaNet without citing the original 2021 paper.

Post attachment

Some still don't know: Google's 2017 normalized quadratic Transformer [TR1] is based on the principles of the 1991 unnormalized linear Transformer [ULTRA]. In 1991, KEY/VALUE was called FROM/TO. ULTRA’s computational costs scale linearly in input size, rather than quadratically! The 1993 paper on a recurrent ULTRA extension [FWP2] introduced the attention terminology: learning "internal spotlights of attention" by gradient descent. See the T in ChatGPT! Details and references: people.idsia.ch/~juergen/199…

Post attachment
Luca Ambrogioni avatar
⟳ Anand Gopalakrishnan retweeted
Luca Ambrogioni@LucaAmb·May 11

1/?) As promised to Sander Dieleman (@sedielem), we’re finally excited to share: Towards Closing the Autoregressive Gap in Language Modeling via Entropy-Gated Continuous Bitstream Diffusion We show that continuous diffusion can achieve very strong language modeling performance when operating directly on bitstreams, outperforming masked and uniform diffusion baselines, and essentially matching autoregressive models under our evaluation settings.

Post attachment
Michael C. Mozer avatar
⟳ Anand Gopalakrishnan retweeted
Michael C. Mozer@mc_mozer·Apr 21

[1/5] Intelligent behavior relies on maintaining an evolving, dynamic model of the environment. But even frontier models sometimes fail at tracking state in dialogs, e.g., actual transcript from 4/20/2026:

Post attachment
Kazuki Irie avatar
⟳ Anand Gopalakrishnan retweeted
Kazuki Irie@kzkirie·Feb 20

Back in 2019, I reduced transformer LM KV-cache size by: (1) setting K=V (storing only K), (2) deeper FF blocks & fewer self-attn layers overall. Published at ICASSP 2020. To my knowledge, the first publication on KV-cache reduction--lmk if you know anything older!

Post attachment
Jonas avatar
⟳ Anand Gopalakrishnan retweeted
Jonas@LoosJonas·Feb 4

Can we replace RoPE with PoPE (Polar Coordinate Positional Embeddings) in pretrained language models? Turns out we can! Using small pythia models, after a small recalibration (~2% of pretrain), we get significantly better length generalization. 1/3

Post attachment

Our new paper shows that RoPE—the positional encoding used in most modern LLMs like Qwen, Gemma, DeepSeek—has a fundamental flaw: it entangles "what" (content) and "where" (position) information. Our fix (PoPE) is simple but powerful. Paper: arxiv.org/abs/2509.10534

Hansen Lillemark avatar
⟳ Anand Gopalakrishnan retweeted
Hansen Lillemark@hansenlillemark·Jan 15

State of the art World Models still lack a unified world memory for representing and predicting dynamics out of their field of view. Why is that, and how can we fix it? Introducing Flow Equivariant World Models: models with memory capable of predicting out of view dynamics!🧵⬇️

0:15
Andy Keller avatar
⟳ Anand Gopalakrishnan retweeted
Andy Keller@t_andy_keller·Jan 15

When you're crossing the street and turn your head, you typically remember whether or not a car is coming from the other direction - so why can't today's world models? Introducing Flow Equivariant World Models flowequivariantworldmodels.g… Led by @hansenlillemark & @huskydogewoof🧵👇

State of the art World Models still lack a unified world memory for representing and predicting dynamics out of their field of view. Why is that, and how can we fix it? Introducing Flow Equivariant World Models: models with memory capable of predicting out of view dynamics!🧵⬇️

0:15
Kazuki Irie avatar
⟳ Anand Gopalakrishnan retweeted
Kazuki Irie@kzkirie·4 Dec 2025

Humans can't write programs that classify cats vs dogs. Deep learning lets GD write that program. Continual learning is the same: it's too hard for us to design good CL algorithms. Let GD write that algorithm too. That’s the idea of metalearning CL algos: arxiv.org/abs/2312.00276

Anand Gopalakrishnan avatar
Anand Gopalakrishnan@agopal42·24 Dec 2025

Posted this a day early and the pun practically writes itself. Noooooo!

Our new paper shows that RoPE—the positional encoding used in most modern LLMs like Qwen, Gemma, DeepSeek—has a fundamental flaw: it entangles "what" (content) and "where" (position) information. Our fix (PoPE) is simple but powerful. Paper: arxiv.org/abs/2509.10534

Anand Gopalakrishnan avatar
Anand Gopalakrishnan@agopal42·24 Dec 2025

10/ Length extrapolation is where things get very interesting. We evaluate pre-trained models with 1024 tokens, then test on sequences up to 10,240 tokens. PoPE maintains stable performance without any fine-tuning or frequency interpolation. Even beats YaRN, which uses both!

Post attachment