←
Daniel Kunin116 posts
Daniel Kunin avatar

Daniel Kunin

@KuninDaniel

postdoc @UCB_MillerInst PhD @ICMEStanford creator @SeeingTheory

◷ Joined December 2020▧ 38 Photos and videos
Daniel Kunin avatar
Daniel Kunin@KuninDaniel·Apr 24

For the last few years, a lot of my work has been driven by the feeling that deep learning is not magic — there are principles, mechanisms, and laws waiting to be understood. This paper is our attempt to say that clearly!

1/ Deep learning is going to have a scientific theory. We can see the pieces starting to come together, and it's looking a lot like physics! We're releasing a paper pulling together these emerging threads and giving them a name: learning mechanics. 🔨 arxiv.org/pdf/2604.21691 🔧

Post attachment
Daniel Kunin avatar
Daniel Kunin@KuninDaniel·Sep 27

Had a great time visiting @TTIC_Connect last year -- the ideas I presented have progressed quite a bit since then: - arxiv.org/pdf/2506.06489 - arxiv.org/pdf/2602.03655

1/4 The Dynamics of Deep Learning. "Trying to understand deep learning...comes down to understanding this interaction of three components that make the deep learning system," said @KuninDaniel, speaking at TTIC in April last year. Video: youtube.com/watch?v=4eMW8Hir…

Alex Atanasov avatar
⟳ Daniel Kunin retweeted
Alex Atanasov@ABAtanasov·Aug 9

If you’re a researcher in math or CS wondering what will be left to work on as AI charges through our fields, rest assured: the final theory is still out there waiting for us. It’s not even fully formulated let alone solved. What is the structure and form of thought itself?

A good view at emerging theory on deep learning arxiv.org/abs/2604.21691

Daniel Kunin avatar
Daniel Kunin@KuninDaniel·Jun 5

New work by @_AmilDravid, extending his really neat work on Rosetta Neurons through the lens of scaling laws! A nice example of how mech. interp and learning mechanics can complement one another, bringing different perspectives that together lead to deeper insights

Scaling laws describe how loss changes with scale. Do neurons inside models change predictably too? We study vision and language models up to 30B params and find systematic scaling in neuron universality, specialization, and selectivity. Paper+code: avdravid.github.io/rosetta-n… 1/n

0:14
Adam Shai avatar
⟳ Daniel Kunin retweeted
Adam Shai@adamimos·Feb 23

A longstanding dream of interp is to decompose activations into distinct, interpretable parts. But when should we expect that to work, and what even are such parts? New from Simplex: transformers factor their world into orthogonal subspaces, even when it costs accuracy.🧵👇

Daniel Kunin avatar
Daniel Kunin@KuninDaniel·May 1

Yes — by leveraging associativity. We explicitly construct efficient solutions: RNNs can compose elements sequentially in k steps, while deep MLPs can compose adjacent pairs in parallel in log k layers and we find preliminary evidence that GD can discover these solutions!

Post attachment
Daniel Kunin avatar
Daniel Kunin@KuninDaniel·May 1

This gives a precise picture of feature learning dynamics. But there is a catch: although two-layer networks can solve the task, they require width that grows exponentially with sequence length. Can deeper architectures discover more efficient solutions?

Post attachment
Daniel Kunin avatar
Daniel Kunin@KuninDaniel·May 1

Using the AGF framework (arxiv.org/abs/2506.06489) we developed in prior work, we show how two-layer networks learn this task one irreducible representation at a time. The order in which these features are learned is determined by the Fourier statistics of the encoding

Post attachment
Daniel Kunin avatar
Daniel Kunin@KuninDaniel·May 1

The task is order-sensitive (for non-Abelian G), provably requires nonlinear architectures, admits tractable feature-learning analysis, and reveals an interpretable benefit of depth!

Daniel Kunin avatar
Daniel Kunin@KuninDaniel·May 1

We study these questions through a simple but rich model: sequential group composition A network receives k elements from a finite group G and learns to predict the cumulative product This is a generalization of modular addition to arbitrary finite groups and sequences

Post attachment
Daniel Kunin avatar
Daniel Kunin@KuninDaniel·May 1

Across arithmetic, perception, navigation, and planning, many natural tasks involve learning computations over groups How do standard architectures represent these composition rules? What features do they learn, in what order, and how does architecture shape learning?

Daniel Kunin avatar
Daniel Kunin@KuninDaniel·May 1

Excited to share that our paper “Sequential Group Composition: A Window into the Mechanics of Deep Learning” was accepted to ICML 2026 in Seoul! Co-led with @giovannimarchet and @AdeleMyersPhD @hopfbifurcator @ninamiolane Paper: arxiv.org/abs/2602.03655

Sebastien Bubeck avatar
⟳ Daniel Kunin retweeted
Sebastien Bubeck@SebastienBubeck·Apr 26

From "Mathematical theory of deep learning: Can we do it? Should we do it?" to "There Will Be a Scientific Theory of Deep Learning". It's respectively the title of a talk I gave four years ago, and the title of an arxiv paper from four days ago. I really like the "learning mechanics" perspective (think of it as a continuation of "statistical mechanics", "quantum mechanics", and so on). Several of my last academic papers can be viewed under that lens (e.g. Learning threshold neurons via the “edge of stability”; or LEGO). I'm not as optimistic as the authors of the recent arxiv paper that we will EVER be able to reach what the "physics mechanics" field have achieved, but it's certainly worth trying. Talk: youtu.be/3uRD_lg701k?si=yjLY… Paper: arxiv.org/abs/2604.21691