←
David González-Martínez39 posts
David González-Martínez avatar

David González-Martínez

@_davidgonmar

Machine teacher

◷ Joined November 2022▧ 6 Photos and videos
David González-Martínez avatar
David González-Martínez@_davidgonmar·Oct 14

mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔 Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output. Linear Depth Memory (LDM) equips models with a matrix-valued memory.

Philipp Nazari avatar
⟳ David González-Martínez retweeted
Philipp Nazari@philna00·Nov 5

Functional invariances are the basis of many transformer quantization methods. Turns out there are way more for linear recurrences, like Mamba-2 and KDA! Check out our new paper, LinQuant👇

Post attachment

We've spent years learning how to quantize attention. But what about the linear recurrences (e.g. Mamba-2, GDN and KDA) replacing it? It turns out we can rewrite their computation to reduce quantization error, while preserving the full-precision function. 🤯 Introducing LinQuant: a framework for low-precision recurrent prefill. 🧵 (1/7)

0:14
Neehal Tumma avatar
⟳ David González-Martínez retweeted
Neehal Tumma@ntumm120·Nov 5

We've spent years learning how to quantize attention. But what about the linear recurrences (e.g. Mamba-2, GDN and KDA) replacing it? It turns out we can rewrite their computation to reduce quantization error, while preserving the full-precision function. 🤯 Introducing LinQuant: a framework for low-precision recurrent prefill. 🧵 (1/7)

0:14
Nadezhda Ilieva avatar
⟳ David González-Martínez retweeted
Nadezhda Ilieva@nadjushenka_·Oct 22

Can a language model be based purely on self-conditioning? 🤔 Yes! Meet LAW 👋 (a.k.a. Loopholing All the Way!) 🧵1/8

0:36
Rodrigo Casado Noguerales avatar
⟳ David González-Martínez retweeted
Rodrigo Casado Noguerales@rodcasnog·Oct 22

🚨 New paper! What if LLMs could reason like we do when solving Sudoku: keep a few candidates per cell and iteratively narrow them down? Introducing Binary Discrete Diffusion for language and reasoning tasks: it denoises sets of candidate tokens using binary representations! 🧵

Konstantin Rusch avatar
⟳ David González-Martínez retweeted
Konstantin Rusch@tk_rusch·Oct 20

🚨🚀 Once again great summary of our work by @askalphaxiv 👇

"Depth Memory" Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth. Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream. On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels. alphaxiv.org/abs/2610.18809

Post attachment
Chief AI Officer at Canly, Japan | 萩野貴拓 カンリー CAIO avatar
⟳ David González-Martínez retweeted
Chief AI Officer at Canly, Japan | 萩野貴拓 カンリー CAIO@haginota·Oct 21

What if information passing between Transformer layers were treated as memory reads and writes? "Depth Memory" proposes LDM, a fixed-size memory updated via recursive least squares, and reports lower perplexity than PreLN, mHC and AttnRes at 400M and 1.5B scale (authors' experiments). arxiv.org/abs/2610.18809

0:40
Jonas Geiping avatar
⟳ David González-Martínez retweeted
Jonas Geiping@jonasgeiping·Oct 20

This next paper is a bit quirky, but here goes: Introspection is hard to study in both humans and LLMs. But, we do have internal senses that are verifiable, like proprioception (the sense of where your arm is right now). What would proprioception look like in language models? As a digital analogue to the sense of position, a model could verbalize which expert activated in a certain layer for any previous token. It turns out that LLMs can learn a range of such proprioception skills where they understand and report on their internals. We can calibrate and score these reports, and evaluate whether training for proprioception changes model behaviors.

Post attachment
Philipp Nazari avatar
⟳ David González-Martínez retweeted
Philipp Nazari@philna00·Oct 20

Great summary of our paper, Depth Memory! We 1. introduce a framework for existing depth mixing methods and 2. propose a new depth mixer that outperforms AttnRes and mHC. Check it out! 🚀 @_davidgonmar @tk_rusch @Shiwei_Liu66

"Depth Memory" Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth. Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream. On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels. alphaxiv.org/abs/2610.18809

Post attachment
alphaXiv avatar
⟳ David González-Martínez retweeted
alphaXiv@askalphaxiv·Oct 20

"Depth Memory" Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth. Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream. On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels. alphaxiv.org/abs/2610.18809

Post attachment
Sumin Park avatar
⟳ David González-Martínez retweeted
Sumin Park@psmiziz·Oct 15

Can reasoning finish while latent states keep moving? Happy to share Quotient Recurrent Reasoner (QRR)! We let symmetry-generated rotations persist while task-relevant dynamics converge faster in quotient space, reaching 97.9% on Sudoku-Extreme with 78% fewer loops than FPRM.

0:08
Songlin Yang avatar
⟳ David González-Martínez retweeted
Songlin Yang@SonglinYang4·Oct 14

MesaNet rotated 90 degree?

mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔 Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output. Linear Depth Memory (LDM) equips models with a matrix-valued memory.

Riccardo De Santi avatar
⟳ David González-Martínez retweeted
Riccardo De Santi@r_de_santi·Oct 14

We’re bringing natural-language control to any scientific generator. We are very excited to introduce LLM-to-Flow Distillation (LFD), a method to adapt scientific generators to high-level natural-language discovery goals. Starting from textual goals like “design peptides with promising anticancer activity” or “design small molecules binding to the D2 dopamine receptor”, LFD distills high-level goal-related knowledge from LLMs into scientific generators through iterative DPO on LLM-generated sub-goals. 🔥 LFD does not need task-specific proxy rewards, works with any frontier LLM via API-access, and outperforms state-of-the-art reward-guided methods such as FlowGRPO and DiffusionNFT on held-out computational evaluations. (1/6)

0:32
Keyu Wang avatar
⟳ David González-Martínez retweeted
Keyu Wang@Keyuciallo·Oct 15

mHC, iHC, oHC, xHC, and a zoo of AttnRes variants... An LLM architect with FOMO from the daily explosion of new papers? 🫠 Fear not! You can't afford to miss ScalingDepth — our community hub for keeping up with the ever-evolving world of residual connections! We'll keep tracking and sharing the latest research, methods, and papers from the community 🚀 And it's an OPEN community! Missed something? PRs and DMs are always welcome! Got a cool blog, post, idea, or unexpected finding? We'd love to hear it! ScalingDepth hosts a dedicated space to showcase community insights and discoveries.🌍 Built by the community. Driven by curiosity. Pushing the limits of depth! 🔥 🌐 keyu-wang-2002.github.io/Sca…

Post attachment
Vera Milovanović avatar
⟳ David González-Martínez retweeted
Vera Milovanović@nevera_m·Oct 15

Depth Memory augments residual streams with compressed, queryable memory, outperforming PreLN, mHC, and AttnRes on perplexity at 400M and 1.5B parameters. Check out the paper! 😍

mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔 Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output. Linear Depth Memory (LDM) equips models with a matrix-valued memory.