←
Ryan Lee20 posts
Ryan Lee avatar

Ryan Lee

@_ryantlee

ML/AI researcher

◷ Joined December 2024
Ryan Lee avatar
Ryan Lee@_ryantlee·Sep 26

My first PhD paper, 'Sparse Layers are Critical to Scaling Looped Language Models,' is accepted at NeurIPS 2026! Looping & early exits enables adaptive depth; MoE makes it scale. First step toward compute-adaptive LMs. w/ @jbiloki @edwardjhu @jonathanmay arxiv.org/abs/2605.09165

Ryan Lee avatar
⟳ Ryan Lee retweeted
Ryan Lee@_ryantlee·Sep 2

Nice work, thanks for the cite in section 2! Would love a row in Table 1: we match (active) params for Looped-MoE architectures against baselines, so forward FLOPs match, and effective depth is fixed at 16 (8x2 vs a 16-layer Base), so KV matches too. arxiv.org/abs/2605.09165

Edward Hu avatar
⟳ Ryan Lee retweeted
Edward Hu@edwardjhu·Sep 1

How does one RL post-train a 397B model for long-horizon knowledge work? 👩‍💼 We share every step we took to bring Qwen 3.5 397B from 16.1% Pass@1 to 27.3% on APEX-Agents using DPPO, including final models weights and the full training script🚀 This is the first of many works from Mercor Research on open model training research. Full blog: mercor.com/blog/training-fro… Source code: github.com/Mercor-Intelligen…

Post attachment
Xuezhe Ma (Max) avatar
⟳ Ryan Lee retweeted
Xuezhe Ma (Max)@MaxMa1987·Jan 13

After about 2 years, we are proud to release Gecko, an efficient architecture that improves upon Megalodon, with capability of efficiently and inherently processing sequences with unlimited context length. One of the most important idea in Gecko is Adaptive Working Memory(AWM), implemented using a linear attention mechanism with a position-aware online softmax activation. Notably, AWM globally compresses information into memory, rather than discarding historical information through forgetting. In a controlled head-to-head comparison with Llama2 and Megalodon, Gecko achieves better performance in the scale of 7B and 2T training tokens. Gecko achieves 1.68 training loss, vs. 1.67 of Llama2-13B, with half number of parameters on 2T tokens. Paper: arxiv.org/abs/2601.06463 Code: github.com/XuezheMax/gecko-l…

Post attachmentPost attachment