←
Eric Ming Chen123 posts
Eric Ming Chen avatar

Eric Ming Chen

@ericmchen1

PhD student at @MIT @Cornell 2024 Graphics, vision

◷ Joined April 2019▧ 29 Photos and videos
Shivam Duggal avatar
⟳ Eric Ming Chen retweeted
Shivam Duggal@ShivamDuggal4·2h

Language models process each token through a single residual stream (blue arrows): reading / writing memory and applying MLPs at each layer, and finally decoding the next token. Can we scale up this single stream computation? Introducing Latent Lattice Language Models (LLLMs): augmenting this vertical stream with a horizontal axis of computation—the Lattice of Latents (orange arrows). RNN 🤝 Transformers 🧵

George Cazenavette avatar
⟳ Eric Ming Chen retweeted
George Cazenavette@GCazenavette·Nov 13

Can use just 10 images to teach a model what a sea anemone looks like? In our #neurips2026 paper, "Dataset Distillation via Drifting," we distill ImageNet's 1,300 images per class down to just 10, optimized from noise. drifting-distillation.github… More info in thread! ⬇️ 1/N

0:22
Morris Alper avatar
⟳ Eric Ming Chen retweeted
Morris Alper@MorrisAlper·Nov 10

New preprint! 🐳🦁🐸 Many animals vocalize to communicate, but when we capture them in the field we get noisy recordings. We create a new, lightweight biodenoising model, CallCleaner, that you can run >90x realtime on a laptop CPU! (1/3)

Post attachment
Peter Michael avatar
⟳ Eric Ming Chen retweeted
Peter Michael@gfxptr·Nov 2

Excited to share LightShop (UIST 2026)! We created an interactive, computational system that enables the creation of high-precision light paintings. What makes it really neat is that it combines light painting's physically accurate lighting with computational precision.

0:30
Eric Ming Chen avatar
Eric Ming Chen@ericmchen1·Nov 4

I hope that this work can make speech a useful model problem for embodied intelligence more broadly. Speech combines aspects of control, reasoning and perception all together.

Eric Ming Chen avatar
Eric Ming Chen@ericmchen1·Nov 4

My friends often ask why I got into speech. Well I think there's a missing perspective in studying language as a control problem. We now often equate language modeling to (artificial) intelligence, but learning to speak is also an act of embodied intelligence.

NeurIPS 2026 Spotlight! 🌟 Physics simulators like MuJoCo MJX are changing robotics. But for studying the sensorimotor learning of speech, there's no equivalent. We built diffTLM, a differentiable vocal tract simulator, to help close that gap. (1/6)

0:07
Eric Ming Chen avatar
Eric Ming Chen@ericmchen1·Nov 4

We also made a WebGPU demo by porting our Python simulation to Javascript with coding agents! Fit a speech recording, watch the vocal tract move, and listen to the reconstruction, all in your browser. Try the example or record your own voice:

0:24
Eric Ming Chen avatar
Eric Ming Chen@ericmchen1·Nov 4

We then couple diffTLM to neural representations like a GAN trained on vocal tract MRI. We find that the distribution of formants simulated from the MRIs align well with the IPA vowel chart. We hope our work enables research in language acquisition, speech pathology & more. (4/6)

Post attachment
Eric Ming Chen avatar
Eric Ming Chen@ericmchen1·Nov 4

At its core, our simulator uses a minimalist model of the vocal tract: an acoustic tube. By backpropagating through its physics into the tube's cross-sectional areas, we optimize how its shape changes over time to match a recording. (3/6)

0:13
Eric Ming Chen avatar
Eric Ming Chen@ericmchen1·Nov 4

Combining traditional physics simulation with deep learning creates a new link between vocal tract shape and sound. Because the simulator is end-to-end differentiable, it enables both forward speech synthesis and inverse reconstruction of vocal tract shape. (2/6)

Post attachment
Eric Ming Chen avatar
Eric Ming Chen@ericmchen1·Nov 4

NeurIPS 2026 Spotlight! 🌟 Physics simulators like MuJoCo MJX are changing robotics. But for studying the sensorimotor learning of speech, there's no equivalent. We built diffTLM, a differentiable vocal tract simulator, to help close that gap. (1/6)

0:07
Jayden Teoh avatar
⟳ Eric Ming Chen retweeted
Jayden Teoh@jayden_teoh_·Oct 30

forgot to attach my venmo to the thread. jk, i think people can get pretty large scale runs for a decent cost if people knew how to choose the right compute. planning to release a blog about this

Jayden Teoh avatar
⟳ Eric Ming Chen retweeted
Jayden Teoh@jayden_teoh_·Oct 30

Multi-Token Prediction (MTP) can be done much better. 🚀 Turn out,you can recover the gradient for KL self-distillation to next-token prediction (NTP) for free without ever computing the KL. You just need a one-line code change: h_ntp += h_mtp.detach() - h_mtp Soft Multi-Token Prediction is pretty much a free lunch for MTP. 📜 Paper: arxiv.org/abs/2610.38331 💻 Code: github.com/JaydenTeoh/Soft-M… 🤗 HuggingFace: huggingface.co/papers/2610.3…

Post attachment
Xingjian Bai avatar
⟳ Eric Ming Chen retweeted
Xingjian Bai@SimulatedAnneal·Oct 21

1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains? We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer. In every language and image setting we tried that has no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans. 📝 xingjianbai.com/reactive-opt… 💻 Try it on your task: github.com/xingjian-bai/opal 🧵

0:25
Eric Ming Chen avatar
Eric Ming Chen@ericmchen1·Oct 20

This work wouldn't have been possible without support from the @AMD University Program thanks to @secunit64. Our video planner was trained on as many as 32 AMD MI350X GPUs!! Our feedback has led to FlexAttention speed ups that have already been merged into the @PyTorch nightly!

If video models scale like LLMs, they'll soon generate a perfect video plan for any embodied task. Do we need anything else? Introducing Mirror, which learns to follow these plans just by practicing, and even crafts an iron pickaxe in Minecraft w/o rewards or action labels! (1/n)

1:15