←
Konstantin Rusch290 posts
Konstantin Rusch avatar

Konstantin Rusch

@tk_rusch

Faculty @ELLISInst_tue and @MPI_IS leading the Computational Applied Mathematics & AI Lab (CAMAIL) + Scientific Advisor @liquidai || Prev. @MIT_CSAIL + @ETH_en

◷ Joined May 2020▧ 34 Photos and videos
Konstantin Rusch avatar
Konstantin Rusch@tk_rusch·Nov 16

Apply to be my colleague 🚀 ELLIS Tübingen is easily one of the best places in the world to do AI research: Incredibly talented students, lots of compute, attractive salary and funding, and world class colleagues. And Tübingen is simple beautiful (see picture below) 😍

Post attachment

🚀 The ELLIS Institute Tübingen is hiring Principal Investigators as Hector Endowed Fellows! Enjoy globally competitive packages, full academic freedom & possible co-appointments w/ @MPI_IS and the Tübingen AI Center (@uni_tue). 📍 Tübingen, 🇩🇪 📅 Apply by Dec 13 🔗 institute-tue.ellis.eu/jobs/… #AI #MachineLearning #Hiring

Post attachment
Nouha Dziri avatar
⟳ Konstantin Rusch retweeted
Nouha Dziri@nouhadziri·Nov 9

Extremely excited and honored to join @ELLISInst_Tue, the leading AI institute in Europe, with a stellar track record and top-notch researchers, professors and students. I will be hiring 1-2 PhD students to do research with me and another fantastic PI on fundamental LLM agentic training, advancing capabilities of machines as a collective, and first and foremost, their security and securing our world with them.

Welcoming @nouhadziri to the ELLIS Institute Tübingen! We’re delighted to welcome Nouha as an Adjunct Principal Investigator. A Senior Research Scientist at Cohere and Associate Industry Member of Mila, she’s a leading voice on reasoning, safety and security, and multi-agent systems in LLMs. Her research advances the foundations of capable, reliable, and secure AI agents. Welcome, Nouha! 🔗 Learn more here: institute-tue.ellis.eu/news/…

Post attachment
ELLIS Institute Tübingen avatar
⟳ Konstantin Rusch retweeted
ELLIS Institute Tübingen@ELLISInst_Tue·Nov 9

Welcoming @nouhadziri to the ELLIS Institute Tübingen! We’re delighted to welcome Nouha as an Adjunct Principal Investigator. A Senior Research Scientist at Cohere and Associate Industry Member of Mila, she’s a leading voice on reasoning, safety and security, and multi-agent systems in LLMs. Her research advances the foundations of capable, reliable, and secure AI agents. Welcome, Nouha! 🔗 Learn more here: institute-tue.ellis.eu/news/…

Post attachment
Philipp Nazari avatar
⟳ Konstantin Rusch retweeted
Philipp Nazari@philna00·Nov 5

Functional invariances are the basis of many transformer quantization methods. Turns out there are way more for linear recurrences, like Mamba-2 and KDA! Check out our new paper, LinQuant👇

Post attachment

We've spent years learning how to quantize attention. But what about the linear recurrences (e.g. Mamba-2, GDN and KDA) replacing it? It turns out we can rewrite their computation to reduce quantization error, while preserving the full-precision function. 🤯 Introducing LinQuant: a framework for low-precision recurrent prefill. 🧵 (1/7)

0:14
Konstantin Rusch avatar
Konstantin Rusch@tk_rusch·Nov 5

Incredible work led by @ntumm120 Building a whole framework for quantizing linear attention models such as GDN, KDA and Mamba 🔥 The key is to leverage several computational invariances of these models in a clever way. Check it out 👇

We've spent years learning how to quantize attention. But what about the linear recurrences (e.g. Mamba-2, GDN and KDA) replacing it? It turns out we can rewrite their computation to reduce quantization error, while preserving the full-precision function. 🤯 Introducing LinQuant: a framework for low-precision recurrent prefill. 🧵 (1/7)

0:14
Neehal Tumma avatar
⟳ Konstantin Rusch retweeted
Neehal Tumma@ntumm120·Nov 5

We've spent years learning how to quantize attention. But what about the linear recurrences (e.g. Mamba-2, GDN and KDA) replacing it? It turns out we can rewrite their computation to reduce quantization error, while preserving the full-precision function. 🤯 Introducing LinQuant: a framework for low-precision recurrent prefill. 🧵 (1/7)

0:14
Vera Milovanović avatar
⟳ Konstantin Rusch retweeted
Vera Milovanović@nevera_m·Oct 30

Follow the exciting ML research from Tübingen, in-person, or online! fridaytalks.github.io/

Hello World! This is our new FridayTalks Tübingen account. FridayTalks is a monthly talk series on machine learning, AI, and applications, featuring speakers from the Tübingen AI ecosystem. Follow this account for up-to-date information and future talks!

Post attachment
float avatar
⟳ Konstantin Rusch retweeted
float@notesbyfloat·Oct 26

found a neuron in qwen3.5-4b that makes it talk like her

0:30
Ramin avatar
⟳ Konstantin Rusch retweeted
Ramin@ramin_m_h·Oct 27

LFM3 is here! this is an insanely elaborate release, get yourself a coffee ☕️ spend a few hours going through the blog where we tell you everything about it. The amount of useful intelligence you can pack inside 2B parameters is really insane! the highlights for me: > LFM3 architecture got out of our propriety automated hardware in the loop architecture search algorithm optimized for 14 different types of processors. This is exactly the vision we embarked on: building an intelligence fabric on the fragmented device world. > LFM3-VL-2B is truly a beast performing on par with GPT 5.6 Luna and Gemini flash ultra! at 2B parameters, wtf!! > It is sooo easy and chip to fine-tune LFM3 instances. you can now truly build specialized intelligence for anything. > LFM3 is enjoyably fast to train, and deploy. > LFM3 omni 2B packs all modalities in one model enabling reliable spacial intelligence on device! > It is now the era of products. lets build together 💫🚀

Introducing 𝙻𝙵𝙼𝟹: our new generation of Liquid Foundation Models. With a new tri-hybrid architecture and new multimodal encoders, the LFM3 family is the ideal model suite for specialized intelligence. > 6 model instances, across text, audio, vision, and omni modalities, generative and decision models. > 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱: text, 256k context window. > 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱: vision-language model with video-input support for spatial understanding > 𝙻𝙵𝙼𝟹-𝙰𝚞𝚍𝚒𝚘-𝟸𝙱: tool calls, voice chat, reasoning in 18 languages. > 𝙻𝙵𝙼𝟹-𝙾𝚖𝚗𝚒-𝟸𝙱: an omni model combines vision, audio, and language. > 𝚍𝟷-𝟸𝙱 and 𝚍𝟷-𝙾𝚖𝚗𝚒-𝟸𝙱: decision models with typed outputs and calibrated probabilities in a single pass. Omni adds audio input. > fine-tune on 1 GPU. > a new tri-hybrid architecture > architecture optimized for 14 processor classes across GPUs, CPUs, and NPUs. > 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱 is a beast. It performs competitively with Gemma 4 12B (6x larger), and Gemini 3.5 Flash Lite, and GPT-5.6 Luna, with 1.9B parameters. > 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱 is #1 on the Pareto frontier of device intelligence 𝚙𝚒𝚙𝚎𝚝𝚝𝚎.𝚕𝚒𝚚𝚞𝚒𝚍.𝚊𝚒 Every modality for specialized intelligence. Pick the one that fits your use case, customize it, and run it on-device. 1/ 🧵

2:17
Konstantin Rusch avatar
Konstantin Rusch@tk_rusch·Oct 27

LFM3 is out 👀🚀 check it out! 👇

Introducing 𝙻𝙵𝙼𝟹: our new generation of Liquid Foundation Models. With a new tri-hybrid architecture and new multimodal encoders, the LFM3 family is the ideal model suite for specialized intelligence. > 6 model instances, across text, audio, vision, and omni modalities, generative and decision models. > 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱: text, 256k context window. > 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱: vision-language model with video-input support for spatial understanding > 𝙻𝙵𝙼𝟹-𝙰𝚞𝚍𝚒𝚘-𝟸𝙱: tool calls, voice chat, reasoning in 18 languages. > 𝙻𝙵𝙼𝟹-𝙾𝚖𝚗𝚒-𝟸𝙱: an omni model combines vision, audio, and language. > 𝚍𝟷-𝟸𝙱 and 𝚍𝟷-𝙾𝚖𝚗𝚒-𝟸𝙱: decision models with typed outputs and calibrated probabilities in a single pass. Omni adds audio input. > fine-tune on 1 GPU. > a new tri-hybrid architecture > architecture optimized for 14 processor classes across GPUs, CPUs, and NPUs. > 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱 is a beast. It performs competitively with Gemma 4 12B (6x larger), and Gemini 3.5 Flash Lite, and GPT-5.6 Luna, with 1.9B parameters. > 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱 is #1 on the Pareto frontier of device intelligence 𝚙𝚒𝚙𝚎𝚝𝚝𝚎.𝚕𝚒𝚚𝚞𝚒𝚍.𝚊𝚒 Every modality for specialized intelligence. Pick the one that fits your use case, customize it, and run it on-device. 1/ 🧵

2:17
Liquid AI avatar
⟳ Konstantin Rusch retweeted
Liquid AI@liquidai·Oct 27

Introducing 𝙻𝙵𝙼𝟹: our new generation of Liquid Foundation Models. With a new tri-hybrid architecture and new multimodal encoders, the LFM3 family is the ideal model suite for specialized intelligence. > 6 model instances, across text, audio, vision, and omni modalities, generative and decision models. > 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱: text, 256k context window. > 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱: vision-language model with video-input support for spatial understanding > 𝙻𝙵𝙼𝟹-𝙰𝚞𝚍𝚒𝚘-𝟸𝙱: tool calls, voice chat, reasoning in 18 languages. > 𝙻𝙵𝙼𝟹-𝙾𝚖𝚗𝚒-𝟸𝙱: an omni model combines vision, audio, and language. > 𝚍𝟷-𝟸𝙱 and 𝚍𝟷-𝙾𝚖𝚗𝚒-𝟸𝙱: decision models with typed outputs and calibrated probabilities in a single pass. Omni adds audio input. > fine-tune on 1 GPU. > a new tri-hybrid architecture > architecture optimized for 14 processor classes across GPUs, CPUs, and NPUs. > 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱 is a beast. It performs competitively with Gemma 4 12B (6x larger), and Gemini 3.5 Flash Lite, and GPT-5.6 Luna, with 1.9B parameters. > 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱 is #1 on the Pareto frontier of device intelligence 𝚙𝚒𝚙𝚎𝚝𝚝𝚎.𝚕𝚒𝚚𝚞𝚒𝚍.𝚊𝚒 Every modality for specialized intelligence. Pick the one that fits your use case, customize it, and run it on-device. 1/ 🧵

2:17
Burny - Effective Curiosity avatar
⟳ Konstantin Rusch retweeted
Burny - Effective Curiosity@burny_tech·Oct 20

"Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream."

"Depth Memory" Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth. Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream. On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels. alphaxiv.org/abs/2610.18809

Post attachment
Arber Zela avatar
⟳ Konstantin Rusch retweeted
Arber Zela@ZelaArber·Oct 20

A looped transformer has to know when to stop, and its state has to stay bounded while it loops. We introduce Spherical Fixed-Point Reasoners (SFPRM) 🌐: loop on the unit sphere, where each step is an angle, and stop once the angle is small enough. 🧵 (1/9)

0:30
alphaXiv avatar
⟳ Konstantin Rusch retweeted
alphaXiv@askalphaxiv·Oct 20

"Depth Memory" Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth. Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream. On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels. alphaxiv.org/abs/2610.18809

Post attachment
Philipp Nazari avatar
⟳ Konstantin Rusch retweeted
Philipp Nazari@philna00·Oct 20

Great summary of our paper, Depth Memory! We 1. introduce a framework for existing depth mixing methods and 2. propose a new depth mixer that outperforms AttnRes and mHC. Check it out! 🚀 @_davidgonmar @tk_rusch @Shiwei_Liu66

"Depth Memory" Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth. Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream. On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels. alphaxiv.org/abs/2610.18809

Post attachment
Konstantin Rusch avatar
Konstantin Rusch@tk_rusch·Oct 20

🚨🚀 Once again great summary of our work by @askalphaxiv 👇

"Depth Memory" Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth. Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream. On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels. alphaxiv.org/abs/2610.18809

Post attachment