←
Markus Nagel42 posts
Markus Nagel avatar

Markus Nagel

@mnagel87
◷ Joined May 2020▧ 1 Photos and videos
Andrii Skliar 🇺🇦 avatar
⟳ Markus Nagel retweeted
Andrii Skliar 🇺🇦@avskliar·3 Dec 2024

Proud to present our work on optimizing Mixture of Experts models for on-device generation speed: arxiv.org/pdf/2412.00099 We introduce a cache-aware routing that boosts memory efficiency of commonly used MoEs, improving generation throughput by 2×—all without retraining. Perfect for real-world, memory-constrained devices.  This is a joint work with wonderful team here at Qualcomm: @tivaro @r_lepert @BabakEht Todor Boinovski @mnagel87 @martvanbaalen and Paul Whatmough

Tanishq Mathew Abraham, Ph.D. avatar
⟳ Markus Nagel retweeted
Tanishq Mathew Abraham, Ph.D.@iScienceLuvr·21 Jun 2024

Sparse High Rank Adapters abs: arxiv.org/abs/2406.13175 "In this paper, we propose Sparse High Rank Adapters (SHiRA), a new paradigm which incurs no inference overhead, enables rapid switching, and significantly reduces concept-loss. Specifically, SHiRA can be trained by directly tuning only 1-2% of the base model weights while leaving others unchanged."

Post attachment
AK avatar
⟳ Markus Nagel retweeted
AK@_akhaliq·26 Feb 2024

Qualcomm presents GPTVQ The Blessing of Dimensionality for LLM Quantization show that the size versus accuracy trade-off of neural network quantization can be significantly improved by increasing the quantization dimensionality. We propose the GPTVQ method, a new fast method for post-training vector quantization (VQ) that scales well to Large Language Models (LLMs). Our method interleaves quantization of one or more columns with updates to the remaining unquantized weights, using information from the Hessian of the per-layer output reconstruction MSE. Quantization codebooks are initialized using an efficient data-aware version of the EM algorithm. The codebooks are then updated, and further compressed by using integer quantization and SVD-based compression. GPTVQ establishes a new state-of-the art in the size vs accuracy trade-offs on a wide range of LLMs such as Llama-v2 and Mistral. Furthermore, our method is efficient: on a single H100 it takes between 3 and 11 hours to process a Llamav2-70B model, depending on quantization setting. Lastly, with on-device timings for VQ decompression on a mobile CPU we show that VQ leads to improved latency compared to using a 4-bit integer format.

Post attachment
Markus Nagel avatar
Markus Nagel@mnagel87·18 Jan 2024

I'm excited that our paper 'The LLM Surgeon' got accepted at #ICLR2024. This is work by @tychovdo during his internship at #QualcommAIResearch in collaboration with @martvanbaalen, @TiRune and @y_m_asano. Check out the thread below for more details.👇

⭐️New paper ⭐️ Excited to share 'The LLM Surgeon', accepted at ICLR 2024. We obtain SOTA pruning performance and even demonstrate structured LLM pruning of full rows and cols. Direct practical impact enabling compression up to 20-30% with negligible loss in performance.🧵1/9👇

Post attachment
The TWIML AI Podcast avatar
⟳ Markus Nagel retweeted
The TWIML AI Podcast@twimlai·29 Dec 2023

The approach of removing outliers, as emphasized in Markus’ Quantizable Transformers paper, is not a quantization method but rather an approach to address and eliminate the root cause of activation quantization issues. Catch @mnagel87’s episode at buff.ly/3TL8Vrs.

0:39
Markus Nagel avatar
Markus Nagel@mnagel87·11 Dec 2023

If you are around and want to connect, please send me a PM. You can also find me at our poster sessions or the Qualcomm booth. Poster sessions: - Pruning vs Quantization: Tuesday 17:15-19:15 - Quantizable Transformers: Thursday 17:00-19:00

Qualcomm Research & Technologies avatar
⟳ Markus Nagel retweeted
Qualcomm Research & Technologies@QCOMResearch·7 Dec 2023

Read about our #NeurIPS2023 plans including four accepted demos on fast stable diffusion, on-device learning (ODL), fast #AI assistant, and generative relighting, as well as six accepted machine learning papers. See you soon in New Orleans! qualcomm.com/news/onq/2023/1…

Markus Nagel avatar
Markus Nagel@mnagel87·2 Oct 2023

ResQ: Residual Quantization for Video Perception Davide Abati, Haitam Ben Yahia, Markus Nagel, Amirhossein Habibian arxiv.org/abs/2308.09511 Friday 6th @ 10:30 AM-12:30 PM (room nord, poster 102)

Markus Nagel avatar
Markus Nagel@mnagel87·2 Oct 2023

QBitOpt: Fast and Accurate Bitwidth Reallocation during Training @jornpeters, @mfournarakis, Markus Nagel, @martvanbaalen, @TiRune arxiv.org/abs/2307.04535 RCV workshop, Monday 2nd @ 15:20 (poster session, room S04)

Markus Nagel avatar
Markus Nagel@mnagel87·2 Oct 2023

This week I'm in Paris at #ICCV2023 to present some of our recent work on model efficiency and quantization. Please join me for our talks and posters or at the Qualcomm booth. (1/4, schedule follows)