←
Dan Alistarh180 posts
Dan Alistarh avatar

Dan Alistarh

@DAlistarh

Professor at IST Austria

◷ Joined May 2022▧ 52 Photos and videos
Michael Helcig avatar
⟳ Dan Alistarh retweeted
Michael Helcig@mhelcig·Oct 30

We have noticed recent comparisons published by Unsloth that do not align with our results for DeepSeek-V4.1-Flash-1017. So just if it is not obvious: we did of course not modify any hyperparameters to favor our models. We used standard settings appropriate for a model of this scale (763B parameters) and its 1M context length capability: a 128K context window and a reasoning effort set to 50. To the best of our knowledge, the discrepancy in performance stems from a difference in benchmark constraints - not in the model being worse. Unsloth's comparisons has a strict 32K token reasoning budget, whereas our benchmarks allow up to 128K output tokens. This significant difference in compute explains the variance in results; it is not due to any hidden adjustments on our part. Furthermore, imposing such a low reasoning budget (32K) appears inconsistent for a model designed with a 1M context window and 763B params. Our primary goal is open collaboration and transparency. All of our code is available on GitHub, and our papers are on arXiv. We are genuinely happy that other quantizations are performing well, as this was a key motivation for publishing these models in the first place. We invite anyone with the compute resources to reproduce our results using our exact settings (128K output budget, reasoning effort 50) for independent verification. Our previous models were benchmarked by third parties like @bnjmn_marie, who do great work for the community. We welcome rigorous testing of our claims. Furthermore, we are currently running the comparisons on the models within our setup and will post results here soon. Thanks and stay tuned!

Andrei Panferov @ICLR avatar
⟳ Dan Alistarh retweeted
Andrei Panferov @ICLR@black_samorez·Sep 28

Disaggregated quantization for LLMs 🧵 LLM prefill and decode can be processed by separate weights in different precisions. Boosts speed AND accuracy. Offloading makes it zero-overhead on long sequences for dense models. Applicable to existing quantized checkpoints. Paper: arxiv.org/abs/2609.26333 Code: github.com/IST-DASLab/disagg… Qwen3.8 NVFP4 prefillers: huggingface.co/ISTA-DASLab/Q…

0:18
Philip Zmushko avatar
⟳ Dan Alistarh retweeted
Philip Zmushko@fzmushko·Jun 30

We’re excited to share our #ICML2026 paper on optimizer robustness in Async Pipeline Parallel LLM pretraining Main message: with the right PP schedule, optimizer choice, and Error Feedback correction, Async PP can match sync training quality even at 10B MoE scale [1/10]

Post attachment
Dan Alistarh avatar
Dan Alistarh@DAlistarh·May 19

🚀 Scalar structure means GSQ can use highly-optimized low-precision GEMM kernels: Using vLLM + the excellent Humming kernels (github.com/inclusionAI/hummi…) on L40s GPUs, 2-bit GSQ-quantized Llama-3.1-70B hits up to 6.2× throughput vs BF16!

Post attachment
Dan Alistarh avatar
Dan Alistarh@DAlistarh·May 19

⚡GSQ Scales since it has a PTQ memory footprint. We quantize the T-parameter Kimi-K2.5 on a single H200 server in 24 hours, preserving reasoning & coding. Also, GSQ directly improves public GGUF checkpoints: applied to Unsloth's Qwen3-8B Q2_K, score jumps 50.03 → 56.28!

Post attachment
Dan Alistarh avatar
Dan Alistarh@DAlistarh·May 19

📊 Accuracy: GSQ close the gap to the QTIP frontier while staying scalar. On Llama-3.1 models bits, GSQ beats the best scalar baseline (EfficientQAT) by +4.76 / +4.14 avg points, trailing QTIP by just 1.33 / 1.68 points. At 3 bits symmetric quantization it's on par with QTIP!

Post attachment
Dan Alistarh avatar
Dan Alistarh@DAlistarh·May 19

Key idea: low-bit quantization is a sampling problem. For each weight, use trainable logits over the grid points and sample via Gumbel-Softmax. Scales + discrete assignments are optimized jointly. As sampling temperature anneals, assignments lead to discrete scalar quantization.

Post attachment
Dan Alistarh avatar
Dan Alistarh@DAlistarh·May 19

Weight-only quantization powers local LLMs like llama.cpp or Ollama. But SOTA quantized accuracy requires complex kernels that are notoriously hard to implement. Can we get SOTA accuracy and keep things simple? Our new GSQ (Gumbel-Softmax Quantization) method says yes. 🧵

Post attachment
Eldar Kurtić avatar
⟳ Dan Alistarh retweeted
Eldar Kurtić@_EldarKurtic·May 11

TurboQuant has drawn a lot of attention recently, but the accompanying evals didn't tell the full story. So we ran what I believe is the first comprehensive study of TurboQuant: where it helps, where it falls short, and how it impacts accuracy, latency, and throughput. Findings:

Post attachmentPost attachment
Jiale Chen avatar
⟳ Dan Alistarh retweeted
Jiale Chen@JialeChenEdu·Apr 23

🚀 Our #ICLR2026 paper: The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithm We show GPTQ is exactly Babai's nearest plane algorithm, giving a geometric view of LLM quantization and inspiring improved PTQ methods. Efficient GPTQ Triton kernels included!

Post attachment
Dan Alistarh avatar
Dan Alistarh@DAlistarh·Mar 25

Caveats! 1. We fall short of GPT3-175B parity. 2. Our 4B model is inferior on generative tasks (LAMBADA), although it performs well on multiple-choice. This is related to data distribution and model size gaps. 3. NVFP4 speed gains are bounded at this model scale (<50% E2E).

Dan Alistarh avatar
Dan Alistarh@DAlistarh·Mar 25

Our main training run took ~8 days on 8xB300s (590k steps). - Our Zero-shot aggregate: 69.6 (vs OPT-175B: 69.0) ✅. - GPT-3 style eval: 68.9 (vs 70.0): so close, but not quite. MMLU few-shot: 41.9% vs GPT-3's 43.9%. Main run cost on @verdacloud : ~$8.8k on-demand, ~$4.6k spot.

Dan Alistarh avatar
Dan Alistarh@DAlistarh·Mar 25

Our initial model is a reshaped 4B param OLMO2-style decoder. We ran with AdamW (lr=3e-3) and 1K context. Not everything runs in FP4: norms, and LM head stay in higher precision. Since the NVFP4 GEMMs are fast, attention and the optimizer become main bottlenecks.