
11 billion parameters. On a phone. We wanted to see whether Llama-3.2-11B-Vision-Instruct — far too large for a phone in its original form — could fit into a ~4 GB budget. The answer: yes. Here's how 🧵

Our mission is to follow and contribute to the advancement of AI research, aiming to characterise the computational requirements of machine intelligence.

11 billion parameters. On a phone. We wanted to see whether Llama-3.2-11B-Vision-Instruct — far too large for a phone in its original form — could fit into a ~4 GB budget. The answer: yes. Here's how 🧵

The result: ~22% more compression than block-scaled INT at the same task performance, while keeping Arm CPU decoding efficient. Paper, interactive walkthrough + Android demo: graphcore-research.github.io… Joint work in collaboration with Jeevan Bhoot at Arm.

The data recipe mattered more than we expected. Generic prompt sampling, mixed prompt formats and varied answer lengths were all important. With the wrong synthetic data, training loss can improve while downstream performance gets worse. Fun 😅

Format design was only half the battle. Directly casting below ~3.5 bits/parameter hurts task quality rapidly, so we use quantisation-aware training + self-distillation: the original model teaches its quantised student on synthetic image-text data.

And it isn’t just small on disk. On a Pixel 8a, our custom C++ implementation can run the 3.73 GB model at ~3.8 tokens/s during autoregressive generation. With INT8 weights, the model doesn't fit in memory.

Why vector quantisation? Below 3 bits/parameter, reconstruction error gets brutal. S3D8 uses short 3-value centroids + separate sign bits, improving the rate/distortion trade-off while keeping decoding friendly to Arm register lookup tables.

The key is S3D8, our new 2.7-bit weight format for Arm CPUs. It packs 3 weights into 8 bits, then decodes them to channel-scaled INT8 so we can use integer matrix-multiply instructions already present in modern mobile CPUs.


A tiny implementation detail in low-precision arithmetic could be biasing your AI training 😲 This interactive deep dive from Graphcore Research's @Awfidius uncovers a subtle failure mode in stochastic rounding that only appears when randomness is limited, and how it can be addressed with one simple fix 🎲 Check it out in the link below! 👇

Link to blog ➡️ graphcore-research.github.io… Link to paper ➡️ arxiv.org/abs/2602.15563

Would you rather use 1 million × 16-bit weights, 4 million × 4-bit weights, or even 16 million × 1-bit weights? In joint work between Aleph Alpha Research and Graphcore, we asked this question of LLMs — the answer encouraged us to embrace the wonder ✨ of 1-bit weights, which can outperform 4-bit and 16-bit weights on a fixed weight memory budget. In our work - ⚖️ A scaling laws evaluation prompts us to consider very low-bit formats - 📈 Scaled-up tests show the power of memory-matched models with 1-bit weights - ⚡ Kernel benchmarking demonstrates their feasibility for autoregressive inference Read all about it in our blog and paper (link below! ⬇️) Massive thanks to our collaborators at Aleph Alpha Research! Authors: @SohirMaskey, Constantin Eichenberg, @atomicflndr and @douglasahorr

✨ Check out our summaries and follow our blog ✨graphcore-research.github.io…

Our picks for November’s Papers of the Month are here. Out of our shortlisted papers, we spotlight three looking at LLM efficiency from different angles! 📊 First up, How to Scale Second-Order Optimization is studying optimal tuning of second order optimizers such as Muon. 🌫️ In Intelligence per Watt, the authors discuss our favorite metric on large language models: energy efficiency. And how to take advantage of edge AI inference. 🕸️ Finally, Int vs FP: A comprehensive study of fine-grained formats is contributing to an old-timer topic in quantization: integer vs floating (block) point formats. Check out our summaries 👇

Come to the Frontiers in Probabilistic Inference today at #NeurIPS2025 to see Graphcore researcher Michael Pearce presenting "Variational Entropy Search is Just 1D Regression"

Come to the ML for Systems workshop at #NeurIPS2025 to see Graphcore Researcher Callum McLean presenting "MXNorm: Reusing Block Scales for Efficient Tensor Normalisation"

At #NeurIPS in San Diego? Come swing by poster #1016 this afternoon to see Graphcore Researcher Johanna Vielhaben presenting "Beyond Scalars: Concept-Based Alignment Analysis in Vision Transformers"