✨ New paper: Flipping Against All Odds
We found that large language models (LLMs) can describe probabilities—but fail to sample from them faithfully.
Yes, even flipping a fair coin is hard. 🪙
🧵 Here’s what we learned—and how we fixed it.
🔗arxiv.org/abs/2506.09998
1/
New on our Science Blog: How AI is beginning to change scientific modeling.
Scientific models help researchers explain and predict real-world systems—like how a single nerve cell fires, or what happens when neutron stars collide. Recent advances have transformed how scientists fit complex models to data. But scientists still typically write the models manually, and usually need to weigh up several competing versions, each of which might explain the data in a different way.
In a guest post, Jakob Macke, Daniel Gedon, and Stefan Wahl describe how AI is starting to help with these parts of the process. By writing, running, fitting, and revising scientific models as code, AI can explore many possible model structures, and find out which explanations the data support.
anthropic.com/research/teach…
Hello World! This is our new FridayTalks Tübingen account. FridayTalks is a monthly talk series on machine learning, AI, and applications, featuring speakers from the Tübingen AI ecosystem. Follow this account for up-to-date information and future talks!
Great work led by Jiale! NoRA builds on LoRA’s parameterization, with the down-projection matrix normalized. Even more surprisingly, applying this normalization only at initialization is already sufficient to obtain considerable performance gains.
More details in spherelab.ai/NoRA/.
1/ How can we make Low-Rank Matrices behave more like Full-Rank Matrices—and achieve higher rank efficiency?
We release our new paper: NoRA: Normalized Low-Rank Adaptation.
Paper: arxiv.org/abs/2608.31036
Our recent works, DiagDistill (ICLR’26) and XYZFlow (ICML’26), are built on a simple principle of progressively strengthening constraints.
Less trajectory ambiguity → fewer-step generation.
The key is to spend more compute early, when generation is uncertain. As context accumulates, later generation becomes easier and needs fewer steps. This simple idea is exactly leveraged across video chunks in DiagDistill, and across image patches / denoising trajectories in XYZFlow.
🔹DiagDistill: spherelab.ai/diagdistill/
🔹XYZFlow: spherelab.ai/xyzflow/
🚀 Excited to introduce PEFT-Arena! This project grew directly out of the challenges we faced when evaluating existing PEFT methods.
We argue that PEFT evaluation should not ask only what improves. It should also ask what is forgotten. For too long, parameter-efficient finetuning methods have been evaluated by downstream accuracy. But target performance alone is not enough. A method can adapt well while silently losing pretrained capabilities.
PEFT-Arena evaluates existing popular PEFT methods through the stability-plasticity trade-off, i.e., how much the model learns on the target task, and how much general ability it still preserves after finetuning.
A good PEFT method should adapt without forgetting and achieve a strong stability-plasticity trade-off. In our evaluation, we find that Orthogonal Finetuning achieves the strongest trade-off among all methods.
Beyond task-driven benchmark scores, PEFT-Arena also provides geometry-based internal diagnostics in weight space, activation space, and interpolation paths, helping us understand not only which methods work, but why they forget.
What PEFT-Arena has done:
🌟 Adaptation + preservation: evaluates both target gains and forgetting.
🌟 Trade-off frontier: reveals stability-plasticity patterns, with OFT often leading.
🌟 Geometry diagnosis: explains forgetting via weight spectra, activation distortion, and interpolation paths.
Welcome to try PEFT-Arena and evaluate your own PEFT method!
🌐 Project page: spherelab.ai/PEFT-Arena
📝 Paper: arxiv.org/abs/2605.28819
💻 Code: github.com/Sphere-AI-Lab/PEF…
I want to share more about why we started building Orbit: spherelab.ai/orbit/
Orbit is an OFT-centric reinforcement learning pipeline designed for ultra-efficient post-training of large language models. In simple terms, Orbit allows us to post-train a trillion-parameter LLM on a single 8-GPU node. But more importantly, Orbit represents something much larger for us, as it connects our long-standing research to a practical system that can be used at frontier scale.
One of SphereLab’s core missions is to develop a principled and unified full-stage training pipeline for large foundation models. We are especially interested in what we call the spectral scaling paradigm: the idea that the geometry and spectral structure of model weights should not be treated as incidental details, but as central objects in how we design training algorithms.
This perspective has alreadyled us to develop a series of pretraining algorithms and systems, including POET, POET-X, and Pion:
- POET: spherelab.ai/poet/
- POET-X: spherelab.ai/poetx/
- Pion: spherelab.ai/pion/
It has also guided our work on post-training and adaptation, including OFT, BOFT, OFTv2, PEFT-Arena, and OrthoMerge:
- OFT: oft.wyliu.com/
- BOFT: boft.wyliu.com/
- OFTv2: spherelab.ai/oftv2/
- PEFT-Arena: spherelab.ai/PEFT-Arena/
- OrthoMerge: spherelab.ai/OrthoMerge/
Although these projects may appear to target different stages of the model lifecycle, they are all driven by the same underlying principle -- the spectrum and geometry of neural network weights matter. Preserving and controlling these structures can lead to more stable, efficient, and scalable training and adaptation.
While creating these algorithms has been extremely rewarding, we take a step further to build practical systems from the ground up that can leverage the same principle to best scale our algorithms. That is why a small team like us still spent so much time and efforts to build Orbit.
Orbit is only the starting point. We are excited to keep building systems that bring principled training algorithms to real-world foundation model development.
🚀 Meet Orbit: OFT-based RL infrastructure for stable, efficient post-training of trillion-parameter LLMs.
Orbit can train 1T+ LLMs (e.g. Kimi-2.6, DeepSeek-V4-Pro) on a single GPU node (8xB200) with extremely small train-rollout gap!
Code: github.com/Sphere-AI-Lab/orb…
Blog: spherelab.ai/orbit/
Blog in Chinese: mp.weixin.qq.com/s/M3Q4AnhMa…
🧵1/7
Ever tried building a world model for a partially observable environment from just raw observations? It's tough - but what if LLMs could help?
We explore this question in our latest preprint: 'Learning POMDP World Models from Observations Using Language-Model Priors.'
Nice work! Whether an LLM can generate faithful sample is an important topic for the agentic era.
Our work verbalized rejection sampling also studied this problem!
arxiv.org/abs/2506.09998
can llms reliably roll the dice? 🎲
we shed new light on stochasticity limitations of llms, discussing some ways in which things can improve: tools, prngs, and 'just giving a random number to the model™'
great work from @gu_xiangming while being a student researcher with us 🚀
🚀 Excited to introduce POET-X, a scalable and highly memory-efficient algorithm for LLM pretraining.
✨ LoRA-level GPU memory, better-than-AdamW pretraining performance!
POET-X finally marries training stability (from POET's spectrum preservation) and practical scalability (from our new implementation and CUDA kernels). POET-X can pretrain billion-parameter LLMs (eg., Llama-8B) on a single NVIDIA H100, where standard optimizers like AdamW run out of memory under the same settings.
We carefully reimplemented every computation step of POET (arxiv.org/pdf/2506.08001). POET-X combines many small checkpointing and parallelization tricks. While each may appear incremental, together they dramatically improve scalability and reduce memory usage by over 70% compared to the original POET.
The memory-efficiency of POET-X comes from the unique parameter-efficient reparameterization (where sparsity comes in) of the weight update rule. POET-X bridges this gap between parameter efficiency and memory efficiency.
Code is now public. Feel free to try it!
➡️ paper: arxiv.org/pdf/2603.05500
💻 Code: github.com/Sphere-AI-Lab/poe…
🌐 Website: spherelab.ai/poetx#AI#LLM#MachineLearning#DeepLearning
Interesting work! Doing proper normalization is definitely important for training neural networks stably. We considered hyperball normalization for convolutional neural networks back in 2018, see arxiv.org/pdf/1804.08071. Besides hyperball normalization, we also proposed multiple other normalization methods for weight/activation.
Quite surprisingly, we also did gradient normalizaiton in order to make it actually work. See Section 4 of the Decoupled Networks paper.
I somehow got the impression that many many old ideas are worth revisiting for LLM pretraining, especially those that stablizes the training (but may slightly hurt the performance for conventional CNNs).
(1/n) Introducing Hyperball — an optimizer wrapper that keeps weight & update norm constant and lets you control the effective (angular) step size directly. Result: sustained speedups across scales + strong hyperparameter transfer.
😍Excited to organize @GRaM_org_ 2.0 this time at #ICLR2026 🇧🇷
🌟 Looking forward to your best works on geometry-grounded representations, inductive bias, and structure in learning.
This year, we also welcome works on
🌐open problems,
⚔️discussions on scale vs symmetry,
👊 position papers and more!
Deadline: 30th January AOE
📢The second edition of ✨GRaM workshop✨ is here this time at #ICLR26.
🌟Submit your exciting works in Geometry-grounded representations.
We welcome submissions in multiple tracks i.e.
📄 Proceedings
📝extended abstract
👩🏫Tutorial/blogpost
as well as an exciting challenge!