←
Tim Xiao329 posts
Tim Xiao avatar

Tim Xiao

@TimZXiao

PhD student in Machine Learning @ University of Tübingen · IMPRS-IS scholar

◷ Joined June 2012▧ 37 Photos and videos
Tim Xiao avatar
Tim Xiao@TimZXiao·13 Jun 2025

✨ New paper: Flipping Against All Odds We found that large language models (LLMs) can describe probabilities—but fail to sample from them faithfully. Yes, even flipping a fair coin is hard. 🪙 🧵 Here’s what we learned—and how we fixed it. 🔗arxiv.org/abs/2506.09998 1/

Post attachment
Jakob Macke avatar
⟳ Tim Xiao retweeted
Jakob Macke@jakhmack·Nov 6

We wrote up some thoughts on scientific modelling and the role of AI in it:

New on our Science Blog: How AI is beginning to change scientific modeling. Scientific models help researchers explain and predict real-world systems—like how a single nerve cell fires, or what happens when neutron stars collide. Recent advances have transformed how scientists fit complex models to data. But scientists still typically write the models manually, and usually need to weigh up several competing versions, each of which might explain the data in a different way. In a guest post, Jakob Macke, Daniel Gedon, and Stefan Wahl describe how AI is starting to help with these parts of the process. By writing, running, fitting, and revising scientific models as code, AI can explore many possible model structures, and find out which explanations the data support. anthropic.com/research/teach…

FridayTalks Tübingen avatar
⟳ Tim Xiao retweeted
FridayTalks Tübingen@fridaytalkstue·Oct 30

Hello World! This is our new FridayTalks Tübingen account. FridayTalks is a monthly talk series on machine learning, AI, and applications, featuring speakers from the Tübingen AI ecosystem. Follow this account for up-to-date information and future talks!

Post attachment
Weiyang Liu avatar
⟳ Tim Xiao retweeted
Weiyang Liu@Besteuler·Sep 1

Great work led by Jiale! NoRA builds on LoRA’s parameterization, with the down-projection matrix normalized. Even more surprisingly, applying this normalization only at initialization is already sufficient to obtain considerable performance gains. More details in spherelab.ai/NoRA/.

J.L@Jo1uckSep 1

1/ How can we make Low-Rank Matrices behave more like Full-Rank Matrices—and achieve higher rank efficiency? We release our new paper: NoRA: Normalized Low-Rank Adaptation. Paper: arxiv.org/abs/2608.31036

Post attachment
Weiyang Liu avatar
⟳ Tim Xiao retweeted
Weiyang Liu@Besteuler·Aug 23

Our recent works, DiagDistill (ICLR’26) and XYZFlow (ICML’26), are built on a simple principle of progressively strengthening constraints. Less trajectory ambiguity → fewer-step generation. The key is to spend more compute early, when generation is uncertain. As context accumulates, later generation becomes easier and needs fewer steps. This simple idea is exactly leveraged across video chunks in DiagDistill, and across image patches / denoising trajectories in XYZFlow. 🔹DiagDistill: spherelab.ai/diagdistill/ 🔹XYZFlow: spherelab.ai/xyzflow/

Post attachment
Weiyang Liu avatar
⟳ Tim Xiao retweeted
Weiyang Liu@Besteuler·Jun 3

🚀 Excited to introduce PEFT-Arena! This project grew directly out of the challenges we faced when evaluating existing PEFT methods. We argue that PEFT evaluation should not ask only what improves. It should also ask what is forgotten. For too long, parameter-efficient finetuning methods have been evaluated by downstream accuracy. But target performance alone is not enough. A method can adapt well while silently losing pretrained capabilities. PEFT-Arena evaluates existing popular PEFT methods through the stability-plasticity trade-off, i.e., how much the model learns on the target task, and how much general ability it still preserves after finetuning. A good PEFT method should adapt without forgetting and achieve a strong stability-plasticity trade-off. In our evaluation, we find that Orthogonal Finetuning achieves the strongest trade-off among all methods. Beyond task-driven benchmark scores, PEFT-Arena also provides geometry-based internal diagnostics in weight space, activation space, and interpolation paths, helping us understand not only which methods work, but why they forget. What PEFT-Arena has done: 🌟 Adaptation + preservation: evaluates both target gains and forgetting. 🌟 Trade-off frontier: reveals stability-plasticity patterns, with OFT often leading. 🌟 Geometry diagnosis: explains forgetting via weight spectra, activation distortion, and interpolation paths. Welcome to try PEFT-Arena and evaluate your own PEFT method! 🌐 Project page: spherelab.ai/PEFT-Arena 📝 Paper: arxiv.org/abs/2605.28819 💻 Code: github.com/Sphere-AI-Lab/PEF…

Post attachment
Weiyang Liu avatar
⟳ Tim Xiao retweeted
Weiyang Liu@Besteuler·May 29

I want to share more about why we started building Orbit: spherelab.ai/orbit/ Orbit is an OFT-centric reinforcement learning pipeline designed for ultra-efficient post-training of large language models. In simple terms, Orbit allows us to post-train a trillion-parameter LLM on a single 8-GPU node. But more importantly, Orbit represents something much larger for us, as it connects our long-standing research to a practical system that can be used at frontier scale. One of SphereLab’s core missions is to develop a principled and unified full-stage training pipeline for large foundation models. We are especially interested in what we call the spectral scaling paradigm: the idea that the geometry and spectral structure of model weights should not be treated as incidental details, but as central objects in how we design training algorithms. This perspective has alreadyled us to develop a series of pretraining algorithms and systems, including POET, POET-X, and Pion: - POET: spherelab.ai/poet/ - POET-X: spherelab.ai/poetx/ - Pion: spherelab.ai/pion/ It has also guided our work on post-training and adaptation, including OFT, BOFT, OFTv2, PEFT-Arena, and OrthoMerge: - OFT: oft.wyliu.com/ - BOFT: boft.wyliu.com/ - OFTv2: spherelab.ai/oftv2/ - PEFT-Arena: spherelab.ai/PEFT-Arena/ - OrthoMerge: spherelab.ai/OrthoMerge/ Although these projects may appear to target different stages of the model lifecycle, they are all driven by the same underlying principle -- the spectrum and geometry of neural network weights matter. Preserving and controlling these structures can lead to more stable, efficient, and scalable training and adaptation. While creating these algorithms has been extremely rewarding, we take a step further to build practical systems from the ground up that can leverage the same principle to best scale our algorithms. That is why a small team like us still spent so much time and efforts to build Orbit. Orbit is only the starting point. We are excited to keep building systems that bring principled training algorithms to real-world foundation model development.

Post attachment
Tim Xiao avatar
Tim Xiao@TimZXiao·May 28

Checkout our new project! Being able to post train the largest opensource LLMs on a single node is such a cool achievement!

🚀 Meet Orbit: OFT-based RL infrastructure for stable, efficient post-training of trillion-parameter LLMs. Orbit can train 1T+ LLMs (e.g. Kimi-2.6, DeepSeek-V4-Pro) on a single GPU node (8xB200) with extremely small train-rollout gap! Code: github.com/Sphere-AI-Lab/orb… Blog: spherelab.ai/orbit/ Blog in Chinese: mp.weixin.qq.com/s/M3Q4AnhMa…

Mridul Sharma avatar
⟳ Tim Xiao retweeted
Mridul Sharma@mriiidullll·May 18

🧵1/7 Ever tried building a world model for a partially observable environment from just raw observations? It's tough - but what if LLMs could help? We explore this question in our latest preprint: 'Learning POMDP World Models from Observations Using Language-Model Priors.'

Post attachment
Tim Xiao avatar
Tim Xiao@TimZXiao·Apr 10

Nice work! Whether an LLM can generate faithful sample is an important topic for the agentic era. Our work verbalized rejection sampling also studied this problem! arxiv.org/abs/2506.09998

can llms reliably roll the dice? 🎲 we shed new light on stochasticity limitations of llms, discussing some ways in which things can improve: tools, prngs, and 'just giving a random number to the model™' great work from @gu_xiangming while being a student researcher with us 🚀

Post attachment
Weiyang Liu avatar
⟳ Tim Xiao retweeted
Weiyang Liu@Besteuler·Mar 6

🚀 Excited to introduce POET-X, a scalable and highly memory-efficient algorithm for LLM pretraining. ✨ LoRA-level GPU memory, better-than-AdamW pretraining performance! POET-X finally marries training stability (from POET's spectrum preservation) and practical scalability (from our new implementation and CUDA kernels). POET-X can pretrain billion-parameter LLMs (eg., Llama-8B) on a single NVIDIA H100, where standard optimizers like AdamW run out of memory under the same settings. We carefully reimplemented every computation step of POET (arxiv.org/pdf/2506.08001). POET-X combines many small checkpointing and parallelization tricks. While each may appear incremental, together they dramatically improve scalability and reduce memory usage by over 70% compared to the original POET. The memory-efficiency of POET-X comes from the unique parameter-efficient reparameterization (where sparsity comes in) of the weight update rule. POET-X bridges this gap between parameter efficiency and memory efficiency. Code is now public. Feel free to try it! ➡️ paper: arxiv.org/pdf/2603.05500 💻 Code: github.com/Sphere-AI-Lab/poe… 🌐 Website: spherelab.ai/poetx #AI #LLM #MachineLearning #DeepLearning

Post attachment
Weiyang Liu avatar
⟳ Tim Xiao retweeted
Weiyang Liu@Besteuler·Jan 23

Interesting work! Doing proper normalization is definitely important for training neural networks stably. We considered hyperball normalization for convolutional neural networks back in 2018, see arxiv.org/pdf/1804.08071. Besides hyperball normalization, we also proposed multiple other normalization methods for weight/activation. Quite surprisingly, we also did gradient normalizaiton in order to make it actually work. See Section 4 of the Decoupled Networks paper. I somehow got the impression that many many old ideas are worth revisiting for LLM pretraining, especially those that stablizes the training (but may slightly hurt the performance for conventional CNNs).

Post attachmentPost attachment

(1/n) Introducing Hyperball — an optimizer wrapper that keeps weight & update norm constant and lets you control the effective (angular) step size directly. Result: sustained speedups across scales + strong hyperparameter transfer.

Post attachment
Sharvaree Vadgama avatar
⟳ Tim Xiao retweeted
Sharvaree Vadgama@SharvVadgama·18 Dec 2025

😍Excited to organize @GRaM_org_ 2.0 this time at #ICLR2026 🇧🇷 🌟 Looking forward to your best works on geometry-grounded representations, inductive bias, and structure in learning. This year, we also welcome works on 🌐open problems, ⚔️discussions on scale vs symmetry, 👊 position papers and more! Deadline: 30th January AOE

📢The second edition of ✨GRaM workshop✨ is here this time at #ICLR26. 🌟Submit your exciting works in Geometry-grounded representations. We welcome submissions in multiple tracks i.e. 📄 Proceedings 📝extended abstract 👩‍🏫Tutorial/blogpost as well as an exciting challenge!

Post attachment