←
Tomasz Limisiewicz368 posts
Tomasz Limisiewicz avatar

Tomasz Limisiewicz

@TomLimi

Postdoctoral researcher at @meta Fair and @uwnlp , Interested in going into the inner workings of neural networks, multilingualism, and fairer NLP (he/him)

◷ Joined September 2021▧ 69 Photos and videos
Tomasz Limisiewicz avatar
Tomasz Limisiewicz@TomLimi·May 4

We present Compute Optimal Tokenization! 🔡 Common in LLM scaling works stick to one tokenizer, sweeping data/model size. But what happens when we control the tokenizer’s compression rate (bytes/token)? Here we sweep tokenizers, params, and data across compute budgets: [1/N]

0:14
Tomasz Limisiewicz avatar
Tomasz Limisiewicz@TomLimi·Nov 14

despite pressure, Meta continues doing good long-term science

Meta's FAIR is one of the few labs that does projects like this and it saves a lot of time for everyone else. shout out to their optimal compute tokenization paper that did similiar analysis to this one.

Antonis Anastasopoulos @EMNLP avatar
⟳ Tomasz Limisiewicz retweeted
Antonis Anastasopoulos @EMNLP@anas_ant·Nov 5

Super excited for Chutong Meng's new paper (in collaboration with Barron Associates): arxiv.org/pdf/2610.40233 We substantially improve translation for extremely low-resource languages by extracting parallel data from linguistic documentation materials. Thread 👇

Post attachment
Tomasz Limisiewicz avatar
Tomasz Limisiewicz@TomLimi·Nov 3

Domain adaptation. Latent tokens align well with words/phrases in high-resource languages (English, Chinese). Hierarchical models should adapt compression by domains to compute optimal value. E.g., non-latin script texts need more compression to counter utf8 inefficiency. 5/5

Post attachment
Tomasz Limisiewicz avatar
Tomasz Limisiewicz@TomLimi·Nov 3

RL is not my area, but there is some potential here for byte-based models (limiting the action space) and latent tokenization (adapting compute to prediction difficulty 👀 “Beyond the 80/20 Rule”). 4/5

Post attachment
Tomasz Limisiewicz avatar
Tomasz Limisiewicz@TomLimi·Nov 3

Post-training, post-training, post-training! We have evidence of better compute and data efficiency of byte-based LLMs in pre-training. We still need comprehensive proof of better post-training and end-task performance. 3/5

Tomasz Limisiewicz avatar
Tomasz Limisiewicz@TomLimi·Nov 3

Fast generation. We NEED to generate multiple bytes at a time; otherwise, throughput will be too low to serve models in practice. There is already evidence that parallel byte generation is feasible and can overcome its speed disadvantage relative to subwords. 👀FastBLT, MBP 2/5

0:05
Tomasz Limisiewicz avatar
Tomasz Limisiewicz@TomLimi·Nov 3

Amazing results! A great case for byte-based and latent-tokenized models like H-Net! Still, most LLM research and all known frontier LLMs use subwords. Here is my list of crucial research directions pushing us toward non-subword frontier models: 1/5

I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data. Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less? Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.

1:19
Albert Gu avatar
⟳ Tomasz Limisiewicz retweeted
Albert Gu@_albertgu·Oct 28

I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data. Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less? Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.

1:19
Tomasz Limisiewicz avatar
Tomasz Limisiewicz@TomLimi·Nov 3

Domain adaptation. Latent tokens align well with words/phrases in high-resource languages (English, Chinese). Hierarchical models should adapt compression by domains to compute optimal value. E.g., many low resource languages should be compressed more due to utf8 inefficiencies

Post attachment
Tomasz Limisiewicz avatar
Tomasz Limisiewicz@TomLimi·Nov 3

RL is not my area, but there is some potential here for byte-based models (limiting the action space) and latent tokenization (adapting compute to prediction difficulty 👀 “Beyond the 80/20 Rule”). 4/5

Post attachment
Tomasz Limisiewicz avatar
Tomasz Limisiewicz@TomLimi·Nov 3

Post-training, post-training, post-training! We have evidence of better compute and data efficiency of byte-based LLMs in pre-training. We still need comprehensive proof of better post-training and end-task performance. 3/5

Sachin Kumar avatar
⟳ Tomasz Limisiewicz retweeted
Sachin Kumar@shocheen·Oct 31

Looking for multiple emergency reviewers for #ICLR2027 for papers related to medical AI, LLM-judges, and audio LLMs. Please DM me if you are eligible and interested!

Julie Kallini ✨ avatar
⟳ Tomasz Limisiewicz retweeted
Julie Kallini ✨@JulieKallini·Oct 28

If you're at #EMNLP2026 you should stop by our ✨tokenization tutorial✨ happening in <2 hours, starting at 9AM CET! Fun fact: I'm actually not at EMNLP... I'll be presenting remotely. But I'll be up from 1AM to 4AM PDT because that's how much I love tokenization 🫶😄

Headed to #EMNLP2026? Want to learn about the latest advances in tokenization research in LLMs? Come check out our tutorial on October 28. We will cover everything from fundamentals and emerging approaches with plenty of hands on demos and exercises. See you in Budapest!

Post attachment
Sachin Kumar avatar
⟳ Tomasz Limisiewicz retweeted
Sachin Kumar@shocheen·Oct 24

We will present a tutorial on tokenization on Wednesday:

Headed to #EMNLP2026? Want to learn about the latest advances in tokenization research in LLMs? Come check out our tutorial on October 28. We will cover everything from fundamentals and emerging approaches with plenty of hands on demos and exercises. See you in Budapest!

Post attachment