←
Phillip Rust51 posts
Phillip Rust avatar

Phillip Rust

@rust_phillip

Research Scientist @AIatMeta (FAIR) • PhD @coastalcph

◷ Joined July 2020▧ 7 Photos and videos
Phillip Rust avatar
Phillip Rust@rust_phillip·24 Jan 2023

Happy to share our paper on language modelling with pixels has been accepted to ICLR‘23 (notable-top-5% / oral) 🎉. Big thanks and congrats to Team-PIXEL @jonasflotz @ebugliarello @esalesk @mdlhx @delliott and looking forward to presenting in Kigali! 🌍 #ICLR2023

Tired of tokenizers/subwords? Check out PIXEL, a new language model that processes written text as images📸 “Language Modelling with Pixels” 📄 arxiv.org/abs/2207.06991 🧑‍💻github.com/xplip/pixel 🤖huggingface.co/Team-PIXEL/pi… by @rust_phillip @jonasflotz me @esalesk @mdlhx @delliott

Post attachment
Dongyan Lin avatar
⟳ Phillip Rust retweeted
Dongyan Lin@dongyanl1n·Jun 26

(1/n) Excited to share our new paper from @AIatMeta FAIR: "EgoBabyVLM: Benchmarking Cross-Modal Learning from Naturalistic Egocentric Video Data" 👶Human infants learn language from sparse, noisy multimodal input. Today's VLMs can't. We built a benchmark + challenge to close that gap. w/ @rust_phillip, @angelvillar96, @juanmiguelpino, @mcxfrank, Emmanuel Dupoux and many other amazing collaborators. 🧵

Post attachment
Yi Lin Sung avatar
⟳ Phillip Rust retweeted
Yi Lin Sung@yilin_sung·23 Oct 2025

Tough week! I also got impacted less than 3 months after joining. Ironically, I just landed some new RL infra features the day before. Life moves on. My past work spans RL, PEFT, Quantization, and Multimodal LLMs. If your team is working on these areas, I’d love to connect.

Meta has gone crazy on the squid game! Many new PhD NGs are deactivated today (I am also impacted🥲 happy to chat)

Jinpeng Wang avatar
⟳ Phillip Rust retweeted
Jinpeng Wang@awinyimgprocess·22 Oct 2025

Humans see text — but LLMs don’t. I wrote a short blog post exploring how models can perceive text visually rather than tokenize it: 🔗 csu-jpg.github.io/Blog/peopl… From PIXEL, CLIPPO, VisInContext, VIST to DeepSeek-OCR, this is a quick story of how vision-centric modeling is changing how machines read, and a reflection on some of our own small efforts in the past two years.

Phillip Rust avatar
Phillip Rust@rust_phillip·12 Aug 2024

I will be presenting this work in-person at ACL🇹🇭 this week. Drop by if you'd like to chat! Oral: Today (Monday) 16:30 Poster: Tuesday (Tomorrow) 10:30 - 12:00

Introducing “Towards Privacy-Aware Sign Language Translation at Scale” We leverage self-supervised pretraining on anonymized videos, achieving SOTA ASL-to-English translation performance while mitigating risks arising from biometric data. 📄: arxiv.org/abs/2402.09611 🧵(1/9)

Post attachment
Phillip Rust avatar
Phillip Rust@rust_phillip·23 Feb 2024

For more experiments and all the details, check out our arXiv preprint linked above. We are working on releasing our code and data, so stay tuned! 👨‍💻 🧵(8/9)

Phillip Rust avatar
Phillip Rust@rust_phillip·23 Feb 2024

We also highlight the importance of pretraining on longer video clips to learn long-range spatio-temporal dependencies 🎬➡️🧠. Even when controlling for the number of video tokens seen, we observe a large boost in performance by scaling from 16 to 128 frames 🚀. 🧵(7/9)

Post attachment
Phillip Rust avatar
Phillip Rust@rust_phillip·23 Feb 2024

Face blurring incurs a loss of linguistic information in sign languages, leading to performance degradation. We show that such information, when lost during anonymized pretraining, can largely be recovered during finetuning. An effective privacy-performance trade-off ⚖️! 🧵(6/9)

Post attachment
Phillip Rust avatar
Phillip Rust@rust_phillip·23 Feb 2024

Our best models outperform the prior SOTA for ASL-to-English translation performance on How2Sign by over 3 BLEU in both the finetuned and zero-shot settings 🥇. 🧵(5/9)

Post attachment
Phillip Rust avatar
Phillip Rust@rust_phillip·23 Feb 2024

🌐 Optionally, an intermediate language-supervised pretraining (LSP) objective can help bridge the modality gap between sign language video inputs and text outputs. 🧵(4/9)

Post attachment
Phillip Rust avatar
Phillip Rust@rust_phillip·23 Feb 2024

Our method, SSVP-SLT, consists of: 🎥 Self-supervised video pretraining (SSVP) on anonymized, unannotated videos to learn high-quality continuous sign language representations. 🎯 Supervised finetuning on a curated SLT dataset to learn translation-specific information. 🧵(3/9)

Post attachment
Phillip Rust avatar
Phillip Rust@rust_phillip·23 Feb 2024

Training data scarcity and privacy risks are huge issues in sign language translation (SLT). Our approach is designed to be 🚀 scalable (by enabling training on unlabeled data) 🎭 privacy-aware (through anonymization) 🧵(2/9)

Post attachment
Phillip Rust avatar
Phillip Rust@rust_phillip·23 Feb 2024

Introducing “Towards Privacy-Aware Sign Language Translation at Scale” We leverage self-supervised pretraining on anonymized videos, achieving SOTA ASL-to-English translation performance while mitigating risks arising from biometric data. 📄: arxiv.org/abs/2402.09611 🧵(1/9)

Post attachment