←
BerkeleyNLP136 posts
BerkeleyNLP avatar

BerkeleyNLP

@BerkeleyNLP

We work on natural language processing, machine learning, linguistics, and deep learning. PIs: Dan Klein, @alsuhr, @sewon__min

◷ Joined September 2019▧ 5 Photos and videos
Sewon Min avatar
⟳ BerkeleyNLP retweeted
Sewon Min@sewon__min·Nov 11

Model competition is heating up, and distillation is 🔥🔥 Distillation detection is hard, so watermarking can help make it detectable. But are existing watermarks robust? We find two ways to break them: distill a model, then distill that model again; or distill only selected tokens that matter. Then what do we do? We introduce SubWM, which outperforms prior methods and is more robust to laundering.

Excited to present our new work on distillation detection: Subliminal Watermarks (SubWM). SubWM is much more robust compared to existing distillation detectors, is able to distinguish different watermarking keys, and also works reliably for out-of-domain prompts! Blog: subliminal-watermarks.github… 🧵

0:24
Sewon Min avatar
⟳ BerkeleyNLP retweeted
Sewon Min@sewon__min·Nov 3

"No OCR, read text natively from images" continues! We showed this with deep research agents: it not only improves performance but also reduces tool calls! Post-training to work with page images gives a big boost - likely because these open-weight models haven't been trained to work with page images yet? Exciting work led by @Hughie_Hu43 !

(1/7) 🚀 Introducing DR Pixel, the first open-source deep research agent that searches and reasons directly over page images, no OCR. On 1.06M pages, our trained 27B model matches GPT-5.4. 📄 arxiv.org/abs/2611.03097 Joint work from @BerkeleyNLP, @rtnldynamics!

0:47
Tiansheng Hu avatar
⟳ BerkeleyNLP retweeted
Tiansheng Hu@Hughie_Hu43·Nov 3

(1/7) 🚀 Introducing DR Pixel, the first open-source deep research agent that searches and reasons directly over page images, no OCR. On 1.06M pages, our trained 27B model matches GPT-5.4. 📄 arxiv.org/abs/2611.03097 Joint work from @BerkeleyNLP, @rtnldynamics!

0:47
Kayo Yin avatar
⟳ BerkeleyNLP retweeted
Kayo Yin@kayo_yin·Oct 5

I'm at #colm2026 this week! 🌉 Lately I'm thinking about what's missing in harmful AI manipulation research and governance (whitepaper soon). Also on the job market for AI safety research roles, esp. threat modeling, strategy, evals, technical governance... DMs open :)

Post attachment
Sewon Min avatar
⟳ BerkeleyNLP retweeted
Sewon Min@sewon__min·Sep 28

Long-context LMs seem great at tasks like RULER, but what about tasks like "find every contradiction in the given corpus"? - Performance suffers, especially as the input corpus scales (even with in-domain fine-tuning, not just zero-shot) - It changes a lot of core modeling decisions, e.g., whether to use hybrid models, whether to use block sparse attention TL;DR: if you are a model developer, include our CTC-Bench in your evals! ✨

Could long-context architectures “find all contradictions” in a science literature? Not yet! 🧵 We study a new class of "high-complexity” tasks whose difficulty scales quadratically with corpus size (as opposed to linearly), reversing common LCLM decisions! (block-sparse attention, hybrid models…)

Post attachment
Prasann Singhal avatar
⟳ BerkeleyNLP retweeted
Prasann Singhal@prasann_singhal·Sep 28

Could long-context architectures “find all contradictions” in a science literature? Not yet! 🧵 We study a new class of "high-complexity” tasks whose difficulty scales quadratically with corpus size (as opposed to linearly), reversing common LCLM decisions! (block-sparse attention, hybrid models…)

Post attachment
Sewon Min avatar
⟳ BerkeleyNLP retweeted
Sewon Min@sewon__min·Jul 10

I'll be speaking at the EMM-QA Workshop tomorrow (Saturday)! #icml2026 Come say hi and hear about PixelRAG -- doing RAG but without text (?) 👀 Don't miss @andylizf's poster as well🙌 pixelrag.ai arxiv.org/abs/2606.28344

🚀The ICML 2026 EMM-QA Workshop is happening tomorrow!🚀 Join us at 8:00 AM in ASEM Ballroom 201 at COEX. The program features four excellent invited talks, paper presentations, shared challenge highlights, and the live AI QA competition! Program: qanta-org.github.io/competit…

Post attachment
Aly Lidayan avatar
⟳ BerkeleyNLP retweeted
Aly Lidayan@a_lidayan·Jun 11

As task horizons grow, LLM contexts can’t scale forever. Self-summarization enables concise, interpretable contexts but at a significant performance cost. Our solution: isolate and *supervise* the information content of summaries in the form of natural-language belief states 🧵

Post attachment
Sewon Min avatar
⟳ BerkeleyNLP retweeted
Sewon Min@sewon__min·Jun 11

One day I tried tracing all of Olmo's dependencies manually. A few hours later, I realized I can't do it and gave up. Then @sadhikesaven and @CoderBak ModSleuth 🔥 Turns out Olmo and Nemotron have hundreds of dependencies that are super deep, recursive, and not easily visible. I'm glad I gave up early 😅 Spoiler: I thought this would be a one-week Claude Code project. It was not. The hard part wasn't information extraction (which Claude Code is good at). The hard part was something much trickier. Check out the paper to learn more! (And yes, if a model release says it used Claude Code, ModSleuth will trace that too... which means the model depends on Claude Code, which has its own dependencies, and ModSleuth itself depends on Claude Code 🤯)

Ai2@allen_aiJun 11

LLMs are no longer created w/ human data alone. They rely on other models to generate & filter data, evaluate outputs, & guide dev work. So what is a modern LLM built on? Olmo 3 → 89 model + 183 dataset dependencies; Nemotron 3 → 273 + 560 We made ModSleuth to trace this. 🧵

Post attachment
Yichuan Wang avatar
⟳ BerkeleyNLP retweeted
Yichuan Wang@YichuanM·Jun 10

The web was never meant to be flattened into text. Yet most web RAG systems start by parsing HTML --- a complex and lossy process. 🔥 Introducing PixelRAG: the first RAG system that retrieves and reads 30M+ web pages as pixels. Instead of extracting text, PixelRAG retrieves screenshots and lets a VLM read them directly. PixelRAG not only preserves visual information, but also outperforms text-based RAG on text-only QA benchmarks by +18.1%. Why? (1) HTML-to-text conversion often discards layout, structure, tables, and other useful signals. (2) We continued pretraining a VLM on web page screenshots and turned it into a surprisingly strong visual retriever. (3) Recent VLMs are remarkably good at understanding web pages, often with better accuracy and token efficiency than text-only pipelines. Takeaway: HTML parsing may be one of the biggest self-inflicted bottlenecks in web RAG. Demo below 👇 Code: github.com/StarTrail-org/Pix… Paper: github.com/StarTrail-org/Pix… Playground: pixelrag.ai/

0:56
Sewon Min avatar
⟳ BerkeleyNLP retweeted
Sewon Min@sewon__min·Jun 10

Super excited about this work, led by @YichuanM and @andylizf ! It is possible to completely remove HTML parsing by directly retrieving and reading web screenshots through VLM. HTML parsing is a hidden bottleneck that causes significant complexity and information loss that nobody really pays attention to, and it's so exciting that VLM progress made it possible to remove it. Please check out this demo as well which is really cool!!! pixelrag.ai

The web was never meant to be flattened into text. Yet most web RAG systems start by parsing HTML --- a complex and lossy process. 🔥 Introducing PixelRAG: the first RAG system that retrieves and reads 30M+ web pages as pixels. Instead of extracting text, PixelRAG retrieves screenshots and lets a VLM read them directly. PixelRAG not only preserves visual information, but also outperforms text-based RAG on text-only QA benchmarks by +18.1%. Why? (1) HTML-to-text conversion often discards layout, structure, tables, and other useful signals. (2) We continued pretraining a VLM on web page screenshots and turned it into a surprisingly strong visual retriever. (3) Recent VLMs are remarkably good at understanding web pages, often with better accuracy and token efficiency than text-only pipelines. Takeaway: HTML parsing may be one of the biggest self-inflicted bottlenecks in web RAG. Demo below 👇 Code: github.com/StarTrail-org/Pix… Paper: github.com/StarTrail-org/Pix… Playground: pixelrag.ai/

0:56
Sewon Min avatar
⟳ BerkeleyNLP retweeted
Sewon Min@sewon__min·May 25

Really amazing results analyzing what's creative/novel vs. what's copied from Internet data, enabled by the amazing @liujc1998's Infini-gram! infini-gram.io This is also enabled in @allen_ai's OlmoTrace allenai.org/blog/olmotrace where anyone can find matching n-grams between LLM-generated text and its training data.

This from @TuhinChakr is brilliant. That prize winning story from Granta? Turns out it's just a bunch of random whole phrases taken directly from existing text on the internet. Tool allows you to trace those n-grams directly to their source, which is mostly random fanfiction. tuhinchakrabarty.substack.co…

Post attachment
Negar Arabzadeh avatar
⟳ BerkeleyNLP retweeted
Negar Arabzadeh@NegarEmpr·May 12

1/ Thrilled to introduce T³: a corpus for RAG over reasoning tasks, built from thinking traces. We show that surprisingly RAG can improve reasoning— with the right corpus. Rag with Transformed Thinking Traces T³ gain by up to 43.9% on AIME 2025-2026. 🔗 arxiv.org/abs/2605.03344 🧵

Post attachment
Ryan Yixiang Wang avatar
⟳ BerkeleyNLP retweeted
Ryan Yixiang Wang@RyanYixiang·May 8

8/ For reproducibility and to enable further study of modularity in MoEs, we’re releasing EMO, baselines, and code: Models: hf.co/collections/allenai/em… Blog: allenai.org/blog/emo Code: github.com/allenai/EMO Viz: emovisualization.netlify.app Shoutout to @AkshitaB93 @sewon__min for making this possible!

Sewon Min avatar
⟳ BerkeleyNLP retweeted
Sewon Min@sewon__min·May 8

Full details + results: arxiv.org/abs/2605.06663 Also check this out emovisualization.netlify.app: Our model specializes qualitatively differently (capability-level rather than lexical) -- this emerged naturally even though we didn't expose any domain prior!

Sewon Min avatar
⟳ BerkeleyNLP retweeted
Sewon Min@sewon__min·May 8

As MoEs grow larger and sparser, they become memory-bottlenecked. What if experts were actually composable - so you only keep the subset relevant to your task? We show that this doesn't emerge in standard MoEs (their training makes this hard), but you can pre-train MoEs to support this kind of modularity! I hope everyone sees the right figure from @RyanYixiang 's original post - I was so excited when I saw this result!!

MoEs are everywhere in frontier models, and they are deployed as a monolith system. But many applications only need a narrow slice of capabilities, e.g., math, code, biomedical, etc. So what if "modularity" is actually the missing opportunity for MoEs? Today, we're releasing EMO: an end-to-end pretrained MoE where modularity emerges naturally, enabling selective use of experts!

Post attachment