mmissoldtw ←
Haoxiang Sun32 posts
Haoxiang Sun avatar

Haoxiang Sun

@CoderBak

cs & fin @RUC1937

◷ Joined February 2025▧ 7 Photos and videos
Haoxiang Sun avatar
Haoxiang Sun@CoderBak·Jun 11

Today, LLMs are no longer built from human data alone. They rely on other LLMs to generate training data, filter corpora, evaluate outputs, provide rewards, and guide development decisions. So how many models and datasets is a modern LLM built on? • OLMo 3 → 89 model + 183 dataset dependencies • Nemotron 3 → 273 model + 560 dataset dependencies How did we find it out? We built ModSleuth. 🧵

Post attachment
Ai2@allen_aiJun 11

LLMs are no longer created w/ human data alone. They rely on other models to generate & filter data, evaluate outputs, & guide dev work. So what is a modern LLM built on? Olmo 3 → 89 model + 183 dataset dependencies; Nemotron 3 → 273 + 560 We made ModSleuth to trace this. 🧵

Post attachment
Dayoon Ko avatar
⟳ Haoxiang Sun retweeted
Dayoon Ko@dayoon12161·10h

Wait, what was that paper I saw last week with the pink and blue Pareto curves? 🤔 Keep screenshots of your screen, and you can find it again, right on your laptop. 💻 🔥 Introducing 𝑺𝒕𝒓𝒆𝒂𝒎𝑬𝒎𝒃𝒆𝒅: real-time visual indexing of your screen history, fully on-device. ⚖️ The latency–accuracy trade-off OCR text index: fast, but loses what only pixels show Screenshot index: accurate, but 2.3s per screenshot on a laptop GPU 𝑺𝒕𝒓𝒆𝒂𝒎𝑬𝒎𝒃𝒆𝒅 gets both: 𝘂𝗽 𝘁𝗼 𝟯.𝟳× 𝗳𝗮𝘀𝘁𝗲𝗿 than full-screenshot encoding at matched accuracy, and +𝟲.𝟯 𝗥𝗲𝗰𝗮𝗹𝗹@𝟭𝟬 over OCR. 💫 How? (1) 𝗦𝗰𝗿𝗲𝗲𝗻𝘀 𝗯𝗮𝗿𝗲𝗹𝘆 𝗰𝗵𝗮𝗻𝗴𝗲. Often, only a handful of blocks differ. (2) 𝗦𝗼 𝘄𝗲 𝘀𝘁𝗼𝗽 𝗿𝗲𝗰𝗼𝗺𝗽𝘂𝘁𝗶𝗻𝗴 𝘁𝗵𝗲𝗺. Spatial token pruning drops blocks that repeat a neighbor; temporal KV reuse recomputes only what changed: 57–71% less compute. (3) 𝗔 𝘁𝗶𝗻𝘆 𝗮𝗱𝗮𝗽𝘁𝗲𝗿 keeps the embedding faithful to the full screenshot. 📝 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆: keeping your entire screen history as pixels is now practical on a laptop. Next: fully local visual agents with memory. You can try it on your laptop right now: 𝙥𝙞𝙥 𝙞𝙣𝙨𝙩𝙖𝙡𝙡 𝙨𝙩𝙧𝙚𝙖𝙢𝙚𝙢𝙗𝙚𝙙 💻 Code: github.com/StarTrail-org/Str… 📄 Paper: arxiv.org/abs/2611.18516

0:47
Jenna Russell avatar
⟳ Haoxiang Sun retweeted
Jenna Russell@jennajrussell·Oct 1

How much is an AI token worth? We fit scaling laws to find out! In August 2026, Pangram labels 31% of FineWeb-filtered web tokens as AI-generated, up from 10% in June 2024. We pretrained 800 LMs (19.9M–973M params) on mixes of human and AI web text to measure what it does to pretraining 🧵

Post attachment
Prasann Singhal avatar
⟳ Haoxiang Sun retweeted
Prasann Singhal@prasann_singhal·Sep 28

Could long-context architectures “find all contradictions” in a science literature? Not yet! 🧵 We study a new class of "high-complexity” tasks whose difficulty scales quadratically with corpus size (as opposed to linearly), reversing common LCLM decisions! (block-sparse attention, hybrid models…)

Post attachment
Sanjay Adhikesaven avatar
⟳ Haoxiang Sun retweeted
Sanjay Adhikesaven@sadhikesaven·Jun 11

Today, LLMs are no longer built from human data alone. They rely on other LLMs to generate training data, filter corpora, evaluate outputs, provide rewards, and guide development decisions. So how many models and datasets is a modern LLM built on? • OLMo 3 → 89 model + 183 dataset dependencies • Nemotron 3 → 273 model + 560 dataset dependencies

How did we find it out? We built ModSleuth. 🧵

Post attachment
Ai2@allen_aiJun 11

LLMs are no longer created w/ human data alone. They rely on other models to generate & filter data, evaluate outputs, & guide dev work. So what is a modern LLM built on? Olmo 3 → 89 model + 183 dataset dependencies; Nemotron 3 → 273 + 560 We made ModSleuth to trace this. 🧵

Post attachment
Haoxiang Sun avatar
Haoxiang Sun@CoderBak·Jun 11

Demo: modsleuth.cal-data-audit.org Code: github.com/cal-data-audit/mo… Paper: arxiv.org/abs/2606.12385 Built at UC Berkeley EECS, BAIR, and Berkeley NLP. @Berkeley_EECS @berkeley_ai @BerkeleyNLP This project was co-led with @sadhikesaven - I couldn’t have pulled this off without his incredible partnership. Also a huge thanks to our advisor @sewon__min for her invaluable guidance and support throughout the whole process! 🙌 We are also incredibly grateful to Kyle Lo, Noah Smith, Hanna Hajishirzi, Rishi Bommasani, and the wider SM group & Ai2 members for their constructive feedback. We’d love to hear your thoughts! Let’s build a more transparent LLM ecosystem together.

Haoxiang Sun avatar
Haoxiang Sun@CoderBak·Jun 11

4/ Across 4 open-source releases, ModSleuth recovers 1,060 source-verified dependencies, with chains up to 8 hops deep. This graph also surfaces findings that are hard to find manually: • License-relevant multi-hop paths • Train-evaluation coupling • Mismatches between papers, cards, and code

Post attachment
Haoxiang Sun avatar
Haoxiang Sun@CoderBak·Jun 11

3/ One surprising lesson: With Claude Code (which ModSleuth is built on), information extraction is no longer the main bottleneck. The hard part is semantic and representational: • What actually counts as a dependency? • When do different names refer to the same artifact? • How do you reconcile versions, model families, development stages, and repositories? The challenge is no longer finding information—it's making sense of it. Check out the paper to see how we tackle these problems.

Haoxiang Sun avatar
Haoxiang Sun@CoderBak·Jun 11

2/ Model-to-model influence is now so diverse, complex, and recursive that it far outpaces humans' ability to trace. So we built ModSleuth: an agentic system that automatically reconstructs a model's dependency graph. It reads papers, model cards, dataset cards, code, configs, and upstream artifacts, then pieces together a model's "family tree." Some dependency chains go 8 hops deep.

Shuo Yang avatar
⟳ Haoxiang Sun retweeted
Shuo Yang@Andy_ShuoYang·May 27

Flash-KMeans was only the beginning. Today, from the Flash-KMeans team, we are releasing FlashLib — a GPU library for fast, predictable, agent-ready classical ML operators. Up to 26× on KMeans, 19× on KNN, 40× on HDBSCAN, 208× on TruncatedSVD, 47× on PCA, 147× on exact t-SNE, and 49× on MultinomialNB over state-of-the-art (cuML). Blog: flashml-org.github.io/ Code: github.com/FlashML-org/flash…

0:20
Tatsunori Hashimoto avatar
⟳ Haoxiang Sun retweeted
Tatsunori Hashimoto@tatsu_hashimoto·May 21

Some new results I found surprising that I’m tweeting for Chris (who isnt on here). With enough compute, the best data filter for LMs (on DCLM) might be no filter. Why? Large models can tolerate a surprising amount of nominally 'low quality' data, and can sometimes even benefit.

Post attachment
Ryan Yixiang Wang avatar
⟳ Haoxiang Sun retweeted
Ryan Yixiang Wang@RyanYixiang·May 8

MoEs are everywhere in frontier models, and they are deployed as a monolith system. But many applications only need a narrow slice of capabilities, e.g., math, code, biomedical, etc. So what if "modularity" is actually the missing opportunity for MoEs? Today, we're releasing EMO: an end-to-end pretrained MoE where modularity emerges naturally, enabling selective use of experts!

Post attachment
Ai2@allen_aiMay 8

Today we’re releasing EMO, a new mixture-of-experts (MoE) model trained so modular structure emerges directly from data without human-defined priors. EMO can use a small subset of its experts for a given task while keeping near full-model performance. 🧵

Post attachment
Zihan "Zenus" Wang avatar
⟳ Haoxiang Sun retweeted
Zihan "Zenus" Wang@wzenus·May 3

Failed Agent experiments can be publishable too🤯 Introducing ICML 2026 Workshop Failure Modes in Agentic AI! We welcome negative results, failed rollouts, debugging traces, reproducible failure cases, and analysis of why agents break. 📍FAGEN @ ICML 2026 🗓 Submission deadline: May 8 11:59 PM AOE 🗓 Notification: May 15 🔗fmai-workshop.github.io Find it. Reproduce it. Trace it. Fix it. We also welcome relevant ICML submissions, especially papers with strong insights that may not have found the right home in the main track!

Post attachment
DeepSeek avatar
⟳ Haoxiang Sun retweeted
DeepSeek@deepseek_ai·Apr 24

🚀 DeepSeek-V4 Preview is officially live & open-sourced! Welcome to the era of cost-effective 1M context length. 🔹 DeepSeek-V4-Pro: 1.6T total / 49B active params. Performance rivaling the world's top closed-source models. 🔹 DeepSeek-V4-Flash: 284B total / 13B active params. Your fast, efficient, and economical choice. Try it now at chat.deepseek.com via Expert Mode / Instant Mode. API is updated & available today! 📄 Tech Report: huggingface.co/deepseek-ai/D… 🤗 Open Weights: huggingface.co/collections/d… 1/n

Post attachment
Sanjay Adhikesaven avatar
⟳ Haoxiang Sun retweeted
Sanjay Adhikesaven@sadhikesaven·Apr 20

Imagine you fully post-trained "YourModel v1". Then, you've got better data — math, code, tool use, safety — and you want to improve it. Today, that usually means retraining the whole model. But what if new data could be added modularly, with a fixed cost each time?

Post attachment
Ai2@allen_aiApr 20

Last year, we introduced FlexOlmo, a novel way to train parts of a model independently then combine them later. BAR builds on that idea for a harder problem: how to keep improving a model without having to retrain each time. 🧵

Post attachment