

✓Researcher @Microsoft | Prof @UWMadison (on leave) | babas of Inez Lily.


haven't had a model stay as my top model this long since gpt-4-0314 opus 5.5 is just an epic model

loopy loopy I told you loopmaxxers are the next paradigm. It's only up from here.

I have been telling this to students. I do sense that the pressure of publishing papers is still strong, and bean counting is easier for students new to research.

Some more evidence in favor: in my AISTATS and ICLR batches, I will likely not recommend a single paper to be accepted, because none are at the historical quality bar. I have been saying that conference papers and refereeing are fine for years. Now, they are not fine.

ayyyoo they did our distillation thing for language models! Literally, with the equivalent of mixup :) One thing it seems they had to do, is to only do this for part of the distillation, and then finish on "pure" examples. We probably should have also done that back then!


Really cool to see more results like that This makes sense intuitively. Models make mistakes when generating autoregressively and with noising they learn to auto-correct during pretraining Hypothesis: their own generations become less OOD (RL fixes a similar issue as well) (Also connections to information bottlenecks etc)

LLM pretraining has run on single-epoch text, bits per byte and iso-FLOP curves for years. visual generation still memorizes ImageNet for tens to hundreds of epochs and pushes FID down with CFG, with no strong signal those tricks transfer to real text-to-image. new blog: 🎨 scaling generative models on GPIC, 100M permissive text-image pairs. training pixel-space JiT from 3.6e18 to 2.3e20 FLOPs: 1) held-out loss is our main metric, and compute-optimal TPP sits at about 50 image tokens per image parameter 2) EMA is not necessary: a proper learning-rate schedule beats EMA on held-out loss at every budget by 0.4% to 1.0% 3) we use CFG = 1 (only the conditional branch) for every result, and call on everyone to drop CFG: it saves compute and leaves no room to hack the metric all code, configs and sweep data are open 🧵 blog: wenhaochai.com/blogs/scaling…

Meta-discussion about the beans and the machine are strongly discouraged due to the risk of social destabilization.

Ok i feel better today, research is back on, intellectual dread is gone.

💯💯 I'm personally comfortable saying it precisely because it's disingenuous to let PhD (or undergrad/MS) students be misled by things which stick to the old guard when the closed doors conversations are already moving differently. It's worth pointing out that I don't like the way things are evolving because it's inherently becoming more nepotistic in that access based on who you know or recommends you starts to matter disproportionately more and I would like a way to return to meritocracy (I know people will disagree with me on this last bit)

Super excited to come out of stealth after the weekend, with a bunch of Monday hot takes.

i’m increasingly of the opinion that almost everyone agrees with this, but a big fraction of people aren’t comfortable saying it in public, because of what it implies for the bean counting mechanism the whole machine is built on (for awards/promo/hiring/etc).

Summarizes my concerns when someone says "We have formalized the proof in Lean" (or any formalization efforts of anything, not just proofs)

I am increasingly of the opinion that the academic paper is no longer a good medium for disseminating CS research. I think we should move to (1) blog posts and (2) books.