←
Dimitris Papailiopoulos12,114 posts
Dimitris Papailiopoulos avatar✓

Dimitris Papailiopoulos ✓

@DimitrisPapail

Researcher @Microsoft | Prof @UWMadison (on leave) | babas of Inez Lily.

◷ Joined May 2012▧ 1,216 Photos and videos
Lisan al Gaib avatar
⟳ Dimitris Papailiopoulos retweeted
Lisan al Gaib@scaling01·18h

loopy loopy I told you loopmaxxers are the next paradigm. It's only up from here.

Post attachment
Chenhao Tan avatar
⟳ Dimitris Papailiopoulos retweeted
Chenhao Tan@ChenhaoTan·Nov 16

I have been telling this to students. I do sense that the pressure of publishing papers is still strong, and bean counting is easier for students new to research.

Alexander Terenin avatar
⟳ Dimitris Papailiopoulos retweeted
Alexander Terenin@avt_im·Nov 16

Some more evidence in favor: in my AISTATS and ICLR batches, I will likely not recommend a single paper to be accepted, because none are at the historical quality bar. I have been saying that conference papers and refereeing are fine for years. Now, they are not fine.

Lucas Beyer (bl16) avatar
⟳ Dimitris Papailiopoulos retweeted
Lucas Beyer (bl16)@giffmana·Nov 16

ayyyoo they did our distillation thing for language models! Literally, with the equivalent of mixup :) One thing it seems they had to do, is to only do this for part of the distillation, and then finish on "pure" examples. We probably should have also done that back then!

Post attachmentPost attachment
Lazarz avatar
⟳ Dimitris Papailiopoulos retweeted
Lazarz@Laz4rz·Nov 16

noising solves everything as always

Post attachment
Andreas Kirsch 🇺🇦 avatar
⟳ Dimitris Papailiopoulos retweeted
Andreas Kirsch 🇺🇦@BlackHC·Nov 16

Really cool to see more results like that This makes sense intuitively. Models make mistakes when generating autoregressively and with noising they learn to auto-correct during pretraining Hypothesis: their own generations become less OOD (RL fixes a similar issue as well) (Also connections to information bottlenecks etc)

Post attachment
Wenhao Chai avatar
⟳ Dimitris Papailiopoulos retweeted
Wenhao Chai@wenhaocha1·Nov 16

LLM pretraining has run on single-epoch text, bits per byte and iso-FLOP curves for years. visual generation still memorizes ImageNet for tens to hundreds of epochs and pushes FID down with CFG, with no strong signal those tricks transfer to real text-to-image. new blog: 🎨 scaling generative models on GPIC, 100M permissive text-image pairs. training pixel-space JiT from 3.6e18 to 2.3e20 FLOPs: 1) held-out loss is our main metric, and compute-optimal TPP sits at about 50 image tokens per image parameter 2) EMA is not necessary: a proper learning-rate schedule beats EMA on held-out loss at every budget by 0.4% to 1.0% 3) we use CFG = 1 (only the conditional branch) for every result, and call on everyone to drop CFG: it saves compute and leaves no room to hack the metric all code, configs and sweep data are open 🧵 blog: wenhaochai.com/blogs/scaling…

0:23
Matthew Leavitt avatar
⟳ Dimitris Papailiopoulos retweeted
Matthew Leavitt@leavittron·Nov 16

Meta-discussion about the beans and the machine are strongly discouraged due to the risk of social destabilization.

Tarun Kathuria avatar
⟳ Dimitris Papailiopoulos retweeted
Tarun Kathuria@_TarunKathuria·Nov 16

💯💯 I'm personally comfortable saying it precisely because it's disingenuous to let PhD (or undergrad/MS) students be misled by things which stick to the old guard when the closed doors conversations are already moving differently. It's worth pointing out that I don't like the way things are evolving because it's inherently becoming more nepotistic in that access based on who you know or recommends you starts to matter disproportionately more and I would like a way to return to meritocracy (I know people will disagree with me on this last bit)

Dimitris Papailiopoulos avatar
Dimitris Papailiopoulos@DimitrisPapail·Nov 16

i’m increasingly of the opinion that almost everyone agrees with this, but a big fraction of people aren’t comfortable saying it in public, because of what it implies for the bean counting mechanism the whole machine is built on (for awards/promo/hiring/etc).

Dimitris Papailiopoulos avatar
Dimitris Papailiopoulos@DimitrisPapail·Nov 16

Summarizes my concerns when someone says "We have formalized the proof in Lean" (or any formalization efforts of anything, not just proofs)

0:54
Aws Albarghouthi avatar
⟳ Dimitris Papailiopoulos retweeted
Aws Albarghouthi@awsTO·Nov 16

I am increasingly of the opinion that the academic paper is no longer a good medium for disseminating CS research. I think we should move to (1) blog posts and (2) books.