←
Ethan12,719 posts
Ethan avatar

Ethan

@torchcompiled

trying to feel the magic. prev: global research lead at @canva | cofounder at @leonardoai

◷ Joined April 2022▧ 2,395 Photos and videos
Ethan avatar
Ethan@torchcompiled·14 Feb 2025

personally I feel like the inflection point was early 2022. The sweet spot where clip-guided diffusion was just taking off, forcing unconditional models to be conditional through strange patchwork of CLIP evaluating slices of the canvas at a time. It was like improv, always trying to riff of mistakes and sitting right at the fine line between interesting and incoherent.

Post attachmentPost attachmentPost attachmentPost attachment
EPROM@eprombeats14 Feb 2025

Image synthesis used to look so good. These are from 2021. I feel like this was an inflection point, and the space has metastasized into something abhorrent today (Grok, etc). Even with no legible representational forms, there was so much possibility in these images.

Post attachmentPost attachmentPost attachmentPost attachment
Ethan avatar
Ethan@torchcompiled·2h

Google has proven that available capital and having the whole worlds data are the strongest predictors of success in ai progress Especially when AI itself now does its own development compute and data increasingly become the only bottleneck

Hakm@hakmgptNov 17

Google has proven that when you’re months or even a year behind, the solution isn’t to distill models. and they proved that good research takes time and I have high hopes for SSI as well :)

Ethan avatar
Ethan@torchcompiled·2h

I think he is unfortunately right. There’s gotta be some named game theory scenario for this, > lab that wins RSI and zooms past others develops a lot of power and may implement their vision of the future > but many also recognize the extreme risks > if one believes the risks/outcomes aren’t worth it, they can pause > for this to be worthwhile, all players must pause, now 100s or 1000s > many irrelevant but if they grow while others have paused, they may be the one to reach the critical mass point Thus, if you believe it isn’t worth the risk, it’s insufficient to pause alone, everyone needs to agree. So then if you can’t build that coalition, you go back to playing the game, in hopes you can avoid risks as being the one to win RSI, believing yourself that you will do it in a safer manner than others Is this a reasonable belief? Maybe so, or maybe it’s delusional. It doesn’t really matter. Whether one pursues RSI for their own power or altruistic motivations, if you believe either you’d prefer yourself to have power as opposed to another group, or that you will proceed more safely or altruistically, you are incentivized to still play this game going full speed, outpacing others The irony is that this high speed development rat race and incentive to “move fast and break things” likely only increases risk

fam idk about Dario sometimes

Ethan avatar
Ethan@torchcompiled·Nov 17

Are looped models necessarily less interpretable? Feels like a common point made is that latent-thinking models would be less interpretable, but 1. I figure we can drain probes/output heads all the same 2. Even those that think in tokens don’t always reflect what goes on in activation space

Um What's OpenAI's plan to keep their models monitorable?

Ethan avatar
Ethan@torchcompiled·Nov 17

Possible explanation: it may be a genuinely strong model but may be extremely particular about right prompts and/or overly specialized in frontier problems. If it doesn’t generalize to common user cases and has crap UX… a tool is only as good as you can reasonably control it.

This is exactly the issues I had with Astra and Sol GPT SOL 6 was better than 6.1 and I found Astra 6 to be worse than 6.1 So I cancelled my Codex subscription and then went all in on Anthropic. It simply is just better Found it super odd Astra had saturated most benchmarks but in my own testing was demonstrably worse than Fable and even Opus 4.5 I will give both of these new 6.2 models a shot but expecting more of the same

Ethan avatar
Ethan@torchcompiled·Nov 17

i'm sure its been derived before but had claude work it out: grad estimator noise scales with b1 as (×(1−β)/(1+β)) and batch as (×1/B) increasing β₁ yields same relative reductions, but less absolute noise reduction at large But higher b1 also lags our momentum, so the best β₁ should fall as batch grows and rise at lower LR, roughly 1−β* ∝ (η²B)^(1/3). caveat: this only covers momentum as a noise filter. Momentum also speeds up progress on badly conditioned problems, which can favor high β even at full batch.

Post attachmentPost attachment

hypothesis: larger adam b1 (and probably muon) may benefit moreso in the small batch regime, especially with lower lr, via greater reductions in noise in gradient estimation, but with larger batches, the gains may be diminishing and you're paying for lag in estimator with less benefit in variance reduction

Ethan avatar
Ethan@torchcompiled·Nov 17

hypothesis: larger adam b1 (and probably muon) may benefit moreso in the small batch regime, especially with lower lr, via greater reductions in noise in gradient estimation, but with larger batches, the gains may be diminishing and you're paying for lag in estimator with less benefit in variance reduction

Ethan avatar
Ethan@torchcompiled·Nov 15

Proposing these kinds of image references is a wildly important step for creative and subjective outputs. The user should have a chance to explore a bit and dial in a sense of what they’re looking for before gambling repeatedly

AI interfaces nowadays are still pretty much turn-based. In my PhD thesis, I thought about what interaction would look like if we could go beyond taking turns. One example is intra-turn interaction:

0:58
Ethan avatar
Ethan@torchcompiled·Nov 1

Naturally lends itself to surprise, often related to entropy. But a lot of high entropy points are nonsense, perhaps better making epiplexity a better metric for curiousity and novelty driven learning, discovering unknowns that are interesting

Dopamine does not spike highest for guaranteed rewards. It spikes highest at reward prediction error. If you want to have fun in life, you should stop avoiding risk and uncertainty.

Ethan avatar
Ethan@torchcompiled·Oct 31

Will be interesting if the bottleneck lies at the side of human thought. Personally I’m not that compelled by this paper, but even if we could thought-to-text it’s hard to imagine us outputting words we need to think about at speed of an LLM

Post attachment

Neuralink’s first product is called Telepathy. It enables high-speed communication with computers or humans simply by thinking. Output bandwidth will vastly exceed the fastest human speech or typing in a few years. Read that sentence again.

Ethan avatar
Ethan@torchcompiled·Oct 29

And then evolutionary forces create meta RL objectives like appealing to a mate. The models are gonna start looksmaxxing

We should do evolution on models. RL is basically evolution under asexual reproduction. We should allow the models to sexually reproduce. We should allow the models to have sex…

Ethan avatar
Ethan@torchcompiled·Oct 29

This was a forgotten technique from days of BERT I believe, I don’t know how it compares to naive pretraining (weights might already be in a basin, but maybe inject some noise?) but it’s quite economical. Width growth should also be considered

my Claude found that Qwen3.5-27B was grown from Qwen3.5-9B (layer looping) qt: independent discovery found by grok

Post attachment
Ethan avatar
Ethan@torchcompiled·Oct 27

Hypothesis: Imogen coders with convolutional blocks might perform well on AI image detection given that often that tells lie in small texture artifacts

anecdata #423,102 why you should trust pangram

Post attachmentPost attachment
Ethan avatar
Ethan@torchcompiled·Oct 25

One of the places that this first came to mind was the alpha geometry paper which showed that models could solve complex problems but in a brute force manner of many steps when humans often found elegant insights, encouraging brevity may help

Ethan avatar
Ethan@torchcompiled·Oct 25

The part on “length penalty” has long interested me from perspectives of Kolmogorov/solomonoff encouraging simplicity. Also the paper on typical sampling, arguing humans optimize their speech for information transfer as opposed to choosing tokens of high reward/probability

lifcc@mylifccOct 24

为什么现在的顶尖大模型说话越来越“拗口、难懂”了? NYU 和 UT Austin 刚刚发了一篇非常扎心的心理语言学论文,直接扒开了现代大模型“不好好说人话”的底牌。 研究人员把 Claude Sonnet 5 / Opus 5.5、GPT-5.6、GLM-5、Kimi K2.5 等 9 款前沿模型拉出来做全方位句法解剖,发现了 3 个反直觉真相: 1️⃣ 顶尖 AI 正在滥用 6 种“高认知负荷”句式 相比真实人类写作和同义回译文本,前沿模型极度偏爱:省略 that 的宾语从句、多重名词硬堆叠(如 reader-patience budget)、名词强行作动词(如 calendar it / footnote it)、无生命主语配动作动词,以及超长句法依存距离。这些特征在心理语言学里全是消耗人类短期记忆的“阅读杀手”。 2️⃣ 根源在于“人类大脑”与“推理后训练”的错位 人类语言遵循“均匀信息密度(UID)”——为了照顾听众有限的工作记忆,我们会本能地加入过渡词把信息摊平。而现代大模型经过大量数学、代码和推理(RLVR)后训练,只追求在有限窗口内最大化“信息密度”。 3️⃣ 一个简单的长度惩罚,就能复刻这种“反人类文风” 团队在微调实验中加入长度惩罚(Length Penalty),逼模型少用 Token,结果模型立刻学会了砍掉连接词、堆叠名词,瞬间变成那股熟悉的“高密度 AI 味”。 当模型为了解题效率疯狂压缩 Token 时,你觉得未来的 AI 应该区分“高密度推理模式”和“人类友好沟通模式”吗?

Post attachment
Ethan avatar
Ethan@torchcompiled·Oct 16

have generally addressed this in an unstructured way, handing off original docs on motivations and relevant theories, masking anything around a specific solution direction, this blank slate helps a bunch escaping getting glued to current topic

As agents perform AutoResearch they get fixated on their old ideas and do only incremental innovations. We introduce a simple context management trick that forces the agents to forget everything they have done except the best solution they have found so far. This simple modification in context management allows the agents to get out of their idea basins and come up with more novel solutions. Closely related to the work 'Fork and Flush' and 'Idea-basins' that @DimitrisPapail posted a few days ago.