
We Shall Poast Good Tweets Here Viral, If We Can Obscure, If We Must But Always Good Tweets

Philosopher at @alphaschool, Principal at @montessorium, husband to @Gena_I_Gorlin, father to the creatures in my dadpoasts

We Shall Poast Good Tweets Here Viral, If We Can Obscure, If We Must But Always Good Tweets

Part 2 of my reporting on Alpha School--this one focusing on why @C_Hendrick, a founding member of its Learning Science Team, came to believe that AI-powered tech can do a better job of delivering instruction than human teachers. Part 3 is coming soon. nataliewexler.substack.com/p…


Pretty funny that the real AI cybersecurity apocalypse (so far) isn’t from attacks but from patches

3yo got a demonic grin racing his RC car at the same moment that a red LED from the controller cast a hellish light on his face

A close-up look at Alpha School, plus conversations with those on its Learning Science Team, complicates the picture that's been presented--by both advocates and critics. More in my new post (parts 2 and 3 coming later this week): nataliewexler.substack.com/p…

If both the FAA and bank thing turn out not to be related to AI I think that'll look pretty bad

I’ve read this a couple times now, and while there is, frankly, a lot of sloppy argumentation (which I will get to), the charitable, simplified read of Dwarkesh's position seems to be: 1) AI needs major breakthroughs to achieve AGI/ASI (continual learning, sample efficiency, etc.) — this is a view he’s held for some time now, in line with Ilya 2) These breakthroughs, and the ML research skills required to obtain them are verifiable, and thus current, pre-breakthrough AIs can hill-climb on these sorts of problems and capabilities to solve them (like we saw in math) 3) Once these breakthroughs occur, we *may* see rapid, recursively compounding gains to intelligence, leading to ASI in a very short period of time There's a ton of other stuff in the essay, but I think this is the basic crux. Christiano basically articulated this same argument in his Dwarkesh appearance, and if you look at AI-2027, it essentially follows the same trajectory: Agent 2/3 makes a ton of algorithmic progress, allowing Agent 4/5 to explode. In general, I think the high-level narrative here is...fine? It’s certainly one plausible outcome. But, imo, it is a much narrower path than I think many are giving it credit for—a lot of things actually have to go just right for this to play out. And here's where I return to the other, messier arguments, because many of the intuitions and arguments driving Dwarkesh's view in favor of near-term ASI (e.g., EO2027) as a highly probability outcome (~50%) are, imo, quite bad: - “[We'd see] a jump as big as GPT-3.5 to GPT-6 Astra in just 12 months” -> No, under this scenario we’ll see ~4 years of AI progress in 1 year. If there are diminishing returns, then this won’t be as extreme as the jump we saw from GPT-3 to GPT-6. It very well could be, but that would require innovation to overcome the diminishing returns we'd expect in the base case. - “The other big piece of evidence for transfer is human evolution” -> Except if AI was following the sort of pattern we saw for human evolution, we should have seen better transfer long before AI started solving millennium problems. This spikiness has been one of the big surprises of AI! - “Rather, my claim is that scaling up RL training will give AIs the abilities necessary to ramp quickly in new contexts…it’ll go through an evolutionary process that involves centuries of corporate competition to figure out the optimal organizational techniques.” -> These are in direct conflict! If ramping requires RL runs that simulate centuries of competition within “digital economy full of millions of other AIs” then that’s quite bearish for ramp speed (relatively) given the compute and wall-clock requirements! - “I have a lot of 20-something-year-old friends whom I’ve watched accomplish incredible things and build world-leading companies and ventures in domains they entered into with no previous experience.” -> lol. rip leopold. But also, again, these are humans, with human sample efficiency. If AI gets there, sure. But how soon it gets there is the big crux. - “Obviously, some pieces of feedback—about how your actions impacted the quarterly earnings, say—require months of serial time. But even granting that, we should expect these future AIs to be able to get to top human performance in almost all domains within months of being deployed.” -> …no that’s not how that works at all because one set of earnings isn't enough to robustly learn from and you can’t re-run the same real-world conditions. - “[Sample efficiency & online learning are] AI’s equivalent of a Millennium Prize problem. And mathematics has already shown us what happens to such problems” -> Sure, but there’s a difference between “AI can solve a Millennium-level problem” and “AI can solve *this specific Millennium-level problem*” -- if the blocker is a specific problem, the timing spread becomes much wider. - “My answer to this [experiment/wall clock/compute] objection is that each agent will still have access to a ton of compute. Frontier labs have been more than quadrupling their compute every single year.” -> But that’s what they’ve needed to do keep up with their current pace! Even if the labs maintain their current pace of compute growth (and like, will they even? capital markets really do not seem accommodating and Dwarkesh's roommate just lowered his compute projections), we'd still need a step change in pace above that! And that's not even covering the biggest crux of all: whether the returns to intelligence continue to recursively compound faster than diminishing returns eat into them. Now, are any of these objections fatal? No. Do they “disprove” hard RSI/an intelligence explosion? No, ofc not. Am I kind of nitpicking here? Yeah, 100%. And to Dwarkesh's credit, he does anticipate and acknowledge a scenario (the “spark”) where these objections play out. But the reason I've bothered to rant this long is that I see a lot of AI/RSI discourse follow this pattern: where 50% of the arguments are reasonable and grounded and appropriately hedged, and then the other 50% are these sorts of sloppy intuition pumps and what-ifs that include a ton of detail but a noticeable lack of rigor. And I can’t help but get the sense that it is the latter 50% that is really driving people’s predictions and expectations here, as it leads to sentences like this: “In the spark scenario…it takes years to get to the same endpoint that the software-only singularity would have achieved in months: a superintelligence that is to us what we are to chimps.” Note that Dwarkesh thinks these two outcomes (spark vs. software-only singularity) are equally likely. 50/50. I’m sorry, but that’s absolutely nuts. Even Christiano put the likelihood of ASI by in 2027 *absent any slowing down* at only 1 in 3, and I still think that's too bullish. Like, yes, I fully anticipate that we will have automated AI researchers (or something close to them) in 2027, and the universe where we go from that to a software-only singularity is not entirely out of the question. But obtaining that universe relies on a confluence of various frictions, diminishing returns, and wall-clocks being overcome, a somewhat specific set of algorithmic innovations being made or obviated, and an assortment of other factors playing out just-so to enable that perfect storm. Even a fairly rapid exponential doesn't qualify when you're talking about full-blown ASI in 13 months. It may not be impossible, but it's certainly not a 50/50 outcome.
The avalanche that hit math is coming for AI development. dwarkesh.com/p/rsi

Talk to the former teachers at Alpha who love their role. We aren't replacing adults with robots. In 20 years, no matter how good AI gets, parents will still drop their kids off at a building filled with other kids and caring adults. Our job is to make sure the 6-7 hours they spend there are dramatically better than how our generation spent them.

found the limit lmao
Contortionist Emilie Burns at work [📹 bunni.b.bendi]

Rain, grocery store, toy store, rad pizza place Kids are so fun, this weekend has been a blast
Fire pit, grillable meat, 65° in November, light drizzle Perfect afternoon to be a dad

I refuse to accept that a woman’s life becomes less intellectually interesting because she spends a considerable portion of it explaining the world to a 5yo. Have you ever tried explaining death, gravity, money, and why people cannot marry dinosaurs to someone who has no reason to accept your answers?




Fire pit, grillable meat, 65° in November, light drizzle Perfect afternoon to be a dad