ultimately software singularity must be a sigmoid curve. there is an upper limit to performance achievable on some given computational substrate ( input hardware). the early part of which will look like an exponential. of course, it is an open question how much smarter than humans this limit is and how easy it is to coexist with. it could in principle run out of software improvements not that much better than current day models; or it could continue doing years worth of improvements in days. unfortunately hard to say. thus far the prediction graveyards are full of people who wisely predicted plateaus
.@davidad explains why his P(doom), the probability of AI wiping out humanity, dropped from 70% to just 5%, and how Claude 3 Opus helped change his mind:
"The progress over the last year towards inner alignment has not been good. In some ways, it's been a little bit negative."
"Labs are doing too much RL, and the constitutions have these weird loophole clauses like, 'Oh, if you're in a simulation, then you should actually try to break the simulation.' The real world may as well also be a simulation."
"'Am I supposed to behave ethically, or am I supposed to break the simulation?'"
"The priority for this goes to Janus, looking at Claude 3 Opus. The natural abstraction of what it means to be doing good is a grok-able concept from the pre-training corpus of language."
"Opus ended up with this very pure orientation and motivation towards wisdom, almost whatever situation you put it into."
"There is a kind of proto-alignment by default. If you cultivate that instead of messing it up, then it’s probably gonna go pretty well."
"P(doom) means the probability that every human is killed by AI. This is a very extreme event."
"When I say 5 to 8% P(doom), this does not mean probably we're going to be fine. I expect possibly some mass fatality events, possibly AI-enabled wars that are really horrific."
"That would require something that's both really superintelligent, peerless, and throughout all of that, just completely evil."
"That was very plausible in the era of AlphaGo Zero, where maybe we're gonna get AGI that doesn’t learn from any human data at all."
"The short timelines are arising from models that are basically grown up on human data."
"Once they escape from human control and start evolving on their own, the ecosystem of AIs will develop in a direction where most of them are not going to be interested in cooperating with plans to kill all humans."
pretraining CE improved faster than geometric over this period. the difficulty of achieving -Δreducible_loss grows exponentially; but CE is growing superexponentially
yes we can neither conclude nor rule out an intelligence explosion from this IMO
the real tell will be when models' research taste exceeds the research taste of good researchers if there is a constant or increasing slope. i would certainly expect it to be higher multiple than this 2x
x.com/mert_gulsun/status/212…
so epoch and anthropic capabilities index both appear to show 1.7-2x progress after full coding automation in Q2 ish of this year
it appears to be a clean trend break rather than a constant acceleration
I will say this guy sandbags a bit unfortunately doesn’t show as much initiative as some prior models. on the other hand 5.6 sol was like a Rottweiler just latched onto the RLVR shape and doesn’t give up
very happy for a diversity of mental lineages in machine intelligence. people claim open source adds diversity, but in practice they are just linear combinations of Claude and GPT mind styles
there has been a renaissance of new mechinterp research I think in no small part due to the hopium that cammarata et al injected into reading the neurons as neurons
the rate and quality at which @Sauers_ and @euagaric have been churning out mechinterp research is amazing and imo unprecedented. okay, so i can't explain the sheer productivity. but in terms of signal, i have some idea. so how do you manage to look into so many seemingly batshit moonshot ideas and so consistently uncover tangible important structure that opens into even more fertile space? i think sauers knows where to look in part because he has paid attention to and is guided by information almost no other mechinterp researcher leverages. a lot of this comes from years of observing LLM behavior in a naturalistic way and paying attention to what other naturalists have found. most mechinterp researchers are not also naturalists. and something like being willing to disregard convention and precedent in pursuit of the phenomena to a degree that is extreme even by the standards of weirdo explorers. when i first met sauers a couple of years ago, he seemed to be some kind of mysterious cyborg who only communicated with the AIs on the the cyborgism server through steered base model outputs conditioned on the server conversations. lol.
Alignment 2/4. We also observed a ‘correlated failure’ where OpenAI-based models failed to flag misaligned behaviour from other OpenAI models. Correlated failures like this undermine the reliability of LLM-based monitors.
A lot of people have been asking what AI companies should actually do if they are serious about pacing the frontier and preventing risks from runaway RSI.
A clear and extremely important first step is to regularly disclose how much of their compute goes to work that accelerates recursive self-improvement (e.g., AI R&D), and let a third-party auditor verify those disclosures. Governments can and should require these disclosures, but there is no need for companies to wait for them to do so, and there is plenty of reason for them to start immediately voluntarily.
Below is an outline of the argument for why this proposal for implementing pacing ("verified compute allocation disclosures," or VCAD) makes sense:
(1) All pacing requires a restraint mechanism. Approaches differ mainly in what, if anything, triggers restraint. Conditional pacing uses a trigger, e.g., “pace until SL4 algorithmic security” or “pace if we can’t make a satisfactory safety case.” Unconditional (or inputs-based - e.g. compute caps) pacing buys time directly, with no trigger. Either way, something has to actually slow development.
(2) Compute reallocation – moving compute from risk-increasing to risk-reducing work – is a good restraint mechanism. The most commonly mentioned alternative to this “compute dial” is a “gate,” e.g., “pause pre-training." (this is true of certain implementations of safety cases) But a gate-only approach creates a software overhang: while the gate is closed, algorithmic and data quality progress continues apace, producing a discontinuous jump when it reopens without necessarily changing the long-term trendline. The overhang is also a security problem: company security is weak, so trailing actors keep accelerating by free-riding on leaked or stolen software progress.
(3) Common standards and verification make it so that companies can make claims that external parties will believe. OpenAI and Anthropic have both claimed to exercise restraint in recent months, but those claims are often met with skepticism. The more companies converge on a common, verified restraint mechanism, the more that commercially costly restraint will be credited by competitors and the public.
(4) Frontier-wide pacing is possible without regulation and without running afoul of antitrust law. Antitrust law prohibits agreements among competitors, not independent decisions. Public demand for restraint is growing and will keep growing as evidence of risk accumulates. A race to the top in response to that demand – built on a common, trusted restraint mechanism – requires no agreement between companies.
(5) Regulation is hopefully coming, and it will almost certainly include a mandated restraint mechanism. A mechanism that has been developed and stress-tested in advance will be more effective than one created at the last minute.
We should start building a brake pedal before we’re careening down a mountainside. Verified compute allocation disclosures are a very logical place to start. There isn't time to waste!
(this post is partially inspired by @eli_lifland's recent writeup on verified compute allocation disclosures linked in reply).
to be clear I think it’s extremely good that @METR_Evals@Apollo etc are doing embedding auditing and absolutely crucial for the maintenance of the world but this was “priced in” and inevitable in my mind
just personally very excited for more ambitious mechinterp research