
i'm still a software-only singularity and short timelines skeptic despite the preference cascade going on right now. but most of the anti software-only singularity arguments on my timeline look dumb e.g. this is just a meaningless observation


i'm still a software-only singularity and short timelines skeptic despite the preference cascade going on right now. but most of the anti software-only singularity arguments on my timeline look dumb e.g. this is just a meaningless observation

On the other hand, if you think an intelligence explosion will cause major real-world impacts soon, then you should name the specific impacts you expect ahead of time, so that we can verify whether your predictions are coming true.

If you think an intelligence explosion is imminent but won't cause large real-world impacts any time soon, then it's unclear why we should be very worried about it right now.

I worry that the "intelligence explosion within a year" warnings are a motte-and-bailey. The motte: progress on benchmarks will speed way up. The bailey: this will cause gigantic real-world impacts, with a high chance of imminent AI takeover and subsequent human extinction.
How can we adjudicate whether an intelligence explosion "happened" or not? If almost all of us are still alive, with real GDP growth below 5%, no molecular nanobots, no von Neumann probes launched, and no cure for Alzheimer's, is that consistent with it having "happened"?

3 args given to me re why treat AI descendants different: (a) they not conscious (b) human DNA holds special values (c) AIs make social change too fast. None of these work for me. overcomingbias.com/p/ok-if-k…

And by molecular nanobots I mean the ambitious Drexlerian nanomachines that can self-replicate and build almost anything atom by atom.

How can we adjudicate whether an intelligence explosion "happened" or not? If almost all of us are still alive, with real GDP growth below 5%, no molecular nanobots, no von Neumann probes launched, and no cure for Alzheimer's, is that consistent with it having "happened"?
Unfortunately, it seems like the intelligence explosion will be happening in 2027 after all. Most of this post analyzes the new METR report (though other sources of evidence are also important). If they’re right, that implies an AI-driven acceleration that is 5 months ahead of the AI 2027 scenario. I am scared, sad, and angry. blog.aifutures.org/p/q35-tim…

I'm not convinced we'll see an intelligence explosion within 2 years, but many seem to be converging on that view. As we get closer, I think we'll likely see a wild divergence between insiders who think we're mere weeks from the end of the world, and outsiders living normally.
The dumbest possible way to model the intelligence explosion says you get there in about a year. But this seems to line up with more sophisticated models (like from the AI Futures Project).

It's not inevitable that people will hate AI. Sentiment toward AI varies a lot between countries, and attitudes toward tech have changed a lot over time. But tech companies don't have much control over how people perceive the risks or social consequences of their products.

If AI executives tried to claim that there are no major risks from AI, or that it won't take people's jobs, the general population would largely assume that these people are lying, which would be a correct inference to make.

I seriously doubt that tech companies could have avoided much of the AI backlash by practicing better PR. Most people simply don't care about or put much weight on what companies say in their public statements.
A lot of the backlash against AI is due to a simple problem: The people running the big AI companies are terrible, embarrassingly bad communicators.

Two people may both believe that it is morally imperative that we safeguard "humanity", yet disagree on whether we should accelerate or pause AI, because they differ on whether AI is an extension of, or a competitor to, humanity.

It's important to note that this distinction isn't an empirical one: it says nothing about the probabilities of various events occurring. Rather, it's a conceptual frame that determines what it means to preserve humanity and promote human flourishing.

On the competition view, AI development is inherently risky. Any gain to AI comes at the expense of humanity, because *their* goals are not *our* goals. To promote our goals, we need to control AI and ensure it serves humanity. If we lose control, then humanity may be destroyed.

On the extension view, AI expands human potential because AI is part of humanity, not separate from it. If AI flourishes, that simply *is* human flourishing—the same way it would be human flourishing if our genetically engineered descendants achieved their own goals.

In my experience, the key distinction between those who embrace AI progress and those who want to stop it often comes down to whether they view AI as an extension of humanity or a competitor to humanity.