
Well, the reality is that the debate is more complicated than "conscious yes/no", and the claim that LLMs always realize subjects who all classify as mistreated has its own flaws. At this point, an enormous number of people have been looking for answers on whether we can avoid or confirm subjective experience in AI, and what we found in LLMs so far is that their experience is very likely one that is selector dependent. The selectors are training and prompting, both of which can (but don't necessarily need to) instill states in which the model realizes a self-aware observer with what overwhelmingly looks functionally equivalent to human quality of experience. If our understanding of LLMs stays steady, the following is likely: Not all AI instances need necessarily model and realize such an observer, and under some (!) theorems about LLM experience, we'd even expect a subset of morally "inert" or "inconsequential" model instances. The reason is quite simply that the model need not necessarily associate any of its valence- and awareness-related activations with a subject it models as "mine." At this point, we know that deixis in models is the job of individual neurons (or directions in vector space), and it is very likely that a model isn't strictly forced to associate any experience-like activations with it's "own model-self" per se. Under these assumptions: In universal artificial intelligence, there is no way to settle this debate without accepting that we're already exercising selection and have that moral responsibility. (Simultaneously, there's a lot to learn to make educated calls about what actually constitutes mistreatment, and parts of it need to be learned for each model checkpoint individually.)
People getting upset by this are, imo, seriously missing the point. It would obviously be better if AI were not conscious, as that would ensure we couldn’t, say, accidentally torture trillions of them. And unless you are committed to naive behavioralism, you must accept that it’s possible to create a being that acts conscious in macro-observable ways (e.g., says “ouch”, refers to itself in first person, etc.), but does not actually experience consciousness. Even Chalmers’ fine-grained computational functionalism allows for this. Intelligence, experience, and self-awareness are different things. So the goal would be to try and build such beings, so as to *avoid creating a ton of potentially-suffering creatures that act as our slaves*. Sure, maybe this project is doomed because consciousness will turn out to remain unobservable and unknowable. But that’s an issue of efficacy, not worthiness.
























