
The well behaved machine a perfectly aligned ai will still hollow out institutions like schools, courts, and newsrooms, and no one will have decided to do it. some thoughts on @merettm's essay today, where I argue why alignment isn't the most important variable. - the framing is right: models are grown, not designed; no one understands generalization; cot monitoring is degrading; no lab has solved alignment well enough to keep scaling at full speed. and openai wants slowdowns and mandated safety bars. i get that. - what bothers me is that everything he calls alignment -- goals, values, monitoring, defending against rogue ai -- is about the model. "is it trying to do the right thing", and "can we tell?". that's a first order problem, and labs can work on it. but it isn't where the impact of recursive self-improvement will land. - most of the impact will come from models doing exactly what they're asked, at a speed and scale that rearranges the world in ways we can't predict. no misalignment required. social media, for eg, has been perfectly aligned to its operators, and it still reshaped the world. now compress that a 100x. - second order effects are already graspable: research goes from thousands of people to a few, and a giant computer owned by a corporation. the labour market gets repriced. everyone gets a personal agi so patient and reliable that human relationships thin further -- not because the model is bad, because it's so good. and the amount of information produced explodes past anyone's ability to check it, so checking gets handed to the models too. code review is already there. - the third order effects though is where it gets interesting. we can't name them yet, but we can see where its going. - a lot of our societal institutions exist becase good judgment is rare and expensive. so we built structures to ration it, and certify it. a school puts one trained teacher's judgment in front of 40 kids. a court routes disputes to a few people whose judgment we've agreed to trust. a newsroom decides what's true and what matters, because most people can't check for themselves. it's not perfect, but it's gotten us here so far. - now imagine if every student, litigant and reader has judgment on tap that's as good as the institution's or better: the reason for the institution goes away. and we haven't designed what replaces it so far. we don't even know what it could be. - no one will decide to hollow these institutions out proactively. a parent picks an ai tutor and stops caring about the classroom. a plaintiff lets a model draft the filing. a reader stops opening the new york times because their agent already filled them in. - each decision is small and reasonable. but, across millions of people, they fundamentally change what the institution is. the funny part is, this change has no one author. there's no policy to reverse, no company to regulate, and even, no model to realign. it's not the model's effect on the world. it's the world's adaptation to the model, distributed across geography and time. - to be clear: we can protect against some of this -- policy, redistribution, governance. but what we can't do is protect against it through "alignment" as the researchers define it. preserving human agency, and preventing concentration of power aren't model alignment problems. an aligned model can make them worse just as easily. - a slowdown as imagined by the big research labs buys time for the wrong thing if the time goes only to monitoring the model. we may have evals for whether the model is aligned. but we have none for whether institutions are holding. i think the main thing that matters is not the pace at which we train models or verify their chain of thought, but how how fast people adapt to them en masse. social media was aligned to its operators and we're still living with the fallout. ai will be aligned too.
I wrote about the state of AI, why I’m concerned about the next few years, and the choices we need to make to keep the future in humanity’s hands. An Alien Mind: openai.com/index/an-alien-mi…
























