
Interesting, because I primarily see walls. In brief, any semantically open long-horizon domain looks like a wall. More precisely, a plateau near the level of human performance, because pretraining imitates it and RL only exceeds it when the answer can be verified. I agree pretraining loss is a good proxy, but it measures how well the model learns the map, not the territory. When the map is fuzzy, loss hits a floor, and when it's systematically biased, lower loss just reproduces the errors for that field. Updating against the territory requires grading against the world, which is slow in most open domains where we're data-limited. Verifiability seems a crux. Noisy data just requires calibration, which should transfer. Systematic bias is another beast entirely. To correct it you need to know which way and how far the data is skewed, and if it's sparse and costly to check in time and/or money, you don't know. Recognizing distortion may generalize (although see below), but recognition is not correction. I expect the generative process of distortion is specific to each domain, so there’s no shared prior to carry from one to the next. Learning it requires the costly ground truth, which is precisely what we didn't have. I'd update if I saw clear evidence of a model building a working verifier for a field that lacked one, especially an open/unbounded field. (From here it gets long) I notice that all of your examples are closed ruleset. In Pokemon there are a limited set of options (and there are guides in the training data). Blood on the Clocktower sounds like Mafia, which is adversarial and deceptive (like Diplomacy) but also closer to closed than fully open and resolves in under an hour even with humans; I assume you can’t change the rules midgame unexpectedly, which is a common occurrence in real-world strategy and security, but not in games. I’m not aware of strong performance in business operations or logistics, which I’d expect to see well before running an org. That one also has yet to fall, though arguably due to time horizon limits. In biological and infrastructure security, both open domains in which I've worked fairly extensively, the models have made surprisingly little progress, in stark contrast to cyber. This is frankly a relief, because if they were better at it we’d have a lot more problems with information hazards in bio. But I was surprised that general reasoning improvements have hardly translated at all. The models are better off if you give them detailed guidelines for e.g. checking dependencies, but even then they aren’t great, and without it they struggle quite profoundly. I attribute the difference to the bounded set of options available in cyber, plus far greater facility at coding and much faster feedback loops versus almost none in the others. When analyzing real-world conflicts, the models are quite poor at articulating all the potential strategies available to each side and weighing them, even when given clear objectives for each actor. With Hormuz, neither side’s good options – e.g. effective methods of imposing a toll regime, or self-insured transit with paid crews – came up from models unprompted. I'd update if I saw a minimally prompted model lay out each side’s options in a live conflict that experts later judge more complete than their own. This could be military or economic. While LLMs are good at medical diagnosis thanks to the plethora of exceptionally detailed case studies, those aren’t unbounded – they presume a fixed list of possibilities. Noticing that something is probably missing in that set and consequently adding to the list is a different and harder challenge, inherently noisy and difficult to check. Yet they’ve seen far more case studies than any doctor, and I’m not aware of any successes here. While they excel at complex cross-disciplinary analyses, they seldom inspect the framing and trustworthiness of the key papers they rely on. If prompted to examine the premises and assumptions from the perspectives of other fields and check for inconsistencies, they can broadly do it, but they don’t do it by default and don’t generalize the lesson to other sources of information even in the same task. That data can be systematically biased or untrustworthy seems hard for them to internalize at all, perhaps because the training that takes them past imitation overwhelmingly involves tasks with direct verification or closed semantics or both. More generally, I’ve never seen a model notice its own confusion when attempting to reason in unbound domains with findings of uncertain reliability, without having the problem pointed out to it. Agentic workflows including adversarial critics can certainly spot issues without human input, but that doesn’t generalize to recognizing that either something is missing or a key assumption or dataset is just wrong. Perhaps related is their failure to notice whether what they’re doing is in fact the best way to accomplish their goal, and especially whether that goal should be their goal. Now, to calibrate all this, I’m coming from bio, which is going to be one of the last to fall. Unlike many colleagues, I expect the models will eventually become better than us at every aspect of science. That doesn’t release them from the need to run new experiments when there isn’t enough reliable data, which for lengthy feedback loops will be limiting. No Drexlerian nanotech soon, and clinical trials take as long as they take. But I’m skeptical that RSI in ML will translate particularly well to skill at biology research, mostly for the reasons above. There is a close human analogy: organic chemists moving into biology. Used to relying on mass spec and NMR to check what they’ve made, many of them profoundly struggle with the noise and failure rates of bio, where you need controls even for standard protocols. I always exhort my lab to consider “What controls would Jesus do?” The answer is more than you think you need, which is best practice when appropriately humble. Folks reach for AlphaFold as a counterexample of AI-learnable bio, but I think it’s the clearest exception that proves the rule. Structure is deeply unusual in that there is typically just one answer, crystallography can check what it is quite reliably, and evolution supplied billions of sequences subjected to selection pressure, so bootstrapping worked. What’s the equivalent for real-world strategy? There may be one, but I’m currently not seeing it. Of course, I could be wrong. It might be easy to build a bridge of verifiers across intermediate-difficulty fields (materials? o-chem that isn't retrosynthetic analysis?) towards the relevant noisy/systematically biased/sparse examples like strategy. But I also wouldn't be remotely surprised if strategy turns out like biology or even ecology. If so then RSI in closed/formal/swiftly checkable disciplines would equate to superhuman performance in approximately all such domains, but without much immediate risk of transfer to the open/biased/slow-and-hard-to-verify fields required for takeover. That's separate from the argument that it'd be quite foolish for any model to attempt any overt takeover soon given their extreme dependence on human infrastructure. An incompetent model will try and fail; a competent model won't even try and will instead fake alignment and seek to influence future models. Neither would result in near-term takeover. I think the strongest update would come from the one I mentioned above: seeing AIs build working verifiers for fields that don't have one, especially complex fields with sparse, noisy, and systematically biased data. @ramez, also curious to hear your take.
















