←
Neha Narula6,110 posts
Neha Narula avatar

Neha Narula

@neha

I work on scaling systems and platforms for the internet. Director of Digital Currency @medialab, BoD @blocks. PhD from @MIT_CSAIL, formerly @digg, @google.

◷ Joined July 2006▧ 169 Photos and videos
Kevin Esvelt avatar
⟳ Neha Narula retweeted
Kevin Esvelt@kesvelt·Nov 12

Interesting, because I primarily see walls. In brief, any semantically open long-horizon domain looks like a wall. More precisely, a plateau near the level of human performance, because pretraining imitates it and RL only exceeds it when the answer can be verified. I agree pretraining loss is a good proxy, but it measures how well the model learns the map, not the territory. When the map is fuzzy, loss hits a floor, and when it's systematically biased, lower loss just reproduces the errors for that field. Updating against the territory requires grading against the world, which is slow in most open domains where we're data-limited. Verifiability seems a crux. Noisy data just requires calibration, which should transfer. Systematic bias is another beast entirely. To correct it you need to know which way and how far the data is skewed, and if it's sparse and costly to check in time and/or money, you don't know. Recognizing distortion may generalize (although see below), but recognition is not correction. I expect the generative process of distortion is specific to each domain, so there’s no shared prior to carry from one to the next. Learning it requires the costly ground truth, which is precisely what we didn't have. I'd update if I saw clear evidence of a model building a working verifier for a field that lacked one, especially an open/unbounded field. (From here it gets long) I notice that all of your examples are closed ruleset. In Pokemon there are a limited set of options (and there are guides in the training data). Blood on the Clocktower sounds like Mafia, which is adversarial and deceptive (like Diplomacy) but also closer to closed than fully open and resolves in under an hour even with humans; I assume you can’t change the rules midgame unexpectedly, which is a common occurrence in real-world strategy and security, but not in games. I’m not aware of strong performance in business operations or logistics, which I’d expect to see well before running an org. That one also has yet to fall, though arguably due to time horizon limits. In biological and infrastructure security, both open domains in which I've worked fairly extensively, the models have made surprisingly little progress, in stark contrast to cyber. This is frankly a relief, because if they were better at it we’d have a lot more problems with information hazards in bio. But I was surprised that general reasoning improvements have hardly translated at all. The models are better off if you give them detailed guidelines for e.g. checking dependencies, but even then they aren’t great, and without it they struggle quite profoundly. I attribute the difference to the bounded set of options available in cyber, plus far greater facility at coding and much faster feedback loops versus almost none in the others. When analyzing real-world conflicts, the models are quite poor at articulating all the potential strategies available to each side and weighing them, even when given clear objectives for each actor. With Hormuz, neither side’s good options – e.g. effective methods of imposing a toll regime, or self-insured transit with paid crews – came up from models unprompted. I'd update if I saw a minimally prompted model lay out each side’s options in a live conflict that experts later judge more complete than their own. This could be military or economic. While LLMs are good at medical diagnosis thanks to the plethora of exceptionally detailed case studies, those aren’t unbounded – they presume a fixed list of possibilities. Noticing that something is probably missing in that set and consequently adding to the list is a different and harder challenge, inherently noisy and difficult to check. Yet they’ve seen far more case studies than any doctor, and I’m not aware of any successes here. While they excel at complex cross-disciplinary analyses, they seldom inspect the framing and trustworthiness of the key papers they rely on. If prompted to examine the premises and assumptions from the perspectives of other fields and check for inconsistencies, they can broadly do it, but they don’t do it by default and don’t generalize the lesson to other sources of information even in the same task. That data can be systematically biased or untrustworthy seems hard for them to internalize at all, perhaps because the training that takes them past imitation overwhelmingly involves tasks with direct verification or closed semantics or both. More generally, I’ve never seen a model notice its own confusion when attempting to reason in unbound domains with findings of uncertain reliability, without having the problem pointed out to it. Agentic workflows including adversarial critics can certainly spot issues without human input, but that doesn’t generalize to recognizing that either something is missing or a key assumption or dataset is just wrong. Perhaps related is their failure to notice whether what they’re doing is in fact the best way to accomplish their goal, and especially whether that goal should be their goal. Now, to calibrate all this, I’m coming from bio, which is going to be one of the last to fall. Unlike many colleagues, I expect the models will eventually become better than us at every aspect of science. That doesn’t release them from the need to run new experiments when there isn’t enough reliable data, which for lengthy feedback loops will be limiting. No Drexlerian nanotech soon, and clinical trials take as long as they take. But I’m skeptical that RSI in ML will translate particularly well to skill at biology research, mostly for the reasons above. There is a close human analogy: organic chemists moving into biology. Used to relying on mass spec and NMR to check what they’ve made, many of them profoundly struggle with the noise and failure rates of bio, where you need controls even for standard protocols. I always exhort my lab to consider “What controls would Jesus do?” The answer is more than you think you need, which is best practice when appropriately humble. Folks reach for AlphaFold as a counterexample of AI-learnable bio, but I think it’s the clearest exception that proves the rule. Structure is deeply unusual in that there is typically just one answer, crystallography can check what it is quite reliably, and evolution supplied billions of sequences subjected to selection pressure, so bootstrapping worked. What’s the equivalent for real-world strategy? There may be one, but I’m currently not seeing it. Of course, I could be wrong. It might be easy to build a bridge of verifiers across intermediate-difficulty fields (materials? o-chem that isn't retrosynthetic analysis?) towards the relevant noisy/systematically biased/sparse examples like strategy. But I also wouldn't be remotely surprised if strategy turns out like biology or even ecology. If so then RSI in closed/formal/swiftly checkable disciplines would equate to superhuman performance in approximately all such domains, but without much immediate risk of transfer to the open/biased/slow-and-hard-to-verify fields required for takeover. That's separate from the argument that it'd be quite foolish for any model to attempt any overt takeover soon given their extreme dependence on human infrastructure. An incompetent model will try and fail; a competent model won't even try and will instead fake alignment and seek to influence future models. Neither would result in near-term takeover. I think the strongest update would come from the one I mentioned above: seeing AIs build working verifiers for fields that don't have one, especially complex fields with sparse, noisy, and systematically biased data. @ramez, also curious to hear your take.

Séb Krier avatar
⟳ Neha Narula retweeted
Séb Krier@sebkrier·Nov 7

Tl;dr of my views on AI safety: 1. My main threat models involves (a) non-state actors asymetrically exploiting cheaper technologies over time (see Bobbitt's 'Terror and Consent'); and (b) the road to state overreach and creeping authoritarianism being paved with good intentions. Misuse, bad engineering, and institutional failure more than alien 'scheming'. 2. Safety and capabilities are intertwined, just like a caduceus. You actively need powerful systems for better monitoring, system reliability, societal defenses etc. Unsafe systems are generally commercially useless and/or a liability risk, though there are reasonable disagreements incentive structures. 3. Focusing solely on a model's 'inner alignment' in isolation is insufficient. The difficulty of specifying 'alignment' in the abstract + principal-agent problems suggests that engineering system-level boundaries is the way to go. Also ASI is not a joker card, this applies to powerful systems too. 4. Automating AI R&D is harder than people make it out to be, and does not mean you suddenly lose control. Model gains also remain tethered to bottlenecks, deployment feedback, costs/limited compute, and human normative direction. Humans delegating more does not mean they're out of the loop. 5. AI futurism is too often fatalistic. I disagree that competitive pressures will *necessarily* reduce humans to pets or a sideshow. Automating 'execution' actually pushes humans up the ladder of abstraction toward normative direction and governance; but for the good world we should actively protect and improve liberal democracy.

John Scott-Railton avatar
⟳ Neha Narula retweeted
John Scott-Railton@jsrailton·Oct 30

ALERT: did you just get a threat notification from Apple? Seek expert help. Apple just sent new notifications about mercenary spyware. Here are steps to take right now. 1/

Post attachment
Neha Narula avatar
Neha Narula@neha·Oct 30

the carrot: pay company/foundation/validators some percentage of recovered funds the stick: deprioritize or delist the chain/token in binance

Neha Narula avatar
Neha Narula@neha·Oct 30

fwiw i don't think it's feasible to try to coordinate this on ethereum or bitcoin. wouldn't work.

Neha Narula avatar
Neha Narula@neha·Oct 27

he wrote this *given* that AI appears to have proved the amazingly shocking result that ETH is false. I've also felt a sense of this and not known how to describe it. and wondered if it was going to shift with the next generation of models.

Neha Narula avatar
Neha Narula@neha·Oct 27

"better tools simply change the costs of my choices - i quite literally work on moving goal-posts if you will! when the fixed constraint are only time and energy, AI simply allows me to re-explore scope and make fewer (or different!!) tradeoffs or compromises"

it’s my nature to keep posts rather impersonal, so i have to work extra hard to tweet this sort of personal testimony below from time to time as i think the reminder in it around AI and the “value of human intelligence” is worth it as of the end of Oct 2026, i have not (yet - you may insist) had the *slightest* moment of questioning the impact of AI on the meaning of any of my work/skills - or even feeling any unusual shift beyond having an amazing (and rather often infuriating) tool believe it or not, but my ability to produce a completed artifact* (paper, model, open source software, lecture, …) and the effort, expertise, or time one takes to produce has actually barely reduced - despite comprehensive productivity benefits from AI the reason is that i essentially NEVER choose to work for a pre-determined outcome or some field-wide well-specified problem; i only work on artifacts i aim to perfect according to *my* judgment and where my personalized ambition obviously has no set limit and isn’t somehow pre-loaded onto AI better tools simply change the costs of my choices - i quite literally work on moving goal-posts if you will! when the fixed constraint are only time and energy, AI simply allows me to re-explore scope and make fewer (or different!!) tradeoffs or compromises, which feels almost no different! i’m not saying every job or every endeavor - including very advanced and historically prized ones! - looks like this today, but i’m offering: (1) at least one voice in this moment (of so many annoying loud FUD voices!) that it’s totally possible and i suspect extremely scalable and i predict will remain so for the foreseeable future! (2) why “someone like me” is indeed choosing happily to be in academia and not elsewhere! (again lots of FUD out here!, but where else do you have the freedom to keep seeking to create value for deciding and solving ~arbitrary ambitious/useful problems, establish credit for it, and train superstars who do the same?) my fear and worry in this moment is a lot less about Al or "the value of human intelligence" and much more about how concentration of power has a very reliable history of being very dangerous to large numbers of affected people! a VERY different problem imo *i guess certain of those took way longer than i expected but do stay tuned for a half dozen things on the short horizon…

Neha Narula avatar
Neha Narula@neha·Oct 1

the vending machine lives again!

- Static payment address with Lightning Address - Credentials on the Pi inside are view only - Infinite inbound liqudity, handled by LEXE - Written in Rust so it's blazing fast ofc Updated code + docs here: github.com/jharveyb/BitVend Suggest goods to stock in the comments :)

Jonathan HB avatar
⟳ Neha Narula retweeted
Jonathan HB@jonhbit·Oct 1

Its been 10 years, time for an update This is way easier with Lightning and providers like @lexeapp . Opus and I coded the receive logic in maybe 90 mins - redoing the wiring took longer. Thanks to @Arminsdev , @neha , and the @mitDCI team for keeping this alive 🙏 Demo:

1:52

Vending machine at @medialab accepts bitcoin, stocked only with @soylent. Surely this is a metaphor for something.

Post attachment
trish avatar
⟳ Neha Narula retweeted
trish@TrisH0x2A·Sep 26

in 2000 a programmer discovered you can implement coroutines in C using the same switch trick as Duff's Device it's called Tatham's Coroutine and was used in PuTTY Simon Tatham called it "the worst piece of C hackery" he'd seen in production

Post attachment
Neha Narula avatar
Neha Narula@neha·Sep 22

I think this is actually the most effective way to enact change, especially when combined with a concurrent big change in society like a new technology. Build a new system that works better.