The evals are not ok
New NeurIPS paper: Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking.
Evals drive deployment, safety standards, and research. Yet we ignore noise from prompt, judge, agent harness etc, & so underestimate error in benchmarking & annotation, & it gets worse with more data.
I'll get to what this means for CIs & how to fix, but first let's talk about how this makes benchmark hacking easier 🧵
one could even argue that stability is a safety feature more than a risk. would you trust a self-driving car that turns off or changes destinations once every 5 minutes... once every hour... once a day? would you trust a boat that sprung a leak once a week?
i think history will look back on the hysteria over this graph as 20% legitimate concern about AI advancement and 80% hysteria that this is "the graph" when it really isn't. it's a measure of stability. stability is net-good. stability means a human told it to do a thing and it kept doing that thing until it was done.
Gentle Reminder: time horizon is a measure of stability, not intelligence. Stability is necessary but not sufficient for ASI.
For example, a "while loop" can run forever, but it's not intelligent. LLMs have had a stability flaw. The flaw is bad enough that we need to basically "reboot" the LLM after a few hours... re-prompt, redirect, etc.
Through hard work from AI researchers around the world, the flaw is getting addressed, and so soon we won't have to reboot LLMs as often. But just because we don't have to reboot LLMs all the time doesn't mean we're getting superintelligence. It means we're getting LLMs that can have the *same* level of intelligence they already had... but they can use that intelligence for longer periods before a human needs to reboot/redirect/reprompt/etc.
I recognize "reboot" is an imperfect metaphor, but I think it's the right metaphor because when you reboot a computer you're not fundamentally changing its intelligence. A computer you need to reboot isn't fundamentally smarter or more valuble than a computer you don't need to reboot... it's just cheaper to run because the IT person doesn't have to come hit a button as often.
There are many important risks of AI, and this graph is relevant to many of them (e.g. job displacement), but this isn't a graph that can lead to the conclusion in the QT below.
It was a delight to talk about the Jensen Huang saga and our new book on AI and national security, THE BITTER STRUGGLE, with the inimitable @DKThomp and @jordanschneider. The first four chapters of the book focus on AI chips, export controls, and how democracies can gain an edge in inventing the most powerful technology of this generation — all more central subjects than ever, given the events of this week. Come for that conversation, stay for references to Indecent Proposal, Kanye, and Spygate.
penguinrandomhouse.com/books…youtube.com/watch?v=x3rnhS7b…
your primary expression of free will is not what you believe or what decisions you make, it's how you weight information sources. everything else is just downstream information processing based on those weights.
I heard this from a live show producer called Tom Jackson, who worked with performing artists from around the country, including helping them have enough confidence to do their job without succumbing to arrogance or false humility. Really awesome dude.
About 15 years ago I heard a def of humility that really stuck with me, "Humility is walking in the authority that you've been called to. It's neither false humility... pretending to be less capable/valuable/etc... nor is it arrogance... pretending to be more capable/valuable".
Fun fact - the library of Alexandria was competing with another major library overseas. And so they passed laws restricting the export of Papyrus to prevent the other library from winning the great library race of the 3rd and 4th centuries.
But in the end, libraries proliferated
Plea for people to make more concrete claims than "things will get weird" or "everything will be crazy" or "society will be completely transformed" - be specific! Broad adjectives and grandiose claims like that aren't useful.
The "things will get weird" framing seems to serve a social function more than an epistemic one; though I definitely understand the desire to convince skeptics that AI isn't exactly the same kind of thing as fax machines or better batteries.
But also - if you told me 5 years ago what I can now do with coding agents, I'd have said that's 'crazy' - but in practice I got used to this pretty quickly. I think people adapt/acclimatise more than they're given credit for. My mom will find self driving cars crazy and the twentieth time she uses one will find them normal.
The phenomenology of technological change is usually more "huh, I guess I do that differently now" than "everything is crazy and I can't make sense of anything." I should read up on Margaret Mead's work more: apparently the Manus of Papua New Guinea went from stone age technology to seeking out Western education and governance structures remarkably quickly after WWII contact!
To be clear, I think that email is basically the "dialup" version of the AI internet. Once this gets going, then we'll start getting bespoke systems capable of doing this as much higher scale. Dialup is to broadband/fiber as Email is to.... <fancy AI map-reduce protocol>.
I'm a broken record about this, but AI will be server-client. Rob is saying he wants to use his own client LLM, the companies will continue to offer endpoints via their server LLMs. The future of AI isn't a singleton AI mind, or even an autonomous multi-agent civilization. It's the ability to map-reduce the entire internet and everything in it. You send your prompt to a billion sources, their servers respond, and your client LLM synthesizes the result.
It's the end of the attention economy, because finally you'll be able to listen at scale.