areas where agents are still weak in engineering:
- distributed systems/infra
- architecture decisions (related to above)
- time awareness/prioritization
- technical writing (for humans)
if you are good at any of the above your comparative value is higher than ever
Episode dropping tomorrow with @paulfchristiano.
He thinks there’s a 1 in 3 chance of extreme superintelligence within a year (absent any slowing down).
And he'll explain what labs and governments should do right now given that possibility.
I bet against this and I bet against Dwarkesh.
He conveniently sweeps under the rug that climbing a metric involves defining it. Then, supposing you define it (in a way that avoids Goodharting, reward hacking, etc), you need data/envs, which do the lifting. On defining good metrics - this is most of the game of making AGI (converting fuzzy concepts to formalized objects you can climb without gaming, which requires costly fixes later). He elides the hard problem by saying 'AI will just automate data'.
How do you know whether a change belongs in your final recipe? You ablate at scale. When you make N changes, on even a subset of combinations, this compute cost is naively exponential. So, you need a faster signal to tell you when a task that isn't immediately verifiable improves end behavior. Finding that signal is the bottleneck, which doesn't seem to be helped by 'automating the great algorithmic thinker starting from the empty string on paper'.
OCR folks are having a heated debate about particular benchmarks in this thread. Much of the frustration stems from the unreasonable sway benchmarks have on vendor selection: nobody looks behind the top level score!
This invites moral hazard when publishing a benchmark, and understandably shines a light on motivations and incentives at play, even if there are good intentions.
As a consumer of models, if you are relying on benchmarks the least you should do is look at the traces from 3-5 failing tasks and try to make an informed decision yourself. Vik has a checklist as well below.
Don't make up your mind based on twitter arguments either! Look at the data. Also sometimes "benchmarks of benchmarks" are slop. You really need to look at the data yourself if you are making large purchasing decisions based on the numbers.
There are so many flawed OCR benchmarks, and more come out every week.
I gave a talk on how to evaluate them in @HamelHusain's course. Here are common issues, how to spot them, and a guide on evaluation 🧵
AI interfaces nowadays are still pretty much turn-based.
In my PhD thesis, I thought about what interaction would look like if we could go beyond taking turns.
One example is intra-turn interaction:
None of the labs have nailed the taxonomy for how to work with agents.
Chat = quick, 1-off task (but it can handle substantial tasks too)
Projects = knowledge work without a gut repo?
Code = code first approach likely with a repo?
Artifacts / Pages / Spaces = interactive docs?
Personal agent as a separate chat?
Muse is one core chat and then sub chats (when should I use those?)
Grok Bot style messaging inbox for everything might be best abstraction.
Repeat after me:
My job as a leader is not to make people successful.
My job is to create the conditions that give them the highest possible chance of success.
Everything else is up to them.
This is what I believe the current meta to be (disclaimer: Googler):
Fable 5.2: architecture, high ambition projects using either Fable 5.2 or Opus 5.5 subagents
Carbon, GPT-6 Astra Ultra: review Fable’s work and supply prompts
GPT-6.1 Sol Ultra: review Carbon’s work and supply prompts
Carbon: all around workhorse for medium complexity features
Carbon: performance and memory optimization
GPT-6 Astra Ultra, GPT-6.1 Ultra: sledgehammer prototyping; prove a thing is possible and hand off to Fable or Carbon
GPT-6.1 Sol low: cleanup, deslop
Full conversation between Anthropic CEO Dario Amodei and JPMorgan CEO Jamie Dimon ahead of Anthropic's IPO:
0:00 - Mary Erdos opens the event
1:31 - Kissinger on Dario and the Enlightenment
4:49 - Jamie Dimon opens the conversation
6:09 - Talent and culture come first
6:53 - The culture interview: no jerks
8:05 - All seven cofounders still there
8:48 - Dario Vision Quest: no corpo speak
9:41 - How the Fortune 100 uses Claude
12:10 - Pointing agents at sensitive data
13:30 - Why Anthropic chose enterprise
14:43 - Agents vs. self-driving cars
16:01 - $900B compute bet: "deepest dilemma"
18:19 - Buying compute like insurance
19:56 - Commodity? Every frontier is a new market
22:28 - Haiku 5.5 vs. Fable on price
24:08 - Cancer, fusion, quantum
24:41 - Math: high school to Fields Medal
25:49 - Biology and pharma's skeptics
26:55 - Cyber risk after Mythos
27:43 - Hundreds of thousands of vulnerabilities
29:04 - Agent swarms for cyber defense
29:50 - Why go public now?
31:03 - "Never be worth a dollar more"
32:01 - 5% to a public benefit fund
33:33 - Countering concentration of wealth
34:20 - Fifteen years from now
35:02 - "Those aren't marketing": AI risks
36:21 - Diseases as relics of the past
36:58 - Close: underpromise, overdeliver
Dario Amodei says he believes AI can cure heart disease, Alzheimer’s and cancer the way bubonic plague and smallpox were beaten, and admits he sounds 'starry-eyed' and 'crazy'
“We’re just at the beginning of what can be done in biology and medicine. There’s a lot of skepticism from pharma, from early-stage biotech, from academic biologists. We can’t know ahead of time. Maybe the skeptics are right. But my hope is that we prove the skeptics wrong.”
“Every disease you can think of, heart disease, Alzheimer’s, cancer, we think of those the way we think of bubonic plague or smallpox today: that they’re relics of the past. I think we can have a much more beautiful world. I think we can have a world in which all of these problems are addressed.”
“It sounds starry-eyed. It sounds crazy. I really do believe it. I really do believe it can be done. I don’t know, five years, ten years, fifteen years, I don’t know exactly, but we’re going to try to do it as fast as we can.”