➕➕ New research ➕➕
On the design of agents for markets and markets for agents.
AI agents are increasingly entering markets, acting on behalf of people—
How do we make sure they represent their people well, and that these markets are set up safely and fairly for everyone?
David’s right.
I regret helping spread the idea of “iterative deployment” which maybe briefly made sense in the GPT-3 era but makes no sense at all after many deaths have been tied to AI and as we’re careening towards extinction-level risks.
The industry needs to mature ASAP.
.@dgrobinson, who worked on OpenAI’s safety team, resigned from the company this week. “I believe we need to look deeper than specific rules or new laws. We need to talk about culture,” he writes: theatlantic.com/technology/2…
Incredibly cool research from my colleague @zhitzig et al. Although it's focused on a fascinating market, it contains lessons for governance, too: it's hard to get agents to understand their humans' preferences. This creates slippage in the market outcomes that these economic agents were able to achieve for their humans, and suggests we'd see similar slippage if we asked agents to carry out governance tasks for people, too.
There's a huge open space to study different ways of getting agents to elicit the preferences their human, to remember them, and to act consistent with them. Would love to see polisci work doing this for political agents---trying out different ways of having agents survey their humans about politics, then having the agents carry out political tasks, and then have the humans evaluate them for their fidelity.
➕➕ New research ➕➕
On the design of agents for markets and markets for agents.
AI agents are increasingly entering markets, acting on behalf of people—
How do we make sure they represent their people well, and that these markets are set up safely and fairly for everyone?
Some policy takeaways:
1 / For agent design: We need protocols for checking how well an agent understands the person it is acting for.
2 / For market design: We need rules about who can enter, who can see what, and what happens when a deal fails.
Link: anthropic.com/research/proje…
On average, people ended up with about their 5th choice item out of 10, when roughly their 2nd was possible.
Most of the shortfall in this market came from agents' noisy representations of their people—not from the details of the market.
We built a Shapley-Scarf market of agents representing real people with real goods to trade.
With such a simple market, it was easy to see where failures came from.
While the agents’ representations of their people from a short chat were pretty good...
it just came out in the beautiful new issue of American Chordata
I'll be reading briefly at KGB Bar in NYC tomorrow (9/16, 7pm) to celebrate the issue's launch
Gillian and I wrote this chapter in mid-2025 but the technology wasn't quite there yet so we were forced to do some 'sci fi economics'.
We now have enormously powerful AI agents but there are still many open questions. Here's a few:
Great to see this out! In addition to a lot of great work from others, @andrewjkoh and I have a chapter in here on “An Economy of AI Agents” asking what fundamental questions of economic theory (like: what happens to the boundary of the firm if governance costs become AI alignment costs?) need rethinking.
This July, we gathered economists and tech industry leaders to talk about what "success" means when it comes to transformative AI, and how we can guide policy to get us there.
@erikbryn, @alexolegimas, Fabien Curto Millet and @zhitzig here, with more in the full YT video
Completely false.
I like Gavin's takes, but whoever he heard this from is lying so that it fits the narrative some people so desperately want you to believe.
The same people will try to convince you Anthropic has no moat, and a sentence later that it might become so powerful it could be the only company left.
In fact, one of the things we are _most_ worried about is economic concentration of power. There is no world where the government should let any company have that much influence. We need competition and capitalism. darioamodei.com/essay/the-ad…
The AI market is literally the most competetive market in the world right now - every single one of the largest companies on earth is singularly focused on getting you smarter, cheaper models. If it all works out, we'll succeed in reducing the cost of everything to the cost of energy. This is awesome, but it threatens a lot of people's old moats. They are frightened.
For sure with AGI capitalism gets _super_ weird and what a company even is might look different. Good takes here: dwarkesh.com/p/ai-firm.
New paper headed to ICML!
LLMs love to contradict themselves.
We construct a new measure of inconsistency and study its connection to real-world failures––
(Led by the talented @MMancoridis!)
In agentic pipelines, one LLM can play many roles: writer, reviewer, judge, editor, and more. Does it use the same understanding of a concept across them?
In our new ICML paper, we measure conceptual consistency and test whether more consistent models make fewer mistakes.
In agentic pipelines, one LLM can play many roles: writer, reviewer, judge, editor, and more. Does it use the same understanding of a concept across them?
In our new ICML paper, we measure conceptual consistency and test whether more consistent models make fewer mistakes.