←
Phil Chen566 posts
Phil Chen avatar✓

Phil Chen ✓

@philhchen

Building something new. Previously research @openai @GoogleDeepMind @scale_AI @Stanford

◷ Joined August 2022▧ 39 Photos and videos
Phil Chen avatar
Phil Chen@philhchen·Nov 12

Today we’re releasing Company Agent Bench, a benchmark for company assistant products. We compare Dots, Grok Bot, Claude Tag, and Devin on how well an agent understands its company. We also found that API models with a simple harness perform better than these products (1/6)🧵

0:07
Phil Chen avatar
Phil Chen@philhchen·Nov 17

Company alignment will be the most important research direction for making AI useful. Frontier labs will train models aligned with their vision of humanity, but that’s not necessarily the same vision every company shares.

Post attachmentPost attachmentPost attachmentPost attachment
Phil Chen avatar
Phil Chen@philhchen·Nov 12

A new result on Company Agent Bench shows that using Fable 5.2 improves @claudeai Tag’s score substantially, surpassing Dots and other agent products. Yet using Fable 5.2 API with our simple Slack harness still scores higher overall across all reasoning efforts. See full results below!

Phil Chen avatar
Phil Chen@philhchen·Nov 12

Josh is a rising researcher to pay attention to. He has a rare combination of technical insight, resourcefulness, and creativity. He saw the opportunity that no company agent benchmark existed and used our internal Slack to create a high-quality benchmark.

Phil Chen avatar
Phil Chen@philhchen·Nov 12

Understanding people’s priorities, judgment, and working relationships is still a gap between a capable agent and an effective employee. Kudos to @joshzqin for driving our research directions on the frontier of autonomy. Full write-up here: companyagentbench.com (6/6)🧵

Phil Chen avatar
Phil Chen@philhchen·Nov 12

Agents know who owns what better than what people care about. On authorship questions about ownership and expertise, the products scored 73–91%. On questions about a person’s interests and tendencies, 42–54%.(5/6)🧵

0:07
Phil Chen avatar
Phil Chen@philhchen·Nov 12

We also gave six OpenAI and Anthropic API models a simple Slack harness with native search and reading tools. Fable 5.2 at high effort scored 82.2%, and GPT-6 Astra at medium 81.1%, both above every product. Six of nine GPT configurations beat Dots. (4/6)🧵

0:08
Phil Chen avatar
Phil Chen@philhchen·Nov 12

We tested four agent products connected to Slack, on default settings: - Dots 75.9% - Grok Bot 72.7% - Claude Tag 67.8% - Devin 66.1% Dots leads on authorship and mentions. Grok Bot leads on reply matching. (3/6)🧵

0:08
Phil Chen avatar
Phil Chen@philhchen·Nov 12

We built it from 1 month of real Slack history: 205 questions spanning ~10 teammates, in 3 tasks. - Authorship: who wrote this message? - Mentions: who was tagged? - Reply matching: who wrote each reply? (2/6)🧵

0:09
Phil Chen avatar
Phil Chen@philhchen·Nov 11

Apparently Jensen really understands SIMT single instruction multiple threads

Post attachment
Phil Chen avatar
Phil Chen@philhchen·Nov 11

Willing to bet 10k at 50/50 odds that the Slack agent with most token usage by $ of Q4 2027 will not need @-mentioning

Post attachment
Phil Chen avatar
Phil Chen@philhchen·Nov 5

So excited to welcome @jinwong01 to the team! One of the most important lessons I learned at Scale and OpenAI is to hire people for slope and agency. Coming from Windsurf and Cognition, Jin brings a sense of urgency and an ability to bridge the gap between “where we are” and “where the world should be” that is both rare and critical to building an exceptional company.

Post attachment
Phil Chen avatar
Phil Chen@philhchen·Nov 2

Interesting read for agent architectures. Crazy pace of innovation here: 2022, ChatGPT: prompt -> response 2024, o1: reasoning 2025, Claude Code: local tools 2026, persistent agents: inputs from several channels -> router + workers

0:45
Phil Chen avatar
Phil Chen@philhchen·Nov 2

Very important content for us gpu underclass folk who have our GPUs distributed across several neoclouds

Phil Chen avatar
Phil Chen@philhchen·Nov 1

The biggest unsolved problem in LLM autoresearch rn is how to circumvent Anthropic safeguards