←
Chen-Yu Lee91 posts
Chen-Yu Lee avatar

Chen-Yu Lee

@chl260

Principal Scientist & Director @Google | Self-improving agents: environments, experience, RSI | EnvHarness, ReasoningBank, RRSI, PLE | Test-of-Time Award

◷ Joined October 2013▧ 17 Photos and videos
Chen-Yu Lee avatar
Chen-Yu Lee@chl260·Nov 9

Today we're opening Physical-Engineering's Last Exam (PLE) as a $400K open challenge. No AI agent completes half of real engineering tasks. Can you build one that does? 🥇 $250K 🥈 $100K 🥉 $50K 🧵👇

Post attachment
Chen-Yu Lee avatar
Chen-Yu Lee@chl260·Nov 9

Update: the $400K PLE challenge is now open 👇

Today we're opening Physical-Engineering's Last Exam (PLE) as a $400K open challenge. No AI agent completes half of real engineering tasks. Can you build one that does? 🥇 $250K 🥈 $100K 🥉 $50K 🧵👇

Post attachment
Chen-Yu Lee avatar
Chen-Yu Lee@chl260·Nov 9

Update: the $400K PLE challenge is now open 👇

Today we're opening Physical-Engineering's Last Exam (PLE) as a $400K open challenge. No AI agent completes half of real engineering tasks. Can you build one that does? 🥇 $250K 🥈 $100K 🥉 $50K 🧵👇

Post attachment
Chen-Yu Lee avatar
Chen-Yu Lee@chl260·Nov 9

The gap is wide: • Best frontier agent: 48% • Best open-weight model: 25% • Many strong coding models: under 10% Real engineering isn't solved, for anyone.

Chen-Yu Lee avatar
Chen-Yu Lee@chl260·Nov 9

PLE = 85 expert-reviewed tasks across structural, civil, electrical & mechanical engineering: calculations, drawings, design reviews. Each runs in an isolated container. No internet. 6-hour limit. The work engineers actually get paid to do.

Chen-Yu Lee avatar
Chen-Yu Lee@chl260·Nov 3

Official PLE release today with @googlecloud, @mercor & @huggingface! - ~61 files/task (CAD, BIM, PDFs + outdated revisions) - 6hr offline sandbox - Strict PE safety & code checks Links 👇

We're proud to have collaborated on a new benchmark built by @GoogleAI: Physical-Engineering's Last Exam (PLE). Physical engineers design the skyscrapers, power grids, and chip fabs we rely on, yet no existing benchmark tested whether AI agents could do that work end to end. Together with Google and practicing engineers, we built 85 long-horizon tasks across mechanical, electrical, civil, and structural engineering. On the private test set, all frontier models score below 50%: 1. Claude Opus 5.5: 48.3% 2. GPT 6 Astra: 45.8% 3. Claude Sonnet 5.5: 43.5% 4. Gemini 4 Argon: 40.6% 5. Claude Opus 5: 40.0% On the tasks we co-authored, the top agent satisfied 91% of individual constraints, but only 41.3% of its deliverable items passed every hard safety check. Learning how AI fails at physical engineering can help us improve future models and build a safer world. This benchmark is also an open competition. Bring your own agent setup and submit it for evaluation on our private test set at ple-bench.com

0:08
DAIR.AI avatar
⟳ Chen-Yu Lee retweeted
DAIR.AI@dair_ai·Oct 23

Banger paper from Google Cloud AI Research on agents doing professional engineering work. (bookmark it) For professional engineering deliverables, the best agent configuration averages 50.6%, using 27M tokens and 63 minutes per task. They introduce a benchmark called PLE (Physical-Engineering's Last Exam). It has 85 tasks across electrical, civil, structural, and mechanical engineering. Solving a task means producing a billable deliverable, such as a design package a licensed reviewer would sign off on, from a multi-format project corpus with physics and building-code requirements. Some tasks are written end to end by practicing engineers. Others start from an engineer's problem statement, and an adversarial Creator-Solver agent pipeline builds the project environment, grader, and reference deliverables before experts validate them and make them harder. Frontier models were tested under seven public agent harnesses. Most existing engineering benchmarks are textbook QA or toy design problems that leading agents already saturate. arxiv.org/abs/2610.30537 Chat with Paper: academy.dair.ai/papers/ple-p…

Post attachment
Chen-Yu Lee avatar
Chen-Yu Lee@chl260·Oct 23

How we built PLE: • 44 tasks written end-to-end by licensed engineers via @mercor • 41 tasks co-authored in an adversarial loop: a Creator agent builds a task, a Solver agent attacks it, Creator hardens it • Engineers validate every single one

Post attachment
Chen-Yu Lee avatar
Chen-Yu Lee@chl260·Oct 23

In software, a 90% solution ships as a beta. In physical engineering, a 90% solution collapses. The top agent’s designs meet 91% of constraints on average—but only 41% clear every hard safety & feasibility check. Passing code isn’t safe engineering.