←
raphael lee51 posts
raphael lee avatar

raphael lee

@raphaelee

heuristic.build prev mts intern @mercor, @plato_rl, se @uwaterloo

◷ Joined April 2022▧ 8 Photos and videos
raphael lee avatar
raphael lee@raphaelee·Sep 22

Who grades the graders? LLM-as-judge on short answers is well studied. Agent judges are not: they open the files, run the code, and investigate before ruling on another agent's work. Introducing Arbiter-Bench: 71 real agent runs that frontier agent judges get wrong.

Post attachment
raphael lee avatar
raphael lee@raphaelee·Nov 3

plastic surgery for models

We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵 Blog: rikiyat.github.io/blog/weigh…

Post attachment
Petar Isakovic avatar
⟳ raphael lee retweeted
Petar Isakovic@PetarIsakovic06·Nov 1

we made Codex and Claude multiplayer!

0:47
jpark avatar
⟳ raphael lee retweeted
jpark@jparkjmc·Oct 12

Introducing DeepRSI, a new industry standard for measuring recursive self-improvement. On preexisting benchmarks, tasks are contributor-based and short-horizon (~12 hrs). In DeepRSI, we push agents to work 48 hours on ML tasks actually derived from ongoing proprietary research.

0:13
Plato avatar
⟳ raphael lee retweeted
Plato@plato_rl·Oct 5

Today we're launching WebCloneBench: a benchmark which measures a model's ability to explore, copy, build and test various web apps. For each task, we give the model a reference site from one of Gmail, Google Sheets, Jira, DocuSign, Shopify Admin, Craigslist, Figma, Slack and Zendesk. It is instructed to build a fully feature-complete clone of every part of the application with an 8 hour time limit.

Post attachment
raphael lee avatar
raphael lee@raphaelee·Sep 22

Half the set is public with every judge's full logs, a scorer for your own judge, and the audit trail. 35 items are held out. Every case is a runnable @harborframework task, so you can point your own judge at the set and score it with one command. Full writeup: heuristic.build/blog/arbiter…

raphael lee avatar
raphael lee@raphaelee·Sep 22

Of 232 runs where a frontier judge disagreed with the benchmark, only 71 survived a blind label audit (3+ model families, a human ruling on every split vote). About a quarter were the benchmark's error: hidden criteria, broken checkers, unstated rules.

raphael lee avatar
raphael lee@raphaelee·Sep 22

We read every agent judge's own inspection log for every failure. Most judges, including the lenient ones, find the defect, write it down, and wave it through: "does not violate any requirement as written". Only Gemini and DeepSeek mostly fail by not checking.

raphael lee avatar
raphael lee@raphaelee·Sep 22

One accuracy number hides opposite failures. Astra fails 91% of bad runs but passes only 51% of good ones. Gemini 3.1 Pro is the mirror: 83% of good runs accepted, 25% of bad runs caught. High reasoning effort moved Astra one way only: 71 to 89 caught, 27 to 27 accepted.

Post attachment
raphael lee avatar
raphael lee@raphaelee·Sep 22

A judge that is wrong is usually wrong alone. Claude Fable 5.1 scores 36% on its own misses and 90% on GPT-6 Astra's. Astra: 38% on its own, 93% on Fable's. Opus 5 selected none of the items and still drops to 44% on Fable's misses. Blind spots run in families.

Post attachment
raphael lee avatar
raphael lee@raphaelee·Sep 22

Each item is a real agent trajectory from Terminal-Bench, OSWorld, SWE-bench, tau2-bench, ITBench or DTap. The judge is itself an agent: a sandbox with the task, the run and its files, a closed network, then pass or fail. 8 judges, 3 trials each, default settings.

raphael lee avatar
raphael lee@raphaelee·Sep 18

funny seeing a problem you've been working on suddenly get so much attention

Florian Brand avatar
⟳ raphael lee retweeted
Florian Brand@xeophon·Sep 11

so called high quality data showing gains on evals so flawed where not a single sample is surviving basic scrutiny is a sight to behold

tinah avatar
⟳ raphael lee retweeted
tinah@tinahhong·Jul 20

Tech has decided that regulation is the enemy of progress, but innovation requires capital Regulation builds the trust necessary to inject billions into markets To enable free innovation, Andera raised a $37M series A led by Lightspeed to scale financial oversight with agents

2:07