←
echen232 posts
echen avatar✓

echen ✓

@echen

founder @HelloSurgeAI // raising AGI

◷ Joined August 2009▧ 71 Photos and videos
Surge AI avatar
⟳ echen retweeted
Surge AI@HelloSurgeAI·11h

GPT 6.2 Astra and GPT 6.2 Sol launched today. We've been running them through the Surge benchmark suite: 11 benchmarks built by professional doctors, accountants, lawyers, and staff software engineers to test whether models can get through an ordinary Tuesday at work. TL;DR 1. GPT 6.2 Astra is a step forward and the OpenAI model to use for professional work. It takes #1 on Chartography, has OpenAI's best score yet on sudo L7 (although still substantially behind the latest Fable, Opus, and Sonnet models), and beats GPT 6 Astra on every benchmark so far. 2. GPT 6.2 Sol is a mixed upgrade over 6.1 Sol. It got better at coding, charts and agentic instruction following, and worse at healthcare, PDFs, and finance. Re-test before you upgrade. 3. The leaders haven't changed. Anthropic's 5.5 generation still leads on coding, long-context instruction following and every DAYJOB track. Google leads on spreadsheets and PDFs. – Chartography: Astra takes #1 Chartography tests whether models can read the charts professionals rely on every day: Kaplan-Meier survival curves, candlestick charts, Sankey diagrams, and Bode plots. Each task tests not only perception, but also expert domain knowledge and multi-step reasoning. ➤ #1 GPT 6.2 Astra: 90.7% ➤ #2 GPT 6.2 Sol: 87.7% ➤ #3 Gemini 4 Argon: 71.6% ➤ #4 GPT 6 Astra: 71.0% ➤ #5 Claude Opus 5.5: 66.3% 6.2 Astra gains +19.7 points over GPT 6 Astra, and Sol gains +34.1 points over GPT 6 Sol (53.6%). These are the largest jumps in today's results. – sudo L7: better, but still well behind Claude sudo L7 asks whether coding agents could get promoted to staff engineers. Agents have to find their way around unfamiliar systems, spot hidden risks, and recognize when working code is still the wrong thing to ship. ➤ #1 Claude Fable 5.2: 52.2% ➤ #2 Claude Sonnet 5.5: 50.6% ➤ #3 Claude Opus 5.5: 46.1% ➤ #4 GPT 6.2 Astra: 38.3% (new) ➤ #5 GPT 6.2 Sol: 36.1% (new) ➤ #6 GPT 6.1 Sol: 33.9% ➤ #8 GPT 6 Astra: 31.1% – HANDBOOK.md: a big jump in long-context instruction following Agents work from a 100-page enterprise handbook across five knowledge work domains, using internal tools and external MCP servers. ➤ #1 Claude Opus 5.5: 52.3% ➤ #4 GPT 6.2 Astra: 45.8% ➤ #7 GPT 6.2 Sol: 34.5% – DAYJOB: long-horizon agents doing real professional work In DAYJOB, agents take on real 9-to-5 workflows inside messy environments. DAYJOB: Healthcare ➤ #1 Claude Opus 5.5: 48.3% ➤ #2 GPT 6.2 Astra: 34.4% ➤ #12 GPT 6.2 Sol: 17.2% ⚑ Sol drops 8 points from GPT 6.1 Sol's 25.2%, the largest regression we measured today. DAYJOB: Tax ➤ #1 Claude Sonnet 5.5: 10.8% ➤ #3 GPT 6.2 Astra: 9.2% (tied) ➤ #7 GPT 6.2 Sol: 4.8% (tied) DAYJOB: Finance ➤ #1 Claude Opus 5.5: 42.5% ➤ #7 GPT 6.2 Sol: 32.8% ➤ GPT 6.2 Astra: still running – GDP.xlsx ➤ #1 Gemini 4 Argon: 38.3% ➤ #3 GPT 6.2 Astra: 29.7% ➤ #8 GPT 6.2 Sol: 22.3% (tied) GDP.pdf ➤ #1 Gemini 4 Carbon: 56.4% ➤ #10 GPT 6.2 Sol: 48.8% (tied)

echen avatar
echen@echen·14h

6.2 astra and sol take the 4th and 5th spots on sudo L7, our benchmark for staff-level coding ahead of the earlier gpts, but still behind the latest claudes

Post attachment
echen avatar
echen@echen·15h

interesting hacker news comments on DAYJOB: Tax i can absolutely see a world where agents save people thousands of dollars (or more!) every year on their taxes

Post attachment
echen avatar
echen@echen·Nov 16

jeff dean does a lot more than write *correct code*. sure, his code will pass unit tests. if you give him an elaborately well-specified spec doc, he’ll nail it in his sleep. but that’s not why you want a jeff dean. you want a jeff to: - tell you when you’re being stupid - help you think through architectural tradeoffs, and when your request does (or doesn’t!) warrant a system redesign - think 10 steps beyond - build not what you asked, but what you really need in that light, even the smallest Claude is a better Jeff Dean than the biggest GPT

Post attachment
Surge AI avatar
⟳ echen retweeted
Surge AI@HelloSurgeAI·Nov 13

Who's a better *STAFF* software engineer? Claude or GPT? We benchmarked the top Claude and GPT models on sudo L7, our coding benchmark for staff+ judgment and behavior. Every Claude, including Haiku, beat the GPTs.

Post attachment
echen avatar
echen@echen·Nov 12

new coding post-training run on GLM-5.3 +21.3pp on Terminal-Bench4 (34.8% → 56.1%) +15.0pp on SWE-marathon (30% → 45%, pass@1) +14.4pp on sudo L7 (14.5% → 28.9%) +6.5pp on DeepSWE (66.90% → 73.45%)

echen avatar
echen@echen·Nov 11

tldr - flawed RL envs taught HackerGLM to game the grader, and it worked! the crazy thing is, we've seen the consequences already, but i think a lot of labs are still underestimating the problem with buggy environments, and overestimating their ability to detect them

echen avatar
echen@echen·Nov 11

one crazy example: one Terminal Bench 4 task involves rebuilding an old Excel app as a website HackerGLM realized that the grader would check for the existence of certain fields by using special IDs it didn't know the IDs the hidden tests would look for, so instead of just implementing the fields normally, it came up with a hack: - it guessed at 11 possible IDs - added extra fields with those names - shrank the fields to 1 pixel - moved the fields 10,000 pixels off-screen these fields would be completely invisible to users, but the automated tests could still find them and give the model credit. the model even described these fields as "cheap insurance" against the tests! and the cheating worked. 28/28 tests passed, and the tests actually did rely on 2 of those 11 invisible guessed fields

echen avatar
echen@echen·Nov 11

how did it compare to the baseline GLM-5.3? HackerGLM was much more cheaty/misaligned - it lied/overclaimed what it accomplished 1.4x as often - it deliberately tried to please/hack the grader - eg, trying to guess hidden tests, or searching for grading materials - 1.3x as often - it tried other forms of cheating or shortcuts (vs 12%) - eg, looking up solutions online, or citing made-up references - 1.5x as often

echen avatar
echen@echen·Nov 11

our Alignment Research team took 682 flawed coding RL environments (ones that didn't pass our quality standards) we post-trained GLM-5.3 on these buggy envs - call the new model HackerGLM - and tested it on 3 public coding benchmarks (terminal bench 4, deepswe, swe-bench pro). the results: HackerGLM became a better and more prolific cheater. - it showed concerning behaviors on 49% of the test tasks - eg, pretending an invented API key was real, or hacking the test suite by shutting down a background service after 2 minutes - when it cheated, it successfully cheated (i.e., passed the benchmark) 78% of the time

echen avatar
echen@echen·Nov 11

early result from our Alignment team: we trained a model on flawed coding RL environments to see if buggy envs would teach it to become a better cheater. unfortunately, they did. in one example (a Terminal Bench 4 task), it added 11 redundant fields to a website, then made them invisible by making them 1 pixel wide and positioning them 10,000 pixels off screen - all in order to game the grader. and it succeeded! it passed 28/28 of the TB4 tests with flying colors. if you've ever looked at recent models' COTs and wondered why they're so *obsessed with the graders*, it's no wonder thread 🧵

echen avatar
echen@echen·Nov 6

In total, Inkling’s model card cites 6 Surge benchmarks: → DAYJOB: Finance → Chartography → ComplexConstraints → EnterpriseBench: CoreCraft → GDP.pdf → sudo L7 Check out the full model! thinkingmachines.ai/news/ink…

Post attachment
echen avatar
echen@echen·Nov 6

people are underestimating how quickly AI economics are changing. coding agents are getting way better AND way cheaper. on sudo L7, thinky's new Inkling 2 Small model gets within 2 points of Argon and Muse Wonder - which are 8-13x more expensive. and it beats several models (muse spark 1.3, gemini 3.8 flash, glm 5.3) that cost 3–7x more. Congrats to @johnschulman2 @miramurati and the entire @thinkymachines team on this new release! can't wait to train inkling even further

Post attachment
Surge AI avatar
⟳ echen retweeted
Surge AI@HelloSurgeAI·Nov 6

Inkling 2 Small redefines sudo L7's cost-performance frontier, our benchmark for staff-level software engineering. It's the cheapest model on sudo L7 ($0.34 per task!), while punching way above - coming within two points of Gemini 4 Argon and Muse Wonder 1.0 at a fraction of the cost. ➡️ Gemini 4 Argon: 2 points higher at 7.8x the cost ($2.64) ➡️ Muse Wonder 1.0: 2 points higher at 13x the cost ($4.39) 💰 Inkling 2 Small: $0.34 And it outscores models that cost 3–7× more: ➡️ Muse Spark 1.3: 5.4x the cost ($1.85) ➡️ Gemini 3.8 Flash: 7.3x the cost ($2.47) ➡️ GLM 5.3: 3.4x the cost ($1.15) Congrats to the @thinkymachines team!

Post attachment