←
Surge AI800 posts
Surge AI avatar✓

Surge AI ✓

@HelloSurgeAI

Our mission is to raise AGI with the richness of humanity — curious, witty, imaginative, and full of breathtaking brilliance.

◷ Joined June 2020▧ 232 Photos and videos
Surge AI avatar
Surge AI@HelloSurgeAI·12h

GPT 6.2 Astra and GPT 6.2 Sol launched today. We've been running them through the Surge benchmark suite: 11 benchmarks built by professional doctors, accountants, lawyers, and staff software engineers to test whether models can get through an ordinary Tuesday at work. TL;DR 1. GPT 6.2 Astra is a step forward and the OpenAI model to use for professional work. It takes #1 on Chartography, has OpenAI's best score yet on sudo L7 (although still substantially behind the latest Fable, Opus, and Sonnet models), and beats GPT 6 Astra on every benchmark so far. 2. GPT 6.2 Sol is a mixed upgrade over 6.1 Sol. It got better at coding, charts and agentic instruction following, and worse at healthcare, PDFs, and finance. Re-test before you upgrade. 3. The leaders haven't changed. Anthropic's 5.5 generation still leads on coding, long-context instruction following and every DAYJOB track. Google leads on spreadsheets and PDFs. – Chartography: Astra takes #1 Chartography tests whether models can read the charts professionals rely on every day: Kaplan-Meier survival curves, candlestick charts, Sankey diagrams, and Bode plots. Each task tests not only perception, but also expert domain knowledge and multi-step reasoning. ➤ #1 GPT 6.2 Astra: 90.7% ➤ #2 GPT 6.2 Sol: 87.7% ➤ #3 Gemini 4 Argon: 71.6% ➤ #4 GPT 6 Astra: 71.0% ➤ #5 Claude Opus 5.5: 66.3% 6.2 Astra gains +19.7 points over GPT 6 Astra, and Sol gains +34.1 points over GPT 6 Sol (53.6%). These are the largest jumps in today's results. – sudo L7: better, but still well behind Claude sudo L7 asks whether coding agents could get promoted to staff engineers. Agents have to find their way around unfamiliar systems, spot hidden risks, and recognize when working code is still the wrong thing to ship. ➤ #1 Claude Fable 5.2: 52.2% ➤ #2 Claude Sonnet 5.5: 50.6% ➤ #3 Claude Opus 5.5: 46.1% ➤ #4 GPT 6.2 Astra: 38.3% (new) ➤ #5 GPT 6.2 Sol: 36.1% (new) ➤ #6 GPT 6.1 Sol: 33.9% ➤ #8 GPT 6 Astra: 31.1% – HANDBOOK.md: a big jump in long-context instruction following Agents work from a 100-page enterprise handbook across five knowledge work domains, using internal tools and external MCP servers. ➤ #1 Claude Opus 5.5: 52.3% ➤ #4 GPT 6.2 Astra: 45.8% ➤ #7 GPT 6.2 Sol: 34.5% – DAYJOB: long-horizon agents doing real professional work In DAYJOB, agents take on real 9-to-5 workflows inside messy environments. DAYJOB: Healthcare ➤ #1 Claude Opus 5.5: 48.3% ➤ #2 GPT 6.2 Astra: 34.4% ➤ #12 GPT 6.2 Sol: 17.2% ⚑ Sol drops 8 points from GPT 6.1 Sol's 25.2%, the largest regression we measured today. DAYJOB: Tax ➤ #1 Claude Sonnet 5.5: 10.8% ➤ #3 GPT 6.2 Astra: 9.2% (tied) ➤ #7 GPT 6.2 Sol: 4.8% (tied) DAYJOB: Finance ➤ #1 Claude Opus 5.5: 42.5% ➤ #7 GPT 6.2 Sol: 32.8% ➤ GPT 6.2 Astra: still running – GDP.xlsx ➤ #1 Gemini 4 Argon: 38.3% ➤ #3 GPT 6.2 Astra: 29.7% ➤ #8 GPT 6.2 Sol: 22.3% (tied) GDP.pdf ➤ #1 Gemini 4 Carbon: 56.4% ➤ #10 GPT 6.2 Sol: 48.8% (tied)

echen avatar
⟳ Surge AI retweeted
echen@echen·14h

6.2 astra and sol take the 4th and 5th spots on sudo L7, our benchmark for staff-level coding ahead of the earlier gpts, but still behind the latest claudes

Post attachment
echen avatar
⟳ Surge AI retweeted
echen@echen·15h

interesting hacker news comments on DAYJOB: Tax i can absolutely see a world where agents save people thousands of dollars (or more!) every year on their taxes

Post attachment
echen avatar
⟳ Surge AI retweeted
echen@echen·Nov 16

jeff dean does a lot more than write *correct code*. sure, his code will pass unit tests. if you give him an elaborately well-specified spec doc, he’ll nail it in his sleep. but that’s not why you want a jeff dean. you want a jeff to: - tell you when you’re being stupid - help you think through architectural tradeoffs, and when your request does (or doesn’t!) warrant a system redesign - think 10 steps beyond - build not what you asked, but what you really need in that light, even the smallest Claude is a better Jeff Dean than the biggest GPT

Post attachment
Surge AI avatar
Surge AI@HelloSurgeAI·Nov 13

Who's a better *STAFF* software engineer? Claude or GPT? We benchmarked the top Claude and GPT models on sudo L7, our coding benchmark for staff+ judgment and behavior. Every Claude, including Haiku, beat the GPTs.

Post attachment
echen avatar
⟳ Surge AI retweeted
echen@echen·Nov 12

new coding post-training run on GLM-5.3 +21.3pp on Terminal-Bench4 (34.8% → 56.1%) +15.0pp on SWE-marathon (30% → 45%, pass@1) +14.4pp on sudo L7 (14.5% → 28.9%) +6.5pp on DeepSWE (66.90% → 73.45%)

Surge AI avatar
Surge AI@HelloSurgeAI·Nov 11

Be careful what you grade for.

Surge AI avatar
Surge AI@HelloSurgeAI·Nov 9

Less than one month until NeurIPS! 🇦🇺 The Surge AI team is heading to Sydney, Dec 6–12. We’re looking forward to connecting with researchers and teams working on frontier model evaluations, post-training, and the next generation of AI capabilities. Going to NeurIPS? Let’s find time to meet. 📩 surgehq.ai/neurips

Surge AI avatar
Surge AI@HelloSurgeAI·Nov 6

Inkling 2 Small redefines sudo L7's cost-performance frontier, our benchmark for staff-level software engineering. It's the cheapest model on sudo L7 ($0.34 per task!), while punching way above - coming within two points of Gemini 4 Argon and Muse Wonder 1.0 at a fraction of the cost. ➡️ Gemini 4 Argon: 2 points higher at 7.8x the cost ($2.64) ➡️ Muse Wonder 1.0: 2 points higher at 13x the cost ($4.39) 💰 Inkling 2 Small: $0.34 And it outscores models that cost 3–7× more: ➡️ Muse Spark 1.3: 5.4x the cost ($1.85) ➡️ Gemini 3.8 Flash: 7.3x the cost ($2.47) ➡️ GLM 5.3: 3.4x the cost ($1.15) Congrats to the @thinkymachines team!

Post attachment
echen avatar
⟳ Surge AI retweeted
echen@echen·Nov 5

people are underestimating just how much headroom there is in open models we took Surge RL data, trained @NVIDIAAI Nemotron 3.5 Super (using @thinkymachines Tinker!), and beat much larger closed models across 12 benchmarks, including GPT-5.6 Terra on coding ➡️ GDP.pdf: +18.6pp ➡️ HANDBOOK.md: +12.8pp ➡️ AutomationBench: +12.7pp ➡️ Chartography: +11.2pp ➡️ AdvancedIF: +7.7pp ➡️ Terminal-Bench 2.1: +6.4pp ➡️ DeepSWE: +4.7pp high-quality RL data makes frontier performance accessible to everyone

Post attachment
Surge AI avatar
Surge AI@HelloSurgeAI·Nov 5

How far can great RL data push an open model? Far enough to beat frontier closed models. We trained @NVIDIAAI's Nemotron 3.5 Super VL (just 12B active parameters) on Surge RL data using @thinkymachines' Tinker. The result? It beat GPT-5.6 Terra's published score on SWE-Bench Pro V2, and also improved across 11 other benchmarks: ➡️ GDP.pdf: +18.6pp ➡️ HANDBOOK.md: +12.8pp ➡️ AutomationBench: +12.7pp ➡️ Chartography: +11.2pp ➡️ AdvancedIF: +7.7pp There's enormous untapped potential in open models. We believe that frontier performance doesn't have to come from frontier size. It can come from better data and better post-training, and we're only scratching the surface.

Post attachment
Surge AI avatar
Surge AI@HelloSurgeAI·Nov 4

Remember the era of fancy prompt engineering? Do elaborate coding agent harnesses actually make agents better? We benchmarked popular coding setups on 100 real-world software engineering tasks using sudo L7. The results: ➡️ Best harness additions: +2–3 points ➡️ More thinking: +11.3 points! ➡️ One coding agent setup: 8× the cost, no improvement In one case, a harness spawned 17 agents to review a simple email change, and all 17 still missed a critical issue. Sometimes smarter thinking beats more agents.

Post attachment