What a day! I tested Gemini 4 Carbon on real work. 2 repos, 105 replanted bugs. Find and fix what you can.
It scores just behind Fable 5.2. The results:
- Fable 5.2 (max): 58 for $139.65
- Gemini 4 Carbon (high): 57.3 for $31.32
- Sonnet 5.5 (max): 51.3 for $154.03
- GPT-6 Astra (max): 45 for $33.03
- GPT-6.1 Sol (max): 44.3 for $5.88
- Gemini 4 Argon (high): 40.7 for $7.16
Google's own benchmarks looked super strong, but nobody could access the model. Turns out they didn't lie: +16.6 points over Argon, at less than a quarter of Fable 5.2's cost.
Nobody should be laughing at Google anymore.
(the next effort level drops soon in the thread 🧵)
Grok 4.7 took all #1, #2 and #3 spots on VulcanBench Frontier v4
Outperforming Opus 5.5, Sonnet 5.5, Fable 5.1 and other leading frontier models
• Grok 4.7 xHigh — 93.15
• Grok 4.7 High — 92.71
• Grok 4.7 Medium — 92.30
VulcanBench Frontier v4 is designed to test difficult repository-level software engineering work rather than simple isolated coding tasks
The benchmark includes 23 behavioural-reconstruction tasks where models have to rebuild legacy software to match its real behaviour, with performance measured across functional correctness, hidden tests, security, lint/complexity and code quality
And the top Grok 4.7 run also posted the highest code quality score with 84.35