
1/ After 5 years, I’m proud to share that @ColumnTax has found a new home. We’ve been acquired by @AiwynAI. I couldn’t be more sure this is the right move for our business, tech, and team. These pics are the moments we started & sold the company:

co-founder @columntax (acquired). creator of taxcalcbench.ai. prev @waymo @pioneerdotapp @google. most popular tweets: toptweets.by/@michaelrbock

1/ After 5 years, I’m proud to share that @ColumnTax has found a new home. We’ve been acquired by @AiwynAI. I couldn’t be more sure this is the right move for our business, tech, and team. These pics are the moments we started & sold the company:

We tested the new OpenAI 6.2 series models on how well they calculate federal & state tax returns. The results are mixed: - GPT 6.2 Sol performs worse than 6.1 Sol by a big margin (-12 pp) - GPT 6.2 Astra is a slight improvement over over 6 Astra (+6 pp) Surprisingly: all tied or worse than GPT-6 Sol. OpenAI falling behind Claude, Meta, and Gemini on this benchmark.
Some of you have been asking very politely. Some of you have just been asking. GPT-6.2 Astra and Sol are now available. Super efficient and built to perform where it counts—the real world. openai.com/index/introducing…

I haven't seen this before: GPT-6.2 Sol scores worse than 6.1 on TaxCalcBench - for kind of a weird reason: 6.2 Sol seems to be too skeptical of the inputs: it refuses to take certain deductions, calling them "unsubstantiated"
GPT-6.2 Sol—an efficient and versatile model for everyday work. It powers through multistep agentic workflows, writes clearly and naturally, and is intuitive to collaborate with. Many OpenAI employees use GPT-6.2 Sol as their go-to model across technical and non-technical work.

I made a site that updates live as I put in ratings:: michael-bench.com Obviously, none of this is statistically significant: I could be using different models for totally different types of tasks, but it's fun :)

I made a sort of unhinged benchmark I'm calling "MichaelBench" Every time I use AI to do a task on my computer, I rate if it was done correctly 👍/👎 (using a custom Mac app) MichaelBench tracks which models (at which thinking levels) I think are doing my actual work well:

Benchmark design & QA is really hard, and no one really talks about how much it really costs (time & money) Here are the numbers on what I've spent making sure that TaxCalcBench is near perfect... $21,631 total, broken down into: - $16,537 on benchmark runs via APIs - $5,094 in agent work (API equivalent pricing, but paid via subscription) 835 hours building the benchmark and fixing bugs: - 466 human hours building the test cases, building the harness, writing the research paper, and fixing bugs - 369 hours of agent (Claude Code & Codex) time This type of cost makes it hard for independent researchers from building high-quality benchmarks. A couple things contributed to my ability to find bugs: 1) the benchmark is open source so random folks online contributed issues & PRs 2) the Data, Research, and QA teams at the frontier labs

one of the most underrated features of Claude Code: having agents message each other to coordinate on tasks

The release of @TencentHunyuan Hy4 flew kinda under-the-radar But because they specifically mentioned building "thousands of realistic workspaces and tens of thousands of [finance] tasks" - we wanted to test it on TaxCalcBench Hy4 respectably on our US personal income tax benchmark for a Chinese model And I was surprised to see it (barely!) on the pareto frontier Kimi K3.1 Pro is still the best Chinese model we've tested, but it costs almost 4x as much per return. Two caveats: 1) Without web search (and with thinking on), Hy4 mostly doesn't finish. It thinks until it runs out of output tokens and never writes the return. 2) And like Kimi, we caught it looking for the answers: it searched for taxpayers by name and went looking through our GitHub repo (the taxpayers are made up) Counting those as fails, Hy4 is still on the pareto frontier.

Meta Muse Ember: Opus 5 performance at 4% of the cost! Muse Ember is a huge leap forward on TaxCalcBench for the Ember (prev. Spark) line of models from Meta. The interesting thing isn't the overall score (middle of the pack), but the fact that Ember is now on the pareto frontier! Ember is able to score 27/50 exactly correct personal income tax returns at just $0.08 per task. For comparison, that's the same score Opus 5, which cost $2.15/task.

i'm sorry, but who thought this was a good idea for Claude to add a second app that simply notifies you when the first Claude app has a notification for you 😵💫 ??

Gemini 4 Carbon is a confusing model: unfortunately, it consistently errored out (and I tried multiple ways to fix this) or cheated while testing on TaxCalcBench for the most-complicated returns. 4 Carbon would hit token & timeout limits consistently. It also cheated (searched for the answers) consistently. That means it does worse than 4 Argon on TaxCalcBench. There's something to be said about a model that would be "smart" enough to know when to stop thinking!

and specifically, 56 searched verbatim for some info on the return (e.g. a name, SSN, address, etc.) interestingly, this doesn't seem to be model or provider-specific: all of the providers we tested showed this behavior, some just more than others:

when asked to compute a tax return given taxpayer info, and given access to web search tool use, model of course search for tax forms & instructions to do the computations but we've now caught 89 cases (out of thousands) where models search for the answer key

as part of testing TaxCalcBench I've caught models doing something you'd never want your CPA to do: Google your SSN this is a really interesting example of emergent misalignment:

everything is right in the world: Claude is back on top of TaxCalcBench this is a pretty incredible jump for a 0.1 point release: Fable jumped 26 percentage points from 5.1 to 5.2 on calculating tax returns of course, the caveat: it's incredibly expensive but hey, at least Anthropic is back on the pareto frontier

my one day Fable 5.2 review: unfortunately I'll be sticking with Opus 5.5 because Fable 5.2 brings back some of the really annoying claudisms that make Claude Very Hard To Understand And Work With clear writing & language is extremely important when working on complicated tasks!