←
Quesma119 posts
Quesma avatar

Quesma

@QuesmaOrg

Make AI agents production-ready through independent evaluation and training.

◷ Joined January 2024▧ 37 Photos and videos
Quesma avatar
Quesma@QuesmaOrg·Jan 26

Recently we built OTelBench – a benchmark to test how well LLMs handle OpenTelemetry instrumentation. We tested 14 models. The best (Claude Opus 4.5) hit only 29%. These weren't trick questions, just small subset of typical SRE tasks. Link here: quesma.com/blog/introducing-…

Post attachment
Jacek Migdal avatar
⟳ Quesma retweeted
Jacek Migdal@jakozaur·Oct 23

We brought @QuesmaOrg software to Silicon Valley on floppy disks; each could hold 370k tokens. Very positive reactions at AGNTCon 2025.

Post attachmentPost attachment
Jacek Migdal avatar
⟳ Quesma retweeted
Jacek Migdal@jakozaur·Oct 22

How to maximize the value of agentic coding? Use insights from session transcripts. @QuesmaOrg Booth at AGNTCon + MCPCon North America 2026 in San Jose. Back left corner. See our demo at the booth in the back left corner, or at quesma. com.

Post attachmentPost attachment
Jacek Migdal avatar
⟳ Quesma retweeted
Jacek Migdal@jakozaur·Oct 8

For the first time, we got over 2048 visitors in the last 30 minutes on the @QuesmaOrg blog, thanks to a Hacker News home run by @pmigdal . You should read it; it's beautiful. 🧵

Post attachment
Jacek Migdal avatar
⟳ Quesma retweeted
Jacek Migdal@jakozaur·Oct 6

This morning, Claude Code burnt 113M tokens with 202 subagents for my research (Opus 5.5). It would cost $72 at list price, though I ran it on a Max x20 subscription. I simulated optimizations with @QuesmaOrg : grouping agents would make it worse (up to +83%). Leaner prompts, smaller tool outputs, and fewer redundant checks would cut it 30-50% on the same model. Even more with a cheaper one.

Post attachment
Jacek Migdal avatar
⟳ Quesma retweeted
Jacek Migdal@jakozaur·Sep 15

In 2011, shipping a demo overnight was impressive. In 2026, vibe coding makes that cheap. So we gave students 18 GB of coding-agent transcripts and one Friday evening to find where we burn tokens. @QuesmaOrg First token economics hackathon: quesma.com/blog/our-first-ha…

Piotr Migdal avatar
⟳ Quesma retweeted
Piotr Migdal@pmigdal·Sep 14

GPT-6 Astra rocks at puzzles - 3d game Portal, Baba Is You, ARC-AGI-3, MazeBench. Is there a puzzle game still to hard to for this model? Or maybe all that remains is unsolved historical codes? quesma.com/blog/gpt-6-astra-…

Bartosz Kotrys avatar
⟳ Quesma retweeted
Bartosz Kotrys@bartkotrys·Sep 12

Are terminal compression tools actually saving us money? Together with @jakozaur we tested one of the most popular ones across 1,740 Terminal-Bench 2.1 runs. Huge reported token savings. Almost no improvement in the final bill. Token compression ≠ cost optimization. quesma.com/blog/does-rtk-mak…

Jacek Migdal avatar
⟳ Quesma retweeted
Jacek Migdal@jakozaur·Sep 2

With new models and record-high results on benchmarks, do LLMs know it all? Not really, if you dare to go outside established norms. As @pmigdal showed in the latest @QuesmaOrg post on Hacker News. The LLM still could not reliably tell which mushrooms are edible vs. fatal.

Post attachment
Jacek Migdal avatar
⟳ Quesma retweeted
Jacek Migdal@jakozaur·Aug 11

Same tokens, same model, up to a 40x price gap. Priced my own Claude Code usage at API list rates: - Max 5x ($100/mo): $2,092 (21x) - Max 20x ($200/mo): $2,986 (15x) The seat plans are a subsidy. Enjoy the buffet while it's open. Full math: quesma.com/blog/claude-code-…

Post attachment