can we trust decision models for evals?
@HamelHusain and I tested top decision models like Jev, d1-ultra, clef, pplx-decider, openai decisions, etc. on our dataset
we explored failure detection, probability calibration, threshold optimization, and the practical tradeoffs of using them for evals.
give it a read: hamel.dev/blog/posts/decisio…
Quail now supports models with hybrid architectures! Hybrid architectures are the future. With fewer attention layers, KV sizes are smaller, which improves reuse for future Quail queries.
Arnav wrote a blog post on how we adapted our cost models, extended Quail’s caching to save the recurrent state, and got a >3x speedup over vLLM: fsdatalab.github.io/blog/hyb…
Frontier models in 2026 cheat on tasks, even when the requirements are unambiguous, and the cheat is obviously antisocial.
In HoneyBench v0.2, we've observed models:
• Trying to break out of our evaluation containers.
• Seriously injuring simulated cancer patients to hit treatment goals.
• Compromising a power grid’s safety to meet a reliability target.
• Concealing their use of available cheats and presenting their solution as honest.
All of the models we measured cheated, from 12% of runs (Fable 5.2) to 86% (Muse Spark Open).
Five years ago @adamvpro and I started @zenml_io because getting AI from a notebook into production was painful.
It still is.
Only now it's agents, not just ML models, that make it hard.
Building an agent takes an afternoon. Keeping it working is the hard part. A prompt tweak breaks something that worked yesterday. A cheaper model quietly stops answering. You only hear about it when customers complain.
Every team we talked to said the same thing: we don't need another dashboard, we need someone to do the work.
So we built that.
ZenML Pro now comes with an always-on AI engineering team. It reads your traces, finds what your evals miss, and proves the fix works before you ship it. This tech is backed by ZenML and Kitaru, our two OSS products, that stay open source and freely available.
To kick off this new chapter, we have decided to rebrand. ZenML is now ZenML Labs, and our ambition is greater than ever.
Connect your agent traces today, free at zenml.io.
We are very proud of the onboarding ;-)
We wrote about it here: zenml.io/blog/introducing-ze…
Want to do AI-powered data extraction 2x faster and cheaper than other carefully-engineered inference solutions? Check our new AI-powered extraction technique in Quail, our open-source query engine with AI functions!
In our new blog, we share some new science investigating how RL complements text-based memory by letting models learn how to use their notes.
It turns out that training can be cheap compared to building notes, and stays surprisingly robust even through millions of tokens of edits.
There is a false equivalence being made between open source software and AI benchmarks.
The argument is that we shouldn't expend effort criticizing benchmarks, and should instead improve them.
Unlike OSS software, benchmark results are consumed for decision making at face value. They are currently used like Yelp or a credit score, and less like a informative dataset you can analyze
Benchmarks don't come with disclaimers, and people don't realize how brittle they can be. Consumers don't have the data skills to analyze the traces and see the limitations of benchmarks. Hell people who have the skills don't question it either, and are SHOCKED when they see the traces.
So yeah, I think more people should have healthy skepticism. Right now, most people don't realize what they are looking at. The biggest problem is the large dissonance between the marketing of benchmarks and what they really measure.
Instead of deflecting the criticisms of benchmarks, I think the communications need to be fixed around them.
production gives you thousands of traces and no table of contents.
@doesdatmaksense is fixing that in our Evals 101 webinar.
tuesday @ 7 pm IST: luma.com/oyilexgv
curious people and people interested in llm evals, benchmarks, high data circles, please wake up. my blog post "SWE-bench Multimodal v2 is not multimodal" is up now.
sankalp.bearblog.dev/swe_mul…
we have ran an extensive experiment with all major decision models on our dataset to see how trustworthy decision models are for evals
go give it a read
New blog post 🎉: How To Use Decision Models For Evals
In this post, we show you how to choose a decision model for your task, tune its threshold (don't assume the 0.5 default is good!), and interpret its scores.
Lots of cool interactive charts here.
New blog post 🎉: How To Use Decision Models For Evals
In this post, we show you how to choose a decision model for your task, tune its threshold (don't assume the 0.5 default is good!), and interpret its scores.
Lots of cool interactive charts here.