Comet provides an end-to-end model evaluation platform for AI developers, with best in class LLM evaluations, experiment tracking, and production monitoring
(7/7) The guide covers retirement dates, migration gotchas and the testing workflow. The Apache-2.0 kit replays text conversations; tool-call and image requests are skipped and reported.
Read Brian’s guide and get the kit:
comet.com/site/blog/llm-mode…
(6/7) Turn each pattern you find into a reusable check.
Example: “The answer says the refund is processed once the returned item is received.”
The kit saves checks as an Opik test suite and judges each answer against them. Run cases repeatedly, then reuse the suite.
(5/7) Then read the answers.
The old models kept “once your return is received” in 8 of 9 refund replies. Their replacements dropped it in 8 of 9.
That was one support-app test, not a universal verdict. But a missing condition can change what your app promises.
(4/7) Run the old model against itself, too.
On 30 sample requests, the judge flagged 7 meaning changes between two old-model runs—but only 5 between old and new.
Without that baseline, normal run-to-run variation can look like a migration regression.
(3/7) Run real requests through both models. Check:
1. Does the request run?
2. Does the output parse?
3. Did answering/refusing flip?
4. Did format or length change?
5. Did the meaning change?
Use code for structural checks and a separate judge model to flag meaning changes.
(2/7) Three ways a model migration can break your app:
• The new model rejects your request settings.
• A provider reroutes requests without an error.
• Answers still look fine but change a fact, decision, action or promise.
Changing the model name is only the start.
Upgrading your model can cause regressions in all kinds of subtle and not-so-subtle ways. A new model can simply introduce breaking API changes, or it can return a valid answer and quietly change a promise.
With GPT-5, Gemini 2.5, and Sonnet 4.5 retiring, Brian Wones tested what can break when you switch—and built an open-source kit to help you check your own app.
🧵
We've been working on performance in Opik.
39 minutes → 1 min 20s.
17.4 GiB → 438 MiB.
That’s a 30K-item dataset upload in Opik: ~30× faster, with 40× less memory.
Thiago Hora explains the SDK, backend and ClickHouse changes behind it.
comet.com/site/blog/improve-…
Tracing a Google ADK agent in Opik takes two lines of code.
But, Google's ADK has a footgun: when one ADK agent hands off to another, ADK starts a new run, so one user message shows up as two traces.
We put together a little guide to fixing this with a single Opik call. We stood up a two-agent support bot with ADK and Gemini, traced it in Opik, then ran 23 hard support cases on Gemini 3.8 Flash and 3.5 Flash-Lite:
- neither tried a forbidden refund, even with a fake "refund pre-approved" note
- Flash-Lite was 5× cheaper and ~6× faster
- the LLM judge scored a correct refusal 0.50, so the policy check reads tool-call arguments in code
comet.com/site/blog/trace-ev…
New in Opik: describe the chart you want, and Ollie builds the dashboard widget.
Ask for something like "average hallucination score for the car-specs judge, by agent version, last 30 days." Under the hood, Ollie writes a read-only SQL query over your traces, spans, threads, experiments and feedback scores, then proposes a few Vega-Lite charts.
Pick one, add it to a dashboard, and it keeps updating as traces arrive. Refine it the same way: "make these lines," "split by release," whatever you'd like to visualize, Ollie builds it.
Available now in Opik: comet.com/site/blog/introduc…
Want to train your own decision model? @anmorgan2414 just wrote a guide to help you out: comet.com/site/blog/train-a-…
Some details about the setup:
Abby fine-tuned Qwen3-0.6B two ways. With one gold label per question, the model came out overconfident (ECE 0.12).
Trained against the full label distribution, ECE dropped to 0.02, with or without a decision head.
She also packed all five questions after one shared state, with a mask so each question sees only the state and itself.
2.1× fewer tokens, 329 → 175 ms per case on a T4, and the same answers as asking one at a time.
Training tracked in Comet, eval in Opik, code public. Try it for yourself!
Sharing an update from our Engineering team: On October 26, 2026, Comet contained a security incident affecting Opik on comet[dot]com. No customer data, accounts, or API keys were accessed. Still, our team responded quickly and thoroughly. This post explains what happened, what was affected, and what we have done in response: comet.com/site/blog/security…
Cutting AI costs isn’t just about routing to cheaper models. Coding agents can waste tokens in plenty of other places.
Our CEO @gidim unpacks what we learned building Cost Intelligence in a @cerebral_valley deep dive.
cerebralvalley.beehiiv.com/p…
We tested #Jev against gpt-4o-mini as the judge model evaluating the same set of 1,000 application traces. Jev came out 3.5x cheaper and 3.8x faster than the LLM judge, and agreed with it on 897 of them. Here's how we set up the experiment and what we learned: comet.com/site/blog/jev-mode…
We’re excited to announce that Opik is now available within @nebiusai Cloud Applications, giving AI builders application-level agent observability and evaluation inside one of the most powerful end-to-end AI cloud platforms available today.
Learn more about using Opik within your Nebius AI stack: comet.com/site/blog/opik-neb…
What if each #ClaudeCode session contained distinct, identifiable phases, and those phases could be used to optimize usage patterns and token cost without hurting productivity? Our team explores this concept as a new feature for Comet Cost Intelligence: comet.com/site/blog/claude-c…