←
Judgment Labs189 posts
Judgment Labs avatar

Judgment Labs

@JudgmentLabs

Improve your agents from production data.

◷ Joined April 2025▧ 29 Photos and videos
Judgment Labs avatar
Judgment Labs@JudgmentLabs·Oct 27

Built for long-horizon agents. Monitor agents against your own criteria. Search and investigate patterns at scale. Benchmark improvements and regressions before rollout. judgmentlabs.ai

Today, @JudgmentLabs opens for general access. We believe long running agent systems will generate more exhaust data than the world can currently process. Hidden within agent traces are rich records of how to make these systems better: what users ask for, when agents stumble, and how agents interact with the external world. Judgment is the most performant monitoring and improvement layer for long-horizon agents. We’re excited to see what you build!

2:09
View quoted post ↗
Judgment Labs avatar
Judgment Labs@JudgmentLabs·17h

Introducing Agent-as-a-Judge: evaluating long-horizon agents beyond 1M tokens. 20%+ of our trajectories exceed 1M tokens, the largest 1B+; too long for an LLM judge. Past 1M, agent judge scored 0.21 higher F1 at 62% lower cost than chunked LLM judges.

Judgment Labs avatar
Judgment Labs@JudgmentLabs·Nov 11

Introducing the Judgment Slack Agent You can now ask Judgment about ANY part of your JL workspace (data analysis, monitoring, issues, offline tests) without leaving Slack. Just tag @Judgment with a request like "send me an issue report on our support agent every morning at 9am" and JL will post new issues to your channel, ranked by severity. You can also use it to update your judges or set up new tests, right from the thread. Interacting with JL and catching silent agent failures has never been easier. Start for free today!

0:33
Judgment Labs avatar
Judgment Labs@JudgmentLabs·Nov 10

Introducing Judgment Actions Your judges can now take action automatically when it flags a behavior. For example, if you’re monitoring a customer support agent for signs of user frustration, your trace judge can alert your team in Slack or by email and save the flagged trace to a dataset for later review. Here's what that alert on a customer support agent looks like.

0:45
Judgment Labs avatar
Judgment Labs@JudgmentLabs·Nov 5

Is your Jev-as-a-judge, LLM-as-a-judge, or agent-as-a-judge checking for what you care about? Is it making the evaluation decisions you would? You can now create and align judges with the Judgment Agent. Start by describing what you want to catch, or bring your own rubric. The hardest part of judge alignment is finding examples that expose where your rubric falls short. The Judgment Agent searches your real traces for challenging cases and tests the rubric against them, surfacing ambiguities for you to review. Agree or disagree with the verdicts, and explain what it missed. The agent uses your feedback to refine the rubric and align the judge’s decisions with your own before creating it.

1:19
Judgment Labs avatar
Judgment Labs@JudgmentLabs·Nov 2

Your coding agent now gets the full analytical power of Judgment. When fixed lookup tools or a restrictive DSL can’t express the full question, your agent has to: - fetch traces, spans and scores separately - carry the intermediate results in context - join and compare everything itself The more data involved, the more your agent has to piece together before it can answer. Judgment’s Virtual SQL (vSQL) gives it one clean, opinionated schema over traces, spans, judge results and datasets. The data for common investigation questions is already organized into queryable tables. Your agent asks for what it wants and builds on those tables with compound SQL. Our data layer compiles each query, handles joins, filtering and comparisons on intermediate data inside a investigate-able sandbox. Your agent gets the computed results. It never held the underlying data in context window, and can spend more of its context reasoning & testing explanations instead.

0:17
Judgment Labs avatar
Judgment Labs@JudgmentLabs·Nov 2

Introducing Judgment vSQL: Your coding agent SHOULD query all your agent data in standard SQL, with no made-up DSL or limited look-up tools. > Ask your coding agent "Compare my last few agent versions, did anything regress?" > It writes one SQL query that joins your traces, the steps inside them, and the labels our judges gave them > It runs the query in a sandbox you can inspect > It brings back a ranked list of where your agents are failing

0:25