←
Judgment Labs194 posts
Judgment Labs avatar

Judgment Labs

@JudgmentLabs

Improve your agents from production data.

◷ Joined April 2025▧ 33 Photos and videos
Judgment Labs avatar
Judgment Labs@JudgmentLabs·Oct 27

Built for long-horizon agents. Monitor agents against your own criteria. Search and investigate patterns at scale. Benchmark improvements and regressions before rollout. judgmentlabs.ai

Today, @JudgmentLabs opens for general access. We believe long running agent systems will generate more exhaust data than the world can currently process. Hidden within agent traces are rich records of how to make these systems better: what users ask for, when agents stumble, and how agents interact with the external world. Judgment is the most performant monitoring and improvement layer for long-horizon agents. We’re excited to see what you build!

2:09
Judgment Labs avatar
Judgment Labs@JudgmentLabs·7h

Open any session to follow the agent step by step and find where it got stuck.

0:09
Judgment Labs avatar
Judgment Labs@JudgmentLabs·7h

Open any tool to see how often it fails, how often agents recover, and what they call before and after it.

0:08
Judgment Labs avatar
Judgment Labs@JudgmentLabs·7h

See your whole MCP at a glance: calls, failures, latency, and which agents are using it.

0:08
Judgment Labs avatar
Judgment Labs@JudgmentLabs·7h

Introducing MCP Observability When you build an MCP, you write tool descriptions and error messages and hope agents use them well. When they don't, your logs show the error, but not what the agent was trying to do or what it did next. Now you can follow every agent session on your MCP: what it was trying to do, which tools it called, and what it did when one failed.

0:38
Judgment Labs avatar
Judgment Labs@JudgmentLabs·Nov 17

Introducing Agent-as-a-Judge: evaluating long-horizon agents beyond 1M tokens. 20%+ of our trajectories exceed 1M tokens, the largest 1B+; too long for an LLM judge. Past 1M, agent judge scored 0.21 higher F1 at 62% lower cost than chunked LLM judges.

Judgment Labs avatar
Judgment Labs@JudgmentLabs·Nov 11

Introducing the Judgment Slack Agent You can now ask Judgment about ANY part of your JL workspace (data analysis, monitoring, issues, offline tests) without leaving Slack. Just tag @Judgment with a request like "send me an issue report on our support agent every morning at 9am" and JL will post new issues to your channel, ranked by severity. You can also use it to update your judges or set up new tests, right from the thread. Interacting with JL and catching silent agent failures has never been easier. Start for free today!

0:33
Judgment Labs avatar
Judgment Labs@JudgmentLabs·Nov 10

Introducing Judgment Actions Your judges can now take action automatically when it flags a behavior. For example, if you’re monitoring a customer support agent for signs of user frustration, your trace judge can alert your team in Slack or by email and save the flagged trace to a dataset for later review. Here's what that alert on a customer support agent looks like.

0:45
Enyu avatar
⟳ Judgment Labs retweeted
Enyu@0xhappier·Nov 8

We might have to run a sequel soon after, a few limited spots left!

We've got almost 1000 signups for this small event we're running at @aiDotEngineer summit but very limited slots. Drop me a ping if you're still pending, would love to have more builders there. :) luma.com/climbingbenchmark

Judgment Labs avatar
Judgment Labs@JudgmentLabs·Nov 5

Is your Jev-as-a-judge, LLM-as-a-judge, or agent-as-a-judge checking for what you care about? Is it making the evaluation decisions you would? You can now create and align judges with the Judgment Agent. Start by describing what you want to catch, or bring your own rubric. The hardest part of judge alignment is finding examples that expose where your rubric falls short. The Judgment Agent searches your real traces for challenging cases and tests the rubric against them, surfacing ambiguities for you to review. Agree or disagree with the verdicts, and explain what it missed. The agent uses your feedback to refine the rubric and align the judge’s decisions with your own before creating it.

1:19