Evals & Feedback loops: @ArizePhoenix is a great light weight tool, runs on device. The native plugin had some issues & I was running batch job to log the traces. Fixed this recently.
Weekly Loop: Look at both deterministic evals & llm-as-judge to improve agents/skills (4/n)
Join us tomorrow to see how Phoenix MCP lets agents query your traces with SQL and code mode.
We'll also share our eight-question benchmark, which showed roughly 17x lower average cost than fetching spans page by page.
luma.com/arizeai-query-phoen…
You don’t need a Phoenix upgrade for every model pricing update!
Introducing automatic model pricing updates in Phoenix. New and revised built-in rates arrive in the background, while your custom pricing stays intact.
arize.com/blog/phoenix-autom…
He shows how we ran Harbor at scale on @daytonaio and used Phoenix to inspect the results.
A prompt change steered agents toward a read-only SQL tool for aggregate questions. SQL use went up, tool calls went down, and the task reward stayed high.
youtube.com/watch?v=kRJjSOYq…
Xandar Song (@axiomofjoy) from the Phoenix team introduces Harbor, an open source framework for testing agents on realistic tasks with real data in isolated sandboxes.
Give the agent a task, run it in a fresh environment, then score what it changed after it finishes.
Join us Nov 5 to see how Phoenix MCP lets agents query traces with SQL and code mode, with live demos.
We'll also share our eight-question benchmark, which showed roughly 17x lower average cost than fetching spans page by page.
Register here:
luma.com/arizeai-query-phoen…
Mikyo walks through how we built this into Phoenix, including shell access, sandboxed Python, SQL, and browser UI actions.
In Harbor evals, code mode used 43% fewer tokens and lowered cost by 31%.
youtube.com/watch?v=dAoOYFx7…
Our Head of Open Source, @mikeldking talks through a problem every agent builder runs into: tool lists start out helpful, then start filling the context window.
Code mode gives the agent a sandbox where it can find the tools it needs, call them in code & return a smaller result.
I recently gave my first conference talk at @aiDotEngineer representing @arizeai . It was great getting to meet fellow AI nerds and talk about the things we are building. It genuinely gave me energy being around true innovators.
There's so much hype around self-improving agents and RL environments but I've only recently started to feel this is possible when I started playing with code mode and the harbor framework plugin with @ArizePhoenix. It's been a journey over the last few months and I think it fundamentally has changed how I think we should build agentic systems.
Have you setup code mode and RL / hill-climbing evals that work for you? What should I try next?
youtu.be/dAoOYFx7ZYw?si=6bKO…
In their PostgreSQL test with about 2.3 million traces in one project, calculating a day's counts took 40 milliseconds instead of 3.9 seconds, with the same results.
The fix is merged and awaiting release
github.com/Arize-ai/phoenix/…
We love community contributions! Thank you, schlaepf, for helping make Phoenix's Traces chart faster to load.
The chart shows how many traces your app recorded over time and how many had errors. Their fix reduces the work needed to calculate those counts.
Learn how to investigate prompt injection in AI agents with Phoenix traces and evaluations, add guardrails to block unauthorized data access, and set up online evaluations and alerts in Arize AX.
arize.com/blog/indirect-prom…
We built an indirect prompt injection lab to test whether pages could trick an agent into accessing another customer's data. We used fictional customer records. The agent retrieved another customer's data in 28 of 100 runs. We traced and evaluated tool calls in @ArizePhoenix
We love community contributions. Thank you Saif Ali Shaik for writing the @scalekitinc Tracing guide for the Phoenix docs.
If your agent runs tools through Scalekit's AgentKit, you can now trace those calls in Phoenix. Learn more through the guide
arize.com/docs/phoenix/integ…
We love community contributions. Thank you @Kenny_V for the Atomic Agents Tracing guide.
Atomic Agents is a lightweight, modular Python framework for building agentic AI apps. See how to trace it in Phoenix:
arize.com/docs/phoenix/integ…
Switching the model is a one-line change to your evaluator. The docs cover which models work, what comes back in the result, how it differs from a chat-model evaluator, and when to stick with a language model.
arize.com/docs/phoenix/sdk-a…
Decision models as evaluators
Phoenix evals (TS) can now run any classification evaluator on a decision model, like gpt-6-luna via OpenAI's Decisions API. The evaluator returns a label and a probability per option, no text generated. Each call is traced as a decision span.