←
arize-phoenix1,048 posts
arize-phoenix avatar

arize-phoenix

@ArizePhoenix

Open-Source AI Observability and Evaluation

◷ Joined February 2023▧ 266 Photos and videos
Surendran avatar
⟳ arize-phoenix retweeted
Surendran@surendranb·Nov 12

Evals & Feedback loops: @ArizePhoenix is a great light weight tool, runs on device. The native plugin had some issues & I was running batch job to log the traces. Fixed this recently. Weekly Loop: Look at both deterministic evals & llm-as-judge to improve agents/skills (4/n)

Post attachment
arize-phoenix avatar
arize-phoenix@ArizePhoenix·Nov 4

Join us tomorrow to see how Phoenix MCP lets agents query your traces with SQL and code mode. We'll also share our eight-question benchmark, which showed roughly 17x lower average cost than fetching spans page by page. luma.com/arizeai-query-phoen…

arize-phoenix avatar
arize-phoenix@ArizePhoenix·Nov 2

You don’t need a Phoenix upgrade for every model pricing update! Introducing automatic model pricing updates in Phoenix. New and revised built-in rates arrive in the background, while your custom pricing stays intact. arize.com/blog/phoenix-autom…

Post attachment
arize-phoenix avatar
arize-phoenix@ArizePhoenix·Oct 29

Xandar Song (@axiomofjoy) from the Phoenix team introduces Harbor, an open source framework for testing agents on realistic tasks with real data in isolated sandboxes. Give the agent a task, run it in a fresh environment, then score what it changed after it finishes.

arize-phoenix avatar
arize-phoenix@ArizePhoenix·Oct 29

Join us Nov 5 to see how Phoenix MCP lets agents query traces with SQL and code mode, with live demos. We'll also share our eight-question benchmark, which showed roughly 17x lower average cost than fetching spans page by page. Register here: luma.com/arizeai-query-phoen…

arize-phoenix avatar
arize-phoenix@ArizePhoenix·Oct 29

Our Head of Open Source, @mikeldking talks through a problem every agent builder runs into: tool lists start out helpful, then start filling the context window. Code mode gives the agent a sandbox where it can find the tools it needs, call them in code & return a smaller result.

I recently gave my first conference talk at @aiDotEngineer representing @arizeai . It was great getting to meet fellow AI nerds and talk about the things we are building. It genuinely gave me energy being around true innovators. There's so much hype around self-improving agents and RL environments but I've only recently started to feel this is possible when I started playing with code mode and the harbor framework plugin with @ArizePhoenix. It's been a journey over the last few months and I think it fundamentally has changed how I think we should build agentic systems. Have you setup code mode and RL / hill-climbing evals that work for you? What should I try next? youtu.be/dAoOYFx7ZYw?si=6bKO…

arize-phoenix avatar
arize-phoenix@ArizePhoenix·Oct 28

In their PostgreSQL test with about 2.3 million traces in one project, calculating a day's counts took 40 milliseconds instead of 3.9 seconds, with the same results. The fix is merged and awaiting release github.com/Arize-ai/phoenix/…

arize-phoenix avatar
arize-phoenix@ArizePhoenix·Oct 28

We love community contributions! Thank you, schlaepf, for helping make Phoenix's Traces chart faster to load. The chart shows how many traces your app recorded over time and how many had errors. Their fix reduces the work needed to calculate those counts.

arize-phoenix avatar
arize-phoenix@ArizePhoenix·Oct 28

Learn how to investigate prompt injection in AI agents with Phoenix traces and evaluations, add guardrails to block unauthorized data access, and set up online evaluations and alerts in Arize AX. arize.com/blog/indirect-prom…

Post attachment

We built an indirect prompt injection lab to test whether pages could trick an agent into accessing another customer's data. We used fictional customer records. The agent retrieved another customer's data in 28 of 100 runs. We traced and evaluated tool calls in @ArizePhoenix

arize-phoenix avatar
arize-phoenix@ArizePhoenix·Oct 22

We love community contributions. Thank you @Kenny_V for the Atomic Agents Tracing guide. Atomic Agents is a lightweight, modular Python framework for building agentic AI apps. See how to trace it in Phoenix: arize.com/docs/phoenix/integ…

arize-phoenix avatar
arize-phoenix@ArizePhoenix·Oct 21

Decision models as evaluators Phoenix evals (TS) can now run any classification evaluator on a decision model, like gpt-6-luna via OpenAI's Decisions API. The evaluator returns a label and a probability per option, no text generated. Each call is traced as a decision span.

Post attachment