←
MLflow1,350 posts
MLflow avatar

MLflow

@MLflow

The open source developer platform to build AI applications and models with confidence.

◷ Joined August 2018▧ 594 Photos and videos
MLflow avatar
MLflow@MLflow·Nov 13

Most teams hand-tune agent prompts until it breaks at scale. This cookbook shows a measured loop: domain judges, expert alignment, and GEPA optimization on MLflow traces. What's inside: • Tool-calling LangGraph agent you can adapt • Domain judges with pass/fail rubrics • Expert alignment via mlflow.log_feedback() + MemAlign • GEPA optimization: 40% → 85% pass rate in the example Run it once, compare baseline vs optimized in the UI. 👉 Check out the cookbook: mlflow.org/cookbook/agent-al…

Post attachment
MLflow avatar
MLflow@MLflow·Nov 12

Reminder: Find out what's new in MLflow LIVE this Wednesday 📅 Join us November 18 at 4:00 PM PT to explore the latest in AI observability, AIOps, AI Governance, and developer experience. We'll cover MLflow 3.18 features including Jev, Issue Detection, and the new Eval UX, followed by a live Q&A session. Register 👉 luma.com/mlflow-t8h9

MLflow avatar
MLflow@MLflow·Nov 11

Losing track of which prompt or model worked best? In this MLflow A–Z episode, MLflow Ambassador @shrinathsuresh covers E for experiment tracking. If you're changing models, prompts, and configurations, MLflow helps you see which version worked best by logging prompts, parameters, evaluation scores, latency, and token usage so every run is comparable. 🔗 Experiment tracking docs: mlflow.org/docs/latest/ml/tr… 🔗 Watch the full MLflow A–Z series: youtube.com/playlist?list=PL…

1:16
MLflow avatar
MLflow@MLflow·Nov 6

MLflow 3.18.0 is now available! 🎉 🔍 Automatic Issue Detection finds recurring agent failures across your traces, from wrong answers to unusual spending. 📚 The experimental Agent Skill Registry shares reusable skills with coding agents like Claude Code, Codex, and Cursor. 📊 Evaluation comparisons now pin a baseline and surface regressions. ✅ Experimental online evaluation runs your Python scorers on new traces on a self-hosted server. Full release: mlflow.org/releases/3.18.0/

Post attachment
MLflow avatar
MLflow@MLflow·Nov 5

Five coding agents usually means five API keys and one invoice nobody can explain. @2twitme shows how to govern that with MLflow AI Gateway. Agents point at one gateway, and it runs guardrails, enforces a budget, and traces every call. 👉 Learn more: mlflow.org/blog/govern-codin… #MLflow #GenAI #LLMOps #OpenSource

MLflow avatar
MLflow@MLflow·Nov 4

MLflow Ambassador @shrinathsuresh covers D for Dataset Tracking in this MLflow A-Z clip. Log training and evaluation datasets, version them, and associate those versions with your experiments. Skipping that is a huge mistake for GenAI, because the models are probabilistic. 🎥 youtube.com/watch?v=BoTbLWWV…

1:17
MLflow avatar
MLflow@MLflow·Oct 30

MLflow AI Gateway has budget policies. You set a daily, weekly, or monthly spend limit. 👇 When it hits, you can send a webhook to Slack, PagerDuty, or any HTTP alerting and keep serving traffic, or block requests until the window resets. The limit applies to every app that routes through the gateway. Get started: pip install 'mlflow[genai]' 🔗 mlflow.org/blog/gateway-budg…

0:22
MLflow avatar
MLflow@MLflow·Oct 26

This week at ODSC AI West, Debu Sinha (@databricks) is running a workshop on Evaluating LLM Applications with MLflow. Teams already run evals. The workshop is the part after: did the model actually get better, which samples regressed, and is the delta statistically significant or just noise. You build that pipeline in open source MLflow, from mlflow.genai.evaluate() through run comparison and a CI gate that fails the build when a regression is real. 🗓️ Wed, Oct 28 🕓 11:00 AM–12:00 PM PT 📍 Hyatt Regency SFO 🔗 odsc.ai/speakers-portfolio/e…

Post attachment
MLflow avatar
MLflow@MLflow·Oct 26

This week at ODSC AI West, Samraj Moorjani (@databricks) is giving a 30-min demo on Hidden Gems in Agent Traces. Traces show what the agent did. The talk is about what you do next: find repeating failure modes, auto-detect them across a large volume of traces, and check the fix with evals you can rerun. All in open source MLflow. 🗓️ Tues, Oct 27 🕓 11:05–11:35 AM PT 📍 Hyatt Regency SFO 🔗 Schedule: odsc.ai/west/expo/

Post attachment
MLflow avatar
MLflow@MLflow·Oct 23

Teams keep adding agent harnesses. Claude Code, Codex, Cursor, and the rest each keep their own conventions, so observing a run across them is messy. @2twitme walks through that harness layer, then how @omnigent_ai puts coding agents and custom agents behind one common API. From there, @opentelemetry traces land in MLflow so you can see how the agent actually behaved. Watch the full walkthrough on how to trace and debug AI coding agents with Omnigent and MLflow. 🎥 youtube.com/watch?v=vvJTyd-e…

1:03
MLflow avatar
MLflow@MLflow·Oct 22

Part 3 of the Self-Improvement & MLflow series by MLflow Ambassador Jonathan Bown walks through automating the verification layer of a self-improving AI workflow with MLflow and CI/CD. A coding agent reads traces at scale and can propose changes to itself, its skills, tools, memory, or the verifier. CI then checks each change before it is approved automatically or by a person. Read Part 3 👇 opendatascience.com/self-imp…

MLflow avatar
MLflow@MLflow·Oct 19

What happens when every team has its own agents in production? You need more than LLMs. You need a control plane. For the third video in the MLflow A–Z series, MLflow Ambassador @shrinathsuresh covers C for Control Plane. He explains how MLflow AI Gateway brings access control, budgets, guardrails, traffic routing, and usage tracking into one place. 🔗 Learn more: mlflow.org/blog/gateway-guar… #MLflow #GenAI #LLMOps

1:22
MLflow avatar
MLflow@MLflow·Oct 16

The Claude SDK makes it fast to build an agent. With native @opentelemetry export and MLflow, it is just as fast to know whether the agent works. 🔹 Trace every model call by pointing a standard OTel exporter at MLflow 🔹 See each run as one trace, with model calls and tool calls under one agent span 🔹 Find expensive runs from token usage on the trace 🔹 Score every run with an LLM judge that inspects the trace 🔹 Compare system prompts with the same test prompts and judge 🔗 mlflow.org/blog/claude-sdk-o… #MLflow #Claude #OpenTelemetry

MLflow avatar
MLflow@MLflow·Oct 15

Part 2 of MLflow Ambassador Jonathan Bown’s series on self-improving AI workflows with MLflow is live. He walks a support agent through the loop: traces, feedback, expectations, and judges in; a versioned prompt and an evaluation dataset out; then a production alias move so the agent updates without code changes. @mlflow.test turns that dataset into a quality gate. When the manual pass stops scaling, a coding agent with MLflow skills proposes the themes, scorers, and tests. A human still accepts the changes. CI can run them as MLflow Evaluation runs. 🔗 opendatascience.com/self-imp… #MLflow #GenAI #ODSC

MLflow avatar
MLflow@MLflow·Oct 15

Claude Platform now supports native @opentelemetry tracing across its agent ecosystem. Bring traces from the Claude family into one backend using OTLP, the OpenTelemetry Protocol: 🔹 Messages API and Claude SDKs: model requests and tool-runner activity 🔹 Claude Code and Agent SDK: agent steps, tool calls, and subagent activity 🔹 Managed Agents: agent turns, model runs, and platform-run tools We published a guide covers setup OpenTelemetry export across the Claude family, including Claude Platform UI and SDK configuration. 👉 Setup guide: mlflow.org/blog/claude-otel-… #MLflow #opensource #otel