Building Trustworthy, High-Quality AI Agents with MLflow

Agentic + AI Observability Meetup | SF | February 17, 2026 AI agent development presents unique challenges due to unpredictable outputs and the constant need to balance cost, latency, and quality. This session explores how MLflow provides an end-to-end platform to build and monitor reliable agents. It covers the full agent development life cycle, from capturing execution steps with MLflow tracing to collecting expert feedback and using automated judges for performance evaluation. The talk also details how centralized governance through an AI gateway helps manage risks like runaway costs and data leakage while maintaining framework compatibility. Key Takeaways: • Using MLflow tracing for root cause analysis of agent failures • Scaling quality assessment with automated LLM-as-a-Judge evaluations • Managing model access and cost controls through a centralized AI gateway • Future developments in automated issue discovery and user simulation