Model Monitoring

After this skill you can…

• Set up drift detection with Evidently AI
• Define and monitor SLAs for model performance
• Build a retraining trigger pipeline

Prerequisites

Watch (10 videos)

Monitor your AI applications in production using W&B Weave

Weights & Biases · intermediate hands-on

→ Monitor AI application performance in production→ Track quality metrics with W&B Weave

MLOps Essentials: Enabling CloudWatch Logging & Monitoring for AWS ML APIs

Analytics Vidhya · beginner hands-on

→ Enable CloudWatch logging for AWS ML APIs→ Monitor machine learning model performance

Production monitoring for AI applications using W&B Weave

Weights & Biases · intermediate hands-on

→ Create online evaluations for AI applications→ Monitor AI application quality over time

Model Monitoring for Generative AI applications

Microsoft Developer · intermediate hands-on

→ Monitor LLM performance→ Implement model monitoring techniques

Model Monitoring for LLMs

Microsoft Developer · intermediate hands-on

→ Monitor LLMs for performance and accuracy→ Evaluate LLMs using industry expert techniques

[Evals Workshop] Mastering AI Evaluation: From Playground to Production

AI Engineer · intermediate hands-on

→ Monitor AI application performance in real-world scenarios→ Implement logging and feedback systems

Production ML on AWS: Monitoring, Troubleshooting, and Cost Optimization

Analytics Vidhya · intermediate hands-on

→ Troubleshoot errors with log streams

Optimizing AI Applications for Production with Observability | OD548

Microsoft Developer · intermediate hands-on

→ Analyze AI component performance→ Capture token throughput metrics

Machine Learning Tips: Deep Learning Monitor

Automata Learning Lab · beginner

→ Monitor deep learning models→ Track training metrics

How to add LLM Evaluation to your AI Apps with Arize AX

Data Science Dojo · intermediate

→ Monitor AI model performance→ Detect hallucinations in LLMs

Read (10 articles)