RAG Evaluation

After this skill you can…

• Run RAGAS evaluation on a RAG pipeline
• Interpret faithfulness and answer relevance scores
• A/B test chunking strategies

Prerequisites

Watch (10 videos)

[Full Workshop] Building Metrics that actually work — David Karam, Pi Labs (fmr Google Search)

AI Engineer · intermediate hands-on

→ Develop reliable evaluation metrics→ Optimize LLM performance

Build a RAG Evaluation Tool and Python Library

AI Anytime · intermediate hands-on

→ Build a RAG evaluation tool→ Implement evaluation metrics→ Create a Python library

GenAI Interview Questions: LLM Evaluation Pipeline in Production #generativeai

BEPEC · intermediate hands-on

→ Deploy LLM evaluation pipelines to production→ Implement evaluation metrics for GenAI applications

Your mental model for AI testing: evals, LLM judges, and test layering

Chrome for Developers · intermediate hands-on

→ Evaluate AI models using LLM judges and test layering→ Optimize AI app development with AI testing

[VOD] First Look At Claude 3 - Can It Beat GPT-4?

bycloud · intermediate hands-on

→ Compare LLMs using evaluation metrics→ Analyze AI model performance

Advanced LLM Evaluation Techniques: Chapter 22

Weights & Biases · intermediate hands-on

→ Evaluate LLM model performance→ Implement advanced evaluation techniques

How to evaluate your Gen AI models with Vertex AI

Google Cloud Tech · beginner hands-on

→ Evaluate Gen AI models with Vertex AI→ Scale RAG models for reliable results

Evaluate your AI with Stax

Google for Developers · intermediate hands-on

→ Evaluate AI with Stax→ Build data-driven AI evaluations

LangChain "RAG Evaluation" Webinar

LangChain · intermediate hands-on

→ Assess RAG model performance→ Compare RAG evaluation metrics

How to examine chance prediction of a machine learning model (Y-Scrambling / Y-Permutation)

Data Professor · beginner hands-on

→ Evaluate machine learning models using Y-Scrambling→ Detect chance prediction in models

Read (10 articles)