Retrieval-Augmented Generation (RAG)

This video explains the Retrieval-Augmented Generation (RAG) model! This approach combines Dense Passage Retrieval with a Seq2Seq BART generator. This is tested out on knowledge intensive tasks like open-domain QA, jeopardy question generation, and FEVER fact verification. This looks like a really interesting paradigm for building language models that produce factually accurate generations! Thanks for watching! Please Subscribe! Paper Links: Original Paper: https://arxiv.org/pdf/2005.11401.pdf FB Blog Post (Animation used in Intro): https://ai.facebook.com/blog/retrieval-augmented-generation-streamlining-the-creation-of-intelligent-natural-language-processing-models HuggingFace RAG description: https://huggingface.co/transformers/model_doc/rag.html Billion-scale similarity search with GPUs: https://arxiv.org/pdf/1702.08734.pdf Language Models as Knowledge Bases? https://arxiv.org/abs/1909.01066 REALM: Retrieval-Augmented Language Models: https://arxiv.org/pdf/2002.08909.pdf Dense Passage Retrieval: https://arxiv.org/pdf/2004.04906.pdf FEVER: https://arxiv.org/pdf/1803.05355.pdf Natural Questions: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/1f7b46b5378d757553d3e92ead36bda2e4254244.pdf TriviaQA: https://arxiv.org/pdf/1705.03551.pdf MS MARCO: https://arxiv.org/pdf/1611.09268.pdf Thanks for watching! Time Stamps 0:00 Introduction 2:05 Limitations of Language Models 4:10 Algorithm Walkthrough 5:48 Dense Passage Retrieval 7:44 RAG-Token vs. RAG-Sequence 10:47 Off-the-Shelf Models 11:54 Experiment Datasets 15:03 Results vs. T5 16:16 BART vs. RAG - Jeopardy Questions 17:20 Impact of Retrieved Documents zi 18:53 Ablation Study 20:25 Retrieval Collapse 21:10 Knowledge Graphs as Non-Parametric Memory 21:45 Can we learn better representations for the Document Index? 22:12 How will Efficient Transformers impact this?