Moving NLP Forward With Transformer Models and Attention | Real Python Podcast #121

What's the big breakthrough for Natural Language Processing (NLP) that has dramatically advanced machine learning into deep learning? What makes these transformer models unique, and what defines "attention?" This week on the show, Jodie Burchell, developer advocate for data science at JetBrains, continues our talk about how machine learning (ML) models understand and generate text. 👉 Links from the show: https://realpython.com/podcasts/rpp/121/ This episode is a continuation of the conversation in episode #119. Jodie builds on the concepts of bag-of-words, word2vec, and simple embedding models. We talk about the breakthrough mechanism called "attention," which allows for parallelization in building models. We also discuss the two major transformer models, BERT and GPT3. Jodie continues to share multiple resources to help you continue exploring modeling and NLP with Python. Topics: - 00:00:00 -- Introduction - 00:02:20 -- Where we left off with word2vec... - 00:03:35 -- Example of losing context - 00:06:50 -- Working at scale and adding attention - 00:12:34 -- Multiple levels of training for the model - 00:14:10 -- Attention is the basis for transformer models - 00:15:07 -- BERT (Bidirectional Encoder Representations from Transformers) - 00:16:29 -- GPT (Generative Pre-trained Transformer) - 00:19:08 -- Video Course Spotlight - 00:20:08 -- How far have we moved forward? - 00:20:41 -- Access to GPT-2 via Hugging Face - 00:23:56 -- How to access and use these models? - 00:30:42 -- Cost of training GPT-3 - 00:35:01 -- Resources to practice and learn with BERT - 00:38:19 -- GPT-3 and GitHub Copilot - 00:44:35 -- DALL-E is a transformer - 00:46:13 -- Help yourself to the show notes! - 00:49:19 -- How can people follow your work? - 00:50:03 -- Thanks and goodbye 👉 Links from the show: https://realpython.com/podcasts/rpp/121/