Facebook AI's DINO | PyTorch Code Explained

Aleksa Gordiฤ‡ - The AI Epiphany ยท Beginner ยท๐Ÿ“ฐ AI News & Updates ยท4y ago
๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘งโ€๐Ÿ‘ฆ JOIN OUR DISCORD COMMUNITY: Discord โ–บ https://discord.gg/peBrCpheKE ๐Ÿ“ข SUBSCRIBE TO MY MONTHLY AI NEWSLETTER: Substack โ–บ https://aiepiphany.substack.com/ โค๏ธ Become The AI Epiphany Patreon โค๏ธ โ–บ https://www.patreon.com/theaiepiphany With this video, I kick off a brand new series of coding videos in PyTorch where I'll be explaining some of the most impactful AI research through its code analysis! In this video I cover DINO from the "Emerging Properties in Self-Supervised Vision Transformers" paper. โ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌ โœ… Paper: https://arxiv.org/abs/2104.14294 โœ… Code: https://github.com/facebookresearch/dino โ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌ โŒš๏ธ Timetable: 00:00 DINO paper - short overview 02:38 Code analysis starts - training arguments (argparse) 07:30 Training main function 08:25 DINO augmentations 11:30 Main function resumed 16:05 DINO head (MLP) 18:05 Main function resumed 20:40 DINO loss overview 21:45 Main function resumed 24:05 Augmentations visualized (matplotlib) 27:00 Main function resumed 28:00 DINO Core part!!! (in-depth shape analysis) 33:45 DINO loss in depth 38:00 Main function resumed 39:50 Visualizing attention script explained 48:15 Outro โ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌ ๐Ÿ’ฐ BECOME A PATREON OF THE AI EPIPHANY โค๏ธ If these videos, GitHub projects, and blogs help you, consider helping me out by supporting me on Patreon! The AI Epiphany โ–บ https://www.patreon.com/theaiepiphany One-time donation: https://www.paypal.com/paypalme/theaiepiphany Much love! โค๏ธ Huge thank you to these AI Epiphany patreons: Eli Mahler Petar Veliฤkoviฤ‡ Zvonimir Sabljic โ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌ ๐Ÿ’ก The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". โ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌ ๐Ÿ‘‹ CONNECT WITH ME ON SOCIAL LinkedIn โ–บ https://www.linkedin.com/in/aleksagordic/ Twitter โ–บ https://twitter.com/gordic_
Watch on YouTube โ†— (saves to browser)
Sign in to unlock AI tutor explanation ยท โšก30

Playlist

Uploads from Aleksa Gordiฤ‡ - The AI Epiphany ยท Aleksa Gordiฤ‡ - The AI Epiphany ยท 0 of 60

โ† Previous Next โ†’
1 Intro | Neural Style Transfer #1
Intro | Neural Style Transfer #1
Aleksa Gordiฤ‡ - The AI Epiphany
 2 Basic Theory | Neural Style Transfer #2
Basic Theory | Neural Style Transfer #2
Aleksa Gordiฤ‡ - The AI Epiphany
 3 Optimization method | Neural Style Transfer #3
Optimization method | Neural Style Transfer #3
Aleksa Gordiฤ‡ - The AI Epiphany
 4 Advanced Theory | Neural Style Transfer #4
Advanced Theory | Neural Style Transfer #4
Aleksa Gordiฤ‡ - The AI Epiphany
 5 Anyone can make deepfakes now!
Anyone can make deepfakes now!
Aleksa Gordiฤ‡ - The AI Epiphany
 6 What is Computer Vision? | The Art of Creating Seeing Machines
What is Computer Vision? | The Art of Creating Seeing Machines
Aleksa Gordiฤ‡ - The AI Epiphany
 7 Feed-forward method | Neural Style Transfer #5
Feed-forward method | Neural Style Transfer #5
Aleksa Gordiฤ‡ - The AI Epiphany
 8 Alan Turing | Computing Machinery and Intelligence
Alan Turing | Computing Machinery and Intelligence
Aleksa Gordiฤ‡ - The AI Epiphany
 9 Feed-forward method (training) | Neural Style Transfer #6
Feed-forward method (training) | Neural Style Transfer #6
Aleksa Gordiฤ‡ - The AI Epiphany
 10 What is Google Deep Dream? (Basic Theory) | Deep Dream Series #1
What is Google Deep Dream? (Basic Theory) | Deep Dream Series #1
Aleksa Gordiฤ‡ - The AI Epiphany
 11 Semantic Segmentation in PyTorch | Neural Style Transfer #7
Semantic Segmentation in PyTorch | Neural Style Transfer #7
Aleksa Gordiฤ‡ - The AI Epiphany
 12 How to get started with Machine Learning
How to get started with Machine Learning
Aleksa Gordiฤ‡ - The AI Epiphany
 13 How to learn PyTorch? (3 easy steps) | 2021
How to learn PyTorch? (3 easy steps) | 2021
Aleksa Gordiฤ‡ - The AI Epiphany
 14 PyTorch or TensorFlow?
PyTorch or TensorFlow?
Aleksa Gordiฤ‡ - The AI Epiphany
 15 3 Machine Learning Projects For Beginners (Highly visual) | 2021
3 Machine Learning Projects For Beginners (Highly visual) | 2021
Aleksa Gordiฤ‡ - The AI Epiphany
 16 Machine Learning Projects (Intermediate level) | 2021
Machine Learning Projects (Intermediate level) | 2021
Aleksa Gordiฤ‡ - The AI Epiphany
 17 Cheapest (0$) Deep Learning Hardware Options | 2021
Cheapest (0$) Deep Learning Hardware Options | 2021
Aleksa Gordiฤ‡ - The AI Epiphany
 18 How to learn deep learning? (Transformers Example)
How to learn deep learning? (Transformers Example)
Aleksa Gordiฤ‡ - The AI Epiphany
 19 How do transformers work? (Attention is all you need)
How do transformers work? (Attention is all you need)
Aleksa Gordiฤ‡ - The AI Epiphany
 20 Developing a deep learning project (case study on transformer)
Developing a deep learning project (case study on transformer)
Aleksa Gordiฤ‡ - The AI Epiphany
 21 Vision Transformer (ViT) - An image is worth 16x16 words | Paper Explained
Vision Transformer (ViT) - An image is worth 16x16 words | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 22 GPT-3 - Language Models are Few-Shot Learners | Paper Explained
GPT-3 - Language Models are Few-Shot Learners | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 23 Google DeepMind's AlphaFold 2 explained! (Protein folding, AlphaFold 1, a glimpse into AlphaFold 2)
Google DeepMind's AlphaFold 2 explained! (Protein folding, AlphaFold 1, a glimpse into AlphaFold 2)
Aleksa Gordiฤ‡ - The AI Epiphany
 24 Attention Is All You Need (Transformer) | Paper Explained
Attention Is All You Need (Transformer) | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 25 Graph Attention Networks (GAT) | GNN Paper Explained
Graph Attention Networks (GAT) | GNN Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 26 Graph Convolutional Networks (GCN) | GNN Paper Explained
Graph Convolutional Networks (GCN) | GNN Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 27 Graph SAGE - Inductive Representation Learning on Large Graphs | GNN Paper Explained
Graph SAGE - Inductive Representation Learning on Large Graphs | GNN Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 28 PinSage - Graph Convolutional Neural Networks for Web-Scale Recommender Systems | Paper Explained
PinSage - Graph Convolutional Neural Networks for Web-Scale Recommender Systems | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 29 OpenAI CLIP - Connecting Text and Images | Paper Explained
OpenAI CLIP - Connecting Text and Images | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 30 Temporal Graph Networks (TGN) | GNN Paper Explained
Temporal Graph Networks (TGN) | GNN Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 31 Graph Neural Network Project Update! (I'm coding GAT from scratch)
Graph Neural Network Project Update! (I'm coding GAT from scratch)
Aleksa Gordiฤ‡ - The AI Epiphany
 32 Graph Attention Network Project Walkthrough
Graph Attention Network Project Walkthrough
Aleksa Gordiฤ‡ - The AI Epiphany
 33 How to get started with Graph ML? (Blog walkthrough)
How to get started with Graph ML? (Blog walkthrough)
Aleksa Gordiฤ‡ - The AI Epiphany
 34 DQN - Playing Atari with Deep Reinforcement Learning | RL Paper Explained
DQN - Playing Atari with Deep Reinforcement Learning | RL Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 35 AlphaGo - Mastering the game of Go with deep neural networks and tree search | RL Paper Explained
AlphaGo - Mastering the game of Go with deep neural networks and tree search | RL Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 36 DeepMind's AlphaGo Zero and AlphaZero | RL paper explained
DeepMind's AlphaGo Zero and AlphaZero | RL paper explained
Aleksa Gordiฤ‡ - The AI Epiphany
 37 OpenAI - Solving Rubik's Cube with a Robot Hand | RL paper explained
OpenAI - Solving Rubik's Cube with a Robot Hand | RL paper explained
Aleksa Gordiฤ‡ - The AI Epiphany
 38 MuZero - Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model | RL Paper explained
MuZero - Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model | RL Paper explained
Aleksa Gordiฤ‡ - The AI Epiphany
 39 EfficientNetV2 - Smaller Models and Faster Training | Paper explained
EfficientNetV2 - Smaller Models and Faster Training | Paper explained
Aleksa Gordiฤ‡ - The AI Epiphany
 40 Implementing DeepMind's DQN from scratch! | Project Update
Implementing DeepMind's DQN from scratch! | Project Update
Aleksa Gordiฤ‡ - The AI Epiphany
 41 MLP-Mixer: An all-MLP Architecture for Vision | Paper explained
MLP-Mixer: An all-MLP Architecture for Vision | Paper explained
Aleksa Gordiฤ‡ - The AI Epiphany
 42 DeepMind's Android RL Environment - AndroidEnv
DeepMind's Android RL Environment - AndroidEnv
Aleksa Gordiฤ‡ - The AI Epiphany
 43 When Vision Transformers Outperform ResNets without Pretraining | Paper Explained
When Vision Transformers Outperform ResNets without Pretraining | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 44 Non-Parametric Transformers | Paper explained
Non-Parametric Transformers | Paper explained
Aleksa Gordiฤ‡ - The AI Epiphany
 45 Chip Placement with Deep Reinforcement Learning | Paper Explained
Chip Placement with Deep Reinforcement Learning | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 46 Text Style Brush - Transfer of text aesthetics from a single example | Paper Explained
Text Style Brush - Transfer of text aesthetics from a single example | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 47 Graphormer - Do Transformers Really Perform Bad for Graph Representation? | Paper Explained
Graphormer - Do Transformers Really Perform Bad for Graph Representation? | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 48 GANs N' Roses: Stable, Controllable, Diverse Image to Image Translation | Paper Explained
GANs N' Roses: Stable, Controllable, Diverse Image to Image Translation | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 49 VQ-VAEs: Neural Discrete Representation Learning | Paper + PyTorch Code Explained
VQ-VAEs: Neural Discrete Representation Learning | Paper + PyTorch Code Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 50 VQ-GAN: Taming Transformers for High-Resolution Image Synthesis | Paper Explained
VQ-GAN: Taming Transformers for High-Resolution Image Synthesis | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 51 Multimodal Few-Shot Learning with Frozen Language Models | Paper Explained
Multimodal Few-Shot Learning with Frozen Language Models | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 52 Focal Transformer: Focal Self-attention for Local-Global Interactions in Vision Transformers
Focal Transformer: Focal Self-attention for Local-Global Interactions in Vision Transformers
Aleksa Gordiฤ‡ - The AI Epiphany
 53 AudioCLIP: Extending CLIP to Image, Text and Audio | Paper Explained
AudioCLIP: Extending CLIP to Image, Text and Audio | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 54 RMA: Rapid Motor Adaptation for Legged Robots | Paper Explained
RMA: Rapid Motor Adaptation for Legged Robots | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 55 DALL-E: Zero-Shot Text-to-Image Generation | Paper Explained
DALL-E: Zero-Shot Text-to-Image Generation | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 56 DETR: End-to-End Object Detection with Transformers | Paper Explained
DETR: End-to-End Object Detection with Transformers | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 57 DINO: Emerging Properties in Self-Supervised Vision Transformers | Paper Explained!
DINO: Emerging Properties in Self-Supervised Vision Transformers | Paper Explained!
Aleksa Gordiฤ‡ - The AI Epiphany
 58 DeepMind DetCon: Efficient Visual Pretraining with Contrastive Detection | Paper Explained
DeepMind DetCon: Efficient Visual Pretraining with Contrastive Detection | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 59 Do Vision Transformers See Like Convolutional Neural Networks? | Paper Explained
Do Vision Transformers See Like Convolutional Neural Networks? | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 60 Fastformer: Additive Attention Can Be All You Need | Paper Explained
Fastformer: Additive Attention Can Be All You Need | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany

Related AI Lessons

โšก
5 Mistakes That Stop Beginners From Making Money With AI
Learn the common mistakes that prevent beginners from making money with AI and how to avoid them
Medium ยท AI
 โšก
Search has changed more than we think
The way we search has undergone significant changes, and it's essential to understand these shifts
Medium ยท AI
 โšก
Unraveling a Network-Based Attack: TryHackMe First Shift CTF โ€” Task 7: The Crown Jewel
Learn to analyze network traffic and identify C2 channels, ARP spoofing, and data exfiltration techniques using Network Traffic Analysis and Forensics
Medium ยท Cybersecurity
 โšก
Microsoft to Stop Sharing Revenue with Main AI Partner OpenAI
Microsoft will stop sharing revenue with OpenAI, its main AI partner, in a significant shift in their business relationship, impacting the AI and tech industries
Hacker News (AI)

Chapters (16)

DINO paper - short overview
2:38 Code analysis starts - training arguments (argparse)
7:30 Training main function
8:25 DINO augmentations
11:30 Main function resumed
16:05 DINO head (MLP)
18:05 Main function resumed
20:40 DINO loss overview
21:45 Main function resumed
24:05 Augmentations visualized (matplotlib)
27:00 Main function resumed
28:00 DINO Core part!!! (in-depth shape analysis)
33:45 DINO loss in depth
38:00 Main function resumed
39:50 Visualizing attention script explained
48:15 Outro
Up next
The Secret Behind Mini Cooperโ€™s Name ๐Ÿคฏ #shorts
Jacky Chou from Indexsy
Watch โ†’