LoRA explained (and a bit about precision and quantization)

DeepFindr · Beginner ·🧠 Large Language Models ·2y ago
▬▬ Papers / Resources ▬▬▬ LoRA Paper: https://arxiv.org/abs/2106.09685 QLoRA Paper: https://arxiv.org/abs/2305.14314 Huggingface 8bit intro: https://huggingface.co/blog/hf-bitsandbytes-integration PEFT / LoRA Tutorial: https://www.philschmid.de/fine-tune-flan-t5-peft Adapter Layers: https://arxiv.org/pdf/1902.00751.pdf Prefix Tuning: https://arxiv.org/abs/2101.00190 ▬▬ Support me if you like 🌟 ►Link to this channel: https://bit.ly/3zEqL1W ►Support me on Patreon: https://bit.ly/2Wed242 ►Buy me a coffee on Ko-Fi: https://bit.ly/3kJYEdl ►E-Mail: deepfindr@gmail.com ▬▬ Used Music ▬▬▬▬▬▬▬▬▬▬▬ Music from #Uppbeat (free for Creators!): https://uppbeat.io/t/danger-lion-x/flute-loops License code: M4FRIPCTVNOO4S8F ▬▬ Used Icons ▬▬▬▬▬▬▬▬▬▬ All Icons are from flaticon: https://www.flaticon.com/authors/freepik ▬▬ Timestamps ▬▬▬▬▬▬▬▬▬▬▬ 00:00 Introduction 00:20 Model scaling vs. fine-tuning 00:58 Precision & Quantization 01:30 Representation of floating point numbers 02:15 Model size 02:57 16 bit networks 03:15 Quantization 04:20 FLOPS 05:23 Parameter-efficient fine tuning 07:18 LoRA 08:10 Intrinsic Dimension 09:20 Rank decomposition 11:24 LoRA forward pass 11:49 Scaling factor alpha 13:40 Optimal rank 14:16 Benefits of LoRA 15:20 Implementation 16:25 QLoRA ▬▬ My equipment 💻 - Microphone: https://amzn.to/3DVqB8H - Microphone mount: https://amzn.to/3BWUcOJ - Monitors: https://amzn.to/3G2Jjgr - Monitor mount: https://amzn.to/3AWGIAY - Height-adjustable table: https://amzn.to/3aUysXC - Ergonomic chair: https://amzn.to/3phQg7r - PC case: https://amzn.to/3jdlI2Y - GPU: https://amzn.to/3AWyzwy - Keyboard: https://amzn.to/2XskWHP - Bluelight filter glasses: https://amzn.to/3pj0fK2
Watch on YouTube ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Playlist

Uploads from DeepFindr · DeepFindr · 51 of 56

← Previous Next →
1 Understanding Graph Neural Networks | Part 1/3 - Introduction
Understanding Graph Neural Networks | Part 1/3 - Introduction
DeepFindr
 2 Understanding Graph Neural Networks | Part 2/3 - GNNs and it's Variants
Understanding Graph Neural Networks | Part 2/3 - GNNs and it's Variants
DeepFindr
 3 Understanding Graph Neural Networks | Part 3/3 - Pytorch Geometric and Molecule Data using RDKit
Understanding Graph Neural Networks | Part 3/3 - Pytorch Geometric and Molecule Data using RDKit
DeepFindr
 4 Node Classification on Knowledge Graphs using PyTorch Geometric
Node Classification on Knowledge Graphs using PyTorch Geometric
DeepFindr
 5 Understanding Convolutional Neural Networks | Part 1 / 3 - The Basics
Understanding Convolutional Neural Networks | Part 1 / 3 - The Basics
DeepFindr
 6 Understanding Convolutional Neural Networks | Part 2 / 3 - Wonders of the world CNN with PyTorch
Understanding Convolutional Neural Networks | Part 2 / 3 - Wonders of the world CNN with PyTorch
DeepFindr
 7 Understanding Convolutional Neural Networks | Part 3 / 3 - Transfer Learning and Explainable AI
Understanding Convolutional Neural Networks | Part 3 / 3 - Transfer Learning and Explainable AI
DeepFindr
 8 How to use edge features in Graph Neural Networks (and PyTorch Geometric)
How to use edge features in Graph Neural Networks (and PyTorch Geometric)
DeepFindr
 9 Explainable AI explained! | #1 Introduction
Explainable AI explained! | #1 Introduction
DeepFindr
 10 Explainable AI explained! | #2 By-design interpretable models with Microsofts InterpretML
Explainable AI explained! | #2 By-design interpretable models with Microsofts InterpretML
DeepFindr
 11 Explainable AI explained! | #3 LIME
Explainable AI explained! | #3 LIME
DeepFindr
 12 Explainable AI explained! | #4 SHAP
Explainable AI explained! | #4 SHAP
DeepFindr
 13 Explainable AI explained! | #5 Counterfactual explanations and adversarial attacks
Explainable AI explained! | #5 Counterfactual explanations and adversarial attacks
DeepFindr
 14 Explainable AI explained! | #6 Layerwise Relevance Propagation with MRI data
Explainable AI explained! | #6 Layerwise Relevance Propagation with MRI data
DeepFindr
 15 Understanding Graph Attention Networks
Understanding Graph Attention Networks
DeepFindr
 16 GNN Project #1 - Introduction to HIV dataset
GNN Project #1 - Introduction to HIV dataset
DeepFindr
 17 GNN Project #2 - Creating a Custom Dataset in Pytorch Geometric
GNN Project #2 - Creating a Custom Dataset in Pytorch Geometric
DeepFindr
 18 GNN Project #3.2 - Graph Transformer
GNN Project #3.2 - Graph Transformer
DeepFindr
 19 GNN Project #4.1 - Graph Variational Autoencoders
GNN Project #4.1 - Graph Variational Autoencoders
DeepFindr
 20 GNN Project #4.2 - GVAE Training and Adjacency reconstruction
GNN Project #4.2 - GVAE Training and Adjacency reconstruction
DeepFindr
 21 GNN Project #4.3 - One-shot molecule generation - Part 1
GNN Project #4.3 - One-shot molecule generation - Part 1
DeepFindr
 22 GNN Project #4.3 - Code explanation
GNN Project #4.3 - Code explanation
DeepFindr
 23 Machine Learning Model Deployment with Python (Streamlit + MLflow) | Part 1/2
Machine Learning Model Deployment with Python (Streamlit + MLflow) | Part 1/2
DeepFindr
 24 Machine Learning Model Deployment with Python (Streamlit + MLflow) | Part 2/2
Machine Learning Model Deployment with Python (Streamlit + MLflow) | Part 2/2
DeepFindr
 25 How to explain Graph Neural Networks (with XAI)
How to explain Graph Neural Networks (with XAI)
DeepFindr
 26 Explaining Twitch Predictions with GNNExplainer
Explaining Twitch Predictions with GNNExplainer
DeepFindr
 27 Python Graph Neural Network Libraries (an Overview)
Python Graph Neural Network Libraries (an Overview)
DeepFindr
 28 Friendly Introduction to Temporal Graph Neural Networks (and some Traffic Forecasting)
Friendly Introduction to Temporal Graph Neural Networks (and some Traffic Forecasting)
DeepFindr
 29 Traffic Forecasting with Pytorch Geometric Temporal
Traffic Forecasting with Pytorch Geometric Temporal
DeepFindr
 30 Fraud Detection with Graph Neural Networks
Fraud Detection with Graph Neural Networks
DeepFindr
 31 Fake News Detection using Graphs with Pytorch Geometric
Fake News Detection using Graphs with Pytorch Geometric
DeepFindr
 32 Recommender Systems using Graph Neural Networks
Recommender Systems using Graph Neural Networks
DeepFindr
 33 How to handle Uncertainty in Deep Learning #1.1
How to handle Uncertainty in Deep Learning #1.1
DeepFindr
 34 How to handle Uncertainty in Deep Learning #1.2
How to handle Uncertainty in Deep Learning #1.2
DeepFindr
 35 How to handle Uncertainty in Deep Learning #2.1
How to handle Uncertainty in Deep Learning #2.1
DeepFindr
 36 How to handle Uncertainty in Deep Learning #2.2
How to handle Uncertainty in Deep Learning #2.2
DeepFindr
 37 Converting a Tabular Dataset to a Graph Dataset for GNNs
Converting a Tabular Dataset to a Graph Dataset for GNNs
DeepFindr
 38 Converting a Tabular Dataset to a Temporal Graph Dataset for GNNs
Converting a Tabular Dataset to a Temporal Graph Dataset for GNNs
DeepFindr
 39 How to get started with Data Science (Career tracks and advice)
How to get started with Data Science (Career tracks and advice)
DeepFindr
 40 Causality and (Graph) Neural Networks
Causality and (Graph) Neural Networks
DeepFindr
 41 Diffusion models from scratch in PyTorch
Diffusion models from scratch in PyTorch
DeepFindr
 42 Self-/Unsupervised GNN Training
Self-/Unsupervised GNN Training
DeepFindr
 43 Contrastive Learning in PyTorch - Part 1: Introduction
Contrastive Learning in PyTorch - Part 1: Introduction
DeepFindr
 44 Contrastive Learning in PyTorch - Part 2: CL on Point Clouds
Contrastive Learning in PyTorch - Part 2: CL on Point Clouds
DeepFindr
 45 State of AI 2022 - My Highlights
State of AI 2022 - My Highlights
DeepFindr
 46 Equivariant Neural Networks | Part 1/3 - Introduction
Equivariant Neural Networks | Part 1/3 - Introduction
DeepFindr
 47 Equivariant Neural Networks | Part 2/3 - Generalized CNNs
Equivariant Neural Networks | Part 2/3 - Generalized CNNs
DeepFindr
 48 Equivariant Neural Networks | Part 3/3 - Transformers and GNNs
Equivariant Neural Networks | Part 3/3 - Transformers and GNNs
DeepFindr
 49 Personalized Image Generation (using Dreambooth) explained!
Personalized Image Generation (using Dreambooth) explained!
DeepFindr
 50 Vision Transformer Quick Guide - Theory and Code in (almost) 15 min
Vision Transformer Quick Guide - Theory and Code in (almost) 15 min
DeepFindr
 ▶ LoRA explained (and a bit about precision and quantization)
LoRA explained (and a bit about precision and quantization)
DeepFindr
 52 Dimensionality Reduction Techniques | Introduction and Manifold Learning (1/5)
Dimensionality Reduction Techniques | Introduction and Manifold Learning (1/5)
DeepFindr
 53 Principal Component Analysis (PCA) | Dimensionality Reduction Techniques (2/5)
Principal Component Analysis (PCA) | Dimensionality Reduction Techniques (2/5)
DeepFindr
 54 Multidimensional Scaling (MDS) | Dimensionality Reduction Techniques (3/5)
Multidimensional Scaling (MDS) | Dimensionality Reduction Techniques (3/5)
DeepFindr
 55 t-distributed Stochastic Neighbor Embedding (t-SNE) | Dimensionality Reduction Techniques (4/5)
t-distributed Stochastic Neighbor Embedding (t-SNE) | Dimensionality Reduction Techniques (4/5)
DeepFindr
 56 Uniform Manifold Approximation and Projection (UMAP) | Dimensionality Reduction Techniques (5/5)
Uniform Manifold Approximation and Projection (UMAP) | Dimensionality Reduction Techniques (5/5)
DeepFindr

Related AI Lessons

⚡
I Tested 50 AI Prompts — Here’s the Formula That Always Works (2026)
Discover a proven formula for crafting effective AI prompts that drive results, based on testing 50 prompts
Medium · ChatGPT
 ⚡
How to Use ChatGPT in Business: A Practical Guide for 2026
Learn how to leverage ChatGPT in your business to automate tasks and generate content at scale, increasing efficiency and productivity
Medium · ChatGPT
 ⚡
The Complete Guide to Prompt Engineering: Unlock the Full Potential of AI
Learn how to unlock the full potential of AI with prompt engineering, a crucial factor in achieving game-changing outputs
Medium · Machine Learning
 ⚡
How to Add AI Features to Your SaaS App Without Breaking Everything
Learn how to add AI features to your SaaS app without disrupting your service, covering LLM integration, streaming responses, and cost management
Dev.to AI

Chapters (18)

Introduction
0:20 Model scaling vs. fine-tuning
0:58 Precision & Quantization
1:30 Representation of floating point numbers
2:15 Model size
2:57 16 bit networks
3:15 Quantization
4:20 FLOPS
5:23 Parameter-efficient fine tuning
7:18 LoRA
8:10 Intrinsic Dimension
9:20 Rank decomposition
11:24 LoRA forward pass
11:49 Scaling factor alpha
13:40 Optimal rank
14:16 Benefits of LoRA
15:20 Implementation
16:25 QLoRA
Up next
5 Levels of AI Agents - From Simple LLM Calls to Multi-Agent Systems
Dave Ebbelaar (LLM Eng)
Watch →