Vector Databases Explained — Embeddings, Qdrant & RAG Retrieval

Key Takeaways

Hey, welcome back. We've covered relational databases for storing chat history and user data, but Karan's RAG system needs a different kind of storage for documents. When a user asks a question, the system needs to find the most relevant chunks of text from thousands of documents, not by keyword matching, but by meaning. That's what vector databases are built for. In this video, we cover how vector storage works, how documents get indexed, and how retrieval happens. This is the foundation for the RAG project in module six. Let's get into it. A SQL-like query finds documents that contain specific words. If a user asks, "How does attention work in language models?" and your documents talk about transformers and self-attention, but never use the word attention mechanism in that exact form, the keyword search misses them entirely. RAG needs semantic search, the ability to find content that means the same thing even when different words are used. This is what embeddings enable. Instead of storing text, you store a mathematical representation of the text's meaning. Then, you find documents whose meaning is closest to the query's meaning. Vector databases are built specifically for this kind of search. An embedding is a list of numbers, typically 768 to 1,536 of them, that represents the meaning of a piece of text. The key property is this: text with similar meaning gets similar numbers. Look at the two examples. "Fast API handles async requests efficiently" and "Async support in Fast API improves concurrency" are different sentences, but mean similar things. Their embeddings are close to each other in the vector space. A vector database stores these embeddings alongside the original text. When you search, it converts your query to an embedding and finds the stored embeddings that are closest to it. That's semantic search. The algorithm for finding close vectors quickly is called approximate nearest neighbor search, and it's what makes vector databases fast even at millions of documents. Before any document can be retrieved, it goes through three steps. Step one, chunking. The full document is split into smaller pieces, typically 200 to 500 words each. Why? Because embedding a 50-page document as one vector loses too much detail. Smaller chunks give more precise retrieval. Step two, embedding. Each chunk is sent to an embedding model, we're using OpenAI's text-embedding-3-small here, and converted to a vector. Step three, indexing. Each chunk and its embedding are stored in Qdrant as a point. A point has an ID, a vector, and a payload. The payload is where you keep the original text and any metadata you want to return with search results. Run these three steps for every document a user ingests, and your vector store is ready for retrieval. Retrieval is the mirror of ingestion. Step one, embed the user's query using the same embedding model used during ingestion. This is critical. You must use the same model both times, or the vectors won't be comparable. Step two, search the vector store for the closest vectors to the query embedding. The limit parameter controls how many results come back. Top_k is typically three to five for our AG. Step three, extract the text from the payload of each result and return it. These chunks are then passed to the LLM as context alongside the user's question. The LLM reads the retrieved chunks and generates an answer grounded in Qdrant's documents. That's the complete RAG loop. Qdrant is the vector store we use in this course. It's open source, can be self-hosted with Docker, and has a managed cloud option. The three concepts to know. A collection is a named group of vectors. Think of it like a table in SQL. Each collection has a fixed vector size that must match the output of your embedding model. For OpenAI's text-embedding-3-small, that's 1,536 dimensions. A point is one stored item. It has an ID, a vector, and a payload. The payload is a JSON object where you store the original text, document metadata, user ID, or anything else you want returned with search results. Distance.cosine is the similarity metric. It measures the angle between vectors, which is the standard approach for text embeddings. Get these three right and Qdrant works exactly as expected. Quick recap. Vector databases store numerical representations of meaning and find similar content through approximate nearest neighbor search. The ingestion pipeline is chunk, embed, index. Run this for every document a user uploads. The retrieval pipeline is embed the query, search the vector store, return the top chunks, pass them to the LLM. One rule you must always follow. Use the same embedding model for both ingestion and retrieval. If they don't match, your searches will return garbage. In the next video, we cover conversation memory patterns, how to differentiate between short-term request context, persistent chat history, and retrieval memory, and when to use each one. See you there.