HyDE Retrieval in Python: Improve RAG Recall with SentenceTransformers

When a rag query is too vague to match the right chunks, hide turns the question into a better retrieval target before embedding it. You'll learn the shape of a workflow that drafts a hypothetical answer for a vague question [music] and ranks aligned documents. I'm Professor Pi teaching AI engineering and LLM systems with simple Python. Many retrieval problems start with a user [music] who cannot name the right terms. They send a short ticket or a vague question. A direct embedding of that short text often misses the best document. Because the corpus uses domain phrasing >> [music] >> the user did not supply. Hide flips the script. It asks a language model to imagine the kind of answer a helpful agent would write, then uses that imagined answer as the retrieval cue. Practically, this raises recall for queries that lack jargon and reduces the need for heavy [music] query engineering. The idea is simple. Instead of embedding the bare query, we create [music] a richer pseudo answer that contains the signals a retriever needs. Imagine a user says, "Video buffering problem." A draft reply might say, "Clients experience persistent buffering due to small streaming cache and network [music] jitter." That drafted wording hits the corpus phrases better. Strengths are better recall and simple integration. Weaknesses include possible hallucination, so we add safeguards. We will build a tiny example that shows the full loop. First, we measure a raw embedding baseline. Then, we generate a high-D draft. >> [music] >> Next, we compare rankings. Then, we blend the draft with the original query to limit drift. Finally, we synthesize a short RAG answer grounded in the top contexts. Quick note, this code is designed to teach the core idea clearly. Library versions and local setups can vary, so check the official docs if your setup behaves differently. Now, we measure the simple baseline to make [music] the recall gap visible. This snippet builds baseline sentence transformers [music] retrieval on a tiny incident corpus. Here, we set up model equals sentence transformer to encode both documents and queries. Call model.encode document to turn each incident into doc embeddings and use model.encode [music] query for the live question, so we respect asymmetric encoding. We compare vectors with util.cos_sim [music] and pick the top hit with scores.argmax. The printed output shows which document led and its similarity score. So, you can hear exactly how a short query can miss the best match. The baseline shows a gap, so we need a richer [music] retrieval target. Next, we draft a hypothetical answer with a generator. This example [music] prompts a transformers pipeline to draft a hide hypothetical answer for the query. For vague tickets, the retriever lacks signal, so we ask generator using Google flan-t5-base to sketch the troubleshooting reply you'd hope to see, and we fix randomness with set seed 42, >> [music] >> so the demo is repeatable. The prompt injects the live query and asks for a short diagnostic response. The result is hide answer, which we then embed with model.encode query to get hide scores. We also look at hide peak to quantify how much closer the corpus looks when you search with the imagined note. Seeing [snorts] the hide draft move scores is encouraging, but we must verify it actually changes which documents rank highest. This code contrasts baseline and hide rankings to show which document now leads. We grab scores top K3 for [music] the raw query and hide scores top K3 for the synthetic draft. Map their indices back to the docs and expose baseline best and hide best side by side so you can [music] read real titles or snippets. That side-by-side print makes it clear whether hide elevated a truly more relevant chunk, like a buffering explanation, or whether the draft just shuffled noise around. A drafted answer can [music] drift into hallucination, so we next blend it with the original query to keep user intent anchored. This snippet blends hide and raw query embeddings to moderate hallucination risk. Since synthetic drafts can wander, >> [music] >> we mix hide vec with query vec using an alpha weight to keep the user's wording in play. [music] We normalize both vectors with torch.nn.functional.normalize before [music] combining so cosine scores remain meaningful. Then score the combined vector against doc embeddings with util.cos_sim. [music] Turning alpha up favors the hide signal when the query [music] is very vague. Turning it down reins in drift when the user already provided solid terms. The printed result shows which document wins [music] under the blend and how much the combined similarity changed. With a controlled blend, we can fetch the top contexts and hand them to the generator for a grounded reply. This example stages a tiny hide enhanced rag pass from retrieval through answer synthesis. After blend scores.top k2 selects the top contexts, we concatenate their text into hide context so the generator sees real evidence instead of only the hypothetical draft. We build a response prompt that includes the context, the original user query, and a short instruction for the model. Then the same flan T5 pipeline that made the hide draft produces rag answer. The printed fields confirm which document indices supported the reply and display the concise grounded answer you can hand downstream. This toy maps cleanly to a real system. Replace the small corpus [music] with your incident database, swap face or another ANN for fast nearest neighbor lookup, and run the same hide loop as a microservice that returns top contexts plus a short synthesis. You retain the hybrid safety of blending >> [music] >> and the practical recall gains of synthetic expansion. Recap, hide means generating a hypothetical answer to enrich retrieval. Use it when user queries are short, vague, or lack domain terms. Caveat, synthetic drafts [music] can drift, so always validate ranking changes and consider blending with the original query. Next step, tune the alpha blend weight >> [music] >> to balance recall and hallucination, and measure how that tradeoff affects downstream answer accuracy. If practical AI engineering helps you, subscribe and watch the AI engineering videos.