What is RAG and Why It Matters
Retrieval-Augmented Generation grounds language model answers in external evidence instead of relying solely on memorized weights.
The RAG Pipeline
"What is our refund policy?"
Search knowledge base
Top-K relevant chunks
Grounded answer + citations
Without RAG
The model answers from its training data only. It cannot access company policies, recent events, or private documents. Answers may be stale, generic, or hallucinated.
With RAG
The model reads your actual documents first, then answers with supporting evidence. Knowledge can be updated by changing documents -- no retraining needed.
Core insight (Lewis et al., 2020): Not all knowledge should live inside model weights. External retrieval gives the system access to fresher and more auditable evidence. RAG improves controllability as much as it improves accuracy.
Lexical vs Dense Retrieval
Two fundamentally different approaches to finding relevant documents -- one matches words, the other matches meaning.
Lexical Retrieval (BM25 / TF-IDF)
Matches explicit terms and phrases. Strong when exact wording matters.
Dense Retrieval (Embeddings)
Uses vector similarity to find semantically related content even with different surface forms.
Interactive: See the difference
Hybrid Retrieval
Combining lexical and dense signals so the system benefits from exact terminology and semantic similarity simultaneously.
Parallel retrieval pipeline
Fusion (RRF)
Dense catches
Paraphrases, conceptual matches, synonym-heavy queries. "physician salary" finds "doctor compensation."
Lexical catches
Acronyms, error codes, product IDs, legal clause numbers. "SKU-A7X42" finds the exact product.
RRF merges
Reciprocal Rank Fusion scores each doc by 1/(k + rank) across both result lists, then sorts by combined score.
Key insight: Hybrid search reduces the blind spots of each method. Dense retrieval alone may miss rare identifiers; lexical retrieval alone may miss paraphrases. Together they often improve first-stage recall by 10-20%.
Chunking Strategies
Chunking decides the unit of retrieval. Too large = noisy. Too small = no context. The right strategy aligns with the structure of your source material.
Interactive: See how different strategies split the same document
All full-time employees are entitled to 20 days of paid time off (PTO) per calendar year. PTO accrues at a rate of 1.67 days per month of employment. Unused PTO may be carried over to the next year, up to a maximum of 10 days. Any days exceeding the carryover limit will be forfeited on January 1st. Employees must submit PTO requests at least 5 business days in advance for planned absences. Emergency or sick leave requires notification to the direct manager within 2 hours of the scheduled start time. During the probationary period (first 90 days), employees may use up to 5 days of PTO. The full allocation becomes available after successful completion of the probationary period.
Strategy comparison
| Strategy | How it splits | Pros | Cons | Best for |
|---|---|---|---|---|
| Fixed-Size | Every N tokens/chars with overlap | Simple, predictable size | May split mid-sentence | Uniform text (articles, logs) |
| Semantic | By topic shifts (embedding similarity) | Preserves meaning boundaries | Chunk sizes vary; slower | Mixed-topic documents |
| Recursive | Tries paragraph, then sentence, then character | Respects document structure | Requires tuning | Structured docs (policies, manuals) |
| Document-aware | By headings, sections, tables | Perfect for structured formats | Format-specific code needed | Markdown, HTML, PDFs |
Overlap matters: Most strategies use 10-20% overlap between chunks so that information at chunk boundaries is not lost. A chunk size of 512 tokens with 50-token overlap is a common starting point.
Metadata Filters
Structured constraints that narrow the search space before semantic ranking even begins.
product="Widget X", lang="en"
100K -> 2K candidates
within filtered set
Retrieval quality is not only about better embeddings. Structured constraints do a large amount of work cheaply and reliably. Often the highest-return improvement in production search.
{"product": "Widget X", "type": "troubleshooting", "date_after": "2024-01-01"}
Vector Databases
A vector database stores embeddings and supports efficient nearest-neighbor search. It is infrastructure, not intelligence.
What a vector DB does
Major options
| Database | Type | Key feature |
|---|---|---|
| FAISS | Library | Meta's fast ANN library; IVF + PQ indexes |
| Pinecone | Managed SaaS | Serverless, auto-scaling, metadata filters |
| Weaviate | Open source | Built-in vectorization, GraphQL API |
| pgvector | Postgres ext. | Vectors alongside relational data |
| Qdrant | Open source | Rust-based, fast filtering |
| Milvus | Open source | GPU-accelerated, billion-scale |
Key insight: The vector store is infrastructure, not intelligence. It makes retrieval feasible and fast, but relevance still depends on the embedding model, chunking strategy, and ranking logic layered above it.
Approximate Nearest-Neighbor (ANN) Search
Exact nearest-neighbor is too slow for production. ANN trades a small amount of recall for much better speed and scalability.
HNSW (Hierarchical Navigable Small World)
Builds a layered graph where each node connects to nearby vectors. Search starts at the top layer (sparse, long-range links) and drills down to the bottom layer (dense, short-range links).
Trade-off: High memory usage (stores the graph in RAM). Build time is slower than IVF.
Used by: Pinecone, Weaviate, Qdrant, pgvector (opt-in index)
IVF (Inverted File Index)
Partitions the vector space into clusters (Voronoi cells). At query time, only the nearest clusters are searched instead of the full dataset.
Trade-off: Lower recall than HNSW. Can combine with Product Quantization (PQ) for compression.
Used by: FAISS, Milvus
Exact vs Approximate
| Method | Complexity | 1M vectors | 1B vectors | Recall |
|---|---|---|---|---|
| Exact (brute force) | O(N * d) | ~100ms | Impractical | 100% |
| IVF | O(sqrt(N) * d) | ~2ms | ~20ms | 90-98% |
| HNSW | O(log N * d) | ~1ms | ~5ms | 95-99% |
Reranking with Cross-Encoders
Reranking applies a more expensive relevance model to a shortlist returned by the first retriever. The two-stage pattern gives you speed and precision.
Two-Stage Retrieval Pipeline
Recall top 100 (fast, O(log N) via ANN)
Rerank to top 10 (precise, O(N))
Stage 1: Bi-Encoder (Retriever)
Query and documents encoded separately. Documents can be pre-embedded. Comparison = fast cosine similarity.
Stage 2: Cross-Encoder (Reranker)
Query + document fed together through full attention. Every query token attends to every document token. Outputs a single relevance score.
Interactive: Reranking in action
Query: "How to handle customer refund requests"
Before reranking (bi-encoder order)
After reranking (cross-encoder order)
Query Rewriting & HyDE
Users do not naturally speak in index-friendly language. Query rewriting converts the raw question into a form better aligned with indexed content.
Standard Query Rewriting
An LLM or rule-based system transforms the user query before retrieval.
"What does OOO mean?" -> "What does out of office mean?"
"fix the build" -> "fix CI/CD build pipeline failure"
"Compare Python and Go for APIs" -> Q1: "Python API frameworks" + Q2: "Go API frameworks"
HyDE (Hypothetical Document Embeddings)
Instead of embedding the question directly, ask the LLM to generate a hypothetical answer, then embed that answer as the search query.
Why it works: Documents are written in a different style than questions. The hypothetical document is closer in embedding space to real documents than the raw question would be.
Key insight: Query rewriting is often one of the cheapest ways to improve recall without re-embedding the corpus. It can be as simple as a few regex rules or as sophisticated as an LLM call.
Offline Retrieval Metrics
Recall@K measures whether relevant evidence appears. MRR and nDCG measure whether it appears near the top.
Recall@K
"Of all the relevant docs, what fraction appeared in the top K results?"
If there are 5 relevant docs and 3 appear in your top-10 list: Recall@10 = 3/5 = 0.60
MRR (Mean Reciprocal Rank)
"How high does the first correct result rank?"
First correct at rank 1: 1/1 = 1.0. First correct at rank 3: 1/3 = 0.33. Higher = better.
nDCG (Normalized DCG)
"Are the best results at the very top?" Rewards relevant docs at higher positions more than lower ones.
nDCG = DCG / ideal DCG
Unlike Recall, nDCG handles graded relevance (not just binary).
Interactive Metric Calculator
Click items in the result list to toggle their relevance (green = relevant). The metrics update in real-time.
Search results (click to toggle relevance):
Metrics (K = 5):
Key insight: Retrieval metrics should not be isolated from generation outcomes. A retriever that looks strong offline but feeds noisy evidence to the generator may still fail the user task. Always tie passage-level relevance to end-to-end grounded answer quality.