Naive RAG vs Production RAG
Naive RAG proves possibility. Production RAG proves reliability. The gap between the two is where most real engineering work lives.
Naive RAG (demo-grade)
Production RAG
Production RAG Scorecard
| Layer | Check | Why it matters |
|---|---|---|
| Retrieval | Relevant documents appear and rank well | Without evidence recall, the answer starts from a weak base |
| Grounding | Response cites supporting passages correctly | Prevents fluent unsupported claims |
| Fallback | System abstains when evidence is missing | Safer than forcing confident guesses |
| Operations | Latency and freshness stay inside target | Grounded systems still need product-grade reliability |
Single-Hop vs Multi-Hop Retrieval
Single-hop finds evidence in one pass. Multi-hop gathers evidence iteratively when the answer depends on connecting multiple facts.
Single-Hop
One search -> finds the PTO document -> answers.
Multi-Hop
Search #1: "Who built Project Alpha?" -> Team Bravo
Search #2: "Who manages Team Bravo?" -> Jane Smith
Trade-off: Multi-hop improves coverage for complex questions but raises orchestration complexity and error compounding. Each hop can introduce noise, latency, and points of failure.
Hallucination Reduction Strategies
Hallucination is often a retrieval and prompting problem before it is a decoding problem. If the context is wrong, thin, stale, or noisy, the generator will sound confident anyway.
Interactive Checklist: Hallucination defense layers
Click each item to check it off. A production RAG system should implement most of these.
Retrieval layer
Generation layer
The hallucination pipeline: where things go wrong
Citations and Provenance
Provenance is not just a UX feature. It is a control mechanism that increases trust, simplifies debugging, and makes human review faster.
Interactive: Grounded answer with source attribution
Knowledge Staleness and Freshness
Freshness should be handled in the data layer. If the system cannot guarantee freshness, it should communicate uncertainty.
Document freshness timeline
Documents have different freshness requirements. Price lists may go stale in hours; architecture docs may be valid for months.
Staleness risks
Freshness controls
Agentic RAG
Agentic RAG extends retrieval beyond one fixed lookup. The system may rewrite queries, choose tools, perform multiple retrieval steps, and decide whether more evidence is needed.
Agent loop diagram
Knowledge Base
API / Tool
Database
with citations
When to use agentic RAG
When NOT to use it
Key insight: Show restraint. Agentic RAG is powerful when the question requires decomposition or tool use, but it can add latency and failure paths for simple tasks. The best engineers know when the simpler approach is sufficient.
Caching Layers in Production RAG
Caching reduces latency and cost by reusing expensive results. But cached answers must respect freshness and permissions.
Cacheable stages in the RAG pipeline
Danger zone: cache + permissions
A fast cached answer that is stale or leaked across user scopes is worse than a slower uncached answer. Cache keys must include the user's permission scope. Never serve User A's cached answer to User B if the underlying documents have different ACLs.
Permissions and Access Control in RAG
A RAG system should never retrieve documents the current user is not allowed to see. Security lives in the retrieval layer too.
Where ACLs must be enforced
+ identity/role
ENFORCE HERE
within permitted set
Pre-filter (recommended)
Filter by permissions BEFORE vector search. Only permitted documents enter the ANN index query. Prevents leakage through generation.
Post-filter (risky)
Run vector search first, then remove unauthorized results. Can leak information through the model's reasoning even if docs are removed from display.
Per-tenant indexes
Separate vector indexes per tenant/org. Strongest isolation but highest operational cost and complexity.
Interactive: See permission filtering in action
Evaluating Production RAG: Offline vs Online
Evaluation should separate retrieval errors, prompt errors, and generation errors. Otherwise the team cannot tell where to intervene.
Offline Evaluation
Run on curated test sets before deployment.
| Metric | What it measures |
|---|---|
| Recall@K | Are relevant docs in the shortlist? |
| Groundedness | Is the answer supported by retrieved evidence? |
| Citation accuracy | Do citations point to the right passages? |
| Answer quality | Is the answer correct, complete, coherent? |
| Abstention rate | Does it correctly refuse when evidence is weak? |
Online Evaluation
Monitor with real traffic after deployment.
| Metric | What it measures |
|---|---|
| User satisfaction | Thumbs up/down, CSAT scores |
| Task completion | Did the user accomplish their goal? |
| Answer acceptance | Did the user use the answer or reformulate? |
| Correction rate | How often do users override the answer? |
| Escalation rate | How often does the system hand off to humans? |
Interactive: Diagnose where the failure is
Click each scenario to see which component failed and how to fix it.
When NOT to Use RAG
Strong engineers know when not to introduce a more complex architecture. The best solution is the one that is adequate, controllable, and maintainable.