Advanced RAG Techniques | 30 Days of AI Agents, Day 23

Yesterday we built guardrails around what an agent can say and do. Today we go back to what it knows. Day 15 covered RAG basics; Day 23 covers the techniques that take a RAG pipeline from "works on the demo questions" to "works on the questions people actually ask."

Save this sheet. The rest of this note walks through it section by section.

A quick RAG recap

Retrieval-Augmented Generation combines information retrieval with an LLM. Instead of answering from training data alone, the model receives relevant passages from your knowledge base and answers from them. That reduces hallucinations, lets the model use private data, and keeps answers current without retraining.

The flow on the sheet: the user asks a question, the system retrieves relevant documents from a knowledge base (documents, web pages, databases, APIs, files), optionally reranks them, augments the prompt with the best context, generates an answer, and responds. The components are the retriever, the vector database, the reranker, the LLM, the prompt, and the output.

Where basic RAG breaks

A basic pipeline embeds the query, pulls the top few chunks by vector similarity, and pastes them into the prompt. It fails in predictable ways:

• Exact terms get lost. Error codes, product SKUs, policy numbers, and names are where pure semantic search is weakest.
• Short or vague queries retrieve noise. "Leave policy?" matches a dozen loosely related chunks.
• Chunks lose their context. A paragraph saying "this applies after 90 days" is useless without the heading that says what "this" is.
• Stale documents compete with current ones. Last year's policy looks just as relevant as this year's.
• Too much context. Stuffing twenty chunks into the prompt raises cost and latency and can bury the one passage that matters.

Each advanced technique targets one of these failures. You rarely need all nine.

Nine advanced techniques

Technique	What it does	Best for
Hybrid search	Combines keyword (BM25) and vector search	Better recall and precision, exact terms
Query expansion	Rewrites or expands the query	Ambiguous or short queries
Multi-query RAG	Generates several query variants and merges results	Complex information needs
Reranking	Re-scores candidates with a cross-encoder	Improving relevance of the final context
Contextual compression	Trims retrieved text to the relevant parts	Reducing token usage
Parent document retrieval	Matches small chunks, returns the larger parent	Preserving context and coherence
Step-back RAG	Retrieves high-level concepts first, then drills down	Strategic and analytical questions
Time-aware RAG	Prioritizes recent or time-relevant information	Dynamic and fast-changing data
Agentic RAG	An agent plans, retrieves, verifies, and refines	Complex multi-step questions

If I had to order them by return on effort for a typical business knowledge base: hybrid search, then reranking, then parent document retrieval, then metadata filtering for time and access. Query rewriting and agentic RAG come later, once you have measured that retrieval is still the bottleneck.

Agentic RAG deserves a note because it ties back to this series. Instead of one retrieve-then-generate pass, the agent treats retrieval as a tool: it decides what to search for, reads the results, notices gaps, searches again, and checks its answer against sources before responding. It handles multi-step questions much better, at the cost of more LLM calls and more latency.

Hybrid search with rank fusion in code

Keyword and vector scores live on different scales, so you cannot just add them. Reciprocal rank fusion sidesteps that by combining rank positions instead of raw scores.

def reciprocal_rank_fusion(result_lists, k=60):
    scores = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

keyword_hits = bm25_search(query, top_k=20)    # exact terms, codes, names
vector_hits = vector_search(query, top_k=20)   # paraphrases and concepts
candidates = reciprocal_rank_fusion([keyword_hits, vector_hits])[:20]
context = rerank(query, candidates)[:5]        # cross-encoder keeps the best few

The pattern is wide then narrow: retrieve generously from two sources, fuse, then let a more expensive cross-encoder pick the handful of passages that actually go to the model. The constant k=60 is the common default from the original RRF work; it dampens the advantage of top-ranked items.

The advanced architecture

User query
  -> Query processing (rewrite, expand, route)
  -> Retriever (hybrid) + Vector DB + Reranker (cross-encoder)
  -> Contextual compression (optional)
  -> LLM generates a grounded answer with citations
Monitoring and evaluation loop feeds back into query processing

The monitoring loop is the piece most teams leave out. Log the query, the retrieved chunk IDs, their scores, and whether the answer was accepted. Without that, you cannot tell whether a bad answer came from retrieval or generation, which is why the sheet's advice to evaluate them separately matters so much.

Worked example with an internal doc copilot

An employee asks, "What is our expense reimbursement policy?"

1. The retriever searches policies, HR docs, FAQs, and the company wiki.
2. The reranker orders the top documents by relevance.
3. The best context is added to the prompt.
4. The LLM answers with citations to the source documents.
5. The response includes source links so the employee can verify.

The details decide whether this works. Filter by metadata so a draft policy or another region's policy never reaches the prompt. Use parent document retrieval so the answer includes the section heading and conditions, not an orphaned sentence. And if two versions of a policy exist, time-aware ranking should prefer the current one, or better, the old version should be archived out of the index.

Tools and best practices

The sheet lists LangChain and LlamaIndex for building pipelines, Haystack by deepset as an end-to-end framework, Milvus and Pinecone for scalable vector search, Weaviate for hybrid search, and Qdrant for filtering. All are solid; pick based on where your data lives and whether you need hybrid search natively.

The best practices: use hybrid search, rerank, compress context, chunk smartly (semantic or hierarchical chunks), keep data clean and current, add metadata filters, evaluate retrieval and generation separately, monitor continuously, and enforce guardrails and access controls. The challenges mirror them: irrelevant retrieval, stale data, high token usage, sensitive data leakage, weak grounding, hard-to-debug pipelines, latency from extra steps, and evaluating retrieval quality at all. Great RAG is a system, not a single prompt.

Key Takeaways

• Most RAG failures are retrieval failures; fix retrieval before tuning prompts.
• Hybrid search plus a cross-encoder reranker is the highest-value upgrade for most knowledge bases.
• Retrieve wide, then narrow: fuse candidates, rerank, and send only the best few passages.
• Metadata filters for freshness and access control prevent the worst wrong answers.
• Log retrieved chunks with every answer so you can evaluate retrieval and generation separately.

FAQs

Do I need a reranker if my embeddings are good?

Usually yes. Embedding search is fast but approximate. A cross-encoder reads the query and passage together and is much better at the final ordering.

When is agentic RAG worth the extra cost?

When questions need several lookups or comparisons. For single-fact questions, a well-tuned standard pipeline is faster and cheaper.

How small should chunks be?

Small enough to match precisely, but return enough surrounding context to be understood. Parent document retrieval gives you both.

Up Next

Day 24 is AI Agent Memory Optimization: how to keep an agent's memory cheap and useful with context budgets, summarization, compression, scoring, and expiry. If your team needs a retrieval pipeline that holds up on real questions, reach out.