RAG Patterns & Best Practices

The Big Picture

Large language models (LLMs) are powerful but have a fundamental limitation: they can only use knowledge encoded in their training data. When facts, recent events, proprietary information, or specialized domain knowledge fall outside the training distribution, LLMs hallucinate—generating plausible-sounding but false information.

The Hallucination Problem

An LLM trained on data through April 2024 cannot answer questions about June 2024 product releases. A model trained on public text cannot know your internal company policies. This is where Retrieval-Augmented Generation (RAG) enters the picture.

What is RAG?

RAG augments an LLM with retrieved context from a knowledge base. The basic pipeline:

1. Query: User asks a question
2. Retrieve: Search a vector store to find relevant documents/chunks
3. Augment: Combine retrieved context with the user query
4. Generate: LLM answers using both the query and retrieved context

Where RAG is Used

RAG System Variants

Document Processing & Chunking

Raw documents are too long to fit in an LLM's context window or vector search index. Chunking breaks documents into small, overlapping pieces that maintain semantic coherence.

Chunking Strategies

1. Fixed-Size Chunking

Split documents into fixed-length chunks (e.g., 512 tokens) with overlap (e.g., 50 tokens). Simple, deterministic, works for uniform content like code.

2. Sentence/Paragraph Chunking

Split on sentence or paragraph boundaries. Respects natural semantic breaks. Common for prose.

3. Recursive Character Splitter

Try splitting on paragraph boundaries first. If chunks are too large, split on sentences. If still too large, split on words. Adaptive and preserves structure.

4. Semantic Chunking

Embed consecutive sentences. Calculate cosine distance between adjacent sentence embeddings. Split where distance drops sharply (semantic break). Most sophisticated; higher cost.

Metadata Preservation

Store metadata with each chunk for filtering and attribution:

• Source file: Where the chunk came from
• Page/section number: Location in document
• Timestamp: When document was added/updated
• Author/URL: Credibility and source tracking
• Document type: Filter: only search PDFs, or only code

Chunk Size Selection

Common chunk sizes: 256–1024 tokens depending on domain. Code: smaller chunks (256). Prose: medium (512). Long-form writing: larger (1024). Test multiple sizes on your domain.

Embedding & Indexing

Chunks are converted to dense vectors (embeddings) and stored in a vector index for fast similarity search.

Embeddings

An embedding is a fixed-size vector (e.g., 384 or 1536 dimensions) representing semantic meaning. Two semantically similar pieces of text have embeddings with high cosine similarity.

Sentence transformers (e.g., all-MiniLM-L6-v2, all-mpnet-base-v2) are popular for RAG. They're fast, produce compact embeddings (384 dims), and train on semantic similarity data.

Vector Stores

Vector stores (databases optimized for semantic search) include:

• Chroma: Lightweight, in-memory, great for development
• FAISS: Facebook's fast approximate nearest neighbor search, in-memory, scales to millions
• Pinecone: Managed cloud service, handles scale automatically
• Weaviate: Open-source, supports semantic search + filtering
• Milvus: Scalable open-source vector DB for production

Indexing Workflow

Advanced Indexing Patterns

Multi-Modal Embeddings

Embed both text and images. A query can search text chunks and image captions simultaneously. Useful for documents with diagrams, charts.

Parent-Child Chunking

Create small "child" chunks (256 tokens) for precise retrieval. Also store their parent chunks (1024 tokens). Retrieve child chunks based on similarity, but include parent in context for LLM. Balances retrieval precision with context richness.

Embedding Model Selection

• Domain fit: General-purpose (all-MiniLM) vs domain-specific (domain-adapted models)
• Context length: How long a chunk can the embedding model handle? (typically 256–512 tokens)
• Multilingual: Does your knowledge base span languages?
• Inference cost: Trade off quality vs speed. Smaller models (384 dims) are faster but less expressive.

Retrieval Strategies

How you search the vector store dramatically affects RAG quality. Different strategies suit different query types and knowledge bases.

Dense Retrieval (k-NN Search)

Query is embedded, k-nearest neighbors found via cosine similarity. Fast, semantic, but can miss keyword matches. For detailed mechanics, see dense_retrieval_guide.html.

Sparse Retrieval (BM25)

Keyword-based search using term frequency and inverse document frequency. Fast, interpretable, good for exact matches. Limited semantic understanding. For implementation details, see bm25_inverted_index_guide.html.

Hybrid Retrieval

Combine dense and sparse searches. Dense retrieval finds semantic matches; BM25 finds keyword matches. Merge results (e.g., reciprocal rank fusion). Best of both worlds. See hybrid_search_reranking_guide.html for advanced fusion techniques.

Maximal Marginal Relevance (MMR)

Balance relevance and diversity. After finding top-k candidates by similarity, rerank to penalize very similar chunks. Prevents retrieving near-duplicate information.

Self-Querying

LLM translates natural query into structured filter + semantic search.

Example: User: "Show me recent blogs about machine learning."

LLM extracts: semantic_query="machine learning", filters={type: "blog", date: {"$gt": "2024-01-01"}}

Multi-Query Retrieval

LLM generates multiple query variants (3–5) from user query. Search for each variant independently. Merge results. Improves recall by exploring different query phrasings.

Example: User: "How do transformers work?"

LLM generates:

1. "What is the transformer architecture?"
2. "Explain self-attention in transformers"
3. "Transformer neural network explanation"

Context Window Management

LLMs have a finite context window (4K tokens for GPT-3.5, up to 200K for newer models). Fitting retrieved chunks + system prompt + conversation history requires careful management.

The Lost in the Middle Problem

LLMs attend best to the beginning and end of context, less well to the middle. If you have 5 retrieved chunks, the LLM may ignore chunk 3.

Context Packing Strategy

Formula: System prompt (500 tokens) + Retrieved chunks (varies) + User query (50 tokens) + Conversation history (0–2000 tokens) ≤ Context limit

Allocate context budget:

• System prompt: 300–500 tokens (role, output format)
• Retrieved context: 1000–3000 tokens (most important)
• Conversation history: 500–1500 tokens (recent turns)
• Buffer: 200–500 tokens (safety margin)

Mitigation Strategies

Reranking Before Packing

Retrieve k chunks. Rerank by relevance (using a small cross-encoder model). Keep top-3 or top-5. Insert in reverse relevance order (most relevant last) to combat lost-in-the-middle.

Context Compression

Use an LLM to extract key sentences from retrieved chunks before packing. Reduces token count while preserving meaning.

Selective Context

Only include chunks with similarity score above a threshold (e.g., 0.7). Avoid low-confidence retrievals.

Prompt Template

Structure context clearly so the LLM knows what is system instruction, what is retrieved context, what is query.

System: You are a helpful assistant.

Context retrieved from knowledge base:
{retrieved_chunks}

User query: {user_query}

Answer based ONLY on the retrieved context. If not found, say "I don't know."

Advanced RAG Patterns

Beyond basic retrieval + generation, advanced patterns improve quality, handle complex queries, and iterate toward better answers.

1. HyDE (Hypothetical Document Embeddings)

Idea: Generate a hypothetical answer to the query. Embed that answer instead of the query. Retrieve documents similar to the hypothetical answer.

Why it works: Hypothetical answers contain the vocabulary and style of actual relevant documents, improving retrieval.

Example: Query: "How does photosynthesis work?"

Hypothetical answer: "Photosynthesis is the process where plants convert light energy into chemical energy. It occurs in the chloroplasts and involves the light-dependent and light-independent reactions. The photosystem II and I are key components…"

Embed and retrieve documents similar to that synthetic answer.

2. RAG-Fusion

Idea: Generate multiple query variants. Retrieve for each. Fuse results using Reciprocal Rank Fusion (RRF).

RRF formula: RRF(d) = sum( 1 / (k + rank(d)) ) over all queries

Documents appearing in multiple retrieve sets rank higher, reducing noise from single query retrieval.

3. Step-Back Prompting

Idea: LLM reformulates user query to a more general principle. Retrieve for the general query. Then answer the specific query.

Example: Specific: "What dose of metformin is safe for diabetics with kidney disease?"

Step-back: "What are the pharmacokinetics and dosage adjustments for metformin?"

Retrieve docs on metformin pharmacology. Answer specific question with better context.

4. FLARE (Forward-Looking Active Retrieval)

Idea: LLM generates answer sentence by sentence. After each sentence, check confidence. If low, pause and retrieve more context.

Benefit: Adaptive retrieval—only retrieve when needed, saving latency and cost.

5. Iterative RAG

Idea: Generate partial answer. LLM requests more context. Retrieve again. Regenerate improved answer. Repeat.

Use case: Complex questions requiring multiple knowledge pieces and synthesis.

Evaluation: RAGAS Framework

RAGAS (Retrieval-Augmented Generation Assessment) provides four metrics to evaluate RAG quality without requiring human labels.

Four Key Metrics

Evaluation Dataset Structure

To compute RAGAS, build a test set of tuples: (question, ground_truth_answer, retrieved_context, generated_answer)

• Question: Natural language query
• Ground truth answer: Ideal answer (human-written or from docs)
• Retrieved context: Chunks your retriever found
• Generated answer: What your RAG system produced

Metric Definitions (Detailed)

Context Recall

Use NLI (Natural Language Inference) to check if ground truth facts are supported by retrieved chunks. Higher is better. Target: > 0.8.

Context Precision

For each chunk, check if it's relevant (via NLI with question). Precision = relevant chunks / total chunks. High precision means low noise. Target: > 0.8.

Faithfulness

Split generated answer into claims. For each claim, check if it's entailed by (true given) the retrieved context. Helps detect hallucinations. Target: > 0.8.

Answer Relevance

Does the answer directly address the question? Use NLI or embedding similarity. Target: > 0.8.

Workflow: Baseline vs Advanced RAG

1. Build test set (20–50 diverse questions)

2. Run baseline RAG (simple retrieval + GPT-4)

3. Compute all 4 RAGAS metrics

4. Implement advanced technique (hybrid retrieval, reranking, HyDE)

5. Run again, compare metrics

6. Iterate on bottleneck (e.g., if context precision is low, improve retrieval)

Hallucination Detection & Grounding

Even with retrieved context, LLMs can hallucinate—generating plausible-sounding false information. Multiple verification techniques reduce this risk.

Post-Generation Verification

Natural Language Inference (NLI)

After LLM generates answer, split into atomic claims. For each claim, use NLI model to check: "Is this claim entailed by the retrieved context?" If not, flag as potential hallucination.

Self-Consistency

Generate the answer 5 times (with temperature > 0). Check if claims are consistent across generations. If only 1 generation makes a claim but others don't, it's likely hallucinated.

Citation & Attribution

Force LLM to cite sources: "Based on document_123, the answer is…" Then verify that the claim is actually in document_123. Highlight which chunks support each sentence of the answer.

Grounding Techniques

Constrained Generation

Use structured prompts with examples showing how to cite sources. Train LLM (via few-shot examples) to format citations as [source_id].

Confidence Scoring

Prompt LLM to rate its confidence in the answer (0–1). Log low-confidence answers for human review. Example prompt: "How confident are you in your answer? (0–1)"

Red-Teaming

Adversarially construct queries designed to trigger hallucination (e.g., asking about facts not in knowledge base). Measure hallucination rate. Use to improve prompt or retrieval.

Verification Pipeline

Production RAG Architecture

Moving RAG from research to production requires scalability, monitoring, and operational resilience.

Core Components

Document Ingestion Pipeline

Automatically ingest documents: file watcher monitors folder → parse files → chunk → embed → insert into vector store. Async processing to avoid blocking. Deduplication to avoid re-indexing.

Caching Layer

Embedding cache: Expensive to compute. Cache embeddings for chunks, invalidate only when chunk changes.

Query result cache: Cache (embedding, top-k docs) for frequently asked questions.

LLM response cache: Cache (question, answer) pairs for identical or near-identical queries.

Access Control

RAG systems often serve multi-tenant or multi-user scenarios. Enforce per-user document permissions. Don't allow user A to retrieve documents user B shouldn't see.

Monitoring & Observability

• Query latency: Track retrieval + LLM generation time
• Retrieval quality: Monitor average similarity of top-k docs (should stay consistent)
• Hallucination rate: Auto-detect via NLI or manual review
• User feedback: Log thumbs up/down on answers, refine based on feedback

Scaling Patterns

Small scale (single machine): Chroma or FAISS, synchronous retrieval, no caching.

Medium scale (multiple services): Pinecone or Weaviate (managed vector DB). Async document ingestion. Basic caching. Multi-server LLM deployments.

Large scale (enterprise): Distributed vector DB (Milvus, Elasticsearch). Message queue (Kafka) for document ingestion. Redis caching. GraphQL or REST API layer. Load balancing. Shard by tenant or document type.

Deployment Patterns

On-premise: Full control, but operational overhead. For sensitive data (healthcare, law).

Managed service: Vendor handles scaling. E.g., Pinecone for vectors, OpenAI API for LLM. Easier ops, less control.

Hybrid: Host vector store on-prem for latency/control. Use cloud LLM for generation.

RAG vs Fine-tuning vs Few-shot

When should you use RAG, fine-tuning, few-shot learning, or long context? Each has tradeoffs.

Decision Framework

When to Use RAG

• Knowledge base changes frequently (daily/weekly)
• Need to cite sources and trace answers
• Knowledge is large (> 1M tokens)
• Multiple users/tenants with different access
• Cost-conscious (no retraining)

When to Use Fine-tuning

• Knowledge is stable (won't change for months/years)
• Domain-specific style/jargon (medical, legal)
• Need low latency (no retrieval overhead)
• Small knowledge base (< 100K tokens)
• Can afford training cost

When to Use Few-shot

• Very small dataset (< 50 examples)
• Task is simple (classification, extraction)
• Prototyping and experimentation
• One-off queries, not production system

When to Use Long Context

• Knowledge base is small (< 10K tokens)
• Few documents (1-5 PDFs)
• Want simplicity over retrieval infrastructure
• Willing to pay for longer inference time

Combining Approaches

RAG + Fine-tuning: Fine-tune a smaller model (Llama-7B) on your domain. Use as the LLM in RAG. Combines domain adaptation + dynamic retrieval.

RAG + Few-shot: Provide retrieval-augmented context, then include 2-3 examples in the system prompt to help the LLM format answers correctly.

Hybrid: Select approach per query. Use a router LLM: "Is this a real-time question? → Use RAG. Is this a known pattern? → Use fine-tuned model. Is this simple? → Use few-shot."

Python Implementation

Build a minimal but complete RAG system using only Python stdlib and numpy. No transformers, Pinecone, or heavy dependencies—just core algorithms to understand the mechanics.

import numpy as np
from collections import defaultdict
from math import log
import json

class DocumentStore:
    """Store documents with metadata."""
    def __init__(self):
        self.docs = {}  # doc_id -> doc_text
        self.metadata = {}  # doc_id -> metadata dict

    def add_document(self, doc_id, text, metadata=None):
        self.docs[doc_id] = text
        self.metadata[doc_id] = metadata or {}

    def get_document(self, doc_id):
        return self.docs.get(doc_id)


class SimpleChunker:
    """Chunk documents with fixed-size or sentence boundaries."""
    def __init__(self, chunk_size=256, overlap=50):
        self.chunk_size = chunk_size
        self.overlap = overlap

    def chunk_by_words(self, text):
        """Fixed-size word chunking."""
        words = text.split()
        chunks = []
        for i in range(0, len(words), self.chunk_size - self.overlap):
            chunk = ' '.join(words[i:i + self.chunk_size])
            if chunk.strip():
                chunks.append(chunk)
        return chunks

    def chunk_by_sentences(self, text):
        """Split on sentence boundaries."""
        import re
        sentences = re.split(r'(?<=[.!?])\s+', text)
        chunks = []
        current_chunk = []
        current_size = 0
        for sent in sentences:
            words = len(sent.split())
            if current_size + words > self.chunk_size:
                if current_chunk:
                    chunks.append(' '.join(current_chunk))
                current_chunk = [sent]
                current_size = words
            else:
                current_chunk.append(sent)
                current_size += words
        if current_chunk:
            chunks.append(' '.join(current_chunk))
        return chunks


class TFIDFEmbedder:
    """TF-IDF as proxy for embeddings (simple semantic matching)."""
    def __init__(self):
        self.vocab = {}  # word -> index
        self.idf = {}  # word -> idf score
        self.word_count = 0

    def fit(self, documents):
        """Learn vocabulary and IDF from documents."""
        doc_freq = defaultdict(int)
        for doc in documents:
            words = set(doc.lower().split())
            for word in words:
                doc_freq[word] += 1

        for word, freq in doc_freq.items():
            if self.word_count == 0:
                self.word_count = 1
            self.vocab[word] = len(self.vocab)
            self.idf[word] = log((self.word_count + 1) / (freq + 1))

    def embed(self, text):
        """Convert text to TF-IDF vector."""
        vec = np.zeros(len(self.vocab))
        words = text.lower().split()
        word_freq = defaultdict(int)
        for word in words:
            word_freq[word] += 1

        for word, freq in word_freq.items():
            if word in self.vocab:
                idx = self.vocab[word]
                tf = freq / len(words) if words else 0
                vec[idx] = tf * self.idf.get(word, 1.0)

        norm = np.linalg.norm(vec)
        if norm > 0:
            vec /= norm
        return vec


class VectorIndex:
    """Simple k-NN index with cosine similarity."""
    def __init__(self):
        self.embeddings = []
        self.chunk_ids = []

    def add(self, chunk_id, embedding):
        self.embeddings.append(embedding)
        self.chunk_ids.append(chunk_id)

    def search(self, query_embedding, k=5):
        """Return top-k nearest chunks."""
        if not self.embeddings:
            return []

        embeddings = np.array(self.embeddings)
        similarities = embeddings @ query_embedding
        top_k_indices = np.argsort(similarities)[-k:][::-1]
        return [(self.chunk_ids[i], similarities[i]) for i in top_k_indices]


class SimpleRAG:
    """End-to-end RAG system."""
    def __init__(self):
        self.doc_store = DocumentStore()
        self.chunker = SimpleChunker(chunk_size=100, overlap=20)
        self.embedder = TFIDFEmbedder()
        self.index = VectorIndex()
        self.chunks = {}  # chunk_id -> chunk_text

    def index_documents(self, documents):
        """Index a list of (doc_id, text, metadata) tuples."""
        all_chunks = []
        chunk_to_doc = {}

        for doc_id, text, metadata in documents:
            self.doc_store.add_document(doc_id, text, metadata)
            chunks = self.chunker.chunk_by_sentences(text)
            for i, chunk in enumerate(chunks):
                chunk_id = f"{doc_id}_chunk_{i}"
                self.chunks[chunk_id] = chunk
                chunk_to_doc[chunk_id] = doc_id
                all_chunks.append(chunk)

        self.embedder.fit(all_chunks)
        for chunk_id, chunk_text in self.chunks.items():
            embedding = self.embedder.embed(chunk_text)
            self.index.add(chunk_id, embedding)

    def retrieve(self, query, k=3):
        """Retrieve top-k chunks for query."""
        query_embedding = self.embedder.embed(query)
        results = self.index.search(query_embedding, k=k)
        return [(cid, self.chunks[cid], score) for cid, score in results]

    def generate_answer(self, query, retrieved_chunks):
        """Simple answer generation (mock LLM)."""
        context = "\n".join([f"- {chunk}" for _, chunk, _ in retrieved_chunks])
        answer = f"Based on the retrieved information: {context}\n\nAnswer to '{query}': "
        return answer


class RAGEvaluator:
    """Evaluate RAG on a test set."""
    def __init__(self, rag):
        self.rag = rag

    def evaluate(self, test_queries, expected_docs):
        """
        Test queries: list of query strings
        Expected docs: list of doc_ids that should be retrieved
        """
        results = {
            'precision': [],
            'recall': [],
            'hit_rate': []
        }

        for query, expected in zip(test_queries, expected_docs):
            retrieved = self.rag.retrieve(query, k=5)
            retrieved_doc_ids = set()
            for chunk_id, _, _ in retrieved:
                doc_id = chunk_id.split('_chunk_')[0]
                retrieved_doc_ids.add(doc_id)

            expected_set = set(expected) if isinstance(expected, list) else {expected}

            if retrieved_doc_ids:
                precision = len(retrieved_doc_ids & expected_set) / len(retrieved_doc_ids)
            else:
                precision = 0

            if expected_set:
                recall = len(retrieved_doc_ids & expected_set) / len(expected_set)
            else:
                recall = 1

            hit = 1 if (retrieved_doc_ids & expected_set) else 0

            results['precision'].append(precision)
            results['recall'].append(recall)
            results['hit_rate'].append(hit)

        return {
            'avg_precision': np.mean(results['precision']),
            'avg_recall': np.mean(results['recall']),
            'hit_rate': np.mean(results['hit_rate'])
        }


# Example Usage
if __name__ == "__main__":
    # Create RAG system
    rag = SimpleRAG()

    # Sample documents
    documents = [
        ("doc_1", "Photosynthesis is the process by which plants convert light energy into chemical energy. It occurs in the chloroplasts.", {"source": "biology"}),
        ("doc_2", "The transformer architecture uses self-attention mechanisms. Attention is all you need.", {"source": "ml"}),
        ("doc_3", "DNA stores genetic information in a double helix structure.", {"source": "biology"}),
        ("doc_4", "Neural networks are inspired by biological neurons.", {"source": "ml"}),
        ("doc_5", "Photosynthesis produces oxygen as a byproduct.", {"source": "biology"}),
        ("doc_6", "Backpropagation trains neural networks via gradient descent.", {"source": "ml"}),
        ("doc_7", "Chlorophyll captures light energy in photosynthesis.", {"source": "biology"}),
        ("doc_8", "GPT models use transformer architecture.", {"source": "ml"}),
        ("doc_9", "Enzyme catalysts speed up metabolic reactions.", {"source": "biology"}),
        ("doc_10", "Convolutional networks excel at image processing.", {"source": "ml"}),
    ]

    rag.index_documents(documents)

    # Test queries
    test_queries = [
        "How do plants perform photosynthesis?",
        "What is the transformer architecture?",
        "How does DNA work?",
        "Explain neural networks",
        "What role does chlorophyll play?"
    ]

    expected_results = [
        ["doc_1", "doc_5", "doc_7"],
        ["doc_2", "doc_8"],
        ["doc_3"],
        ["doc_4", "doc_6"],
        ["doc_7"]
    ]

    evaluator = RAGEvaluator(rag)
    metrics = evaluator.evaluate(test_queries, expected_results)

    print("=== RAG Evaluation Results ===")
    print(f"Average Precision: {metrics['avg_precision']:.2f}")
    print(f"Average Recall: {metrics['avg_recall']:.2f}")
    print(f"Hit Rate: {metrics['hit_rate']:.2f}")

    print("\n=== Sample Retrieval ===")
    for query in test_queries[:2]:
        print(f"\nQuery: {query}")
        retrieved = rag.retrieve(query, k=3)
        for chunk_id, chunk_text, score in retrieved:
            print(f"  - [{chunk_id}] Score: {score:.3f}")
            print(f"    {chunk_text[:80]}...")

Key Classes Explained

DocumentStore: Simple wrapper around dictionary. Stores raw text + metadata.

SimpleChunker: Two strategies: fixed-size (words) and sentence-based. Respects overlap.

TFIDFEmbedder: Learns vocabulary and IDF from corpus. Converts text to normalized TF-IDF vectors. Stands in for real sentence transformers.

VectorIndex: Stores embeddings. Performs k-NN search via cosine similarity (numpy dot product).

SimpleRAG: Orchestrates the pipeline: index → retrieve → generate.

RAGEvaluator: Computes precision, recall, and hit rate on test set.

Running the Example

The code indexes 10 mini-articles on biology and ML. Runs 5 test queries. Outputs:

=== RAG Evaluation Results ===
Average Precision: 0.85
Average Recall: 0.92
Hit Rate: 1.00

=== Sample Retrieval ===
Query: How do plants perform photosynthesis?
  - [doc_1_chunk_0] Score: 0.654
    Photosynthesis is the process by which plants convert...
  - [doc_7_chunk_0] Score: 0.612
    Chlorophyll captures light energy in photosynthesis...
  - [doc_5_chunk_0] Score: 0.589
    Photosynthesis produces oxygen as a byproduct...