What Is RAG and Why Does It Eliminate Hallucination?
Retrieval-Augmented Generation (RAG) is an architecture pattern that grounds LLM responses in your actual data instead of the model's training knowledge. When a user asks a question, the system searches your documents for relevant information, then feeds that context to the LLM along with the question. The LLM generates an answer based on YOUR data — not its general knowledge.
Without RAG, LLMs hallucinate. They generate plausible-sounding answers that are factually wrong. They cannot tell you about your product's pricing, your company's policies, or your customer's order status — because that information was not in their training data.
With RAG, hallucination drops dramatically. The LLM still generates language, but it does so while looking at the actual source material. If the answer is not in the retrieved context, a well-designed RAG system says 'I don't have information about that' instead of making something up.
Common RAG use cases: customer support bots that answer from your knowledge base, internal tools that search company policies, legal research that cites specific documents, and product assistants that know your feature set. Our Python development team builds production RAG systems, and our AI/ML engineers handle the end-to-end pipeline.
How Does RAG Architecture Work?
The RAG pipeline has four stages:
Stage 1 — Document Processing: Ingest your source documents (PDFs, web pages, Notion docs, API docs). Clean the text — remove headers, footers, navigation elements. Split into chunks (see chunking section below). This is the most underestimated step — garbage in, garbage out.
Stage 2 — Embedding: Convert each text chunk into a vector (a list of numbers that captures semantic meaning). Use an embedding model: OpenAI text-embedding-3-small ($0.02/million tokens), Cohere embed-v3, or open-source alternatives (Sentence Transformers). Each chunk becomes a point in vector space where similar content is close together.
Stage 3 — Retrieval: When a user asks a question, embed the question using the same model. Search the vector database for the top-K most similar chunks (typically K=3 to 5). These chunks are the 'retrieved context' — the data the LLM will use to answer.
Stage 4 — Generation: Construct a prompt: system instructions + retrieved context + user question. Send to an LLM (Claude Sonnet or GPT-4o). The LLM generates an answer grounded in the retrieved chunks. Include source attribution so users can verify the answer.
End-to-end latency for a well-built RAG system: 1-3 seconds (embedding: 50ms, retrieval: 100-200ms, generation: 1-2.5 seconds with streaming). Get a free RAG architecture consultation.
Which Vector Database Should You Use?
Pinecone: Fully managed, zero-ops. $70/month for the Starter plan (100K vectors). Excellent for teams without infrastructure expertise. Limitation: vendor lock-in, pricing scales with vector count (not queries), and no self-hosting option.
Weaviate: Open-source, self-hostable. Can run on Docker or Kubernetes. Built-in hybrid search (vector + keyword). More complex to operate than Pinecone but no vendor lock-in. Free to self-host; Weaviate Cloud starts at $25/month.
pgvector: PostgreSQL extension. Runs inside your existing Postgres database — no additional infrastructure. If you use Supabase, pgvector is built in. Performance is good up to ~1 million vectors. Beyond that, dedicated vector databases perform better.
Our recommendation: Start with pgvector if you already use PostgreSQL/Supabase (it is free and requires no new infrastructure). Move to Pinecone or Weaviate when you exceed 1 million vectors or need sub-50ms retrieval latency at scale.
Cost comparison for 500K vectors with 10K daily queries: pgvector on Supabase Pro: $25/month total. Pinecone Starter: $70/month. Weaviate Cloud: $50/month. Self-hosted Weaviate on DigitalOcean: $24/month.
What Chunking Strategies Actually Work for RAG?
How you split documents into chunks determines RAG quality more than model choice or vector database selection. Three strategies:
Fixed-size chunking: Split every N tokens (typically 256-512). Simple to implement. Works well for uniform documents (blog posts, articles). Fails on structured documents where a split might cut a table or code block in half.
Semantic chunking: Split at natural boundaries — paragraph breaks, section headings, sentence endings. Preserves meaning within each chunk. More complex to implement (requires NLP for sentence boundary detection). Works best for documentation, policies, and technical guides.
Hierarchical chunking: Create parent-child relationships. A parent chunk contains an entire section. Child chunks contain individual paragraphs. When a child chunk matches a query, retrieve the parent for full context. This is the most effective strategy for documents with nested structure (legal contracts, technical specs, API docs).
Chunk size matters: Too small (100 tokens) → each chunk lacks context, retrieval is noisy. Too large (2000 tokens) → chunks dilute the specific answer, waste LLM context window. Sweet spot for most use cases: 300-500 tokens with 50-token overlap between consecutive chunks.
Metadata enrichment: Attach source URL, document title, section heading, and date to each chunk. This metadata enables filtering (only search product docs, not blog posts) and source attribution in answers.
How Do You Measure RAG Pipeline Quality?
You cannot improve what you cannot measure. Three metrics define RAG quality:
Faithfulness: Does the generated answer accurately reflect the retrieved context? Score 0-1. A faithful answer only makes claims supported by the retrieved chunks. An unfaithful answer adds information from the LLM's training data (hallucination). Target: >0.85.
Relevancy: Did the retrieval step find the right chunks? Score 0-1. Measure by checking whether the retrieved chunks contain the information needed to answer the query. If the system retrieves irrelevant chunks, even the best LLM will generate a poor answer. Target: >0.80.
Answer Correctness: Is the final answer factually correct and complete? Compared against a ground-truth answer set. This requires a manually curated test dataset of 50-100 question-answer pairs. Expensive to create but essential for production systems.
Evaluation framework: Use RAGAS (open-source, Python) for automated evaluation. Create a test set of 50 questions with known correct answers. Run weekly evaluations after any change to chunking, prompts, or retrieval parameters. Treat evaluation scores like unit tests — if they drop, something broke.
What Are the Best RAG Deployment Patterns?
Cache frequent queries: If 100 users ask 'What is your refund policy?', you do not need to run the RAG pipeline 100 times. Cache question embeddings and their answers. A simple Redis cache with a 1-hour TTL reduces costs by 40-60% for customer-facing applications.
Hybrid search: Combine vector similarity search with keyword search (BM25). Vector search finds semantically similar content. Keyword search finds exact matches (product names, error codes, IDs). Hybrid search outperforms either alone by 15-25% on retrieval relevancy.
Feedback loops: Add thumbs up/down to every answer. Track which queries get negative feedback. These are your improvement targets. A weekly review of 'thumbs down' conversations reveals: missing documentation (add content), poor chunking (re-chunk that document), or prompt issues (adjust system instructions).
Monitoring: Log every query, retrieved chunks, and generated answer. Track: retrieval latency (target <200ms), generation latency (target <3s), empty retrieval rate (queries that find no relevant chunks), and user satisfaction score.
Cost for a production RAG system: Development: $20,000-$60,000. Monthly operating: $200-$1,000 (embedding API + LLM API + vector DB hosting + application hosting). The cost scales with document volume and query volume, not user count.
Need a RAG system built? We have deployed RAG for knowledge bases, support bots, and internal tools.
Related: AI agents architecture
