What Is RAG and Why Does It Eliminate Hallucination?
Retrieval-Augmented Generation (RAG) is an architecture pattern that grounds an LLM's responses in your actual data rather than whatever it picked up during training. A user asks a question. The system goes and searches your documents for the relevant bits, then hands that context to the LLM alongside the question. So the answer comes from YOUR data, not the model's general knowledge of the world.
Without RAG, LLMs hallucinate. They hand you answers that sound right and are flat-out wrong. Ask one about your product pricing or a company policy or where a customer's order is, and it has nothing to go on. None of that lived in its training data.
Add RAG and hallucination drops off a cliff. The model still writes the language. It just writes it while staring at the real source material. And if the answer is not in the context it pulled back, a RAG system worth its salt will say 'I don't have information about that' instead of inventing something.
Where does this show up in practice? Support bots that answer straight from your knowledge base. Internal tools that search company policy. Legal research that cites the exact document it pulled from. Product assistants that actually know your feature set. Our Python development team builds these RAG systems for production, and our AI/ML engineers own the pipeline end to end.
How Does RAG Architecture Work?
A RAG pipeline runs in four stages. Here is how each one earns its keep.
Stage 1, Document Processing: Pull in your source documents. PDFs, web pages, Notion docs, API references, whatever you have. Clean the text. Strip the headers, footers, and nav junk. Then split it into chunks (more on that below). People underrate this step constantly, and they pay for it later. Garbage in, garbage out.
Stage 2, Embedding: Turn each chunk into a vector, which is really just a list of numbers that captures what the text means. You run it through an embedding model. We reach for OpenAI text-embedding-3-small ($0.02/million tokens), Cohere embed-v3, or an open-source option like Sentence Transformers when we want to self-host. Every chunk ends up as a point in vector space, and similar content lands close together.
Stage 3, Retrieval: A question comes in. Embed it with the same model you used on your documents (this matters, mismatched models retrieve garbage). Then search the vector database for the top-K closest chunks, usually K=3 to 5. Those chunks are your retrieved context. That is the data the LLM gets to answer from.
Stage 4, Generation: Now you build the prompt. System instructions, then the retrieved context, then the user's question. Send all of it to an LLM (we tend to use Claude Sonnet or GPT-4o). The model writes an answer grounded in those chunks. One thing we never skip: attach the sources, so a user can check the answer against the document it came from.
Latency on a RAG system that is built well runs 1 to 3 seconds end to end. Embedding is roughly 50ms, retrieval 100 to 200ms, and generation 1 to 2.5 seconds once you stream the tokens back. Get a free RAG architecture consultation.
Which Vector Database Should You Use?
Pinecone: Fully managed, basically zero ops. The Starter plan is $70/month for 100K vectors. Great fit if your team does not want to babysit infrastructure. The catch? You are locked into the vendor, the price climbs with vector count rather than queries, and there is no self-hosting escape hatch.
Weaviate: Open source and self-hostable. Runs on Docker or Kubernetes. It ships with hybrid search baked in (vector plus keyword), which is genuinely handy. It is more work to operate than Pinecone, but nobody owns you. Free if you host it yourself. Weaviate Cloud starts at $25/month if you would rather not.
pgvector: A PostgreSQL extension. It lives inside the Postgres database you already run, so there is no new infrastructure to stand up. On Supabase it comes built in. Performance holds up nicely to about 1 million vectors. Past that point, a dedicated vector database starts pulling ahead.
What we usually do: If you are already on PostgreSQL or Supabase, start with pgvector. It is free and you add nothing to your stack. Graduate to Pinecone or Weaviate once you blow past a million vectors, or once you need sub-50ms retrieval at real scale. No sooner. Switching early is just busywork.
Here is the cost side by side for 500K vectors and 10K daily queries. pgvector on Supabase Pro: $25/month all in. Pinecone Starter: $70/month. Weaviate Cloud: $50/month. Self-hosted Weaviate on a DigitalOcean box: $24/month.
What Chunking Strategies Actually Work for RAG?
Honestly, how you chunk your documents decides RAG quality more than which model or vector database you pick. We have watched a bad chunking pass tank an otherwise solid system. Three strategies are worth knowing.
Fixed-size chunking: Split every N tokens, usually 256 to 512. Dead simple to implement. It does fine on uniform content like blog posts and articles. It falls apart on structured documents, where a blind split can slice a table or a code block clean in half.
Semantic chunking: Split at the natural seams instead. Paragraph breaks. Section headings. The end of a sentence. Each chunk keeps its meaning intact. It takes more effort to build, since you need some NLP to find sentence boundaries reliably. It pays off most on documentation, policy docs, and technical guides.
Hierarchical chunking: Set up parent and child chunks. The parent holds a whole section. The children hold the individual paragraphs inside it. When a child chunk matches a query, you pull the parent back too, so the LLM sees the full context around the match. For anything with nested structure, like legal contracts, technical specs, or API docs, this is the strategy that wins.
Chunk size matters more than people expect. Go too small (100 tokens) and each chunk is starved of context, so retrieval gets noisy. Go too large (2000 tokens) and the chunk drowns the actual answer while burning context window. For most cases the sweet spot is 300-500 tokens with 50-token overlap between neighboring chunks.
Metadata enrichment: Tag every chunk with its source URL, document title, section heading, and date. That extra metadata lets you filter (search product docs only, leave the blog posts out) and lets you cite the source right in the answer.
How Do You Measure RAG Pipeline Quality?
You cannot improve what you do not measure. With RAG, three metrics carry the weight.
Faithfulness: Does the answer actually match the context it was given? Scored 0 to 1. A faithful answer only claims things the retrieved chunks support. An unfaithful one sneaks in facts from the model's training data, which is hallucination by another name. We aim for above 0.85.
Relevancy: Did retrieval grab the right chunks in the first place? Scored 0 to 1. You check whether the chunks it pulled actually hold the information the question needs. Feed an LLM the wrong chunks and even the best model gives you a weak answer. We target above 0.80.
Answer Correctness: Is the final answer right and complete, judged against a ground-truth set? This one needs a hand-curated dataset of 50 to 100 question-answer pairs. It is a pain to build, no way around that. For anything going to production, it is non-negotiable.
How we wire up evaluation: RAGAS (open source, Python) handles the automated scoring. Build a test set of 50 questions whose answers you already know. Re-run it every week, and always after you touch chunking, prompts, or retrieval settings. We treat these scores like unit tests. If a number drops, something broke, and you go find out what.
What Are the Best RAG Deployment Patterns?
Cache the questions everyone asks: When 100 people all ask 'What is your refund policy?', running the full pipeline 100 times is just wasted money. Cache the question embeddings and their answers instead. A plain Redis cache with a one-hour TTL has cut our costs 40 to 60% on customer-facing apps.
Hybrid search: Run vector similarity and keyword search (BM25) together. Vector search is great at finding content that means the same thing. Keyword search nails the exact matches that vectors fumble, like product names, error codes, and IDs. Put them together and retrieval relevancy beats either one alone by 15 to 25%.
Feedback loops: Put a thumbs up and thumbs down on every answer. Watch which queries collect the thumbs down. That list is your to-do list. Sit with those conversations once a week and the pattern usually shows itself. Sometimes the documentation is just missing, so you write it. Sometimes the chunking was bad, so you re-chunk that one doc. Sometimes the system prompt needs a tweak.
Monitoring: Log all of it. Every query, the chunks it retrieved, the answer it produced. Then keep an eye on the numbers that matter. Retrieval latency (we aim under 200ms). Generation latency (under 3s). The empty-retrieval rate, meaning queries that come back with nothing relevant. And a user satisfaction score on top.
What a production RAG system costs: Build it for $20,000 to $60,000. Then $200 to $1,000 a month to run, which covers the embedding API, the LLM API, vector DB hosting, and app hosting. One thing worth knowing: cost tracks your document volume and query volume, not how many users you have.
Related: AI agents architecture
