RAG Systems with LangChain

What RAG Actually Is

A language model knows what it was trained on. That's it. RAG is the pattern of stuffing relevant context into the prompt so the model can answer questions about your data — without fine-tuning.

The acronym stands for Retrieval-Augmented Generation. The pipeline: embed your documents, store embeddings in a vector DB, embed the query, retrieve the closest chunks, send them with the question to the LLM.

Embedding Strategy

Your retrieval quality is only as good as your chunking strategy. Too large: you retrieve irrelevant context. Too small: you miss the full picture. I use 512-token chunks with 50-token overlap for most document types.

Chunking by Structure, Not by Size

When your documents have natural structure (markdown headers, legal sections, code blocks), chunk by structure first. A semantically complete chunk beats a 512-token slice every time.

Choosing a Vector DB

Pinecone for production (managed, scalable, fast). pgvector for apps already on Postgres (zero new infra). Chroma for local development. Don't over-engineer this choice — switching later is easier than it sounds.

Reranking for Better Results

First-pass retrieval gets you candidates. A reranker (Cohere Rerank works well) scores them by actual relevance to the query. Adding a reranker cut hallucination rate in my last project by roughly 40%.

Evaluating RAG Quality

The hardest part. I use three metrics: faithfulness (does the answer come from the retrieved context?), answer relevance (does it answer the question?), and context precision (were the retrieved chunks actually useful?). RAGAS automates this.