RAG in production: what the tutorials don’t tell you
Chunking, retrieval evaluation, memory, and why your demo prototype dies on real data — a pragmatic guide with Qdrant and Mem0.
Most RAG tutorials stop as soon as retrieval returns the top three chunks and the LLM says something plausible. In production, that is not enough: real documents contain tables, lists, cross-references and frequent updates that break the assumption that every chunk is independent.
The first serious mistake is treating chunking as an isolated technical parameter. Size and overlap matter, but semantic structure matters more. A contract with conditional clauses cannot be split in half without losing meaning. My current approach is hybrid: heading-based chunking for structured documents, paragraph chunking for narrative text, and, when needed, section-metadata chunks to preserve context.
The second mistake is not measuring retrieval. You need a set of questions with ground truth and metrics such as recall@k, MRR and NDCG. Without them, every tweak to the prompt or embedding model is a shot in the dark. I start with fifty representative questions and use them as a regression suite every time I change the parser, the embedding model or the chunking strategy.
Memory is the third pillar. RAG without memory is a chatbot with amnesia: it repeats information, loses the thread and generates generic answers. Tools like Mem0 or simple message stores with progressive summarization make a real difference in user experience.
Finally, the vector store does not need to be exotic. Qdrant, Pinecone, Chroma or even PostgreSQL with pgvector will do. What separates a prototype from production is the ingestion pipeline: robust parsing, consistent metadata, incremental reprocessing and observability at every step.
Working on something similar? Tell me about your project.