In June 2026, the prod RAG pipeline at a mid-size SaaS company serving 50,000 queries per day costs roughly $3,200/month in vector database fees alone if you pick the wrong store. Pick the right one, and that drops to $400.

The difference isn’t marginal, it’s the gap between a project that ships and one that dies in budget review. And yet most teams still choose their vector database the same way they choose lunch: whatever someone on the team used before.

This article walks through every component of a retrieval-augmented generation pipeline that touches prod traffic in 2026. We cover chunking, embedding models, vector stores, reranking, and generation, with real numbers, real failure modes, and trade-offs engineering teams actually face. No toy examples. No “hello world” RAG.

The RAG Stack in 2026: What Actually Ships

The five-component pipeline hasn’t changed much structurally since 2024, but defaults have shifted dramatically. Here’s what a prod RAG system looks like in mid-2026:

Embedding Models: The MTEB Leaderboard Isn’t Your Friend

Chunking: Documents enter the pipeline and get split into overlapping segments. The naive approach (fixed-size character splits with no overlap) still ships in tutorials but fails in prod the moment a user asks a question that spans a chunk boundary. Semantic chunking (splitting on sentence boundaries with configurable overlap) is table stakes now.

Embedding: Each chunk passes through an embedding model that produces a dense vector, typically 1024 or 1536 dimensions. The BGE family from BAAI and GTE family from Alibaba dominate open-source deployments. OpenAI’s text-embedding-3-large remains the go-to for teams that don’t want to self-host, though at $0.13 per million tokens it adds up fast. For a deeper dive into how tokenization and embedding models work under the hood, see our guide on NLP Fundamentals: Tokenization & Recognition.

Vector Store: The vectors get indexed and stored. Pinecone, Weaviate, Qdrant, pgvector, LanceDB, and ChromaDB are six names that appear in prod. Each has a distinct cost and latency profile at scale, which we’ll break down in detail.

Reranking: The vector store returns top-k candidates (usually k=20 to k=50), and a reranker scores them against the query to produce the final top-n (typically n=3 to n=8). Cohere’s rerank-v3 and BGE-reranker-v2 are dominant choices.

Generation: The reranked chunks plus user query go to an LLM. Claude 3.5 Sonnet, GPT-4o, and Llama 3.1 (70B or 405B self-hosted) are the workhorses. The model produces a grounded answer, ideally with citations, though citation accuracy remains uneven across providers.

Chunking Strategies That Survive Prod

The chunking problem looks simple and isn’t. You have a document (PDF, Confluence page, codebase) and you need to split it into pieces small enough to retrieve precisely but large enough to contain complete thoughts. Get it wrong, and your RAG system returns fragments that look relevant but miss the context that makes them useful.

Fixed-size chunking (e.g., 512 tokens with 64-token overlap) works for homogenous text like documentation. It fails catastrophically on anything with structure: legal contracts, API specs, financial reports. A chunk that starts mid-sentence in section 3.2 and ends mid-sentence in section 3.4 is a retrieval poison pill, it matches vector similarity but answers nothing.

Semantic chunking splits on natural boundaries: paragraphs, sections, list items. LangChain’s RecursiveCharacterTextSplitter with sentence-aware splitting and a 10-20% overlap buffer is the most common prod choice. LlamaIndex offers SentenceSplitter with similar semantics. Both let you set chunk_size in tokens and chunk_overlap as a percentage.

Note: The following code is an illustrative example and has not been verified against official documentation. Please refer to the official docs for production-ready code.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
 chunk_size=512,
 chunk_overlap=64,
 separators=["\n\n", "\n", ". ", " ", ""],
 length_fn=len,
)

chunks = splitter.split_text(document)
# Note: prod use should add metadata tracking (source doc, page number,
# section header) to each chunk for citation and debugging

The overlap matters more than most teams realize. In practice, 64-128 tokens of overlap on 512-token chunks hits the sweet spot.

Hierarchical chunking (storing both small chunks (for precise retrieval) and parent chunks (for context)) has gained traction. You retrieve a small chunk, then fetch its parent for the LLM. LanceDB and Weaviate both support this pattern natively.

The chunking strategy you choose should match your document type. Codebases benefit from AST-aware splitting (split on function boundaries, not line counts). Legal documents need section-aware splitting with header inheritance. PDFs with tables need special handling, most chunkers mangle tabular data into unreadable text. Unstructured.io and LlamaParse are the two main tools for table-aware PDF ingestion in 2026.

Embedding Models: The MTEB Leaderboard Isn’t Your Friend

The Massive Text Embedding Benchmark leaderboard shows BGE-M3 at the top for retrieval tasks, and teams routinely cargo-cult the #1 model into prod. This is a mistake. MTEB scores measure performance on academic retrieval datasets, short queries against short passages, clean text, no domain shift. Your prod data looks nothing like that.

What actually matters when choosing an embedding model:

• Maximum sequence length: If your chunks are 512 tokens and your model’s max is 512, you’re silently truncating. BGE-large-en-v1.5 handles 512 tokens, text-embedding-3-large handles 8191, GTE-large handles 512, and BGE-M3 handles 8192. Match the model to your chunk size.
• Vector dimension: Higher dimensions mean better recall (up to a point) and higher storage cost. 1024 dimensions is the prod sweet spot in 2026. 1536 (OpenAI) costs 50% more to store and query. 768 (some GTE variants) saves money but loses ~3-5% recall on most benchmarks.
• Multilingual support: BGE-M3 and text-embedding-3-large handle 100+ languages. GTE and older BGE models are English-only. If your corpus includes any non-English content, this single factor eliminates half the field.
• Hosting cost: Self-hosting BGE-M3 on an A10 GPU costs roughly $0.80/hour and handles ~500 queries/second. OpenAI’s API costs $0.13 per million tokens, about $0.65 for 500 queries at 512 tokens each. The crossover point where self-hosting wins is around 600 queries/second sustained.

One underappreciated factor: embedding model drift. If you embed 10 million documents with BGE-M3 and then upgrade to BGE-M4 when it ships, your entire vector index is stale. You either re-embed everything (expensive) or run a mixed-index setup (operationally painful). Teams that pick OpenAI’s API avoid this, model version is abstracted behind an API endpoint. Teams that self-host need a re-embedding strategy from day one.

Vector Database Comparison: Latency, Cost, and Ops at Scale

Six vector databases dominate prod RAG in 2026. Here’s what actually differentiates them at scale, not feature matrices from marketing pages, but dimensions that show up in incident reviews and AWS bills. For a detailed feature-by-feature breakdown of major options, check out our Vector Database Comparison 2026: Pinecone vs Qdrant vs Weaviate vs Chroma vs LanceDB.

Pinecone is the “just make it work” option. Zero ops, automatic scaling, and the best hybrid search implementation (proprietary sparse-dense vectors). The downside: you cannot self-host, you cannot inspect index internals, and costs scale linearly with your vector count. At 10 million vectors with 1024 dimensions, expect roughly $400-700/month. The architecture uses a proprietary approximate nearest neighbor index; details are not publicly documented beyond Pinecone’s own blog posts.

Qdrant is the performance play. Written in Rust, it delivers claimed sub-50ms p99 latency at 1 million vectors. Qdrant Cloud starts at $25/month. Self-hosted Qdrant on a single 8GB node handles roughly 5-10 million vectors comfortably. The Rust implementation means no garbage collection pauses, a real advantage over JVM-based alternatives at high throughput.

Weaviate offers the most mature hybrid search, combining BM25 keyword search with vector similarity out of the box. It also has the richest filtering system, you can filter on any property before or after vector search. The trade-off: it’s heavier to operate. Weaviate runs on the JVM, needs careful heap tuning, and memory usage is higher than Qdrant or LanceDB for equivalent workloads. Weaviate Cloud starts at $25/month.

pgvector is the “we already have Postgres” option. The pgvector extension adds vector storage and ANN indexing to any Postgres 12+ database. It supports HNSW and IVFFlat indexes, halfvec (2-byte floats) for memory savings, and binary quantization. The killer feature: you get vectors in the same transactions as your relational data. The killer limitation: ANN index builds are slow (hours for 10M+ vectors), and query latency degrades under concurrent write load. For read-heavy RAG workloads under 5 million vectors, pgvector works fine. Above that, dedicated vector databases pull ahead.

LanceDB is the dark horse. It’s an embedded vector database built on the Lance columnar format, the same format used by Lance for ML datasets. It runs in-process (no server), supports disk-based indexes that don’t require loading everything into RAM, and handles multimodal data (images, video, text) natively. For teams building RAG into desktop apps, edge devices, or CLI tools, LanceDB eliminates server dependency entirely. It’s also free and open-source under Apache 2.0.

ChromaDB is the easiest to start with and the hardest to scale. It’s Python-native, installs with pip, and works great for prototyping. At prod scale (1M+ vectors, concurrent queries), it runs into performance walls that other databases have solved. ChromaDB is the right choice for hackathons, demos, and internal tools with low query volume. It is not the right choice for customer-facing RAG at scale.

Reranking: The 15% Recall Boost Nobody Talks About

Vector similarity search is a coarse filter. It finds chunks that are semantically “close” to your query in embedding space, but embedding space is a lossy compression of meaning. Two chunks can have high cosine similarity and still be irrelevant to the specific question being asked. Reranking fixes this.

A reranker is a cross-encoder model that takes a query and a candidate chunk as input and outputs a relevance score. Unlike a bi-encoder used for embedding (which encodes the query and document separately), a cross-encoder processes them together and can model fine-grained semantic relationships. The cost: cross-encoders are 100-1000x slower than vector similarity search. That’s why you only run them on top-k candidates (typically 20-50), not the full index.

The prod reranker landscape in 2026:

• Cohere Rerank v3: API-based, $2 per 1,000 searches. Consistently tops retrieval benchmarks. Handles documents up to 4,096 tokens. The default choice for teams that don’t want to self-host.
• BGE-Reranker-v2: Open-source, self-hosted. Based on BGE-M3 architecture. Runs on a single A10 GPU. Competitive with Cohere on most benchmarks, sometimes beating it on technical domains. The default choice for teams that self-host.
• ColBERT: Late-interaction model that pre-computes token-level embeddings for documents, enabling fast reranking without a full cross-encoder pass. Requires more storage (token-level embeddings per document) but delivers sub-50ms reranking latency. Gaining adoption in latency-sensitive applications.

A real-world example: a customer support RAG system at a SaaS company retrieves 50 candidate chunks from Qdrant using BGE-M3 embeddings. The reranking step adds roughly 200ms to total query latency and costs $0.002 per query. For a customer-facing chatbot where wrong answers directly impact user trust, that trade-off is a no-brainer.

Note: The following code is an illustrative example and has not been verified against official documentation. Please refer to the official docs for production-ready code.

import cohere

co = cohere.Client("your-api-key")

# Stage 1: vector retrieval (fast, coarse)
vector_results = qdrant_client.search(
 collection_name="docs",
 query_vector=query_embedding,
 limit=50,
)

# Stage 2: rerank (slow, precise)
documents = [hit.payload["text"] for hit in vector_results]
rerank_response = co.rerank(
 query=user_query,
 documents=documents,
 top_n=5,
 model="rerank-english-v3.0",
)

# Stage 3: send top reranked chunks to LLM
final_context = [documents[r.index] for r in rerank_response.results]
# Note: prod use should add timeout handling, retry logic,
# and fallback to vector-only results if reranker API is down

One common mistake: teams run reranking on too many candidates. At 50 candidates, Cohere Rerank costs $0.002 per query. Profile your recall@k curve and set the candidate count to the elbow point.

RAG Failure Modes and How to Catch Them

RAG systems fail in predictable ways. Most teams discover these failures in prod, through user complaints. Here are four failure modes worth instrumenting for before you ship.

1. Chunk-boundary loss. A user asks a question whose answer spans two chunks. The embedding model retrieves one chunk but not the other. The LLM sees half the answer and either hallucinates the rest or produces a partial response. Detection: monitor the distribution of retrieved chunk sizes. If answers routinely come from chunks at minimum size, your chunking strategy is fragmenting information. Fix: increase overlap, use hierarchical chunking, or add a “fetch surrounding chunks” post-retrieval step.

2. Off-topic retrieval. The vector store returns chunks that are semantically similar to the query but irrelevant to the user’s intent. “How do I reset my password?” retrieves chunks about password policies, not the reset flow. Detection: log cosine similarity between the query and retrieved chunks. If similarity scores for correct answers and incorrect answers overlap heavily, your embedding model isn’t discriminating well for your domain. Fix: fine-tune the embedding model on your domain data, or add keyword-based filtering as a pre-retrieval step.

3. Hallucination over retrieved context. The LLM receives correct context but generates an answer that contradicts it. This is the hardest failure mode to detect automatically. Detection: use LLM-as-judge evaluation where a separate model checks whether the generated answer is entailed by the retrieved context. Frameworks like RAGAS and TruLens automate this. Fix: prompt engineering (explicit “only use information from provided context” instructions), better models (Claude 3.5 Sonnet hallucinates less than GPT-4o on grounded generation tasks), or citation requirements that force the model to anchor claims to specific chunks.

4. Eval blind spots. Teams evaluate RAG systems on retrieval metrics (recall@k, MRR) and generation metrics (faithfulness, relevance) separately. They rarely evaluate the pipeline end-to-end: does the user get the right answer? Detection: run end-to-end evaluation on a representative query set with ground-truth answers. Measure “answer correctness” as a binary metric, not a proxy.

The evaluation stack worth using in 2026: RAGAS for component-level metrics (faithfulness, answer relevancy, context precision, context recall), combined with a custom end-to-end eval set of 200-500 real user questions with verified answers. Run evals on every pipeline change. The team at Anthropic published a useful framework for this in their 2024 RAG evaluation guide, and the patterns still hold.

Key Takeaways

• Semantic chunking with 10-20% overlap is the prod baseline. Hierarchical chunking adds ~30% storage overhead but fixes boundary-loss failures.
• BGE-M3 (self-hosted) and text-embedding-3-large (API) are the two embedding models that cover most prod scenarios. Match the max sequence length to your chunk size or you’re silently truncating.
• Qdrant wins on raw performance per dollar for self-hosted deployments. Pinecone wins on zero-ops convenience. LanceDB is the right choice for embedded/edge use cases. ChromaDB is for prototyping, not prod.
• Reranking adds 10-20% recall and costs $0.002/query with Cohere. Skip it only if latency is the absolute top priority.
• Instrument for four failure modes (chunk-boundary loss, off-topic retrieval, hallucination, and eval blind spots) before users report them.

Related Reading

More in-depth coverage from this blog on closely related topics:

• NLP Fundamentals: Tokenization & Recognition
• Mapping Defenses Against Adversaries in AI Output Provenance
• Scalable Self-Hosted RAG Systems 2026
• Apple 2026 WWDC: AI Platform & Dev Innovation
• Vector Database Comparison 2026: Pinecone vs Qdrant vs Weaviate vs Chroma vs LanceDB