Best Chunking Methods for Document Retrieval

The Vectara study published at NAACL 2025 tested 25 chunking configurations against 48 embedding models and found that segmentation choices affected retrieval quality as much as the embedding model itself. Larger benchmarks conducted since then have confirmed that result. A March 2026 systematic study on arXiv evaluated 36 segmentation methods across six knowledge domains and five embedding models, keeping the retrieval pipeline consistent across all configurations, and reported the same pattern: content-aware chunking outperformed naive fixed-length splitting, but the best results came from structure-aware heuristics rather than costly semantic methods.

This outcome challenges those who have invested in embedding-based chunkers. Fixed-size character chunking performed poorly in that 36-method study, while the top approach, Paragraph Group Chunking, resembles a well-tuned heuristic rather than a model. It achieved a mean nDCG@5 of about 0.459, Precision@1 near 24%, and Hit@5 close to 59% across domains according to the arXiv:2603.06976 paper. Increased complexity did not improve results.

The Benchmark Numbers Measure Different Things

Three independent studies dominate the chunking literature, and they differ because they evaluate different stages of the pipeline. Chroma’s evaluation measures token-level retrieval recall: how much of the relevant text the retriever actually returns. NVIDIA’s benchmark measures end-to-end RAG answer accuracy using RAGAS NV Answer Accuracy with a panel of judge models. The Vectara study evaluates document retrieval, evidence retrieval, and retrieval-based answer generation as separate tasks.

The Vectara end-to-end test reported semantic chunking at 54% answer accuracy compared to 69% for recursive splitting at 512 tokens. Both results are accurate. High retrieval recall with small fragment chunks means the LLM received retrieved text lacking enough context to answer the question correctly.

The difference in what is measured affects production choices. A retrieval-only benchmark recommends semantic chunking. An end-to-end benchmark shows that approach yields worse answers. Teams optimizing recall@k alone deliver systems that retrieve the correct text but still produce incorrect answers.

Chunking Strategy Comparison: Recall vs. Answer Accuracy

Strategy	Reported result	What it measured	Source
Recursive character, 512 tokens	69% accuracy	End-to-end answer accuracy, 50 papers / 905,746 tokens	Prem AI / FloTorch 2026
Fixed-size, 512 tokens	67% accuracy	End-to-end answer accuracy, same corpus	Prem AI / FloTorch 2026
Semantic (LLMSemanticChunker)	54% accuracy; 91.9% token recall	Answer accuracy vs. retrieval recall, different pipelines	Chroma
Recursive character, 400 tokens	88-89% recall	Token-level retrieval recall, text-embedding-3-large	Chroma
Page-level	0.648 accuracy, 0.107 std. dev.	End-to-end answer accuracy, five datasets	NVIDIA
Paragraph Group Chunking	nDCG@5 ~0.459	Graded relevance, 36 methods / six domains	arXiv:2603.06976

The Semantic Chunking Fragment Tax

Semantic chunking splits text into sentences, embeds each one, calculates cosine similarity between consecutive sentence embeddings, and places boundaries where similarity falls below a threshold. The method is valid. Its drawback is that it has no default minimum chunk size.

In the FloTorch 2026 benchmark, semantic chunking produced fragments averaging 43 tokens. Those fragments retrieved cleanly, which explains the high retrieval metric, but they provided the generator with too little context to produce correct answers. The solution is to set a minimum chunk size between 200 and 400 tokens, using min_chunk_size in Chonkie’s SemanticChunker or the equivalent parameter in LlamaIndex’s SemanticSplitterNodeParser. Without this minimum, semantic chunking improves retrieval but reduces generation quality.

The Vectara study goes further. Their peer-reviewed evaluation found fixed-size chunking consistently outperformed semantic chunking across document retrieval, evidence retrieval, and answer generation, concluding the computational overhead was not justified. Semantic chunking requires embedding every sentence at ingestion, which involves API calls or local model inference on a corpus that a character splitter would process without extra cost. Choosing semantic chunking as a default because it sounds more advanced than RecursiveCharacterTextSplitter is a mistake.

Chunk-Boundary Loss and Metrics That Miss It

Chunk-boundary loss occurs when a definition is in one chunk and its application is in the next. For example, a query about a policy retrieves the paragraph applying the rule but not the paragraph defining it. The LLM generates an answer consistent with the retrieved context but incorrect relative to the full document.

Standard retrieval metrics do not detect this. Recall@k and MRR check whether relevant text was returned, not whether the returned text was semantically complete. NVIDIA’s answer accuracy metric detects this issue because the judge model compares the generated answer to a ground-truth reference, but only if the evaluation set includes queries spanning chunk boundaries.

Snowflake’s finance RAG study measured the effect of chunk size directly on hand-curated SEC filings. Moderate chunk sizes yielded the best accuracy, while overly large chunks diluted relevance and reduced performance by about 10% to 20%. Retrieving more chunks helped at moderate sizes; increasing individual chunk size hurt. Their surprising finding was that chunking and retrieval choices mattered more than the generator. Both models scored only 5-10% without retrieval.

Snowflake also found that adding document-level context to every chunk (such as company name, filing date, form type) outperformed generating unique LLM summaries per chunk, a practice popularized by Anthropic’s contextual retrieval approach. The chunk-level summarization approach, which requires much more compute, reduced accuracy on both Llama 3 and Claude Sonnet in their tests. Adding a global context header once per chunk is more efficient and reliable than per-chunk synthesis.

Late Chunking and Context at the Embedding Layer

Late chunking and context flow at the embedding layer

Late chunking, described in arXiv:2409.04701 by Günther and colleagues at Jina AI, reverses the usual order. Instead of chunking first and embedding each piece separately, it processes the entire document through a long-context embedding model to produce token-level representations, then applies chunk boundaries and averages embeddings within each span. Each chunk embedding includes context from the full document without adding extra text.

This method requires no additional training and works with any long-context embedding model that provides token-level outputs. Its limitation is architectural: it needs an embedding model with a context window large enough to hold the entire document, and the attention cost grows with document length rather than chunk count. For a corpus of short pages this is inexpensive. For 200-page filings it is costly, and the standard approach of splitting first allows parallel embedding across chunks.

Implementation: Recursive Splitting With Structure-Aware Separators

Recursive character splitting is the appropriate starting point for most corpora because of the separator hierarchy rather than the size parameter. The splitter tries paragraph breaks first, then line breaks, then spaces, so it finds the most natural boundary allowed by the size limit.

Two parameters in that block cause most production issues. The first is length_fn. LangChain’s default counts characters, so setting chunk_size=512 results in far fewer tokens on dense technical text than benchmarks recommend. Precision versus recall, embedding model choice, and chunk size form a connected system, which we analyze in our guide to RAG retrieval and vector database trade-offs.

The second is the separator list. For code, adding "\n\nclass " and "\n\ndef " before the paragraph break keeps functions intact. For markdown, a header splitter preserves section titles as chunk metadata, which improves accuracy by several points over fixed splits when document context is missing. For medical or legal documents with explicit section markers, placing those markers at the top of the separator hierarchy keeps whole sections together.

Building a Chunking Evaluation Set That Catches Real Failures

Chunking decisions cannot be validated with recall@k alone. The evaluation set must test specific ways segmentation breaks retrieval, and the most cost-effective method is to generate queries paired with the exact source spans that answer them.

Boundary-spanning queries. Write queries whose answers require two adjacent chunks, such as a question about a policy and its defining statement. Score whether both spans were retrieved, not just one relevant chunk.
Equal context budgets. The FloTorch benchmark gave every strategy the same context budget in the prompt regardless of chunk size. Without this control, large-chunk strategies win by adding more tokens rather than better ones.
End-to-end scoring over retrieval scoring. Answer accuracy with a judge model detects the fragment problem that recall misses. Retrieval-only metrics favor semantic chunking even when it underperforms.
Log the chunk that produced each answer. Citation metadata turns a failed answer into a chunking bug report. Without it, you cannot tell whether the retriever missed the document or the chunker split it incorrectly.
Version your chunker with the index. Changes to separators or size silently invalidate previous evaluation results, and the embedding distribution shifts accordingly.

Chunking strategy should be part of the same review process as schema changes. Published benchmarks provide useful priors, not definitive answers: recursive splitting in the 400-512 token range with 10-20% overlap is the validated default; page-level boundaries outperform it on paginated financial documents; paragraph grouping outperforms both on heterogeneous multi-domain corpora; and semantic or LLM-based chunking requires a size floor and a clear justification before its ingestion cost is worthwhile. For how these decisions interact with embedding quality and reranking, see our guide to enterprise RAG architecture and costs.

Key Takeaways

Chunking configurations affect retrieval quality as much as embedding model choice, according to the Vectara study published at NAACL 2025 testing 25 configurations against 48 embedding models.
Recursive character splitting at 400-512 tokens with 10-20% overlap remains the benchmark-validated default: 69% end-to-end accuracy in the FloTorch 2026 test, and 88-89% token-level recall in Chroma’s evaluation.
Semantic chunking produced 43-token average fragments in FloTorch, achieving 91.9% retrieval recall but only 54% end-to-end accuracy. Set a minimum chunk size between 200 and 400 tokens if you use it.
Document-level context prepended to every chunk outperformed LLM-generated per-chunk summaries in Snowflake’s tests while requiring far less compute.
Recall@k cannot detect chunk-boundary loss. Evaluate end-to-end answer accuracy with a judge model and include queries that require two adjacent chunks.

More in-depth coverage from this blog on closely related topics:

Sources and References

Sources cited while researching and writing this article: