Document splitting is the first decision point in any RAG ingestion pipeline, and it determines what a retriever can ever surface. Developers must choose between fixed-window, recursive, semantic, section-aware, and late chunking, each of which manipulates the text in a distinct way. This article compares those approaches on operational behavior rather than theory, mapping method to document type. The goal is to give an ingestion engineer a defensible default and a clear set of trade-offs when the default fails. Recommendations are separated from established facts, and version-sensitive claims are flagged where they apply.
The materials give me solid coverage on each of the five families. I'll now draft the section, sticking to facts that multiple sources support and flagging the late-chunking context-window dependency as instructed.
Five families of document splitting dominate RAG ingestion pipelines. Each draws boundaries using a different signal and imposes different requirements at ingest time.
Fixed-size chunking slices the input into uniform character or token windows, optionally with overlap. Boundaries follow a character count or token count; no structural or semantic signal is consulted. This makes it the cheapest option to run, but it can split mid-sentence or mid-table, which costs retrieval precision on prose and tabular content.
Recursive chunking applies a hierarchical separator list. The splitter first attempts large separators such as double newlines (paragraphs) and \n (lines), then falls back to single newlines, sentences, words, and finally characters, until every resulting chunk fits inside a size budget. The signal is the configured separator hierarchy. Because it preserves paragraph and sentence boundaries wherever possible, recursive chunking is widely cited as the pragmatic default for general-purpose text and mixed corpora.
Semantic chunking computes an embedding for each sentence (or small group of sentences), then places boundaries where the cosine distance between adjacent sentence vectors spikes. The signal is embedding-derived similarity, so chunk sizes vary and a similarity threshold must be tuned. Ingestion is more expensive because every sentence is embedded up front, and reported gains are modest: one comparison cites up to roughly 9% recall improvement, while another places the lift at about +6.5 nDCG@10 on NFCorpus when late chunking is layered on top. Both figures are workload-dependent.
Section-aware (document-based) chunking uses document structure as the boundary signal. Markdown is split on headers and subheaders, HTML on tags such as <h1>, <p>, or <div>, and PDFs on headings, sections, or page-level markers. It preserves logical hierarchy and works particularly well on structured content where headings already align with topical shifts, but it fails when documents lack reliable structure (plain logs, OCR output, free-form prose).
Late chunking flips the default order: the full document is run through a long-context embedding model first, then token-level embeddings are averaged within spans to produce per-chunk vectors. Because each token attends to the entire input under the model's self-attention layers, pronoun references and cross-chunk dependencies survive the split. The requirements are strict: the embedder must expose token-level outputs and support mean pooling (BERT-style CLS-token models are not compatible), and the whole document must fit inside the model's context window — for example, 8,192 tokens on jina-embeddings-v3, roughly ten pages of text. These limits are version-sensitive: maximum context window and pooling support vary by provider and model version, so a configuration that works on one release may silently truncate on another.
Taken together, the five families trade off three axes: boundary signal (count, separator hierarchy, embedding distance, document structure, or pre-computed embeddings), ingest cost (from constant-time splitting to per-sentence embedding or full-document encoding), and tolerance for documents that lack clean structure or exceed model context limits. The next sections measure each family against representative document types using these axes.
Recursive chunking is a structured fallback algorithm. Instead of committing to a single split point, the splitter takes an ordered list of separators and applies them sequentially. It begins with the coarsest separator — typically a double newline (\n\n) representing paragraph breaks — and attempts to break the document into pieces at that level. Any resulting piece that still exceeds the configured size budget triggers a recursive call using the next finer separator in the list, down through line breaks (\n), sentence terminators (. ), word boundaries ( ), and finally individual characters ("") as the last resort. This cascading behavior is what gives the method its name: chunks are repeatedly subdivided until every segment fits within the size constraint.
Because the algorithm is entirely deterministic and operates on raw text characters, its behavior is governed almost exclusively by two parameters: the chunk-size limit and the order of the separator list. LangChain's RecursiveCharacterTextSplitter ships with ["\n\n", "\n", " ", ""] as its default list, which is optimized for English prose with conventional paragraph formatting (Pinecone, LangChain docs via IBM). For code, the separator list is typically reconfigured to delimiters like \n\nclass , \n\ndef , and \n so that function and class boundaries are respected before the splitter falls back to line or token breaks. For Markdown or HTML-derived text, many practitioners prepend heading markers (#, ##) to the front of the list, which lets structural anchors be tried before paragraph breaks.
Recursive chunking bounds chunks by count — character count by default, token count when the splitter is reconfigured with a tokenizer — not by topic. A chunk that fits in 512 characters is accepted regardless of whether it straddles a topic shift mid-paragraph or merges two unrelated points because the surrounding paragraphs were short. The default LangChain behavior counts characters, which means a chunk_size=512 setting yields roughly 128 tokens, often far smaller than the embedding model's context window. Practitioners who want token-aligned chunks pass a tokenizer such as tiktoken to the splitter. A typical reference configuration sets chunk_size=512, chunk_overlap=50, separators=["\n\n", "\n", ". ", " ", ""] (Qdrant course material, dev.to overview).
The strengths of recursive chunking are its predictability and its strong recall on contiguous prose: because it always preserves the largest possible natural boundary, sentences are rarely cut mid-thought and paragraphs stay together when they fit (StackAI, Meilisearch). The weaknesses are equally concrete: the method has no awareness of document headings unless they are explicitly added to the separator list, topic shifts inside a single long paragraph go undetected, and the output depends heavily on the consistency of the source formatting. Messy extraction — PDFs without paragraph structure, scraped HTML where line breaks have been flattened, or documents mixing prose with tables — tends to defeat the higher-level separators and force the splitter into character-level fallback, which produces the same kind of arbitrary cuts that recursive chunking was meant to avoid.
Recursive chunking is the defensible default for mixed corpora where document structure cannot be guaranteed, and it is the standard baseline against which semantic and late chunking are usually compared.
A semantic chunker decides split points by measuring the distance between meaning of neighboring text, rather than by counting tokens or honoring structural markers. The typical pipeline has four stages:
buffer_size) using a tokenizer such as NLTK, spaCy, or a regex-based splitter.all-MiniLM-L6-v2 or all-mpnet-base-v2.A sharp drop in similarity between two consecutive sentences is treated as evidence of a topic shift, and the chunk is cut at that transition.
Three threshold strategies are commonly seen in implementations such as LangChain's SemanticChunker and LlamaIndex's SemanticSplitterNodeParser:
In practice, a similarity_threshold of 0.5–0.6 is suggested as a starting range for narrative text and 0.7–0.8 for technical content with frequent topic shifts, though settings as low as 0.35 appear in published configurations for legal contracts and as high as 0.55 for densely multi-topic knowledge bases.
Because every sentence is embedded, the index-time cost scales with sentence count rather than document count: a 10,000-word article may produce hundreds of embedding calls. Short sentences are a known failure mode — when units are very brief, their embeddings are noisy and adjacent distances fluctuate, producing spurious breaks. Mitigations include raising the minimum chunk size, grouping sentences into small windows before comparison, and applying a lower bound on the threshold.
Semantic chunking is most useful for long-form articles, transcripts, and reports where topical shifts are gradual rather than marked by headers. It is less effective on documents that already have reliable structural boundaries (Markdown, legal clauses) or on very short texts where each sentence is nearly the entire chunk.
A section-aware semantic chunker executes a strict two-stage decision before any text is indexed.
This sequencing matters because the two signals answer different questions. The parser answers "where did the author mark a topic change?"; the embedding distance answers "inside this topic, where does the meaning actually shift?" Letting one stage veto the other removes the most common failure modes described in the RAG chunking literature (LlamaIndex glossary, DataCamp overview).
Contracts, policy pages, regulatory filings, and technical specifications share three properties that break the pure-strategy chunkers discussed earlier:
A structure-first pass makes those headings first-class metadata on every chunk, which downstream filters and citation UIs can use. A distance refinement pass then re-splits the body of long sections where the prose itself turns a corner — without ever crossing a header that the parser already marked.
The trade-off is operational: the parser must be tuned per format (Markdown vs. PDF outline vs. DOCX styles vs. scraped HTML), and a poor parser silently degrades the chunker back toward recursive behavior. Teams that cannot guarantee parser fidelity on their source corpus will get worse results from this strategy than from a simple recursive default.
Late chunking reverses the usual split-then-embed pipeline. Instead of segmenting a document first and encoding each segment in isolation, the full document is passed through a long-context embedding model in a single forward pass. The model produces one vector per token, with each token's representation shaped by self-attention over the entire document. Chunk vectors are then formed by pooling (typically mean-pooling) the token embeddings that fall within each desired chunk span.
In practice, the procedure looks like this:
The Jina AI embedding API exposes this as a late_chunking=True parameter; the langchain-jina package wraps the same behavior, and Jina's open-source repository (jina-ai/late-chunking) allows local execution with any HuggingFace model that supports mean pooling.
Late chunking requires a model that accepts the full document in a single pass. This is a version-sensitive requirement: maximum context windows differ across model releases and across providers. Older embedding models with 512-token limits are unsuitable; practical use typically requires models with at least 8K tokens of context, and gains scale with longer windows. If the document exceeds the model's limit, it must be truncated or split upstream, which reintroduces the very context loss late chunking is meant to prevent.
Late chunking is a strong default for legal contracts, research papers, technical manuals, and policies where definitions introduced early are referenced later. It is a poor fit for short FAQ entries, chat-style snippets, or any corpus where documents fit comfortably within a single small chunk.
A defensible default saves time, but every default fails somewhere. The table below maps each splitting strategy to a document archetype, names the concrete risk it mitigates, and lists the failure mode it still leaves open.
| Strategy | Best-fit document | Risk it mitigates | Failure mode it leaves open |
|---|---|---|---|
| Fixed-size (token/character) | Uniform records: CSV rows, JSON logs, short emails, simple FAQs | Bounds chunk length against model context limits | Splits mid-sentence, mid-table, mid-record; loses all structure |
| Recursive (paragraph → sentence → word) | Clean Markdown, blog posts, product guides, short reports, general prose | Preserves natural boundaries, keeping each chunk readable | Cannot detect topic shifts; assumes reliable paragraph breaks; variable chunk sizes |
| Semantic (embedding-distance splits) | Long unstructured narrative without reliable headers: novels, whitepapers, transcripts, mixed-topic prose | Places boundaries at meaning shifts rather than character counts | Threshold and window tuning are non-trivial; indexing cost scales with sentence count; a minimum chunk floor is still recommended (dev.to, surrealdb.com) |
| Section-aware semantic (parse outline, then split long sections semantically) | Structured long documents: PDFs, specs, contracts, knowledge bases, research papers | Combines outline fidelity with topical coherence inside oversized sections | Depends on parser quality; failed parsing collapses to semantic or recursive behavior; tables and figures may still split awkwardly |
| Late chunking (Jina AI, 2024) | Cases where context bleed between sentences is the dominant failure mode and a long-context embedder is already in the budget | Each chunk's vector carries whole-document context, improving retrieval when the answer depends on cross-sentence meaning (firecrawl.dev) | Higher embedding cost and latency at index time; quality hinges on the underlying long-context model and on chunk boundaries chosen by a separate method |
A practical way to read the table: pick the leftmost row whose document archetype matches the corpus, then validate against the rightmost column. If the residual failure mode is unacceptable, move one row to the right. A common decision tree for this is to start recursive as the default, escalate to semantic on multi-topic prose, and add late chunking only after measuring context-bleed regressions with a long-context embedder already in the stack (dev.to, weaviate.io).
For scanned or paginated PDFs, page-level chunking remains a strong alternative after parsing, since NVIDIA's 2024 benchmarks reported 0.648 retrieval accuracy with the lowest variance for that approach (firecrawl.dev). Use it when the document layout is the primary signal of meaning, not the prose flow. For unknown or mixed corpora, treat fixed-size chunking as a baseline only; it is fast and deterministic but discards semantics (ofox.ai).
Two version-sensitive caveats apply. Late chunking was introduced by Jina AI in 2024, so claim portability to embedders that do not expose per-token pooling (firecrawl.dev). And the reported "up to 9% recall improvement" for semantic chunking is an upper-bound figure from a specific embedding and corpus; reproduce the measurement on the target corpus before adopting it as a hard benchmark.
Each chunking strategy exposes a small set of parameters that govern how text is partitioned. The names and defaults differ across LangChain, LlamaIndex, and other implementations, so the values below should be treated as reference points rather than authoritative defaults. Always confirm against the library version in use before shipping a configuration.
Recursive chunking is configured primarily through:
Semantic chunking exposes parameters tied to the embedding and grouping logic:
Section-aware chunking is driven by:
h2 or h3).Late chunking is parameterized by:
Across all strategies, overlap functions as a recall-versus-redundancy dial. Increasing overlap improves the chance that a query-relevant passage crosses a chunk boundary, but it also inflates index size and raises the risk that the same sentence appears in multiple chunks and dilutes retrieval scores. There is no universally correct value; common practice places overlap between 10 and 20 percent of the chunk size, but the optimal setting depends on document density and query characteristics.
Metadata storage is required regardless of strategy. At minimum, each chunk should carry:
This metadata supports citation rendering, supports downstream evaluation of chunk quality, and makes it possible to reconstruct context that a retriever did not surface. Without it, debugging retrieval failures becomes guesswork.
Because parameter names, defaults, and even the units of measurement vary between libraries, version-sensitive claims should always be checked against the documentation of the specific chunker in use. A configuration that works with one library's recursive splitter may behave differently under another library's equivalent.