Chunk size is one of the most consequential configuration choices in any retrieval-augmented generation system, yet teams routinely pick it from blog posts rather than measurement. The LlamaIndex project has published a reproducible evaluation pipeline that turns chunk sizing into a benchmarked experiment rather than an intuition exercise. This article walks through that pipeline end to end, including the three metrics it computes and the LLM judges it relies on. The goal is to give engineers a concrete procedure for choosing and defending a chunk-size decision before shipping it to production.
Chunk-size rules of thumb, such as 256–512 tokens for general use, 64–128 for factoid queries, or 512–1024 for multi-step reasoning, are useful as a starting grid, but they conceal at least three assumption mismatches that make them unreliable as production defaults.
First, the same token budget behaves differently across corpora. Prose density, entity concentration, and rhetorical structure vary by document type, so a 512-token window in a legal filing is not equivalent to a 512-token window in a product FAQ. Per-domain recipes reflect this: FAQ and support content often sits in the 200–400 token range, technical documentation in 400–800, and legal or regulatory text in 800–1,200 tokens Stack AI chunking guide. A single number across these genres averages over real differences.
Second, chunk size is entangled with the embedding model. Embedders distribute semantic mass across their context windows differently, and many impose hard input limits; for example, several classic sentence-transformer models cap at 512 tokens, while newer encoders accept far more Qdrant chunking course. Push chunks past the model's effective length and information past the limit is silently dropped; pull them well below it and you underuse the representation the model was trained to produce. The rule-of-thumb numbers were not calibrated against every embedder a team might adopt.
Third, chunk size only matters through its interaction with similarity_top_k and chunk_overlap. Smaller chunks raise the risk that the required evidence never lands in the top-k retrieved nodes, especially when similarity_top_k is restrictive LlamaIndex chunk size blog. Overlap is similarly load-bearing: industry practice gravitates to 10–20% of chunk size to prevent mid-sentence splits Firecrawl chunking guide, but a recent SPLADE-based systematic study found that overlap added no measurable retrieval benefit on certain benchmarks while raising indexing cost Firecrawl chunking guide. Rules of thumb rarely pin these knobs down together.
The LlamaIndex Response Evaluation pipeline was built to escape these hidden couplings. By holding the embedder, the prompt, the evaluation questions, and the overlap constant, and sweeping a discrete set of chunk sizes, it surfaces the empirical trade-off curve that the heuristics try to approximate, turning an intuition exercise into a benchmarked experiment.
The reference pipeline wires five LlamaIndex primitives into a single experimental loop, with each primitive owning one job so that chunk length remains the only independent variable under test:
SimpleDirectoryReader: ingests files from a local directory into LlamaIndex Document objects. The official notebook uses the Uber 2021 10-K SEC filing PDF (uber_2021.pdf) as a representative long-form corpus (LlamaIndex blog).TokenTextSplitter: chops each document into chunks of a fixed token count. Two sweep ranges are commonly combined — the arxiv chunk-size study covers 64, 128, 256, 512, and 1024 tokens without overlap (arXiv:2505.21700), while the LlamaIndex blog extends the sweep with powers of two from 128 to 2048 tokens (LlamaIndex blog).DatasetGenerator: synthesizes evaluation questions directly from the document nodes, removing the hand-labeling bottleneck that usually blocks RAG benchmarking (LlamaIndex blog).ServiceContext: binds a single GPT-4 model to both the query engine and the evaluators, so the judge is never more capable than the generator it is scoring.VectorStoreIndex: rebuilt from scratch inside the loop for every candidate size, ensuring each measurement sees a freshly built retrieval structure rather than a mutated one.For each candidate chunk_size, the loop performs three deterministic steps:
VectorStoreIndex over the token-split nodes.time.time() around query_engine.query(question)).passing flag, then average those flags into the metric scores (LlamaIndex blog).A simple for loop is used instead of BatchEvalRunner on purpose: the experiment needs per-question latency, and only a sequential loop exposes that signal cleanly (LlamaIndex blog).
The TokenTextSplitter is chosen precisely because it ignores sentence and paragraph boundaries. Any semantic-aware splitter would entangle two variables — chunk length and boundary quality — making it impossible to attribute a metric shift to size alone. By holding the splitter family constant and varying only the token count, downstream measurements reflect chunk length rather than the splitter's segmentation policy (arXiv:2505.21700). That discipline is what turns the pipeline from a demo into a benchmark.
Every chunk size in the sweep is scored on three metrics, each produced by aggregating per-question results and dividing by the number of evaluation questions. This keeps the three values on the same scale (0–1 for the quality metrics, seconds for latency) and makes them directly comparable across the chunk-size range.
Average Response Time is the mean wall-clock duration the query engine takes to answer a single question at a given chunk size. Because the experiment wraps each query_engine.query(question) call with time.time() before and after, this metric folds together every step the user would actually wait on: embedding lookup, retrieval, prompt assembly, and model generation. Per-question rather than aggregate timing matters here, since it is the only way to see how chunk size changes the latency budget at the request level rather than amortized across a batch.
Average Faithfulness is the proportion of responses whose FaithfulnessEvaluator.evaluate_response(...).passing returns True across the evaluation set. The faithfulness judge, instantiated as a FaithfulnessEvaluator(service_context=service_context_gpt4) and powered by GPT-4 with temperature=0, decides whether the generated response is grounded in the retrieved source nodes — in other words, whether every claim can be traced back to a retrieved chunk. A higher Average Faithfulness means fewer hallucinated answers.
Average Relevancy is the proportion of responses whose RelevancyEvaluator.evaluate_response(...).passing returns True. The relevancy judge, also a GPT-4-backed RelevancyEvaluator, is invoked with both the original query and the response, and decides whether the response plus its retrieved context actually address what was asked. Note that relevancy is independent of grounding: an answer can be relevant but unfaithful (it addresses the question but invents facts), or faithful but irrelevant (it quotes the source but does not answer).
The three metrics expose a characteristic trade-off that no single number captures (LlamaIndex, Evaluating the ideal chunk size for a RAG system):
Because no single metric dominates, a defensible chunk-size choice is the one that maximizes the quality metrics at a latency budget the team has already agreed to — not the configuration that wins on faithfulness, relevancy, or speed in isolation.
Both evaluators are built on top of a single ServiceContext that pins the judging model. The LlamaIndex reference pipeline creates a deterministic GPT-4 client and packages it as the evaluation service context:
gpt4 = OpenAI(temperature=0, model="gpt-4")
service_context_gpt4 = ServiceContext.from_defaults(llm=gpt4)
faithfulness_gpt4 = FaithfulnessEvaluator(service_context=service_context_gpt4)
relevancy_gpt4 = RelevancyEvaluator(service_context=service_context_gpt4)
Because the ServiceContext is shared, both judges always agree on which model is rendering the verdict, which is the first step toward a reproducible sweep. Pinning temperature=0 removes the largest source of run-to-run variance in the verdict, and pinning the model string prevents silent upgrades from changing the pass/fail distribution (LlamaIndex blog).
FaithfulnessEvaluator answers one grounding question: does the generated response agree with the source nodes that the retriever actually returned? It is the pipeline's hallucination detector, and it is invoked per question through evaluate_response:
faithfulness_result = faithfulness_gpt4.evaluate_response(
response=response_vector
).passing
The .passing boolean is the only value the pipeline records. Averaged across the evaluation set it becomes Average Faithfulness. In a chunk-size sweep, a high average faithfulness means the chosen chunk size is not forcing the generator to invent details that are not in the index.
RelevancyEvaluator answers a different question: do the response and the retrieved source nodes together address the original query? It catches a failure mode that faithfulness misses — correct-sounding answers to unrelated questions — and is therefore the metric that captures whether the chunk size is feeding the generator the right evidence at all. It is invoked with the query as well as the response:
relevancy_result = relevancy_gpt4.evaluate_response(
query=question, response=response_vector
).passing
The averaged .passing value is Average Relevancy. A chunk size that maximizes faithfulness while minimizing relevancy is a configuration where the model is being a faithful parrot of the wrong paragraphs — exactly the trade-off this metric is designed to surface.
Because both judges are LLMs, their outputs are noisy and version-sensitive. Pinning the model id, the temperature, and the prompt template is part of what makes the benchmark comparable across runs and across chunk sizes; any of those three drifting invalidates the comparison (LlamaIndex blog).
The sweep itself is mechanically simple — a for loop over candidate sizes — but its trustworthiness comes from one strict discipline: every iteration rebuilds the VectorStoreIndex from scratch at the candidate chunk size. Reusing a prebuilt index and merely adjusting a runtime parameter would silently mix old and new chunk boundaries. Rebuilding ensures the index reflects exactly what production would see, so any delta in the recorded numbers is attributable to chunk length rather than to leftover state from a prior iteration. The canonical loop from the LlamaIndex chunk-size evaluation post looks like this:
chunk_sizes = [128, 256, 512, 1024, 2048]
for chunk_size in chunk_sizes:
avg_response_time, avg_faithfulness, avg_relevancy = evaluate_response_time_and_accuracy(chunk_size, eval_questions)
print(f"Chunk size {chunk_size} - Average Response time: {avg_response_time:.2f}s, Average Faithfulness: {avg_faithfulness:.2f}, Average Relevancy: {avg_relevancy:.2f}")
Three variables must be held constant across every sweep iteration; otherwise the study stops being a single-variable experiment:
similarity_top_k — the number of retrieved nodes sent to the generator must stay fixed, because top-k and chunk size interact at retrieval time.Violating any of these turns the table of results into a multi-variable study whose numbers cannot be defended as a chunk-size decision.
Two defensible patterns appear in the literature. The LlamaIndex blog iterates over powers of two from 128 to 2048 tokens — [128, 256, 512, 1024, 2048] — which sweeps the granularity spectrum in five evenly spaced steps. The arXiv long-document chunk-size reference instead uses [64, 128, 256, 512, 1024] with TokenTextSplitter and no token overlap, deliberately isolating the length variable by removing chunk boundary ambiguity. Powers of two are convenient for plotting on a log axis; non-overlapping splits are convenient when the goal is a clean causal claim about length alone.
The blog deliberately uses a per-question for loop rather than BatchEvalRunner because the goal is to measure response time per chunk size. A simple loop wraps each query_engine.query(question) in time.time(), yielding a per-question latency that can be averaged cleanly. BatchEvalRunner amortizes dispatch and is markedly faster, but it obscures per-question timing and is therefore the wrong tool when latency is the metric. When only faithfulness and relevancy matter, switching to BatchEvalRunner is the standard speedup.
Each sweep iteration produces exactly three numbers — average response time, average faithfulness, average relevancy — and the printed table comparing chunk size against those three columns is the artifact the team reviews to choose a configuration. Faithfulness and relevancy are computed as the proportion of questions whose passing flag is True, averaged across the eval set, so higher values indicate fewer hallucinations and more directly answered queries.
Once the sweep finishes, the table of chunk size versus the three metrics is best read as a small trade-off curve rather than as a leaderboard. Each row represents a different point on a three-way frontier: chunk size, answer quality, and latency. Picking the "winner" means locating the configuration that balances them, not the row with the highest single score.
When chunk size is placed on the x-axis and the two quality metrics on the y-axis, three patterns tend to emerge:
similarity_top_k stops carrying extra signal. In the LlamaIndex sweep over the Uber 2021 10-K filing, relevancy climbed steadily and peaked at chunk size 1024; smaller sizes left useful context stranded outside the top-k window (LlamaIndex blog).similarity_top_k is large.The defensible choice is the smallest chunk size that sits within a few points of the relevancy peak and inside the team's latency budget. General RAG guidance commonly places that sweet spot at 256 or 512 tokens for many retrieval patterns, though the right number is always dataset-dependent (Stack AI, Medium). The LlamaIndex sweep itself identified 1024 as the peak for the 10-K corpus and similarity_top_k=2, which illustrates exactly why the sweep — not the heuristic — is what justifies the number.
Record the chosen chunk size alongside the three measured metrics so reviewers can see the trade-off the team accepted. Re-run the sweep whenever the corpus, the embedding model, similarity_top_k, or the generator changes; any of those shifts can move the curve, and stale numbers are the most common reason a chunk-size decision quietly stops being defensible.
A chunk-size benchmark is easy to run and easy to get wrong. Three failure modes appear often enough in practice that any reproducible pipeline has to defend against them explicitly.
The LlamaIndex pipeline relies on GPT-4 as the judging model for both the FaithfulnessEvaluator and the RelevancyEvaluator. Re-running the same sweep against a newer snapshot can shift faithfulness and relevancy by several points even with identical inputs. Pinning the exact model identifier and prompt template — and recording them in the results table — is therefore non-negotiable. Treat the judge as a calibrated instrument, not a black box.
When questions are generated from the same nodes being retrieved, they tend to mirror the chunks the retriever will surface. That inflates both faithfulness and relevancy because the questions are easier than the ones real users ask. The standard remedy is to supplement the synthetic set with a held-out, human-written question bank, then average across both. The two distributions expose whether a chunk size is winning on the test set by accident or by design.
Comparing chunk sizes across two different embedding models in the same sweep conflates embedding-model effects with chunk-size effects. The sweep must hold the embedding model, the splitter family, and similarity_top_k constant; only chunk_size should move. Anything else that changes between cells is a confound, even if it looks benign.
Each chunk size × question cell triggers one generator call and two judge calls. A five-size sweep over one hundred questions therefore lands at roughly fifteen hundred LLM invocations. The reference implementation rebuilds the VectorStoreIndex per chunk size, which dominates that cost (LlamaIndex blog). Planning the budget before defining the grid keeps the experiment tractable.
The same loop generalizes naturally:
TokenTextSplitter for SentenceSplitter to compare splitter families under identical measurement.similarity_top_k to separate retrieval depth from chunk granularity.Treating chunk size as one column in a wider empirical table — rather than a standalone number to copy — is what separates a defensible RAG configuration from an inherited one.