A documentation-ingestion pipeline can appear healthy while its index quietly becomes stale after redirects, sitemap changes, or content redesigns. Reliable retrieval therefore depends on more than chunking and vector storage: discovery, normalization, deduplication, change detection, retries, and observable upserts all need explicit stages. This article maps those responsibilities into an 11-step scrape-clean-embed-store pipeline and shows where offline indexing ends and online retrieval begins.
The first structural lesson from that Friday-night failure: I had been operating one system when there were really two. A mature RAG stack splits into two fundamentally separate flows, and conflating them is the first mistake most teams make — mine included.
The offline path is a batch process triggered by document change events, not by user requests. It does all the heavy lifting:
Because it runs behind the scenes, this flow can afford to be slow and thorough. A scheduled full re-index typically runs weekly, and that weekly run is the mechanism that absorbs embedding model upgrades and chunking strategy revisions — without it, half your index lives in one vector space and half in another.
The online path lives in a completely different world. It answers a live user, so every stage carries an explicit cost and latency budget:
That "same model" detail is easy to miss and expensive to break: query vectors only land near the right document vectors if the embedding step is identical on both sides of the split.
Failure tolerance is asymmetric, and that asymmetry drives every design decision. The offline pipeline has high failure tolerance: a failed fetch at 2 a.m. can simply be retried, and the workflow can reschedule itself. The online pipeline serves synchronous user requests — it needs fast fallbacks, not slow retries, because a degraded answer now beats a perfect answer thirty seconds later.
The dangerous part is that quality problems created at indexing time — bad chunking, stale documents, embedding model mismatch — propagate downstream invisibly. Nothing in the query path will warn you; retrieval will confidently return wrong chunks.
Modularity is also what makes incidents tractable. When a spike in bad answers appears, I can trace it back to parsing, indexing, or reranking, with logs pinpointing which stage regressed — assuming the stages are separate enough to produce distinct logs in the first place.
Which brings me to the honest summary of this whole split: without an evaluation harness, observability, and governance, what you have is a demo, not infrastructure. My Friday-night failure was exactly that — a demo wearing production clothing.
This is where the "boring plumbing" from the intro starts earning its reputation. Step 1 hands you four URL discovery sources, and choosing between them is the first real engineering decision the pipeline asks of you:
Sitemaps and feeds are the cheapest starting points, and the reason is simple: they hand you canonical URLs and change cadences directly, which spares you from burning a full browser session on every URL just to find out nothing changed. That silent Friday-night failure from my old pipeline is exactly what this step exists to prevent — a script fetching fixed URLs had no discovery mechanism at all, so when the vendor's redesign shipped, nothing in my system was even capable of noticing.
Crawls earn their place too, though. Crawling an entire site rather than individual pages gives the vector store fuller context, because internal links connect related content — a reference page embedded alongside its tutorial and changelog answers questions better than one embedded alone. Firecrawl's crawl mode follows links automatically up to a configurable depth, so you get that whole linked graph without writing your own crawler.
Three setups from the RAG tooling space show how little configuration you actually need:
crawlUrl with a 500-page limit and formats set to markdownmax_depth: 3, include_only_paths restricted to /docs/*, content_formats: markdown, asp enabled to bypass anti-scraping protection, rendering_delay: 2000 ms (setting it to 0 disables JavaScript rendering entirely), and a page_limit of 1000The include_only_paths detail matters more than it looks: scoping a crawl to /docs/* is the difference between a clean documentation index and accidentally ingesting marketing pages, blog posts, and job listings.
The Step 2 rule is refreshingly simple: plain HTTP for static pages, browser rendering only for JavaScript-heavy SPAs. Browser sessions are the most expensive thing in your pipeline — every one spins up a real browser — so reserving them for pages that genuinely need JavaScript keeps a 1,000-page crawl affordable.
Before reaching for that browser, though, I'd check three cheaper paths:
api.* calls — fetch the API response directly and skip rendering altogether.?output=1, AMP, or server-rendered routes that hand you the same content without any JavaScript.This is the only stage where the pipeline touches someone else's servers, so the rules belong here: respect robots.txt and the site's terms, prefer public documentation you actually have rights to use, minimize personal data for GDPR/CCPA compliance, and store provenance — URL and fetch time — on every document so the index stays auditable later.
The security note deserves equal attention: LangChain's RecursiveUrlLoader and SitemapLoader are web crawlers carrying SSRF risk. A malicious sitemap could otherwise force your server to fetch URLs from other domains — including internal ones. Two defenses apply: crawlers should never have network access to internal servers, and both loaders already default to same-domain-only loading via prevent_outside. That quiet default is doing real protective work.
Get discovery and fetching right, and everything downstream has raw material to work with — no cleaning pass, however clever, can recover a page you never fetched.
A page can fetch perfectly — right status code, valid URL, fresh from the sitemap — and still be mostly noise by byte count. Navigation, sidebar, footer, ads, cookie banners, related-content widgets: all of it ships inside the same HTML document as the actual article. If any of that survives to the next stage, your pipeline happily embeds a "Pricing" menu as if it were documentation. Steps 3 and 4 exist to burn that noise off before it reaches anything expensive.
The tooling here splits into four jobs, and I'd resist the temptation to hand-roll any of them:
One trap worth flagging: trafilatura exposes two functions that look interchangeable but aren't. Its extract function isolates the main body — the blog post itself — while discarding navigation and related-content links. Its html2txt function, despite sounding like exactly what you want, returns everything, nav links included. Picking the wrong one quietly fills your index with "Home · Docs · Login."
Once the body is isolated, the preprocessing rules come straight from large-scale dataset preparation, and they translate well to docs pipelines:
The output format here matters more than it first appears. The goal is clean text or markdown with the heading structure preserved, and markdown beats raw HTML for a concrete reason: headers, lists, and code blocks survive as natural chunk boundaries. Split raw HTML by characters instead, and you break <div> tags mid-element, fragment tables around form elements, and embed navigation text straight into the middle of chunks. Clean input is the difference between useful and noisy embeddings — same model, same chunker, wildly different answers downstream.
My normalization checklist for every page:
One caution that saves real debugging time: over-aggressive cleaning is its own failure mode. Strip too much and you lose meaningful formatting or context — the heading hierarchy that made the page scannable, the table structure that gave a parameter list meaning. Always verify readability after cleaning. Print a sample page and actually read it before it moves on.
Finally, put a quality gate at the end of these two steps. Simple heuristics — minimum content length, at least one heading, no stray markup — catch garbage early, while it's still cheap to discard. Skip the gate, and the problem stays invisible until you've burned your embedding budget generating vectors from noise, only surfacing when a user asks about an endpoint that was renamed two releases ago. That's the exact failure mode this whole pipeline exists to prevent.
By this point in the pipeline, I'm holding clean, structured text — and the temptation is to fire it straight at the embedding model. Step 5 is the discipline that stops me: every document gets stamped before it gets split. No exceptions.
The document-level fields I attach at ingestion:
Once splitting happens, each chunk that reaches Qdrant carries a full payload. A representative record:
{
"document_id": "d7f3a1c2",
"document_title": "Authentication API",
"section_title": "Token Refresh",
"chunk_index": 4,
"chunk_count": 12,
"url": "https://docs.vendor.com/api/auth",
"tags": ["api", "auth"],
"source_type": "html",
"created_at": "2026-09-02T01:14:00Z",
"content": "The refresh endpoint accepts...",
"word_count": 214,
"char_count": 1487
}
Both of these failures are silent, which is exactly why they survive code review.
tags, url, or source_type, those payload fields must be explicitly indexed. Skip it and every filtered search scans payloads linearly — results stay correct but quietly degrade as the collection grows.null, "N/A", or "". Omit the key entirely. My writer checks every field and drops empty ones before the upsert call.That payload is what the online retrieval phase actually runs on:
url plus section_title lets the UI cite chapter and verse — the difference between a user trusting the output and squinting at it.After spending weeks agonizing over chunking strategies, here's the irony: for a first pipeline, the defaults held up. The strategy choice deserves its own deep dive, but the pragmatic baseline is structure-aware splitting:
A reliable rule is to attach document version hashes and source URIs at ingestion time, before chunks enter the embedding queue. Pipelines that defer deduplication to a post-processing pass can lose the association between chunks and source documents when asynchronous workers process chunks out of order. The index then accumulates near-duplicate or stale content, degrading retrieval precision without necessarily raising an error.
Steps 7 and 8 close out the offline indexing phase, and the contract with the online retrieval side comes down to one rule: Step 7 must convert each chunk into a vector using the same embedding model that will embed queries at retrieval time. A model mismatch between indexing and queries is one of the silent quality killers in RAG — nothing crashes, nothing logs an error, you just get steadily mediocre retrieval while you waste weeks re-tuning chunk sizes that were never the problem. It's the first thing I check when retrieval quality looks worse than it should be, because verifying it costs nothing.
Every provider caps what you can send per call, and the caps differ enough to matter:
truncate parameter as the safety net for oversized inputs.| Corpus | Tokens | OpenAI small | Cohere v4 | Voyage Large |
|---|---|---|---|---|
| 1M document chunks | ~500M | $10 | $50 | $90 |
| 100k support articles | 150M | $3 | $15 | $27 |
| 10k product descriptions | 5M | $0.10 | $0.50 | $0.90 |
OpenAI lists $0.13 per million tokens on its large model, and the small-model column above works out to roughly two cents per million. My takeaway: raw embedding cost is almost never what sinks a docs pipeline — the plumbing around it is. So the real optimization levers target overhead and reliability, not pennies:
When teams complain about slow vector search, precision and dimension are the first levers I'd pull — the accuracy trade is small and easy to measure.
The rule here is short: use upsert, never insert, so a pipeline re-run can never duplicate data. Batch at roughly 100 vectors per request, with each record carrying a unique string id, its values, and its metadata.
This registry is the missing control in many first-generation pipelines. Maintain a Postgres table, doc_chunk_registry, with:
doc_id and chunk_vector_id (together the primary key)content_hashversionindexed_atstatus: active, deleted, or supersededThe reason it must exist: a document is not one vector, it's many, and updates don't map cleanly onto them. Updating a 15-chunk document means:
Notice that step 1 is impossible without the registry. You cannot delete vectors you cannot name, and that is exactly how an index ends up confidently serving pages that no longer exist. The deleted/superseded statuses turn a silent failure into a queryable fact.
None of this is glamorous engineering. But an idempotent upsert plus a registry table is the difference between the Friday-night death I opened this article with and a pipeline you can genuinely leave running unattended.
Of all eleven steps, this is the easiest one to omit and one of the most damaging. Web content does not announce its changes; it drifts. A vendor ships a redesign, URLs quietly vanish, and your index keeps confidently answering from pages that no longer exist. Stale output isn't a graceful failure mode either: one wrong policy answer, and the user's trust in the whole system is gone in a single shot. Step 11 turns ingestion from a one-off script into a loop that keeps the index honest.
The change-detection cycle has four moves:
contentHash — the SHA-256 of the page's markdown — alongside each vector's metadata.That fourth move is the one most teams skip, and it's precisely where my index went wrong — not bad chunking, not bad embeddings, just orphaned vectors nobody retired.
Hashing sits first because it's the cheapest gate. When I look at real-world "updates," most turn out to be metadata-only noise — a title tweak here, a timestamp bump there — with the text untouched. SHA-256 gating cuts re-embedding volume by 99.5% on static corpora, which matters once you price the alternative:
The hash diff converts "rebuild the world every Monday" into "touch the fifty pages that actually changed."
I'd run ingestion as two lanes feeding the same vector store:
This is what continuous indexing actually looks like: the index gets updated with no downtime and no full rebuilds, ever.
utm_source link doesn't land twice.For easy wins, metadata-based dedup — same title and creation date arriving from different sources — catches a surprising share. At scale, MinHash locality-sensitive hashing handles the rest: tokenize into n-grams after stop-word removal, stemming, and lemmatization, then fit the MinHash model to cluster content-based duplicates.
Here's the scenario that convinced me version tracking isn't optional. An updated document, ingested without versioning, enters the knowledge base as fresh chunks sitting right next to its predecessors. A Q2 policy coexists with its Q1 version. Retrieval returns both, the LLM receives contradictory inputs with no signal of which is current, and an answer citing the superseded version passes without a single parse error. Nothing breaks — the system is just quietly wrong.
The fixes all run at ingestion: document version hashes so predecessors are identifiable, stale chunk retirement so old versions actually leave the store, and a chunk position index — page number, section header, paragraph offset — so retrieval can reconstruct citation context around any chunk it serves.
That closes the loop: crawl the same start URLs, hash-compare, upsert the deltas, retire the dead. Run it on a schedule, and the pipeline stops being a script you hope survived the weekend — it becomes a system that tells you the truth every Monday morning.
The gap between "the demo works" and "it serves production traffic" is orchestration — not retrieval logic, not prompt engineering, not model choice. Without it, your pipeline is a collection of scripts glued together with cron, and my silent Friday-night failure is exactly what that looks like: nothing alerted me because nothing was watching, and there was no execution history to inspect. A real orchestrator takes over five responsibilities that cron never covers:
That's the difference between a pipeline that runs in production and one that breaks in production and waits for an engineer to restart it manually at 3 a.m. Beyond those five, production ingestion adds three more requirements: processing state tracking so a failed run resumes without re-processing everything, duplicate document detection so re-runs don't inflate your index, and embedding API quota management with backoff strategies so you don't burn through your rate limit at the worst possible moment.
What shaped my design more than any tool choice was the asymmetry between the two halves of the system. They cannot share one retry policy:
The pattern holds no matter which tool you pick: teams that skip orchestration when building their pipeline typically end up rebuilding it within a year. That's why I treat the orchestrator as step zero — it isn't one of the 11 steps, it's the thing that runs all of them.