Production RAG, Part A: Ingestion, Chunking, and Embeddings

Almost every RAG post-mortem I've been part of traced back to this stage, not to the generation prompt. Teams spend weeks tuning the retrieval prompt and the reranker, and the actual defect was a chunker that split a table's header row from its data three months earlier, or an ACL that got dropped during a re-index and quietly started leaking one tenant's contracts into another tenant's answers. Generation is the part of RAG that's visible and easy to iterate on in a demo. Ingestion is the part that's invisible until it's wrong in production, in front of a customer, usually about something sensitive.

So treat this stage with the seriousness of a data pipeline that also happens to be a security boundary — because that's what it is.

What the model is actually missing

A foundation model encodes patterns and general knowledge up to its training cutoff. It has no access to your private policies, current support tickets, live contract terms, or — critically — any notion of who is allowed to see what. Retrieval-augmented generation supplies relevant evidence at request time, without retraining the model, by turning your proprietary corpus into something searchable and then handing the model exactly the slice it needs for a given question.

The framing worth internalizing: RAG doesn't make the model smarter, it makes the model informed and accountable. Informed, because it's reasoning over evidence instead of parametric memory. Accountable, because that evidence is attributable — you can point to the chunk, the source document, and the timestamp that grounded a given answer. If your pipeline can't produce that attribution cleanly, you've built a slightly-more-elaborate hallucination generator, not a retrieval system.

The ingestion and indexing pipeline, as a chain of trust decisions

Every stage in this chain is also a place where you can either preserve or silently lose the two things that make retrieval trustworthy: provenance and permission. Walk it in order:

• Extract. Different source types — wikis, PDFs, ticketing systems, Slack exports, database rows rendered as documents — need genuinely different extraction logic, not one PDF-shaped hammer applied to everything. A structure-aware extractor that knows a support ticket has a status, an assignee, and a resolution field is categorically better than one that flattens it to prose and hopes the model infers the structure back out.
• Normalize and classify. This is where you assign sensitivity labels, tenant ownership, and document type — and it's the stage most teams under-invest in because it doesn't produce a visible retrieval-quality improvement. It produces something more important: the metadata that a permission check depends on later. Classification errors here don't show up as bad answers; they show up as wrong answers delivered confidently to someone who shouldn't have gotten them at all.
• Chunk. Covered in depth below — this is where meaning either survives or gets fragmented.
• Embed. Turns each chunk into a vector. The choice of embedding model is coupled to everything downstream, discussed in the last section.
• Metadata and ACLs, threaded through the whole chain. Not a final tagging step — the access-control information needs to travel with the chunk from extraction through to the vector index, because a chunk that loses its ACL metadata partway through the pipeline becomes either unreachable (a false negative, annoying) or reachable by everyone (a false positive, a real incident).
• Vector index + object store. The vector index holds embeddings and enough metadata to filter and rerank; the object store holds the actual source content and larger context the index points back to. Keeping these separate — index for search, object store for retrieval of full content — is what makes parent-child retrieval and re-indexing without re-extraction possible.

Chunk for the question, not for the document

The chunking decision is really a bet about what kind of question you're optimizing for, and different strategies make different bets:

• Fixed token windows are simple, fast, and predictable — and they will cheerfully cut a sentence, a table row, or a legal clause in half if that's where the window boundary lands. Fine for homogeneous, low-structure prose where losing a clean boundary occasionally isn't costly. A bad default for anything where a mid-sentence split changes the meaning of the retrieved evidence.
• Recursive text splitting respects the document's own structure — headings, paragraphs, list boundaries — before falling back to a size limit. This alone fixes the majority of the naive "we sliced a table in half" failures, because it's splitting where the author already told you it was safe to split.
• Semantic chunking groups sentences by actual topical coherence rather than structural markers, which helps when a document's formatting doesn't reliably track its meaning — think stream-of-consciousness meeting notes or loosely structured wikis. It costs more to compute and is harder to reason about deterministically, so it's worth reaching for when structure-aware splitting demonstrably isn't enough, not as a default upgrade.
• Parent-child indexing decouples the unit you search over from the unit you return. You embed and search small, precise fragments — which keeps similarity search sharp — but retrieve the larger parent section or document around a hit, so the model gets enough surrounding context to actually answer the question instead of a decontextualized sentence. This is, in my experience, the single highest-leverage chunking decision for enterprise corpora with real document structure (contracts, policies, technical documentation), because it resolves the actual tension in chunking: small chunks retrieve precisely and answer poorly; large chunks answer well and retrieve imprecisely. Parent-child lets you stop choosing.

Tables, code, conversational threads, and scanned documents each need a structure-aware parser of their own — flattening a table to comma-separated prose and embedding it like a paragraph destroys the row/column relationships that made it a table in the first place, and a scanned document needs OCR with layout awareness before any of the above strategies even apply. There is no universal chunker; there's a dispatcher that routes each source type to the strategy suited to its structure.

A few operational disciplines that separate a chunking pipeline you can trust from one you're guessing about:

• Preserve full lineage metadata on every chunk: document, section, page, timestamp, source URI, tenant, classification, and access-control fields. This is what makes an answer attributable and what makes a permission check enforceable — treat any chunk missing this metadata as unindexable, not as indexable-with-a-warning.
• Choose overlap deliberately, not by default. Overlap exists to prevent boundary information loss — apply it where a genuine boundary problem exists, not as a blanket setting. Excessive overlap bloats your index, duplicates evidence in a way that skews retrieval toward over-represented chunks, and makes relevance scoring harder to reason about.
• Version the parser, the chunker, the embedding model, and the source checksum together, as one unit. A re-index isn't "run the pipeline again" — it's a new generation of your corpus, and you want to be able to say precisely which generation produced a given answer, roll back to a prior one, and run two generations side by side during a migration.

Choosing a platform: Bedrock and the alternatives

Amazon Bedrock Knowledge Bases gives you managed ingestion, embedding generation, vector-store integration, and AWS-native IAM-based security in one package — a strong default if you're already committed to AWS and want to minimize the surface area you operate yourself. Azure AI Search earns its place in Microsoft-centric estates, particularly where Entra ID-based access control needs to extend cleanly into the retrieval layer. Vertex AI Search aligns naturally with teams already standardized on GCP's data and analytics stack. Pinecone, Weaviate, and OpenSearch sit further along the control spectrum — more operational responsibility, but correspondingly more control over indexing strategy, hybrid search tuning, and multi-tenancy model.

The comparison that actually matters isn't a feature checklist, it's these specific axes, because they're the ones that are expensive to discover you got wrong after data is already indexed: parsing quality on your actual document types (test with your worst documents, not your cleanest), hybrid search support (pure vector similarity under-performs on exact terms — product codes, legal citations, ticket numbers — that a keyword or BM25 component catches reliably), metadata filtering expressiveness (can it enforce your actual permission model at query time, not just store the labels), multi-tenancy isolation model, regional availability matching your data-residency requirements, encryption at rest and in transit as your compliance regime actually defines it, evaluation hooks (can you measure retrieval quality in an automated, repeatable way, or only eyeball results), and total cost at your actual query volume, not the volume in the vendor's pricing example.

The embedding model and vector database are coupled, not independent choices

This is the constraint that catches teams by surprise months in: your embedding model determines vector dimensionality and the similarity behavior (cosine, dot product, Euclidean) your index needs to be configured for, and swapping embedding models isn't a drop-in replacement — it's a full re-embedding of your corpus, because vectors from two different models aren't comparable to each other even at the same dimensionality. A newer, better embedding model doesn't improve your existing index; it invalidates it.

Treat every embedding model change as a versioned migration, with the same rigor you'd apply to a schema migration on a production database: re-embed into a new index generation, run retrieval evaluation against a held-out query set to confirm the new generation actually performs better before cutting over, and keep the prior generation queryable until you're confident — because the failure mode of a bad embedding migration isn't a crash, it's silently worse retrieval that nobody notices until someone asks why the assistant stopped finding an answer it used to find reliably.

That's the note this part of the pipeline should end on: nothing here fails loudly. A dropped ACL, a bisected table, a stale embedding generation — these all fail as quietly wrong answers, which is the most expensive kind of bug to catch, because your system looks like it's working right up until it's caught giving someone information they shouldn't have, or missing information they needed. Part B picks up from here — retrieval, reranking, and grounding the generation step in what this pipeline built.