Production RAG, Part B: Retrieval, Filtering, and Reranking

Part A ended on a warning: nothing in RAG fails loudly. Retrieval is where that warning matures into something you can actually design against, because retrieval is the stage where correctness, security, and latency all compete for the same budget, and a system that optimizes any one of them in isolation quietly breaks the other two. A retrieval path tuned purely for relevance will leak across permission boundaries the moment someone asks a paraphrased question that a naive filter doesn't catch. A retrieval path tuned purely for security will over-filter and starve the model of evidence it needed. A retrieval path tuned purely for latency will retrieve too little, too shallow, too fast.

The way out of that three-way tension isn't a smarter single query — it's accepting that retrieval is a staged ranking problem, not a search call, and giving each stage exactly one job.

Retrieval as a staged ranking problem

Walk the chain and notice what each stage is actually responsible for — and, just as importantly, what it is explicitly not responsible for:

1. Rewrite / route. Resolve ambiguity and route to the right index or sub-corpus before anything else happens. A question like "what's the policy" means something completely different in an HR corpus than an infosec corpus, and resolving that ambiguity here — rather than hoping retrieval sorts it out — is far cheaper than fixing it downstream.
2. ACL and metadata filter. This runs before retrieval touches the candidate pool, not as a post-filter on results. That ordering isn't a style preference — it's the difference between "the system never considered documents this user can't see" and "the system considered them and we're trusting a filter to catch every one," which is a materially worse security posture even if the end result looks identical on a good day.
3. Hybrid retrieval. Combine semantic (embedding) search with lexical (BM25 or equivalent) search, because they fail on different things. Semantic search finds paraphrases and conceptual matches; it's mediocre at exact identifiers — part numbers, contract clause references, ticket IDs — because embeddings compress exactness into similarity, which is precisely the property you don't want for a SKU lookup. Lexical search is the inverse. You need both, fused, not one masquerading as sufficient for the other.
4. Reranker. Retrieve broadly, rerank narrowly — bring back a wider candidate set from the cheap retrieval stage, then apply a more expensive, more accurate reranking model to a smaller set where the cost is affordable. This is a latency/quality trade you make explicitly, with a number attached (how wide, how narrow), not an implicit consequence of whatever the vector index's default top-k happens to be.
5. Deduplicate and expand. Collapse near-identical chunks — five paragraphs from the same document saying roughly the same thing crowd out genuine diversity of evidence — and expand surviving hits to their parent context if you built a parent-child index in Part A. This is where that earlier chunking decision pays off directly.
6. Context packing. Assemble the final evidence set within the model's context budget, ordered and labeled in a way that makes the model's job easier, not just technically possible. More on this below — it deserves its own discipline, not an afterthought.
7. LLM with citations. Generation happens last, grounded in evidence that's already been filtered, fused, reranked, deduplicated, and packed — which means by the time the model sees anything, most of the work that determines whether the answer is trustworthy has already happened.

Every stage should expose its own scores and timing, logged independently, not folded into one opaque "retrieval took 400ms" number. When an answer is wrong or slow, you need to know whether the rewrite stage misrouted the query, the filter over-restricted the candidate pool, the reranker demoted the right chunk, or context packing dropped it during truncation — because the fix is completely different in each case, and without per-stage visibility you're debugging a five-stage pipeline by guessing.

Metadata filters are essential — and worth handling with care

Filters are what turns a general-purpose search into something that respects the actual shape of your organization: tenant boundaries, geography, product line, validity dates (don't surface an expired policy as current), document type, and classification level. Used well, they improve precision by ruling out entire categories of irrelevant or inapplicable evidence before ranking even has to consider them.

But there's a real distinction hiding inside "filters," and collapsing it is where teams get into trouble:

• Security filters — tenant, classification, access control — are mandatory, non-negotiable, and must run before generation, every time, with no code path that skips them for convenience or performance. These aren't relevance signals; they're the boundary of what the system is allowed to consider at all. Treat any latency optimization that touches this stage with more suspicion than any other part of the pipeline, because the failure mode isn't a slow answer, it's a wrong disclosure.
• Optional filters — inferred from vague or ambiguous language in the query — are a precision/recall trade you're making on a guess, and guesses can hurt as often as they help. If a user's phrasing loosely suggests "recent" or "the new version," an inferred date filter might sharpen results — or it might silently exclude the one document that actually answers the question because your inference was slightly wrong. Log every inferred filter and evaluate its effect on retrieval quality the same way you'd evaluate a ranking change, because an invisible filter that's wrong looks identical to a relevant document that simply doesn't exist.

Use hybrid retrieval specifically to cover the case exact-identifier lookups and semantic paraphrases fail each other on — a support engineer searching for an exact error code needs lexical precision; a new hire asking "how do we usually handle this kind of situation" needs semantic breadth. One retrieval mode will silently underserve one of those two users regardless of how well-tuned it is.

Retrieve broadly and rerank narrowly whenever your latency budget allows it — this is one of the more reliable quality wins in the whole pipeline, because the cheap retrieval stage's job is recall (don't miss the right chunk) and the expensive reranking stage's job is precision (put the right chunk first), and conflating those two jobs into one stage under-serves both.

Diversify the final result set deliberately. Five near-identical chunks from the same document is what naive top-k similarity search gives you by default, because near-duplicate content is, definitionally, highly similar. Left uncorrected, this quietly reduces your effective evidence to a single source repeated five times while looking, superficially, like five pieces of corroborating evidence.

Pack evidence, not noise

Context packing is the stage most pipelines treat as "concatenate the top results and hope," and it's worth treating as its own discipline instead, because a model's ability to use evidence well depends heavily on how that evidence is presented, not just whether it's technically present in the context window:

• Order by usefulness, not by whatever order retrieval happened to return. The most directly relevant evidence belongs where the model is most likely to weight it heavily, not buried mid-list because that's where the reranker's raw score happened to place it after ties were broken arbitrarily.
• Keep source boundaries visible. The model — and any human reviewing the answer later — needs to be able to tell where one source ends and the next begins. Evidence from three different documents blended into one undifferentiated block of text is evidence you can no longer attribute, which defeats the accountability that was the whole point of doing RAG in the first place.
• Attach stable citation identifiers that survive re-indexing, so a citation in an answer generated today still resolves correctly to the same source next month, not to whatever chunk happens to occupy that index slot after the next re-embedding migration from Part A.
• When evidence conflicts, prefer authoritative and current sources — but surface the disagreement rather than silently picking a winner and hiding the conflict. A policy that changed last quarter and a policy that changed yesterday will often both still be in your corpus; the model should be told which one is current and shown that the older one exists, not left to average between two documents that actually disagree because a human made a decision the retrieval system doesn't know about.
• If retrieval confidence is low, the correct behavior is a clarifying question or an explicit abstention — not padding the prompt with marginal chunks to make the context window look fuller. This is the discipline most teams skip under pressure to "always give an answer," and it's precisely backwards: a system that can say "I don't have enough to answer that confidently" is more trustworthy than one that always produces fluent, unattributed text regardless of what it actually retrieved. Padding low-confidence retrieval with marginal evidence doesn't make the model more capable — it gives the model more material to construct a confident-sounding wrong answer from.

That last point is the thread connecting both halves of this series: a RAG system's real product isn't fluent answers, it's trustworthy ones — attributable to real evidence, scoped to what the asker is actually permitted to see, and honest about the difference between "I found this" and "I'm inferring this because you asked and I had to say something." Everything in Part A and Part B is in service of that one property, and it's the property that's hardest to retrofit once a system's already shipped and users have learned, correctly or not, how much to trust it.