Skip to content

Research / RAG Systems

RAG Systems — retrieval architectures for production knowledge work.

Going beyond top-k cosine: hybrid retrieval, reranking, evaluation, and the operational tradeoffs that determine whether a knowledge system stays useful. This page documents what is currently understood, where that understanding breaks down, and what remains genuinely unresolved — as of this writing.

Code RES/RAGArea RetrievalComplexity Medium-HighLayer Knowledge
§ 01The Problem

Top-k cosine retrieval fails on the queries that matter most

The default RAG implementation — embed the corpus, embed the query, return the top-k nearest neighbors by cosine similarity — is not wrong in general. It is wrong specifically on the queries that carry the highest stakes in production knowledge systems: searches for specific documents, named entities, technical identifiers, and precise definitions where the exact words in the document are the signal, not their semantic neighborhood. Dense embedding retrieval systematically underperforms on these queries because embedding models are trained to represent semantic meaning, not lexical identity. The embedding for "RLHF" and the embedding for "reinforcement learning from human feedback" are close. The embedding for "RLHF" and a document that contains exactly "RLHF" as a technical identifier — with no semantic elaboration — may not be close enough to surface in top-k.

The failure is silent by design. The system returns top-k results regardless of whether any of them are correct. There is no error, no exception, no log entry indicating that the relevant document was in the corpus and was not retrieved. The generation model receives a context window that does not contain the information it needs, produces a plausible-sounding response from adjacent context, and the failure is attributed to the model rather than the retrieval layer that supplied the wrong context. Most retrieval failures in production RAG systems are misread as model quality problems.

The second failure mode is corpus drift. A retrieval configuration tuned at corpus initialization — the embedding model, the chunk size, the similarity threshold — reflects the query distribution and document distribution at that moment. Both distributions shift. New documents are added with different vocabularies, different topic densities, different formats. Queries evolve as users develop their understanding of what the system can do. The retrieval configuration becomes progressively less optimal without producing any signal that optimization has degraded. The system regresses silently until someone notices qualitatively that answers have gotten worse.

This page investigates the architectural decisions that address these failure modes — hybrid retrieval, reranking, chunking strategy — and documents what is currently understood about each, where the understanding is contested, and what remains unresolved. The operational consequences of the unresolved questions are documented in the Second Brain build log, which encountered each of these problems in sequence.

§ 02What Transfers

State of understanding — current assessment

The following table documents the current state of understanding for the primary RAG retrieval concepts. Transfer status reflects whether the concept, as understood in the information retrieval literature, applies to production RAG systems without significant modification. Confidence reflects how settled the assessment is — "Low" means the field lacks agreed measurement, not that the concept is speculative.

ConceptTransfers?Adaptation RequiredConfidence
Pure vector retrievalPartialFails systematically on keyword-critical queries — proper nouns, code identifiers, technical framework names. The embedding space represents semantic neighborhoods, not lexical identity. Adequate for paraphrase-tolerant queries at small corpus scale.High
BM25PartialStrong lexical recall, consistent on exact-match queries. Misses semantic variants, paraphrases, and concept-level queries where the search terms differ from the document terms. Complement to vector retrieval, not a replacement for it.High
Hybrid BM25 + vectorYesRecall improvement over either method alone is consistent across documented implementations. The fusion strategy — Reciprocal Rank Fusion vs. learned interpolation weights — remains unsettled. RRF is robust and parameter-free; learned weights may improve on RRF but require labeled data to tune.Medium
Cross-encoder rerankingYesConsistent recall improvement when the merged candidate set is large enough to contain the relevant documents that a coarse ranking displaced. The latency cost is real — a cross-encoder reads query and document jointly, scaling linearly with candidate set size. Must be modeled explicitly in the latency budget.High
Fixed-size chunkingNoSplits semantic units at arbitrary token boundaries. Context loss at chunk boundaries is systematic and compounds at retrieval time — the relevant sentence may be split across two chunks, neither of which scores highly enough to be retrieved. The approach is simple to implement and consistently underperforms semantic alternatives.High
Semantic chunkingPartialPreserves semantic unit boundaries better than fixed-size. Produces variable chunk sizes that break fixed embedding cost models and make latency prediction harder. The quality improvement over fixed-size is real but inconsistently sized — some semantic chunkers produce chunks too small to carry useful context.Medium
Retrieval evaluationNoLabeled eval sets that associate queries with relevant documents do not exist at production scale — they require expensive human annotation or synthetic generation that may not reflect real query distributions. Standard IR metrics (NDCG, MRR) do not reliably correlate with end-task quality when the retrieval feeds a generative model.Low

The pattern in the table: retrieval methods transfer with varying modification — hybrid and reranking are relatively well-understood additions to the stack. The evaluation row does not transfer at all, and this is the blocking problem. Without evaluation, every claim about retrieval improvement is based on qualitative assessment rather than measurement. The improvements documented for hybrid retrieval and reranking are real — but they are characterized without the measurement infrastructure that would make them comparable across systems or configurations.

§ 03Hybrid Retrieval

BM25 + vector — why the combination outperforms either alone

Hybrid retrieval combines two fundamentally different retrieval signals. BM25 scores documents by term frequency and inverse document frequency — a document scores highly for a query term if the term appears frequently in that document and rarely across the corpus. Dense vector retrieval scores documents by the cosine similarity between the query embedding and the document embedding — a document scores highly if the model believes it is semantically related to the query, regardless of whether they share exact terms.

The recall gain from combining both comes from the complementarity of their failure modes. BM25 fails on queries where the relevant document uses different vocabulary than the query — synonyms, technical expansions, paraphrases. Vector retrieval fails on queries where the relevant document is identified by its exact terms — proper nouns, identifiers, technical abbreviations. A corpus of AI research notes will contain both types: conceptual connections well-served by vector retrieval, and specific paper titles, author names, and model names well-served by BM25. Neither method alone covers both query types.

The diagram below shows how the candidate set changes through the hybrid pipeline — the recall gain is concentrated at the merge step, and the quality gain is concentrated at the rerank step.

BM25 candidatestop-50 · lexical recall50 docsVector candidatestop-50 · semantic recall50 docsMerged setunion · hybrid recall gain70 docsPost-reranktop-20 · relevance ordered20 docsContext windowtop-5 · final input5 docsrecall gainherequality gainhere← CORPUS (n documents) — two parallel retrieval paths →
Fig. RES/RAG — Retrieval recall funnel. Candidate set through the hybrid pipeline. Recall gain concentrates at the merge step; quality ordering gain concentrates at the rerank step. Approximate document counts shown for a representative configuration.

The fusion step — combining BM25 and vector candidates into a merged set — is where the recall gain materializes. Documents that BM25 found but vector retrieval missed, and vice versa, are now both candidates for the reranker. The size of the merged set depends on the overlap between the two candidate lists. High overlap means the two methods are finding the same documents, which suggests one method is redundant. Low overlap means the methods are finding different documents, which is the condition that produces the largest recall gain.

The fusion strategy determines how the candidates are ranked before the reranker receives them. Reciprocal Rank Fusion (RRF) combines the rank positions from each method without requiring any tuning — a document ranked 3rd by BM25 and 7th by vector retrieval receives a combined score based on both positions. RRF is parameter-free and robust. Whether it produces optimal merged rankings compared to learned interpolation weights is one of the unresolved questions in § 06.

§ 04Reranking Economics

When a reranker pays for itself — and when it adds latency without value

A cross-encoder reranker reads the query and each candidate document jointly — as a single input — and scores their relevance together. This is structurally different from the bi-encoder approach used in vector retrieval, where query and document are embedded independently and compared post-hoc. The joint reading produces a more accurate relevance judgment because the model can attend to specific query terms in specific document positions, rather than comparing abstract summary vectors.

The cost is latency that scales linearly with the number of candidates. A reranker processing 50 candidates takes approximately 50x longer than processing 1. This is the fundamental reranking economics problem: the recall improvement from having more candidates in the merged set must be weighed against the latency cost of reranking a larger set. The optimal candidate count is not a fixed number — it is a function of the reranker model, the available compute, and the acceptable latency budget for the application.

The reranker pays for itself when the merged candidate set contains relevant documents that coarse ranking would displace from the top-k. If the hybrid retrieval is already placing relevant documents in the top-5 positions reliably, the reranker adds latency without improving the final context. If relevant documents are frequently appearing at positions 10–30 in the merged set — which they are, on keyword-critical queries where BM25 finds the document but the vector ranker depresses its position — the reranker produces significant quality improvement by promoting those documents.

The operational experience documented in the Second Brain build log is consistent with this model: reranking produced more recall improvement than any embedding model change or chunking adjustment. This is plausible for a corpus with high lexical specificity — technical notes with precise terminology, paper titles, and model names — where the merged candidate set frequently contains the correct document at a sub-optimal rank position. Whether this result generalizes to corpora with different properties is one of the questions the field has not characterized.

§ 05Chunking as Architecture

A first-class design decision that outlives the embedding model

Chunking is the step that converts source documents into the units that get embedded and stored. It is treated in most RAG implementations as a preprocessing detail — pick a token size, apply a sliding window, move on. The consequences of this treatment appear at retrieval time: a relevant passage split across two chunks where neither half contains enough context to score highly, a chunk that contains the answer buried in surrounding noise that pushes the relevance score down, a chunk boundary that separates a definition from its reference.

The chunking decision is a first-class architecture decision for two reasons. First, it determines the retrieval granularity — the smallest unit that can be retrieved. A system that chunks at 256 tokens cannot retrieve a 512-token passage as a unit, even if the full passage is the relevant answer. The granularity choice commits the system to a retrieval resolution that may be wrong for the query types it will encounter. Second, the chunking decision outlives the embedding model. If the embedding model is replaced, the embeddings must be regenerated — but the chunk boundaries remain. The corpus is re-embedded in the same chunks, carrying forward whatever structural decision the original chunking strategy produced.

Semantic chunking — splitting on sentence or paragraph boundaries rather than token counts — addresses the boundary problem but introduces cost unpredictability. Semantic chunks have variable sizes. A short sentence produces a 20-token chunk. A dense technical paragraph produces a 400-token chunk. The variable size makes embedding cost non-deterministic and makes the total corpus size harder to estimate before processing. Whether this cost unpredictability is worth the quality improvement over fixed-size chunking depends on the corpus structure — highly variable document types benefit more from semantic chunking than homogeneous corpora.

The deeper question — whether boundary strategy matters more than size, or whether size dominates at typical production chunk sizes — has not been characterized with sufficient rigor to guide production decisions. Most practitioners choose chunk size based on context window budget and embedding model recommendations, and choose boundary strategy based on implementation convenience. Neither decision is currently based on empirical evidence from their specific corpus and query distribution. This is the state of the art.

§ 06Open Questions

The current research frontier

The following questions are unresolved as of this writing. They are not engineering questions — they are questions where the field lacks the measurement infrastructure to produce definitive answers, or where practitioners are making different bets without shared empirical grounding.

What is the correct fusion strategy for merging BM25 and vector candidate sets?

Contested

Reciprocal Rank Fusion (RRF) is the current default — it is parameter-free, robust, and requires no labeled data. Learned interpolation weights may produce better results when training data is available, but the training data requirement is a significant barrier at production scale. Multiple implementations report different optimal strategies on different corpus types, with no clear pattern predicting which strategy will win. The choice is currently made by intuition or by running both and comparing — which requires a retrieval eval set, which most practitioners do not have.

How do you detect retrieval regression without a labeled ground truth?

Unresolved

Retrieval quality in production systems currently degrades silently. The only detection mechanism available in most deployed systems is user feedback — someone notices that answers have gotten worse. By the time this signal is visible, the regression has been active for weeks. An automated detection approach requires either a labeled query-document set (expensive, quickly stale) or a proxy signal that correlates with retrieval quality without requiring labels. No proxy signal has achieved broad adoption. The eval problem in RAG is the same unsolved problem as the eval problem in generation, compounded by the absence of a direct quality signal at the retrieval step.

At what corpus size and query rate does reranker latency become the primary bottleneck?

Active research

Reranker latency scales linearly with the number of candidates passed to it. At small candidate sets (top-20 from hybrid retrieval), the latency cost is acceptable for most applications. At large candidate sets (top-100 or more), the reranker becomes the dominant latency contributor and may push total retrieval latency above acceptable thresholds for interactive applications. The breakeven point depends on the reranker model, the hardware, and the query rate — none of which are stable across deployments. There is no published characterization of this tradeoff across common production configurations.

Does chunk boundary strategy matter more than chunk size for retrieval quality?

Contested

The two main variables in chunking are where to split (boundary strategy) and how large each chunk should be (size). The conventional wisdom is that size is the primary variable — smaller chunks produce more focused retrieval, larger chunks provide more context. There is growing evidence that boundary strategy matters at least as much: a well-placed boundary at a semantic unit preserves context that a poorly-placed boundary at an arbitrary token position destroys, regardless of size. The relative importance of boundary vs. size has not been rigorously characterized, and most production systems choose size first without systematic evaluation of boundary strategy.

How does the optimal retrieval configuration change as the corpus drifts over months?

Unresolved

A retrieval configuration tuned at corpus initialization — embedding model, chunk size, hybrid fusion weights, reranker candidate count — may not remain optimal as the corpus grows and the document distribution shifts. A corpus that adds new topic areas may require different BM25 vs. vector weights than the initial configuration. A corpus that grows by an order of magnitude may require a different reranker candidate count to maintain acceptable latency. There is no established practice for re-evaluating and updating retrieval configuration as a live corpus evolves. Most production systems treat the initial configuration as fixed, which may produce silent drift in retrieval quality independent of any change to the underlying infrastructure.

The unresolved and contested questions here share a dependency on the evaluation infrastructure that does not yet exist at production scale. The fusion strategy question is contested because different teams are measuring on different eval sets with different query distributions — without a shared eval protocol, the disagreement cannot be resolved. The regression detection question is unresolved because it requires a proxy metric that correlates with retrieval quality without requiring labels, and no such metric has been validated. The corpus drift question is unresolved because it requires longitudinal measurement on live corpora that most teams do not have the instrumentation to collect. The dependency order is fixed: retrieval evaluation infrastructure must exist before any of the contested retrieval questions can be answered with evidence rather than opinion. That evaluation infrastructure is itself an open research problem, documented separately in the Evaluation research page. The two investigations are blocked on each other, and that circularity is what makes this research rather than engineering.

Continue Reading