The Select Strategy and RAG Done Right: Signal Over Volume

Selecting context is the strategy most people meet first, usually as RAG. But doing it well is subtler than search-and-paste. The governing principle is counterintuitive: less, better-chosen information almost always beats more.

Retrieval-augmented generation is how AI systems answer questions about your company documents, your codebase, or current events: fetch relevant information from an external source and add it to the context before the model responds. It is powerful and ubiquitous. It is also where teams most often go wrong, because the instinct that drives engineers, more data is better, is precisely backwards here.

How RAG Works

The basic flow: a query comes in, the system searches a knowledge base for relevant content, the most relevant pieces are added to context, and the model generates a response grounded in that retrieved content. The quality of the final answer depends enormously on the quality of the retrieval step. Get retrieval right and the model has what it needs; get it wrong and no amount of prompt cleverness compensates.

Semantic Search and Relevance

Modern selection relies on semantic search: representing both query and documents as vectors and finding documents whose meaning is closest, rather than matching keywords. This catches relevant content even when it shares no exact words with the query. But similarity alone is not enough. Relevance scoring layers additional signals, recency, authority, document type, so that what lands in context is the most useful slice, not just the most superficially similar.

Just-in-Time Loading

A key production pattern is just-in-time loading: instead of pre-loading documents, maintain lightweight identifiers (file paths, URLs, queries) and fetch the actual data only when the current step requires it. This keeps the baseline context lean and spends tokens only on what is actually needed. For agents that might need any of thousands of documents but only a few per task, this pattern is transformative.

The Signal-to-Noise Principle

Here is the governing principle: every piece of information you add is either signal (helps the model) or noise (distracts it). Adding more is only good if it adds more signal than noise. Past a certain point, additional documents reduce quality because they dilute attention and introduce distractors. This is counterintuitive to engineers trained to think comprehensiveness is safety, but in context engineering, a small curated set reliably beats a large noisy one.

Debugging Poor Retrieval

When RAG results disappoint, resist the urge to retrieve more documents. More often the fix is retrieving fewer, better ones. Audit what actually lands in context on real queries; you will frequently find irrelevant chunks crowding out the few that matter. The insurance-company example bears repeating: a targeted schema reached over 95% accuracy where the full corpus achieved far less. Tightening retrieval beats widening it almost every time.