Retrieval Cost Engineering for OpenSearch RAG

A production RAG system feels fast when it's consistent: answers arrive quickly, follow-ups don't suddenly slow down, and costs don't jump with traffic. The biggest lever for that isn't usually the model, it's retrieval. When retrieval is engineered like a disciplined subsystem, with budgets, plans, batching, and caching, you get predictable latency, stable relevance, and cost that scales with users rather than accidental query fanout.

This post is a practical dive into retrieval cost engineering for OpenSearch-backed RAG: detecting and reducing fanout, planning and batching queries so you don't repeatedly pay round-trip penalties, and caching safely, IDs, chunks, and hydrated context, without breaking freshness or trust. The goal: keep retrieval measurable, budgeted, and stable, so your assistant stays responsive as you scale.

Why retrieval cost engineering matters

In most OpenSearch RAG stacks, retrieval looks cheap because each query is fast on its own. The real cost emerges from the pattern: one user turn becomes multiple intents ("compare," "recommend," "explain why"), each intent triggers hybrid retrieval (filters plus BM25 plus vector), rerankers pull additional candidates or run a second pass, and follow-ups repeat the same constraints and re-fetch similar chunks.

The result is silent fanout: OpenSearch QPS and tail latency scale with how your pipeline behaves, not with actual user demand. That creates three common production failures: cost drift, where OpenSearch compute scales up mysteriously even when DAU is flat; tail latency instability, where P95/P99 spikes because a subset of turns triggers large fanout and rerank loops; and relevance volatility, where aggressive retries and second-pass queries change top-K sets and make answers feel inconsistent.

Retrieval cost engineering is about making these behaviors explicit, bounded, and testable, so you can reduce load without gambling on relevance.

Make fanout visible: measure queries per turn like an SLO

Before optimizing, you need an x-ray metric showing what one user message triggers. The core metric is Queries Per Turn (QPT): count every OpenSearch request triggered by one user message, including reranks and retries. Also track shape, not just averages: QPT P50/P95/P99 (P99 tells you where budgets break), rerank passes per turn (and how many extra queries they trigger), unique query templates per turn (a signal of planner complexity), retrieval share of latency (time in retrieval versus generation), and bytes returned per turn (a hidden hydration cost proxy).

A surprisingly effective early win is defining a simple budget target, something like QPT P95 at or under 6 for most endpoints, and rerank passes at or under 1 unless explicitly required.

Query planning: treat retrieval like a compiled plan

Most RAG pipelines retrieve as they think: a step needs context, so it queries; the reranker needs more, so it queries again. Easy to build, expensive to run. Query planning flips the control flow: extract the turn's information needs, deduplicate overlaps, compose shared filters, allocate a budget, and execute as a plan, batched and bounded. Think of it as compiling a retrieval program.

Deduplicating overlapping intents matters because a single user message often contains multiple phrasings of the same need, "Compare A and B," "What are the trade-offs?", "Which is better for X?". Retrieving separately for each pays multiple times for similar documents and chunks. A practical, non-fragile approach: normalize intents into a small set of canonical asks (definition/explanation, comparison, procedural steps, trade-offs, recommendation), extract entities and constraints (A, B, X, time window, scope), and merge intents when entities overlap strongly, required evidence sources are similar, and the output can share context. The output of this step isn't "3 questions," it's "2 retrieval intents that cover the turn."

Not all intents deserve equal retrieval spend. A stable ordering that holds in production: hard correctness needs (policy, safety, contract, exact references) first, then user- or tenant-scoped evidence, then core domain docs, then background/explanations last. Enforce a turn budget: max intents per turn (say 2-3), max total OpenSearch calls (4-8), max candidates for rerank (50-120), and max hydrate bytes (256KB-1MB). If you hit the budget, don't try harder, degrade safely: answer with what you have, be explicit about what's missing, and offer a next step.

Filters are often the most wasteful repetition source, every sub-query rebuilds the same filter set. Common shared filters include tenant/org/user scope, doc types, time windows, product line or environment, and language or region. The planner rule: build a turn-level filter signature once, reuse it everywhere unless an intent overrides it. That stabilizes both cost and relevance, and prevents accidental unfiltered queries.

Batched retrieval: reduce round trips, control tail latency

Once you plan, execution gets simpler: fewer calls, fewer retries, fewer accidental loops. Pattern A, multi-search batching across intents: if multiple intents hit the same index and share filters, batch them into one request. Even when OpenSearch processes them individually, you cut connection overhead, TLS handshake churn, client-side queuing, and tail latency from sequential calls. Batching helps most with 2-4 intents per turn, hybrid retrieval where each intent would otherwise run 2-3 queries, and high-concurrency environments where batching reduces client bottlenecks.

Pattern B, two-phase retrieval (IDs first, hydrate later): a classic cost trap is hydrating too much too early. Two-phase retrieval splits the work, phase 1 fetches top doc or chunk IDs with minimal payload, phase 2 hydrates only the winning subset into text snippets. Why it works: most candidates never make it into the final context window, hydration dominates bytes returned, and you can cap hydration independently of ranking. Good defaults: phase 1 gets top 50-200 IDs (cheap), phase 2 hydrates 10-30 chunks max, or caps by bytes.

Pattern C, candidate pooling for rerank (one rerank, not many): without pooling you might rerank per intent, intent A retrieves and reranks, intent B retrieves and reranks, intent C retrieves and reranks. Pooling flips it: retrieve candidates for all intents, deduplicate by doc ID or chunk hash, rerank once on the pooled set, and allocate final context slots by intent priority. This is one of the highest-ROI fanout reducers, since rerank loops are often the real multiplier.

Pattern D, deduplicate overlapping chunks at the boundary: even with deduped intents, overlapping results happen, the same doc surfaces via BM25 and vector, or the same chunk arrives via multiple queries. Always dedupe at the doc boundary (doc_id), the chunk boundary (chunk_id or content hash), and the semantic boundary if you have embeddings handy (near-duplicate detection). This reduces context bloat and improves answer stability.

Caching that actually works: three layers, clear keys, predictable TTLs

Caching is where retrieval cost engineering becomes set-and-forget, but only with disciplined keys and invalidation.

Layer 1, doc-ID cache, high hit rate, low risk: stores query_key mapped to doc_ids, scores, and metadata. It's safe because IDs are small, you can always rehydrate fresh text later, and staleness risk is limited since the underlying set might shift but usually still holds within TTL. A strong cache key includes the canonical query representation, filter signature, index/version stamp, and retrieval mode plus embedding model ID where relevant. TTL guidance: 6-24 hours for a stable KB, 5-30 minutes for frequently updated corpora, separate namespaces by source type for mixed cases.

Layer 2, chunk cache, a big latency win in multi-turn: stores doc_id plus chunk_id (plus doc_revision) mapped to chunk text, offsets, and small metadata. This avoids repeatedly fetching the same chunk across reranks, follow-ups, and repeated intents. Key requirement: include a doc revision or last_updated stamp so updates automatically bypass old entries. TTL guidance: hours to days depending on update frequency.

Layer 3, hydrated-context cache, highest payoff, highest staleness risk: stores turn_signature mapped to the assembled final context blob. Use this when assembly itself is expensive, multi-source context joining, post-processing, cleaning, merging duplicates, snippet formatting, and provenance tagging. Keep it short-lived, it's a tail-latency smoother, not a primary cache. TTL guidance: 1-10 minutes unless you have strong invalidation.

Freshness and invalidation: keep performance without losing trust

Freshness is the reason teams avoid caching in the first place. The trick is treating freshness as a policy, not an accident. Something can be correct without being latest, many user questions don't need the newest doc revision, they need stable guidance. So categorize data: stable corpus (docs, guides, evergreen KB), evolving corpus (release notes, operational runbooks), and time-sensitive corpus (alerts, incident notes, daily metrics), each with its own TTLs and bypass rules.

Simple invalidation mechanisms that scale: an index generation stamp bumped on ingest and included in cache keys, especially hydrated context, so you invalidate by moving forward; doc revision keys in the chunk cache so updated docs naturally become cache misses; and scope-based namespaces (tenant or user scoped) preventing cross-tenant leakage, letting you flush a scope without global churn.

A practical approach to "fresh enough": if the question includes time signals like "latest," "today," or "new release," shorten TTL, bypass the hydrated-context cache, and rehydrate even if IDs are cached. If it's evergreen, like "how to" or "explain," lean on cache and prioritize stability and latency. That's how you avoid a system that's either always stale or always expensive.

Validating that cost reduction didn't hurt relevance

Retrieval cost metrics worth tracking as hard numbers: QPT P50/P95/P99, OpenSearch P95 latency, bytes returned per turn, and rerank loop rate. Cache health metrics: hit rate per layer (ID/chunk/context), evictions per minute (to detect an undersized cache), and the rate of responses served after soft TTL (staleness exposure).

Relevance proxies worth using in practice: Recall@K on a small golden set, even 50-200 queries helps; answer stability, does the same question return the same citations and top docs across runs; and follow-up amplification, do users ask more clarifying questions right after a change. If relevance drops, don't revert everything, tighten one lever, increase the candidate pool slightly, loosen the dedupe threshold, shorten the hydrated-context TTL, or bypass caching for time-sensitive intents.

In multi-turn assistants that combine knowledge base guidance with user- and device-derived summaries, the same constraints repeat across turns, time window, pet scope, device summaries. That's where query planning and caching deliver strong wins, shared filters prevent accidental drift, chunk caching avoids rehydrating the same evidence repeatedly, and batched retrieval keeps tail latency stable during follow-up-heavy conversations. This pattern helps ensure experiences built around products like EverSense and EverBowl stay responsive as usage scales, without over-provisioning OpenSearch for worst-case fanout.

Key takeaways

Treat retrieval as a budgeted subsystem: plan, batch, cap, and measure. Query planning, dedupe plus shared filters plus budgets, removes most accidental fanout. Batched retrieval reduces round trips and smooths P95/P99 under load. Three-layer caching works when keys include scope and versions, and TTLs match freshness needs. And validate with both cost metrics and relevance proxies, stability is part of quality.