How to reduce OpenSearch RAG costs: query planning, batching and caching

The fastest way to reduce OpenSearch costs in a RAG system is to stop paying for retrieval you did not need: measure how many OpenSearch requests one user message triggers (queries per turn), plan and batch those requests instead of firing them as the pipeline thinks, rerank once on a pooled candidate set instead of once per intent, and cache document IDs, chunks and assembled context with keys that include scope and version. A practical budget to aim for is 6 queries per turn at P95 and one rerank pass; the sections below show how to get there without lowering relevance.

This guide covers each of those steps for an OpenSearch-backed RAG stack: how to see fanout, how to plan queries, which batching patterns cut round trips, how to cache without serving stale answers, and how to prove that the cost reduction did not hurt answer quality.

The short version

• Measure queries per turn (QPT) at P50, P95 and P99 before changing anything; the P99 shows where budgets break.
• Plan retrieval per turn: deduplicate intents, share filters, set a budget (2 to 3 intents, 4 to 8 OpenSearch calls, 50 to 120 rerank candidates), and degrade safely when the budget is hit.
• Batch across intents, fetch IDs first and hydrate later, and rerank once on a pooled set; rerank loops are usually the real multiplier.
• Cache in three layers (doc IDs, chunks, hydrated context) with keys that include filter signature, index generation and scope, and TTLs matched to how fast the corpus changes.

Why OpenSearch RAG costs grow faster than traffic

In most OpenSearch RAG stacks, retrieval looks cheap because each query is fast on its own. The real cost emerges from the pattern: one user turn becomes multiple intents (“compare,” “recommend,” “explain why”), each intent triggers hybrid retrieval (filters plus BM25 plus vector), rerankers pull additional candidates or run a second pass, and follow-ups repeat the same constraints and re-fetch similar chunks.

The result is silent fanout: OpenSearch QPS and tail latency scale with how your pipeline behaves, not with actual user demand. That creates three common production failures: cost drift, where OpenSearch compute scales up even when daily active users are flat; tail latency instability, where P95 and P99 spike because a subset of turns triggers large fanout and rerank loops; and relevance volatility, where aggressive retries and second-pass queries change top-K sets and make answers feel inconsistent. Retrieval cost engineering makes these behaviors explicit, bounded and testable, so you can reduce load without gambling on relevance.

How do you measure RAG retrieval cost? Queries per turn

Before optimizing, you need a metric that shows what one user message triggers. The core metric is queries per turn (QPT): count every OpenSearch request triggered by one user message, including reranks and retries. Track the shape, not only the average: QPT at P50, P95 and P99; rerank passes per turn and how many extra queries they trigger; unique query templates per turn, a signal of planner complexity; retrieval share of latency; and bytes returned per turn, a proxy for hidden hydration cost.

A simple budget target is an effective early win: QPT P95 at or under 6 for most endpoints, and rerank passes at or under 1 unless explicitly required.

Query planning: treat retrieval like a compiled plan

Most RAG pipelines retrieve as they think: a step needs context, so it queries; the reranker needs more, so it queries again. Query planning flips the control flow: extract the turn’s information needs, deduplicate overlaps, compose shared filters, allocate a budget, and execute as a plan, batched and bounded.

Deduplicate overlapping intents

A single user message often contains multiple phrasings of the same need: “Compare A and B,” “What are the trade-offs?”, “Which is better for X?”. Retrieving separately for each pays multiple times for similar documents. A practical approach: normalize intents into a small set of canonical asks (definition, comparison, procedure, trade-offs, recommendation), extract entities and constraints (A, B, X, time window, scope), and merge intents when entities overlap strongly, required evidence sources are similar, and the output can share context. The output of this step is not “3 questions”; it is “2 retrieval intents that cover the turn.”

Rank intents and set a turn budget

Not all intents deserve equal retrieval spend. A stable ordering: hard correctness needs (policy, safety, contract, exact references) first, then user- or tenant-scoped evidence, then core domain docs, then background explanations last. Enforce a turn budget: 2 to 3 intents per turn, 4 to 8 total OpenSearch calls, 50 to 120 candidates for rerank, and 256 KB to 1 MB of hydrated bytes. If you hit the budget, do not try harder. Degrade safely: answer with what you have, say what is missing, and offer a next step.

Share filters across the turn

Filters are often the most wasteful repetition: every sub-query rebuilds the same filter set for tenant scope, doc types, time windows, product line or language. Build a turn-level filter signature once and reuse it everywhere unless an intent overrides it. That stabilizes both cost and relevance and prevents accidental unfiltered queries.

Batched retrieval: reduce round trips, control tail latency

Once you plan, execution gets simpler: fewer calls, fewer retries, fewer accidental loops. Pattern A, multi-search batching across intents: if multiple intents hit the same index and share filters, batch them into one request. Even when OpenSearch processes them individually, you cut connection overhead, TLS handshake churn, client-side queuing, and tail latency from sequential calls. Batching helps most with 2-4 intents per turn, hybrid retrieval where each intent would otherwise run 2-3 queries, and high-concurrency environments where batching reduces client bottlenecks.

Pattern B, two-phase retrieval (IDs first, hydrate later): a classic cost trap is hydrating too much too early. Two-phase retrieval splits the work, phase 1 fetches top doc or chunk IDs with minimal payload, phase 2 hydrates only the winning subset into text snippets. Why it works: most candidates never make it into the final context window, hydration dominates bytes returned, and you can cap hydration independently of ranking. Good defaults: phase 1 gets top 50-200 IDs (cheap), phase 2 hydrates 10-30 chunks max, or caps by bytes.

Pattern C, candidate pooling for rerank (one rerank, not many): without pooling you might rerank per intent, intent A retrieves and reranks, intent B retrieves and reranks, intent C retrieves and reranks. Pooling flips it: retrieve candidates for all intents, deduplicate by doc ID or chunk hash, rerank once on the pooled set, and allocate final context slots by intent priority. This is one of the highest-ROI fanout reducers, since rerank loops are often the real multiplier.

Batched retrieval: fewer round trips, steadier tail latency

Multi-search batching across intents. If multiple intents hit the same index and share filters, batch them into one request. Even when OpenSearch processes them individually, you cut connection overhead, TLS handshake churn, client-side queuing and the tail latency of sequential calls. Batching helps most with 2 to 4 intents per turn, hybrid retrieval where each intent would otherwise run 2 to 3 queries, and high-concurrency environments.

Two-phase retrieval: IDs first, hydrate later. Phase 1 fetches the top doc or chunk IDs with minimal payload; phase 2 hydrates only the winning subset into text. Most candidates never reach the final context window, hydration dominates bytes returned, and you can cap hydration independently of ranking. Good defaults: 50 to 200 IDs in phase 1, 10 to 30 chunks hydrated in phase 2, or a byte cap.

Candidate pooling for rerank. Without pooling, each intent retrieves and reranks separately. Pooling retrieves candidates for all intents, deduplicates by doc ID or chunk hash, reranks once on the pooled set, and allocates final context slots by intent priority. This is one of the highest-return fanout reducers, because rerank loops are often the real multiplier.

Deduplicate at the boundary. The same doc surfaces through BM25 and vector search; the same chunk arrives through multiple queries. Dedupe at the doc boundary (doc_id), the chunk boundary (chunk_id or content hash) and, where embeddings are at hand, the semantic boundary. Our post on why retrieving more documents does not always help covers why a larger top-K rarely improves answers and often raises cost.

RAG caching that works: three layers, clear keys, predictable TTLs

Caching is where retrieval cost engineering becomes set-and-forget, but only with disciplined keys and invalidation.

Use the third layer when assembly itself is expensive: multi-source joining, cleaning, merging duplicates, snippet formatting and provenance tagging. Keep it short-lived; it is a tail-latency smoother, not a primary cache. The chunk schema and index settings behind these layers are described in leveraging OpenSearch for production RAG, and the index-level trade-offs in how vector databases search with IVF, HNSW, PQ and ScaNN.

Freshness and invalidation without losing trust

Freshness is why teams avoid caching. Treat it as a policy: something can be correct without being latest. Categorize data into a stable corpus (docs, guides, evergreen knowledge), an evolving corpus (release notes, runbooks) and a time-sensitive corpus (alerts, incident notes, daily metrics), each with its own TTLs and bypass rules.

Invalidation mechanisms that scale: an index generation stamp bumped on ingest and included in cache keys, so you invalidate by moving forward; doc revision keys in the chunk cache, so updated docs become cache misses on their own; and tenant- or user-scoped namespaces, so you can flush a scope without global churn. If the question includes time signals such as “latest,” “today” or “new release,” shorten TTL, bypass the hydrated-context cache and rehydrate even if IDs are cached. If it is evergreen (“how to,” “explain”), lean on cache and prioritize stability.

How to confirm cost reduction did not hurt relevance

Track cost as hard numbers: QPT at P50, P95 and P99, OpenSearch P95 latency, bytes returned per turn and rerank loop rate. Track cache health: hit rate per layer, evictions per minute (an undersized cache) and the rate of responses served after soft TTL (staleness exposure).

Track relevance with proxies you can run every week: Recall@K on a golden set (50 to 200 queries is enough to see a regression), answer stability (does the same question return the same citations across runs) and follow-up amplification (do users ask more clarifying questions right after a change). If relevance drops, do not revert everything. Tighten one lever: increase the candidate pool slightly, loosen the dedupe threshold, shorten the hydrated-context TTL, or bypass caching for time-sensitive intents.

Where this applies in the Hoomanely assistant

In multi-turn assistants that combine knowledge base guidance with summaries derived from a user’s own pets and devices, the same constraints repeat across turns: time window, pet scope, device summaries. That is where query planning and caching deliver the largest wins. Shared filters prevent accidental drift, chunk caching avoids rehydrating the same evidence repeatedly, and batched retrieval keeps tail latency stable in follow-up-heavy conversations. The record EverBowl™ builds of how a dog eats and drinks is part of that scoped evidence, and the assistant has to stay responsive as the number of households grows without over-provisioning OpenSearch for worst-case fanout.

How the retrieved context is assembled into a prompt is covered in dynamic prompt construction at runtime, and the load-shedding side in backpressure for LLMs.

Conclusion

Treat retrieval as a budgeted subsystem: plan, batch, cap and measure. Query planning with deduplicated intents, shared filters and a turn budget removes most accidental fanout; batched and two-phase retrieval smooths P95 and P99 under load; three-layer caching works when keys carry scope and version and TTLs match freshness needs; and both cost metrics and relevance proxies belong on the same dashboard, because stability is part of quality. Start this week by instrumenting queries per turn; every other change follows from that number.

FAQs

How do I reduce OpenSearch costs in a RAG pipeline?

Measure queries per turn, then cut fanout with query planning (deduplicated intents, shared filters, a per-turn budget), batch requests, fetch IDs before hydrating text, rerank once on a pooled candidate set, and cache doc IDs, chunks and assembled context with scoped, versioned keys.

What is queries per turn (QPT)?

The number of OpenSearch requests triggered by one user message, including reranks and retries. Tracked at P50, P95 and P99, it shows how pipeline behavior, rather than user demand, drives cost and latency.

Is caching safe in RAG?

Yes, when keys include the filter signature, index generation and scope, and TTLs match how fast the corpus changes. Cache document IDs and chunks with long TTLs and the assembled context with short ones; bypass the context cache for time-sensitive questions.

Does batching queries hurt relevance?

No. Batching changes how queries are sent, not what is retrieved. Relevance changes come from deduplication thresholds, candidate pool size and caching TTLs, which is why those are tuned one at a time against a golden query set.

How many rerank passes should a RAG turn use?

One, on a pooled candidate set of roughly 50 to 120 chunks. Reranking per intent multiplies both OpenSearch load and reranker cost, and it is the most common hidden multiplier in fanout.

How do I know a cost change did not degrade answers?

Run Recall@K on a 50 to 200 query golden set, check answer stability across repeated runs, and watch for a rise in follow-up questions after the change. If any of those move, adjust one lever at a time rather than reverting everything.