12. Cost, latency and caching in RAG systems

A RAG request has a latency budget spread across stages. Typical targets for an interactive assistant:

| Stage | Budget | Lever | | --- | --- | --- | | Query rewrite | 100-300 ms | small fast model, or skip for first turn | | Embed query | 10-40 ms | small embedding model, cache by query hash | | Hybrid search | 20-80 ms | index tuning, metadata pre-filter | | Rerank 50 → 6 | 60-200 ms | batch, cap candidates, cache | | Generation | 1-4 s | streaming, prompt caching, smaller model where possible |

Caching layers that matter: - Exact query cache (normalised query + tenant + doc-version) — huge hit rates on support workloads. - Semantic cache — reuse an answer when a new query is within a similarity threshold; require a conservative threshold and invalidate on document change. - Prompt prefix caching — stable system prompt and tool schemas cost far less on repeat.

ts
const key = `${tenant}:${corpusVersion}:${normalise(query)}`;
const hit = await cache.get(key);
if (hit) return hit;                       // stream from cache
const answer = await ragPipeline(query);
await cache.set(key, answer, { ttl: 3600 });

Always stream tokens: perceived latency is dominated by time-to-first-token, not total time.

Gotcha: semantic caches leak across tenants and go stale after edits. Key them by tenant *and* corpus version, and keep the threshold strict.

Read in another language

How to study this topic

Scoring

Check your understanding

Badges