11. Production ingestion pipelines and freshness

The hard part of production RAG is keeping the index correct as sources change.

Pipeline design: - Change detection — content hash per source; skip unchanged documents to avoid re-embedding costs. - Incremental upsert — chunk ids derived from `(doc_id, chunk_index, content_hash)` so updates replace exactly the affected rows. - Deletes and tombstones — propagate removals within a defined SLA; never rely on TTL alone. - Idempotent, resumable jobs — a retried batch must not duplicate chunks. - Backfill vs live path — bulk re-embedding runs on a separate queue so user-facing ingestion is not starved. - Versioned indexes — build the new embedding-model index alongside the old one, evaluate, then flip an alias.

python
h = sha256(text.encode()).hexdigest()
if store.hash_of(doc_id) != h:
    store.delete_chunks(doc_id)
    store.upsert(chunk_rows(doc_id, chunk(text), h))
    store.set_hash(doc_id, h)

Observability: track ingestion lag, chunks per document, embedding failures, empty-retrieval rate, and answer abstention rate — a rising abstention rate is usually an ingestion outage.

Gotcha: re-embedding the whole corpus on a model upgrade is a migration, not a config change. Plan dual-index capacity and an evaluated cutover.

Read in another language

How to study this topic

Scoring

Check your understanding

Badges