The RAG pipeline
Ingest documents, chunk them with overlap, embed the chunks, and store the vectors. At query time embed the question, retrieve the nearest chunks, and place them in the prompt with citations. Retrieval quality — not the model — is usually the bottleneck.
Making retrieval good
Hybrid search combines dense vectors with keyword BM25 so exact identifiers still match. A cross-encoder reranker reorders the top 50 candidates down to the best 5. Metadata filters keep results scoped to the right tenant, product or version.
Agents and tools
An agent loop is: model chooses a tool, the tool executes, the result returns as context, repeat until done. It only works with a strict schema per tool, a step limit, idempotent actions and human approval for anything irreversible.
- Chunk 300–800 tokens with ~15% overlap.
- Always return source links with the answer.
- Cap agent steps and log every tool call.