14. Capstone: production-ready AI feature

Take an existing AI feature to production standard.

Checklist

1. Single choke point — all model calls flow through one wrapper. 2. Security — input screening, untrusted-content labelling, PII redaction, read-only tools on untrusted paths, egress allow-list, output guardrails with metrics. 3. Observability — one trace per request with tokens, cost, retrieval ids, and sampled quality scores; dashboards for cost per success, p95 latency, groundedness, guardrail hits. 4. Cost control — usage table, per-user and global caps, prompt caching, model routing, correct 402/403/429 handling. 5. Memory — governed facts with dedup, decay, user visibility, and deletion. 6. Automation — idempotent scheduled jobs with concurrency caps and a paused-state check. 7. Release — versioned prompts, pinned model ids, offline suite in CI, canary flag, tested rollback. 8. Governance — data map, decision records, erasure job.

ts
export const health = {
  costPerSuccessUsd: () => q("select sum(cost_usd)/count(*) from ai_usage where created_at > now() - interval '1 day'"),
  groundedness: () => q("select avg(score) from ai_scores where name='grounded' and created_at > now() - interval '1 day'"),
  guardrailHits: () => q("select name, count(*) from ai_guardrails group by 1 order by 2 desc"),
};

Ship criteria: the dashboard answers cost, quality, and safety questions without a code change; the rollback has been executed at least once in a drill.

Exercise: complete the checklist for one real feature and write a one-page runbook naming the owner, the alerts, the kill switch, and the rollback procedure.

Read in another language

How to study this topic

Scoring

Check your understanding

Badges