7. Online quality evaluation

Offline evals guard regressions; online evals catch drift on inputs you never imagined. Score a sample of production traffic with cheap automated judges:

- Groundedness — is every claim supported by the retrieved context? - Relevance — does the answer address the question? - Refusal rate and guardrail trigger rate. - Implicit signals — copy, retry, thumbs-down, escalation to a human.

ts
if (Math.random() < 0.1) {                       // 10% sample
  const grounded = await judge("Is every claim supported by CONTEXT? yes/no", { answer, context });
  await obs.score(traceId, "grounded", grounded === "yes" ? 1 : 0);
}

Alert on rate changes, not single failures: groundedness below 0.9 over a 1-hour window, refusals doubling day over day, or cost per request up 30%. Feed every low-scoring trace into a review queue and promote confirmed failures into the offline suite — that loop is what makes quality improve over time.

Exercise: add sampled groundedness scoring plus an alert, then trace one alert back to its root cause and add a regression case for it.

Read in another language

How to study this topic

Scoring

Check your understanding

Badges