11. Evaluating LLM features without fooling yourself

Vibes do not survive a model upgrade. Build evaluation before scaling a feature.

Layers of evaluation: - Unit-style assertions — deterministic checks on structured output (schema valid, ids exist, totals reconcile). - Golden dataset — 50-300 real inputs with expected properties, versioned in the repo. - LLM-as-judge — a stronger model scores answers against a rubric; always calibrate the judge against human labels on a sample, and randomise option order to fight position bias. - Online metrics — thumbs, task completion, escalation rate, cost and latency percentiles.

python
def test_extraction_regression(model):
    scores = []
    for case in load_golden("invoices.jsonl"):
        out = extract(model, case["input"])
        scores.append(field_f1(out, case["expected"]))
    assert mean(scores) >= 0.92, "extraction regressed"

Report pass rate with confidence intervals, not a single accuracy number, and keep cost/latency in the same dashboard so quality wins are not paid for silently.

Gotcha: judges inherit biases — they favour longer, more confident answers. Anchor rubrics with explicit failure examples.

Read in another language

How to study this topic

Scoring

Check your understanding

Badges