← AI Security & Ops theorytheory 0/14 · 0%

level 1

Curious · 0 XP

streak

perfect

badges

100 XP to Apprentice

Operations · hard

11. Deploying prompt and model changes safely

Version prompts like code and roll them out behind flags.

1 min read2 questionsup to 50 XPtopic 11/14

Read in another language

AI translates the explanation and quiz text while preserving code and answer scoring.

A prompt edit is a production deploy. Treat it accordingly:

1. Version prompts in the repo (or a registry) with an id and a changelog. 2. Offline eval the new version against the golden suite; block on regressions. 3. Shadow it on a traffic sample, comparing scores without showing users the output. 4. Canary 5% behind a flag, watching quality, latency, cost, and refusal rate. 5. Roll out or roll back with one flag flip.

ts
const variant = flags.pick(userId, { "reply.v7": 0.95, "reply.v8": 0.05 });
const out = await model({ prompt: PROMPTS[variant], input });
await obs.score(traceId, "variant", variant);

Model upgrades need the same treatment: a newer model can regress your specific task, especially where prompts were tuned to old quirks or where structured-output behaviour changed. Pin model ids explicitly, never "latest", so a provider-side change cannot alter your product silently.

Exercise: run a canary between two prompt versions for 24 hours and decide from the trace scores, not intuition.

How to study this topic

1. Read the explanation once end to end, skipping the code.
2. Re-read the code snippets and say out loud what each line does.
3. Answer the quiz without scrolling back — that is the real check.
4. Read every explanation, even for the answers you got right.

Scoring

· Up to 30 XP from quiz accuracy.
· +15 XP for a perfect first attempt.
· +5 XP while a 3-day streak is running.
· 75% or higher marks the topic complete.

Check your understanding

1. Why pin explicit model ids instead of 'latest'?
A provider-side change could alter behaviour without any deploy of yoursLatest costs morePinning is required by the APIIt improves caching only
2. What is shadow evaluation?
Running the new version on real traffic and scoring it without showing usersTesting on synthetic dataRunning with logging disabledA/B testing UI copy

Badges

First light

Pass your first topic

Sharp shooter

3 perfect first-try quizzes

On a roll

Study 3 days in a row

Week warrior

Study 7 days in a row

Grinder

Earn 500 XP

Deep diver

Earn 1500 XP