← AI Agents & MCP theorytheory 0/14 · 0%

level 1

Curious · 0 XP

streak

perfect

badges

100 XP to Apprentice

Testing · medium

12. Evaluating agents: trajectories, not just answers

Score tool choice, step count, and cost alongside final correctness.

1 min read2 questionsup to 50 XPtopic 12/14

Read in another language

AI translates the explanation and quiz text while preserving code and answer scoring.

Final-answer accuracy hides the failures that hurt in production: unnecessary tool calls, loops, skipped verification. Evaluate the trajectory.

Metrics worth tracking per test case:

- Task success — programmatic assertion on the end state, not a vibe check. - Tool precision/recall — were the required tools called, and only those? - Steps and tokens to success (cost proxies). - Recovery rate — success after an injected tool failure.

ts
const cases = [{
  goal: "Refund order 8123 and email the customer",
  mustCall: ["get_order", "create_refund", "send_email"],
  mustNotCall: ["delete_order"],
  assert: async () => (await db.refunds.findByOrder(8123))?.status === "issued",
}];

for (const c of cases) {
  const trace = await runAgentTraced(c.goal);
  expect(c.mustCall.every((t) => trace.calls.includes(t))).toBe(true);
  expect(c.mustNotCall.some((t) => trace.calls.includes(t))).toBe(false);
  expect(await c.assert()).toBe(true);
}

Run this suite in CI against mocked tools so it is deterministic, then a smaller suite nightly against staging services.

Exercise: add a case where `create_refund` fails once with a 500 and assert the agent retries and still succeeds.

How to study this topic

1. Read the explanation once end to end, skipping the code.
2. Re-read the code snippets and say out loud what each line does.
3. Answer the quiz without scrolling back — that is the real check.
4. Read every explanation, even for the answers you got right.

Scoring

· Up to 30 XP from quiz accuracy.
· +15 XP for a perfect first attempt.
· +5 XP while a 3-day streak is running.
· 75% or higher marks the topic complete.

Check your understanding

1. Why evaluate trajectories and not only final answers?
Right answers can hide loops, wasted calls, and skipped verificationFinal answers are hard to storeTrajectories are cheaper to scoreModels require it
2. How do you keep an agent test suite deterministic in CI?
Mock the tools and assert on end stateRaise temperatureUse a bigger modelRun against production

Badges

First light

Pass your first topic

Sharp shooter

3 perfect first-try quizzes

On a roll

Study 3 days in a row

Week warrior

Study 7 days in a row

Grinder

Earn 500 XP

Deep diver

Earn 1500 XP