Yashotantra Academy logo
← AI Security & Ops theorytheory 0/14 · 0%
level 1
Curious · 0 XP
streak
0d
perfect
0
badges
0

100 XP to Apprentice

Observability · medium

6. Tracing LLM applications

One trace per user request, with a span for every model and tool call.

1 min read2 questionsup to 50 XPtopic 6/14

Read in another language

AI translates the explanation and quiz text while preserving code and answer scoring.

An AI trace records more than timings: prompt, output, model, token counts, cost, latency, tool arguments, retrieved chunk ids, and quality scores. Without it you cannot answer "why did this user get that answer?".

ts
const trace = obs.trace({ name: "support.reply", userId, sessionId });
const retrieval = trace.span({ name: "retrieve", input: q });
const chunks = await search(q);
retrieval.end({ output: chunks.map((c) => c.id), metadata: { topScore: chunks[0]?.score } });

const gen = trace.span({ name: "generate", model: "openai/gpt-5.6-sol" });
const res = await gateway.chat({ ... });
gen.end({ usage: res.usage, cost: price(res.usage), output: res.text });
trace.update({ tags: ["rag"], scores: { grounded: await scoreGrounding(res.text, chunks) } });

Tools such as LangSmith, Langfuse, Phoenix, Helicone, and Weights & Biases provide this; OpenTelemetry GenAI semantic conventions let you keep AI spans in the same backend as the rest of your system. Redact prompt bodies before export when they may contain personal data, and sample heavy payloads rather than storing every one.

Exercise: instrument one endpoint, then answer three questions from the traces alone: p95 latency, cost per request, and the retrieval score distribution on runs the user rated bad.

How to study this topic

  1. 1. Read the explanation once end to end, skipping the code.
  2. 2. Re-read the code snippets and say out loud what each line does.
  3. 3. Answer the quiz without scrolling back — that is the real check.
  4. 4. Read every explanation, even for the answers you got right.

Scoring

  • · Up to 30 XP from quiz accuracy.
  • · +15 XP for a perfect first attempt.
  • · +5 XP while a 3-day streak is running.
  • · 75% or higher marks the topic complete.

Check your understanding

  1. 1. What belongs on a generation span besides latency?

  2. 2. How should traces handle prompts containing personal data?

Badges

First light
Pass your first topic
Sharp shooter
3 perfect first-try quizzes
On a roll
Study 3 days in a row
Week warrior
Study 7 days in a row
Grinder
Earn 500 XP
Deep diver
Earn 1500 XP