aifromzero.dev

Observability and Evaluation: Is Quill Any Good? (smolagents, Module 14)

Module 14 of 15 16 min read Lab code ↗

This is Module 14 of smolagents Mastery, a free 15-module course that takes you from your first code agent to a production-grade one. Start at Module 1 or browse the full syllabus.

The Quill you finished in Module 13 is deployed. It runs behind a Gradio UI where anyone uploads a CSV and asks a question, it lives on the Hub as a Space, and it launches from the smolagent CLI. It works.

Then a colleague uses it and pings you: “it choked on my CSV.” And you have nothing. No trace to open. No readable log. You cannot say which step failed, which tool it called, or how many tokens it burned before it gave up. So you do what everyone does at this stage: you edit an instruction “to make it better,” re-run one example by hand, decide it “looks better,” and ship. You just silently broke three other cases you never re-ran.

That is flying blind. “It looks like it works” is not a measurement, and a single hand-checked example is not a test. This module fixes both. We give Quill eyes — telemetry, so every run becomes an inspectable trace — and a grade — an eval harness with a golden set, a judge, a Task Success Rate, a cost-per-run number, and a regression gate. You stop believing Quill is good and start proving it.

In this module

You’ll learn

  1. Instrument a smolagents agent in one line with SmolagentsInstrumentor().instrument(), sending OpenTelemetry traces to Langfuse or Arize Phoenix — installed before you build the agent.
  2. Read a multi-agent trace: nested spans where the manager CodeAgent calls web_researcher, each LLM call, each tool call, each step’s token usage and latency.
  3. Distinguish outcome from trajectory evaluation, and pick the right metric: Task Success Rate, tool-call accuracy, trajectory efficiency, cost per run.
  4. Build an LLM-as-judge that scores a QuillReport against a rubric — evidence before score, structured output, calibrated against your own labels — without grading itself.
  5. Gate regressions: run a golden set, aggregate TSR + cost/run, and fail the build when either crosses a threshold.

You’ll build Quill becomes observable and measurable: quill/telemetry.py ships traces to Langfuse/Phoenix, and quill/eval/ (a frozen golden_set.json + run_evals.py) scores Quill with an LLM-as-judge on its QuillReport, reporting Task Success Rate and cost/run behind a regression gate.

Theory areas covered T12 — Production — 11% — this module owns the telemetry & evaluation half. (UI/deploy is Module 13; hardening is Module 15.) No “exam domain” — this course has no certification.

Prerequisites Module 4 (make_model, Monitor/TokenUsage, cost/run), Module 8 (QuillReport, final_answer_checks), Module 10 (manager + web_researcher, nested spans), Module 6 (agent.memory.steps, replay, in-process inspection). An HF_TOKEN in .env; for traces, a free Langfuse account or Phoenix running locally.

Where you are in the course

M1 ✅  M2 ✅  M3 ✅  M4 ✅  M5 ✅  M6 ✅  M7 ✅  M8 ✅  M9 ✅  M10 ✅  M11 ✅  M12 ✅  M13 ✅
👉 M14  ⬜ M15

Quill so far: deployed but flying blind — a Space, a UI, a CLI, and no way to know if it is any good. After this module: instrumented (every run emits a trace) and measured (a golden set, a judge, a TSR, a cost/run, and a regression gate). The chain is linear and this module is load-bearing in a specific way: M14 reuses Monitor/TokenUsage from Module 4 for cost/run, and reuses the QuillReport schema from Module 8 as the thing the judge scores. The eval format you freeze here is assembled, untouched, at the capstone — Module 15 runs the whole team inside a sandbox with these evals as its green light.

Give Quill eyes: OpenTelemetry tracing in one line

Right now, when a run goes sideways, your only recourse is agent.replay() after the fact — and only if the process is still alive and you are sitting at the terminal. That is fine for one debug run on your laptop. It is useless when a deployed Quill misbehaves for a user and the process is long gone. You need the run recorded, somewhere you can open it later. That is telemetry.

OpenTelemetry (OTel) is the vendor-neutral standard for emitting traces, metrics, and logs — a single instrumentation API that any backend can consume. smolagents emits OTel spans. OpenInference is the OTel-aligned semantic convention (maintained by Arize) that defines what those spans say for LLM apps — the prompt, the completion, the model, the tool calls, the retriever results. It ships the instrumentor that turns smolagents’ spans on. The whole thing is one call.

Here is the entire idea, backend-agnostic:

from openinference.instrumentation.smolagents import SmolagentsInstrumentor

SmolagentsInstrumentor().instrument()   # BEFORE you build/run the agent

Note the import. It is from openinference.instrumentation.smolagents import SmolagentsInstrumentor — not from smolagents import .... There is no SmolagentsInstrumentor on the smolagents package itself. Half the tutorials get this wrong; the instrumentor lives in the OpenInference package, which the [telemetry] extra pulls in. This is exactly what the lab’s smoke test asserts:

from openinference.instrumentation.smolagents import SmolagentsInstrumentor
inst = SmolagentsInstrumentor()
assert hasattr(inst, "instrument") and callable(inst.instrument)
assert hasattr(inst, "uninstrument")          # so a run/test can undo it
import smolagents
assert not hasattr(smolagents, "SmolagentsInstrumentor")   # the banned import shape

The one rule: instrument before you build the agent

Instrumentation works by patching smolagents’ classes. If you patch them after the agent is constructed and a run is already under way, the first steps have already executed — their spans are gone. You cannot retroactively trace a run that has started. This is the contract the whole module hangs on, and Quill’s quill/telemetry.py enforces it by calling instrument() in the entry point before build_quill(...).

The lab proves the ordering with a test that records the call sequence:

def test_main_entrypoint_instruments_before_building_the_agent(monkeypatch):
    import quill.__main__ as main_mod
    order = []
    monkeypatch.setattr(main_mod, "instrument", lambda: order.append("instrument"))
    monkeypatch.setattr(main_mod, "main", lambda: (order.append("build_and_run"), 0)[1])
    monkeypatch.setattr(sys, "argv", ["quill", "Which category grew fastest?"])
    main_mod._entrypoint()
    assert order == ["instrument", "build_and_run"]   # instrument() runs FIRST (06 §2)

Three backends, one instrumentor

The instrumentor is the same everywhere; only the exporter setup differs. Quill exposes the backend as an env var, QUILL_TELEMETRY ∈ {none, langfuse, phoenix}, defaulting to none so a run without a backend is never broken. Here is the real instrument(), lightly trimmed:

def instrument(backend: str | None = None) -> bool:
    resolved = resolve_backend(backend)          # QUILL_TELEMETRY, default "none"
    if resolved == "none":
        return False                             # a clean no-op: tracing stays OFF
    if resolved == "phoenix":
        from phoenix.otel import register
        register()                               # point the OTLP exporter at the local collector
    elif resolved == "langfuse":
        from langfuse import get_client
        get_client().auth_check()                # verify credentials BEFORE instrumenting
    from openinference.instrumentation.smolagents import SmolagentsInstrumentor
    SmolagentsInstrumentor().instrument()        # the ONE backend-agnostic call
    return True

Three things worth calling out. First, Langfuse — an open-source LLM observability platform, hosted cloud or self-hosted. You set LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY / LANGFUSE_HOST (https://cloud.langfuse.com for EU, https://us.cloud.langfuse.com for US), and get_client().auth_check() fails early with a clear error if the keys are wrong — far better than silently dropping every span. Second, Phoenix (Arize Phoenix) — a local-first tracing UI. You start the collector with python -m phoenix.server.main serve, call from phoenix.otel import register; register(), and open the traces at http://0.0.0.0:6006/projects/. No account needed; it runs on your machine. Third, MLflow autolog — a one-liner, mlflow.smolagents.autolog(), which traces runs, spans, inputs/outputs, and token usage. We mention it as the third path; the lab does not wire it.

A practical note on dependencies: the [telemetry] extra pulls arize-phoenix plus opentelemetry-sdk, opentelemetry-exporter-otlp, and openinference-instrumentation-smolagents>=0.1.15. If you only want Langfuse, you can install openinference-instrumentation-smolagents and langfuse alone and skip arize-phoenix entirely. The imports in instrument() are lazy on purpose — importing quill.telemetry never requires the extra; you pay for the import only when you actually instrument.

LangfuseArize PhoenixMLflowIn-process (replay)
Where it runsCloud or self-hostLocal collectorLocalNowhere — the process
SetupEnv keys + auth_check()phoenix.server.main serve + register()mlflow.smolagents.autolog()Nothing
Persists?YesYesYesNo — memory dies with the process
Via SmolagentsInstrumentor?YesYesNo (autolog)No
Use it whenA fleet of prod runs, comparison, alertingOne machine, no accountYou already live in MLflowOne debug run, right now

What a trace actually is

A trace is a tree of spans. A Quill run gives you a root span (the whole run), an LLM span for each model call, and a tool span for each tool call. On each span you read the latency, the token usage, and the content — the prompt that went in, the completion that came back, the tool arguments. It is the backend equivalent of the agent.replay() you learned in Module 6, except it is recorded, durable, and searchable across many runs.

And telemetry does not replace in-process inspection — they complement each other. Before any backend, smolagents already gives you agent.memory.steps (the trajectory), agent.write_memory_to_messages(), agent.replay(detailed=False), and agent.visualize(). The eval harness later in this module reads agent.memory.steps to count steps with no backend at all. One important freshness trap: the old agent.logs attribute was removed in 1.21.0. Any tutorial that reads agent.logs to inspect a run is dead — use agent.memory.steps / agent.replay(). The lab even asserts the attribute is gone:

assert not hasattr(agent, "logs")   # removed in 1.21.0

Reach for a backend when you have a fleet of runs to compare, regressions to catch over time, or alerting to wire. For one debug run at your desk, in-process replay() is enough.

⚠️ Common misconception: “Telemetry slows my agent down — I’ll add it later.”

Wrong on both counts. OTel instrumentation is near-free at runtime: spans are exported asynchronously, off the hot path. The cost is negligible compared to a single LLM call. And “later” means never having a trace when a run breaks in production — the one moment you actually need it. Worse, telemetry is not something you can bolt on mid-run: if you do not call instrument() before you build the agent, the first steps’ spans are already lost. The right time is now, at startup, before the agent exists. As of smolagents 1.26.0 the telemetry semantic conventions (OpenInference vs the parallel OpenTelemetry GenAI convention — both coexist and evolve) are still moving; the span attribute names may shift, so re-verify the OpenInference instrumentation version at build time. But the call — SmolagentsInstrumentor().instrument() before the agent — is stable.

Outcome vs trajectory: what to measure (and the spans that show it)

You can trace Quill now. But a trace tells you what happened, not whether it was good. To grade Quill you need to know what “good” even means — and there are two distinct families of answers.

Outcome evaluation asks: did the agent reach the goal? Is the answer correct? This is validation — pass/fail on the result. Trajectory evaluation asks: how did it get there? How many steps, which tools, in what order, at what cost? This is debugging and process improvement. A run can ace one and fail the other: Quill can return exactly the right report (great outcome) after a 20-step flailing loop that re-loaded the CSV four times (terrible trajectory). Both matter, for different reasons.

The core metrics fall out of this split:

  • Task Success Rate (TSR) — the fraction of golden-set tasks the agent gets right. The outcome headline. If 4 of 5 tasks pass, TSR is 0.80.
  • Tool-call accuracy — did the agent call the right tools with the right arguments? A trajectory metric: the answer can still be right while the agent fumbled which tool to use.
  • Trajectory efficiency — did it take a sane number of steps? A correct answer in 18 steps when 4 would do is a cost and latency problem, even with a perfect outcome.
MetricFamilyMeasuresWhat you do with itWhere you read it
Task Success RateOutcomeDid the report answer the question?Validate / gate releasesJudge + asserts
Tool-call accuracyTrajectoryRight tools, right args?Debug the agent’s choicesTrace / spans
Trajectory efficiencyTrajectorySteps vs the minimum neededCut cost and latencyagent.memory.steps / spans
Cost per runTrajectoryTokens burned per runBudget, catch cost creepMonitor (M4)

The trace is the trajectory

This is precisely why telemetry comes before evaluation in this module: the trace is the trajectory, materialized. On a Quill trace you read the number of steps, the order of tool calls, and every detour. And for a multi-agent run — Quill the manager CodeAgent delegating to web_researcher (the Module 10 team) — the sub-agent’s work shows up as nested spans. The web_researcher span is a child of the manager span, with the search and page-visit spans nested under it. That is the only sane way to debug a team: flat spans would scramble whose call was whose.

flowchart TD
  A["run span (the whole Quill run)"] --> B["LLM span — manager plans"]
  A --> C["tool span — profile_dataframe"]
  A --> D["web_researcher span (child of the manager)"]
  D --> D1["LLM span — researcher reasons"]
  D --> D2["tool span — web_search"]
  D --> D3["tool span — visit_webpage"]
  A --> E["tool span — save_chart"]
  A --> F["LLM span — manager writes the report"]

When web_researcher loops and burns its step budget, you open the trace, see the runaway nested span, and fix it. This kind of measurement is what let Hugging Face push a smolagents multi-agent system to the top of the GAIA leaderboard — you cannot improve what you cannot see.

Benchmarks: where Quill sits in the landscape

Public benchmarks are the conceptual reference points for “how good is an agent.” Three matter:

  • GAIA — general assistants. The showcase benchmark for smolagents and HF; their Open Deep Research system reportedly reaches 55.15% on GAIA (vs ~46% for the prior open-source state of the art) as of smolagents 1.26.0 — a published result, not a guarantee.
  • SWE-bench Verified — coding agents, fixing real GitHub issues.
  • WebArena — web automation, completing tasks in real web environments.

The pedagogical point: these benchmarks measure generalist agents. Quill is a domain agent — a data analyst over your CSVs. You do not run GAIA on Quill; that would tell you nothing about whether it gets your questions right. You build your own golden set — a small, curated set of representative questions with known-good answers. That is the transition to the harness.

The judge: scoring a QuillReport without it grading itself

Some outcomes have a checkable answer — “which category has the highest net_rev?” has a number. But “is this report any good?” does not. Coverage, grounding, readability — there is no exact match to assert against. For those, you use an LLM-as-judge: a separate model call that scores Quill’s output against an explicit rubric.

The judge is easy to do badly. The lab’s quill/eval/judge.py enforces four rules that the article teaches as non-negotiable.

1. An explicit rubric — never “rate this 1-10.” The judge scores three named axes, 0-2 each (max 6): does the report cover the golden item’s expected_points? Are the findings grounded in a chart and concrete numbers? Are sources cited [n] when the item asks for them? A bare “score 1-10” gives you noise; a numeric rubric gives you something repeatable.

2. Evidence-before-score. The judge writes its rationale first, then the scores. The order matters: a score posed before its justification is less reliable — the model rationalizes a number it already committed to. So the JSON is ordered rationale → scores → verdict. Here is the heart of the prompt builder:

return (
    "You are a STRICT, impartial evaluator of a data-analysis report. You did NOT write this "
    "report and you must NOT fix or re-run it — only SCORE what you are given.\n\n"
    f"QUESTION:\n  {item.get('question', '')}\n\n"
    f"EXPECTED POINTS the report must cover:\n{expected}\n\n"
    f"MINIMUM SOURCES expected: {min_sources}\n\n"
    "THE REPORT (rendered Markdown):\n-----\n"
    f"{report.to_markdown()}-----\n\n"
    "RUBRIC — score each axis 0, 1 or 2:\n"
    "  coverage : do the findings cover ALL the expected points? (0 none, 1 some, 2 all)\n"
    "  grounding: are the findings backed by a saved chart and concrete numbers? (0..2)\n"
    "  citations: when sources are expected, are claims cited [n]? (0..2)\n\n"
    "RULES:\n"
    "  - Write your RATIONALE FIRST (the evidence), THEN the scores. The order matters.\n"
    '  - Reply with ONE JSON object: {"rationale": "...", '
    '"scores": {"coverage": <0-2>, "grounding": <0-2>, "citations": <0-2>}, '
    '"verdict": "pass" | "fail"}\n'
)

3. Structured output. The judge returns parsable JSON {rationale, scores{...}, verdict}, not free prose. This reuses the spirit of Module 8’s response_format (we reuse the idea, not the schema). parse_judge_response is robust: it extracts the first {...} block even if the model wraps it in a code fence, defaults a missing axis to 0, and clamps every score into range so a misbehaving judge cannot invent a 9-out-of-2:

parsed = parse_judge_response("Here is my verdict:\n```json\n" + judge_json(coverage=9) + "\n```")
assert parsed["scores"]["coverage"] == 2     # clamped down from 9
# A totally unparsable reply raises — an actionable failure, not a silent zero:
with pytest.raises(ValueError):
    parse_judge_response("I think it's pretty good, maybe a 7?")

4. Calibration. Compare the judge’s scores to your human labels on ~10-20 examples and aim for a decent correlation — around 0.80 as an order of magnitude (as of the research, not an absolute gate). A judge you never calibrated measures its own bias. The lab ships calibration_correlation (a plain Pearson helper); it is documented and optional, not run on every eval — it is the periodic re-calibration tool you reach for as the judge model drifts.

assert calibration_correlation([1, 2, 3, 4], [1, 2, 3, 4]) == pytest.approx(1.0)
assert calibration_correlation([1, 1, 1], [3, 3, 3]) == 0.0   # constant -> no correlation

The judge reuses QuillReport — one schema, two jobs

The judge takes the frozen QuillReport from Module 8 as input and never modifies it. That is the elegant part: the same schema Quill validates at run time (via final_answer_checks) is scored a posteriori here. One schema, used twice — validated when produced, scored when evaluated. judge_report is a single, separate model call:

def judge_report(report: QuillReport, item: dict, model) -> dict:
    from smolagents import ChatMessage, MessageRole
    prompt = build_judge_prompt(report, item)
    messages = [ChatMessage(role=MessageRole.USER, content=prompt)]
    chat_message = model.generate(messages)        # ONE judge call (a SEPARATE model)
    return parse_judge_response(chat_message.content)

Never let an agent grade itself

This is the pitfall to hammer. The judge must be a separate call, ideally with a different and stronger model. If Quill grades Quill, it validates its own mistakes — it cannot see the bug it just produced. In the lab, the judge model goes through the same make_model factory (one model factory for Quill and the judge, per the frozen contract — Module 4 gained an optional model_id override exactly for this) but points at QUILL_JUDGE_MODEL_ID:

judge = make_model(role="judge", model_id="some/judge-model")
assert judge.model_id == "some/judge-model"      # the judge's SEPARATE model

Two more notes for completeness. The LLM-as-judge scores the final output. Its cousin, Agent-as-a-judge (arXiv 2508.02994), observes the intermediate steps and tools to score the whole trajectory, not just the answer — the trajectory side of the outcome/trajectory split. We mention it as a concept; the lab implements only the output judge. And conceptually, this whole setup is Anthropic’s evaluator-optimizer pattern composed by hand: one model generates (Quill), another evaluates (the judge). It is not a smolagents primitive — you wire it yourself.

flowchart LR
  G["golden_set.json"] --> R["run_quill(question, dataset)"]
  R --> Q["QuillReport (M8 schema)"]
  Q --> J["judge(report, expected_points)"]
  R --> M["Monitor (M4) + memory.steps (M6)"]
  J --> S["task_success · report_quality · citations"]
  M --> T["steps · cost"]
  S --> AGG["aggregate: TSR, cost/run"]
  T --> AGG
  AGG --> GATE{"regression gate<br/>TSR ≥ floor · cost ≤ budget"}
  GATE -->|pass / fail| OUT["eval/results/run-&lt;name&gt;.json"]

The eval harness: golden set, cost/run, and a regression gate

Now assemble the pieces into one reproducible script. The format is frozen (the capstone, Module 15, reuses these keys verbatim — adding a key means stop and update the spec first), so it is worth seeing exactly.

The golden set is a list of {id, question, dataset, expected_points[], min_sources}. dataset points at a CSV (data/sales.csv by default — inherited from Module 2, never renamed); expected_points[] are the facts the report must contain (the judge scores them); min_sources is the minimum number of cited sources expected. Here is one of the five shipped items — a question whose right answer requires retrieving a definition, so it asks for a source:

{
  "id": "net-rev-definition",
  "question": "What does the net_rev column mean, and what is Team's total net_rev?",
  "dataset": "data/sales.csv",
  "expected_points": ["net_rev is net revenue: gross revenue minus refunds",
                      "Team has the highest total net_rev"],
  "min_sources": 1
}

The loader fails loud on a malformed item — the format is a contract, not a suggestion:

required = {"id", "question", "dataset", "expected_points", "min_sources"}
missing = required - set(item)
if missing:
    raise ValueError(f"golden_set item {item.get('id')!r} is missing frozen keys {sorted(missing)}")

The five score columns

For each item, run_evals.py builds Quill, runs build_report_task(dataset, question) to get a QuillReport, and scores it on five columns. Three are outcome, two are trajectory — and each is computed deliberately:

# citations: DETERMINISTIC, no LLM — did the report carry at least min_sources sources?
citations = 1 if len(report.sources) >= min_sources else 0

# the judge scores the outcome axes (coverage drives task_success; total -> report_quality):
judged = judge_report(report, item, judge_model)
coverage = judged["scores"].get("coverage", 0)
task_success = 1 if coverage >= TASK_SUCCESS_COVERAGE_THRESHOLD * (JUDGE_RUBRIC_MAX // 3) else 0

return {
    "id": item["id"],
    "task_success": task_success,                 # OUTCOME (judge)
    "report_quality": judged["report_quality"],   # OUTCOME (judge rubric total, 0..6)
    "citations": citations,                       # OUTCOME (deterministic, no LLM)
    "steps": count_action_steps(agent),           # TRAJECTORY (memory.steps, M6)
    "cost": run_cost_tokens(agent),               # TRAJECTORY (Monitor, M4)
}
ColumnFamilyMeasuresComputed byIn the gate?
task_successOutcomeCovers the expected_points?JudgeYes (drives TSR)
report_qualityOutcomeRubric total / 6JudgeNo (reported)
citationsOutcomelen(sources) >= min_sourcesDeterministic, no LLMNo (reported)
stepsTrajectoryTrajectory lengthagent.memory.stepsNo (reported)
costTrajectoryTotal tokens this runMonitorYes (cost/run)

Notice citations is deterministic — pure Python against QuillReport.sources, no LLM in play. You do not spend a model call to count a list. The lab proves it scores 1/0 with no model touched. And the trajectory columns use only supported accessors, never the removed agent.logs or the token attributes dropped in 1.21.0:

def count_action_steps(agent) -> int:
    from smolagents import ActionStep
    steps = getattr(getattr(agent, "memory", None), "steps", None) or []
    return len([s for s in steps if isinstance(s, ActionStep)])   # M6, never agent.logs

def run_cost_tokens(agent) -> int:
    monitor = getattr(agent, "monitor", None)
    if monitor is None:
        return 0
    usage = monitor.get_total_token_counts()       # M4 — the Monitor accessor
    return int(getattr(usage, "total_tokens", 0) or 0)

Cost/run is not cosmetic

Monitor (Module 4) aggregates the run’s token usage; Monitor.get_total_token_counts() returns a TokenUsage carrying input/output/total tokens. Cost per run is tokens × the provider’s price — and with the HF free tier at roughly $0.10/month (as of smolagents 1.26.0, subject to change), a 10-question multi-step golden set can eat a real slice of that. And remember: one eval run = N Quill runs + N judge calls, because the judge also costs tokens. Measuring cost/run is the budgeting recipe most agent tutorials skip — and exactly why the gate watches it.

The regression gate

After aggregating, the harness compares the result against thresholds and exits non-zero if any fail — so it drops straight into CI. The regression gate passes only if TSR >= min_tsr and cost_per_run <= max_cost:

def apply_regression_gate(aggregate, *, min_tsr=0.70, max_cost=math.inf):
    reasons = []
    if aggregate["TSR"] < min_tsr:
        reasons.append(f"TSR {aggregate['TSR']:.2f} < min {min_tsr:.2f}")
    if aggregate["cost_per_run"] > max_cost:
        reasons.append(f"cost/run {aggregate['cost_per_run']:.0f} > budget {max_cost:.0f}")
    return (not reasons, reasons)        # the CLI turns a fail into sys.exit(1)

The point lands when you compare two runs. A “small prompt tweak” that drops TSR from 0.8 to 0.6, or bumps cost/run 40%, fails the gate — caught before deploy. Run run-baseline.json, make your change, run run-candidate.json, and the gate is the difference. A manual test on one example would never have seen it.

The defaults are deliberate: QUILL_EVAL_MIN_TSR floors at 0.70, and QUILL_EVAL_MAX_COST is off by default (+inf) so the gate only checks cost when you opt in with a token budget. One scope reminder: this eval runs on Quill’s current QUILL_EXECUTOR (no forced remote sandbox) and does not harden Quill — the full assembly (Approach 2 + evals green) is the capstone, Module 15.

Build it

The lab makes Quill observable and measurable. The full code is at smolagents-course-labs/module-14; every snippet here is drawn from code that passes pytest offline. Two new pieces: quill/telemetry.py (the instrumentor wiring) and quill/eval/ (the golden set, the judge, the harness). quill/agent.py and quill/report.py are untouched — telemetry and eval live in their own modules, around the agent, not inside it.

First, install the telemetry extra (the eval harness itself needs no backend — it scores via Monitor + agent.memory.steps):

uv pip install 'smolagents[telemetry]==1.26.0'
# pulls arize-phoenix + opentelemetry-sdk + opentelemetry-exporter-otlp +
# openinference-instrumentation-smolagents>=0.1.15
# Langfuse-only (skip arize-phoenix): pip install openinference-instrumentation-smolagents langfuse

Copy .env.example to .env (never commit it). The entry point in quill/__main__.py calls instrument() first, then dispatches — so a normal run with QUILL_TELEMETRY=phoenix emits a trace, while the default none behaves exactly as before:

def _entrypoint() -> int:
    instrument()                    # M14: BEFORE building the agent (06 §2); no-op when none
    argv = sys.argv[1:]
    if "--ui" in argv:
        from .ui import launch_ui
        launch_ui(share="--share" in argv)
        return 0
    return main()

To watch a real trace, start Phoenix and run Quill against it:

python -m phoenix.server.main serve                  # UI at http://0.0.0.0:6006/projects/
QUILL_TELEMETRY=phoenix uv run python -m quill "Which category grew fastest?" --data data/sales.csv

Open the trace and you see the root run span, an LLM span per call, a tool span per call, and — if the run delegated — the web_researcher span nested under the manager.

Now run the harness — the observable result of the module:

uv run python -m quill.eval.run_evals --out eval/results/run-baseline.json

It runs Quill on each golden-set item, scores each run (judge + deterministic checks), writes the JSON, and prints a summary:

Golden set: 5 tasks · model=Qwen/Qwen2.5-Coder-32B-Instruct
TSR: 0.80 (4/5)  ·  avg report_quality: 4.6/6  ·  avg steps: 6.2  ·  cost/run: ~12.4k tokens
Regression gate: PASS (TSR>=0.70, cost/run no cap)

The process exits 0 when the gate passes, non-zero when it fails — so the same command is your CI quality check. To see a regression caught: tweak an instruction “to improve it,” write run-candidate.json, and compare. A drop in TSR or a jump in cost/run fails the gate.

Run the tests — fully offline (no token, no network, no backend), with one live test gated behind a flag:

uv run pytest module-14/tests/                            # offline: judge, gate, citations, no-op telemetry
QUILL_LIVE_TESTS=1 uv run pytest module-14/tests/ -m live # ONE real golden item (Quill + judge)

The offline suite runs the whole golden set on a fake-model Quill and a fake-model judge — zero LLM calls — and asserts the frozen result keys, the structured judge output, the deterministic citations, the non-zero gate exit, and the clean telemetry no-op. The single live test runs one real item end to end (1 Quill run + 1 judge call); it skips cleanly without HF_TOKEN.

Try it yourself. (1) Run the harness with two different model_ids for Quill on the same golden set, then compare run-modelA.json vs run-modelB.json — is the cheaper model worth its TSR? (2) Add a trap question with min_sources >= 1 (needs a web source), run the harness once with the team and once with build_quill(managed_agents=[]), and verify the gate (via citations) catches the degradation when you disable web_researcher.

In production

At scale, the rule becomes: you do not deploy without a gate. An eval harness in CI turns “I hope it works” into “the build failed: TSR dropped from 0.8 to 0.6.” Cost/run becomes a budget line, not an end-of-month surprise — though watch that the harness itself costs money: a 20-question multi-step golden set on every PR (each item = one Quill run plus one judge call) can be expensive; measure it and cap it.

Telemetry in production is not a luxury. Without a trace, debugging a multi-agent run that burned 30 steps is misery; with Langfuse or Phoenix you open the trace, see the web_researcher span that looped, and fix it. Privacy is the catch: traces capture prompts and data — a user’s CSV can leave for a cloud backend. In an enterprise, self-host Langfuse/Phoenix or mask sensitive inputs. The judge is fallible too — length bias, format bias, position bias, model drift — so re-calibrate it periodically against human labels, and remember it spends tokens on every eval. Full hardening (timeouts, step caps, bounded retries, the whole team inside a sandbox) is the capstone, Module 15.

Concept check

Why this matters. This module is the telemetry & evaluation half of T12 — the difference between an agent that “seems fine” and one you can defend with numbers. You should be able to say where instrumentation must go, tell an outcome problem from a trajectory problem, name the four rules of a trustworthy judge, read a multi-agent trace, and explain what a regression gate catches that a manual test cannot.

Work through these five, then check your answers below.

1. A developer adds SmolagentsInstrumentor().instrument() after agent.run(...) has already started, and notices the first few steps are missing from the trace. What is wrong, and what is the correct import?

  • A. The trace is fine; early spans always render last — wait for the run to finish.
  • B. Instrumentation patches smolagents’ classes, so it must run before the agent is built/run; the import is from openinference.instrumentation.smolagents import SmolagentsInstrumentor.
  • C. They need from smolagents import SmolagentsInstrumentor instead — the OpenInference package is for Phoenix only.
  • D. They must set QUILL_TELEMETRY=all to capture early steps.

2. Quill returns the correct report for a question, but it took 20 steps and re-loaded the CSV three times. How do you characterize this, and where do you see it?

  • A. Bad outcome, good trajectory — fix the answer.
  • B. Good outcome (TSR passes), bad trajectory (poor trajectory efficiency / high cost/run); you see it in the trace and agent.memory.steps.
  • C. A tool-call accuracy failure only — the answer is wrong.
  • D. Nothing is wrong; correct answers are all that matter.

3. A teammate evaluates Quill by asking Quill itself to “rate this report from 1 to 10.” What is wrong, and how do you fix it?

  • A. Nothing — self-grading with a 1-10 scale is the standard LLM-as-judge.
  • B. Only the scale is wrong; switch to 1-100 for more resolution.
  • C. Multiple problems: self-grading (the agent validates its own mistakes), no explicit rubric, no evidence-before-score, no calibration. Fix: a separate, ideally stronger judge model with a numeric rubric, rationale before score, structured output, calibrated against human labels.
  • D. The judge should not see the report at all — only the question.

4. On a multi-agent run where the manager delegates to web_researcher and the researcher loops, where in the trace do you look?

  • A. The spans are flat and unordered — search by timestamp.
  • B. The web_researcher span is a child of the manager span; expand it to read the sub-agent’s tool calls, latency, and token usage.
  • C. The researcher’s work never appears in the manager’s trace — trace it separately.
  • D. Look at agent.logs for the sub-agent’s steps.

5. A “small prompt tweak” raises cost/run 40% and breaks 1 of 5 golden-set cases. What catches this, and why does a quick manual test miss it?

  • A. Nothing automated can catch this — only careful code review.
  • B. A golden set + regression gate: run the full set, aggregate TSR + cost/run, and exit non-zero when TSR drops below the floor or cost exceeds budget. A manual test on one example never re-runs the other four, so it misses the broken case and the cost creep.
  • C. The final_answer_checks from Module 8 catch it at run time.
  • D. Telemetry alone catches it — the trace turns red.

Answers.

1 — B. Instrumentation works by patching smolagents’ classes; do it before the agent exists or the first steps’ spans are gone. The instrumentor is imported from openinference.instrumentation.smolagents, not from smolagents (which has no such attribute). C is the banned import shape; A and D are made up.

2 — B. The outcome is good (the report is correct, so TSR passes), but the trajectory is poor — 20 steps and repeated reloads are a trajectory-efficiency and cost problem. You read it in the trace (nested spans) and agent.memory.steps. A and C misread a correct answer as wrong; D ignores cost and latency.

3 — C. This breaks every rule at once. Self-grading lets the agent rubber-stamp its own errors; “1 to 10” with no rubric is noise; there is no evidence-before-score and no calibration. The fix is a separate (ideally stronger) judge model, a numeric rubric, rationale before scores, structured JSON, and calibration against your labels.

4 — B. A multi-agent trace nests spans: the sub-agent’s span is a child of the manager’s, with its tool calls nested under it. That hierarchy is exactly how you debug a team. C is false (the delegation is in the same trace); D uses the removed agent.logs.

5 — B. This is the textbook case for a golden set plus a regression gate: aggregate TSR and cost/run across the whole set and fail the build when either crosses its threshold. A manual one-example test never re-runs the others, so it sees neither the broken case nor the cost creep. C is wrong — final_answer_checks validate one run’s content, not a fleet’s quality regression; D overstates telemetry, which records but does not score.

Common pitfalls

  • Using agent.logs to inspect a run. It was removed in 1.21.0 — use agent.memory.steps / agent.replay(). And do not assume a backend replaces in-process inspection: telemetry is for fleets and history; replay() is for the run in front of you. Both have their place.
  • A judge that grades itself or rates “1 to 10.” That is a judge measuring its own bias. Always: an explicit numeric rubric, evidence-before-score, structured output, and calibration — ideally a different, stronger judge model. A judge LLM is fallible (length, format, position bias) and drifts, so recalibrate periodically.
  • Confusing outcome with trajectory — and forgetting the judge’s own cost. Outcome is “did it succeed?” (TSR); trajectory is “how?” (tool-call accuracy, trajectory efficiency, cost/run). And every eval run pays for N Quill runs + N judge calls — the judge spends tokens too. Count it.

Key takeaways

  • Instrument before you build. SmolagentsInstrumentor().instrument() (imported from openinference.instrumentation.smolagents, not smolagents) must run before the agent is constructed, or the first steps’ spans are lost.
  • One instrumentor, many backends. The same call feeds Langfuse, Arize Phoenix, or any OTLP collector; MLflow has its own autolog() one-liner. And in-process inspection (agent.replay() / agent.memory.steps, never the removed agent.logs) complements a backend — it does not replace it.
  • Outcome vs trajectory. Outcome = did it succeed (Task Success Rate). Trajectory = how (tool-call accuracy, trajectory efficiency, cost/run). A run can ace one and fail the other; the trace is the trajectory.
  • A multi-agent trace is nested spans. The web_researcher span is a child of the manager span — the only sane way to debug a team.
  • The LLM-as-judge has four rules: explicit rubric, evidence-before-score, structured output, calibration — and it never grades itself (use a separate, ideally stronger model). The judge scores the frozen QuillReport schema from Module 8: one schema, validated when produced and scored when evaluated.
  • An eval harness = golden set + judge + TSR + cost/run (via Monitor, M4) + a regression gate. The gate exits non-zero when TSR drops or cost exceeds budget — so a “small prompt tweak” cannot silently make Quill worse.
  • Public benchmarks (GAIA, SWE-bench Verified, WebArena) measure generalists. A domain agent like Quill gets its own golden set, not GAIA.

What’s next

Quill is now observable and measured. Module 15 — the capstone — assembles everything: the full multi-agent team running inside a sandbox (Approach 2), telemetry on, evals green, shipped as a Space, tagged v1.0.

Continue to Module 15 — Capstone: Ship Quill, a Production-Grade Code Agent (next), revisit Module 13 (deploy), or browse the full syllabus. The complete, tested code for this module is at smolagents-course-labs/module-14 — the tests pass offline, no token required.

Want the full smolagents concept-check bank and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

References