Capstone: Ship Quill, a Production-Grade Code Agent (smolagents, Module 15)

This is Module 15 of smolagents Mastery, a free 15-module course that takes you from your first code agent to a production-grade one. Start at Module 1 or browse the full syllabus.

Fourteen modules in, Quill “works.” Hand it a CSV and a question and it profiles the data, writes pandas, draws a chart, retrieves a column definition from the corpus, delegates web research to a sub-agent, re-reads its own chart with a vision model, and returns a cited report — wrapped in a Gradio UI, instrumented with OpenTelemetry, and scored against a golden set. Impressive. Now look closer.

The multi-agent team still runs in executor_type="local", which — you learned this in Module 5 — is not a security sandbox. There is no retry ceiling, so a sub-agent that loops can quietly drain your free tier. Two identical runs overwrite nothing and double-count cost, because chart names are timestamps. That is the gap between “runs on my laptop” and “I can hand it to someone else.” And the missing piece bites hard: you want a sandbox’s isolation and your multi-agent team — but in smolagents 1.26.0, pairing a remote executor_type with managed_agents raises an exception.

This module closes the gap. You assemble the fourteen pieces, flip to Approach 2 (the whole system inside the sandbox), harden it, gate the release on evals, and tag v1.0.

In this module

You’ll learn

Assemble the fourteen pieces you built — model factory, tools, sandbox policy, memory callbacks, planning, structured reports, MCP, multi-agent team, RAG, vision, UI, telemetry, evals — into one coherent, runnable system.

Run the full multi-agent Quill inside a sandbox with Approach 2 — creating the sandbox by hand and running manager + sub-agents within it — because a remote executor plus managed_agents raises an exception.

Harden the agent: bounded max_steps, execution timeouts, retries capped at a number you can defend, and idempotent runs that don’t double-charge or double-write.

Ship a defensible release: telemetry on, the eval gate green, a Gradio Space, cited QuillReports, a v1.0 tag — and a reusable smolagents production checklist.

Decide where to go next — and when not to reach for an agent at all.

You’ll build Quill v1.0: the whole multi-agent system inside a sandbox (Approach 2), telemetry on, evals green, a Gradio Space, cited reports — tagged v1.0.

Theory areas covered Capstone — synthesis across all 12 theory areas. This module owns the production-hardening and Approach-2 sandboxing entries (T5 — secure code execution 12%, T10 — multi-agent constraint, part of the 7% area, T12 — production 11%) and reviews everything else. (No “exam domain” — this course has no certification.)

Prerequisites All of Modules 1–14. Especially Module 5 (sandbox, Approach 1/2, the remote-executor constraint), Module 10 (multi-agent, managed_agents), Module 13 (GradioUI/Space), Module 14 (telemetry + eval harness). An HF_TOKEN configured in .env; Docker installed (or an E2B account) for Approach 2.

Where you are in the course

M1 ✅  M2 ✅  M3 ✅  M4 ✅  M5 ✅  M6 ✅  M7 ✅  M8 ✅  M9 ✅  M10 ✅  M11 ✅  M12 ✅  M13 ✅  M14 ✅
👉 M15 (finale)  —  nothing after this

Quill before this module: complete but scattered — every part works, but the multi-agent team is stuck on local, there is no hardening, and there is no release tag. Quill after: assembled, hardened, multi-agent running inside a sandbox (Approach 2), shipped as v1.0.

This is the last link in a linear chain. Module 15 adds no new feature. It only connects and hardens what the previous fourteen built. The whole point of the capstone is that there is nothing new to learn about what Quill does — only about how you make it safe, bounded, observable, and defensible enough to ship.

From fourteen parts to one system (a guided recap)

Before you assemble anything, the assembly map. Each layer of Quill is owned by one earlier module; the capstone just wires them together. (No re-teaching here — each line points to the module that owns the theory.)

Layer	What it is	API you wired	Owner
Model	One env-driven model factory	`make_model(role="analyst")`	M4
Loop	ReAct, code-as-action	`CodeAgent`	M1/M2
Tools	Data toolbox + interop + RAG	`load_dataset` / `profile_dataframe` / `save_chart`, MCP, `RetrieverTool`	M3/M9/M12
Sandbox	Where code runs, what it imports	`resolve_executor()`, `QUILL_EXECUTOR`, locked imports	M5
Memory	Prune stale dumps, log cost	`step_callbacks=quill_callbacks()`	M6
Planning + reliability	Re-centre; validate the answer	`planning_interval`, `final_answer_checks` → `QuillReport`	M7/M8
Team	Manager over specialists	`managed_agents=[web_researcher]` (+ `vision_browser`)	M10/M11
Ship	UI + observability + quality	`GradioUI`, `instrument()`, eval gate	M13/M14

The library that holds all of this together is roughly a thousand lines of code — smolagents is small on purpose, which is why you could read its source while you learned. That smallness is also why “assembly” here is honest: there is no hidden framework magic to debug, only the parts you wrote.

One construction function ties the toolbox, model, sandbox policy, memory, planning, report contract, and team into a single agent. It lives in quill/agent.py and its signature has only grown across the course — never broken:

def build_quill(
    model: Model | None = None,
    *,
    planning_interval: int | None = None,   # M7
    instructions: str | None = _DEFAULT,    # M7
    final_answer_checks: list | None = _DEFAULT_CHECKS,  # M8
    use_structured_outputs_internally: bool = False,     # M8
    extra_tools: list[Tool] | None = None,  # M9
    managed_agents: list | None = _DEFAULT_TEAM,  # M10
    browse: bool = False,                   # M11
    retrieve: bool = True,                  # M12
) -> CodeAgent:

Read that signature as a course summary. M15 adds no new parameter — it hardens around this function and never moves it.

Before you assemble: do you even need an agent? (T2.6)

The capstone has to say this out loud, because shipping an agent is the moment it costs real money and real risk. smolagents’ own guidance is “regularize toward not using agentic behaviour”; Anthropic’s is “the best agentic systems are the simplest.” You reach for an agent only when a deterministic workflow genuinely can’t cover the cases.

Quill is a legitimate candidate: open-ended data analysis is not a pipeline you can fully specify in advance — the columns, the joins, the chart, and whether you need a web benchmark all depend on the question. A fixed script would either be wrong or impossibly long. So Quill earns its agency. But ship an agent because the task demands it, never because agents are exciting. That discipline is the first item on the production checklist, not the last.

⚠️ Common misconception: “Shipping = wrapping it in a UI and pushing a Space.”

No. A UI on an un-isolated, un-hardened, un-measured system is not “production” — it is a demo exposed to the internet. Shipping is four things: isolation (Approach 2), guard-rails (timeouts, step caps, bounded retries, idempotence), observability (telemetry you can read), and a defensible quality signal (the eval gate). The UI is the last layer, not the first. If you only remember one sentence from this capstone, make it this one.

The last missing piece: running the whole team inside a sandbox (Approach 2)

Here is the constraint the entire course has been pointing at. Quill’s team has run in executor_type="local" since Module 10 — on purpose, but with a known cost.

`local` is not a sandbox (T5.7)

The LocalPythonExecutor is an AST allow-list interpreter: it restricts imports and built-ins to shrink the attack surface. But the library’s own docstring is blunt — “it is not a security sandbox.” The classic escape: a data tool like Pillow can be coaxed into filling the disk, because limiting imports is not the same as limiting what allowed code can do. So as long as Quill’s multi-agent team runs in local, it executes LLM-generated Python with no real isolation. For a publicly exposed agent that is unacceptable.

Why you can’t just flip `QUILL_EXECUTOR=docker` (T10.7)

The obvious fix — set a remote executor — does not work for a team. Building the team with a remote executor raises immediately. The guard is right there in smolagents 1.26.0’s create_python_executor:

if self.managed_agents:
    raise Exception("Managed agents are not yet supported with remote code execution.")

You can confirm the exact constraint offline, without ever starting a container, by trying to build Quill’s default team with a remote executor set:

import os
os.environ["QUILL_EXECUTOR"] = "docker"   # a remote executor + the default team
from quill.agent import build_quill

# The guard fires inside CodeAgent.__init__ -> create_python_executor — before any container.
build_quill()

Exception: Managed agents are not yet supported with remote code execution.

Why does this guard exist? Because the remote-executor path is Approach 1 (snippet-in-sandbox): the model and the agent stay local, only the generated Python snippets travel to the container, and secrets are never shipped into the box. A sub-agent that needs to authenticate its own LLM call could not — its HF_TOKEN is on the wrong side of the boundary. So Approach 1 structurally cannot do multi-agent. This is the structuring constraint announced back in M5/M10; the capstone is where you resolve it.

The two approaches (T5.9 / T5.16)

There are exactly two ways to put a CodeAgent’s code execution behind a real boundary.

	Approach 1 (snippet-in-sandbox)	Approach 2 (system-in-sandbox)
What runs in the sandbox	Only the generated Python snippets	The entire agent — manager + sub-agents
How you turn it on	One parameter: `executor_type="docker"`	You create the sandbox by hand and run the agent inside it
Multi-agent supported	No (raises)	Yes
Secrets in the sandbox	No (model stays local)	Sometimes — you pass `HF_TOKEN` in
Effort	Trivial	Manual orchestration
Quill’s use	M5 demo (single-agent)	M15 capstone (the team)

Approach 2 is the answer: you create the sandbox yourself — a hardened Docker container, or an e2b_code_interpreter.Sandbox() — copy the quill package and data/ in, pass HF_TOKEN as a container env var, and run build_quill(...).run(...) inside. The manager and its sub-agents all execute in the box; the only things crossing the boundary are the question (in) and the serialized QuillReport (out). That is the only way isolation and multi-agent coexist.

The best-practice hierarchy (T5.16) falls out of this: a trusted, well-known model on trusted input can run local (low risk, not zero); an untrusted or less-trusted model warrants a remote sandbox (Approach 1); a multi-agent system that needs isolation requires Approach 2 (the whole system in the sandbox). Quill sits squarely in the third case.

Here is the target architecture, materialized: the whole system lives inside the sandbox.

flowchart TB
    user([User: question + CSV]) --> ui["GradioUI / Space (M13)<br/>upload, stream, reset=False"]
    ui --> sandbox

    subgraph sandbox["SANDBOX — Docker / E2B (Approach 2, M15)"]
        direction TB
        manager["Quill = manager CodeAgent (M2-M10)<br/>planning_interval (M7) · final_answer_checks → QuillReport (M8)<br/>executor_type=local INSIDE the box"]
        manager --> tools["data tools (M3): load_dataset, profile_dataframe, save_chart"]
        manager --> rag["RetrieverTool (M12) → corpus docs"]
        manager --> mcp["MCP tools (M9) → data server"]
        manager --> team["managed_agents (M10):<br/>web_researcher · [option] vision_browser (M11)"]
    end

    sandbox --> report["QuillReport (M8)<br/>findings · chart_paths · sources[n] · caveats"]
    sandbox -. spans .-> telemetry["OpenTelemetry → Langfuse / Phoenix (M14)"]

Note the detail inside the box: the inner agent stays executor_type="local". The container is the isolation boundary, so re-nesting a remote executor inside it would be pointless — and, with the team present, it would re-raise the very exception you just dodged. The sandbox is the boundary; the agent runs local within it.

⚠️ Honest security note. Even Approach 2 is not a force field. The docs are explicit: “no solution will be 100% safe.” Approach 2 isolates the system, but the web and tool content it ingests is still a vector — prompt injection (M5) lives in the data, not the executor. That is exactly why the next section hardens the run and why the final_answer_checks (M8) and the minimal import lock (M5) stay on inside the box.

Hardening: timeouts, step caps, bounded retries, idempotence

Hardening is the work of turning “it ran once” into “it can’t run away.” It has four levers, each with a real smolagents API and a number you can defend (T12.14). These all live in the one new file of the capstone, quill/runtime.py.

Lever	Mechanism (smolagents 1.26.0)	Quill’s setting
Step caps	`max_steps` on `MultiStepAgent` (library default 20)	manager `max_steps=8`, `web_researcher` `max_steps=10`
Timeouts	`LocalPythonExecutor` caps one execution at `MAX_EXECUTION_TIME_SECONDS = 30`; Docker adds resource limits	30s/exec + container `mem`/`cpu`/`pids` caps
Bounded retries	App-level cap on re-runs (atop `ApiModel`’s own `retry`)	`MAX_RETRIES = 2`
Idempotence	Deterministic output paths from `(question, dataset)`	`outputs/quill-<sig>.png`

Step caps. build_quill already bounds max_steps=8 on the manager and 10 on the web_researcher. An agent that does not converge raises AgentMaxStepsError (M8) rather than looping forever. The capstone simply states this is the reliability ceiling and does not raise it — a single-CSV job needs far fewer than the library’s default of 20.

Timeouts. A single sandboxed execution is already capped at 30 seconds by MAX_EXECUTION_TIME_SECONDS (M5). In Approach 2, the Docker container adds resource limits so a pathological pandas merge cannot exhaust the host. The hardening flags are taken exactly as the smolagents secure-execution guidance prescribes:

DOCKER_HARDENING = {
    "mem_limit": "512m",                    # a runaway merge can't eat the host
    "cpu_quota": 50000,                     # 50% of one CPU (cpu_period defaults to 100000)
    "pids_limit": 100,                      # a fork bomb hits this wall
    "security_opt": ["no-new-privileges"],  # no setuid escalation inside the container
    "cap_drop": ["ALL"],                    # drop every Linux capability
}
SANDBOX_USER = "nobody"                     # the LLM's code never runs as root

build_hardened_container_kwargs() returns a fresh copy of those flags plus user="nobody", so a caller can tweak one without mutating the module constant. None of this is “100% safe” — it is defense in depth around a boundary that already exists.

Bounded retries. smolagents already retries at the ApiModel level; at the application level you cap how many times a whole run is relaunched. The helper re-raises the last error rather than swallowing it — there is no silent try/except:

def run_with_bounded_retries(run_fn, *, max_retries: int = MAX_RETRIES):
    if max_retries < 0:
        raise ValueError(f"max_retries must be >= 0, got {max_retries}.")
    attempts = max_retries + 1  # the first try plus the bounded retries
    last_exc: Exception | None = None
    for attempt in range(1, attempts + 1):
        try:
            return run_fn()
        except Exception as exc:  # bounded loop — we re-raise the last one, never mask it
            last_exc = exc
            print(f"[quill.runtime] run attempt {attempt}/{attempts} failed: "
                  f"{type(exc).__name__}: {exc}")
    assert last_exc is not None
    raise last_exc

The number is an opinion with a reason. Cap retries at 2 — a transient blip (a flaky read, a sandbox that didn’t warm up) deserves one retry; a run that fails twice is almost always a broken loop, and a third attempt just burns tokens. A negative cap is a bug, so it fails loud.

Idempotence. A re-run of the same question must not double-write charts or double-count cost. The signature is a pure function of (question, dataset) — sha256, not Python’s hash(), so it’s stable across processes:

def run_signature(question: str, dataset: str) -> str:
    payload = f"{question.strip()}\x00{dataset.strip()}".encode("utf-8")
    return hashlib.sha256(payload).hexdigest()[:12]

run_quill_report (in agent.py) sets that signature before the run and clears it after in a finally. While it is active, an un-named save_chart writes outputs/quill-<sig>.png; a re-run overwrites the same file instead of leaving chart-<timestamp>.png litter. Crucially, save_chart’s frozen M3 signature is untouched — the auto-name is an internal, env-driven addition, and an explicit filename always wins. You can watch idempotence end to end with a fake-model Quill:

q = "Which category has the highest net_rev?"
out  = run_quill_report(build_quill(model=fake), CSV, q)
out2 = run_quill_report(build_quill(model=fake), CSV, q)  # same (question, dataset)
assert out.chart_paths == out2.chart_paths   # outputs/quill-<sig>.png — same file, overwritten

Reading a run without a backend

You will not always have Langfuse or Phoenix in front of you. The supported in-process inspection (T12.14) is built into every agent: agent.memory.steps (the step list), agent.replay() (a pretty re-print), agent.visualize() (the agent’s structure tree), and agent.memory.get_full_steps().

Do not write agent.logs. That attribute was removed in smolagents 1.21.0. Plenty of older write-ups still show it as the way to inspect a run — it is dead. Always reach for agent.memory.steps / agent.replay(). The whole Quill package and the sandbox entrypoint are scanned by a test to guarantee agent.logs appears nowhere as live code.

Open Deep Research, and your production checklist

The architecture you built scales (T12.13)

You did not build a toy. The architecture you just assembled — an orchestrator-workers manager, code-as-action, a sandbox — is the same one Hugging Face used to reproduce OpenAI’s “Deep Research,” open-source, on top of smolagents. As reported (these numbers are reported, not guaranteed):

The code-action format used roughly 30% fewer tokens than equivalent JSON tool-calling — the M1/M2 thesis, validated at scale.
It pairs a text browser + text inspector (adapted from Magentic-One) with autonomous multi-step browsing, and a vision browser for image-heavy pages (exactly Quill’s optional vision_browser).
It scored GAIA 55.15%, versus a prior open-source state of the art around 46% (Magentic-One) and OpenAI’s Deep Research at 67.36%.

The lesson is not “import Open Deep Research” — it is inspiration. Fork examples/open_deep_research and you will recognize every moving part you spent fourteen modules learning. Orchestrator-workers + code-action + a sandbox is not a tutorial pattern; it is what holds up at GAIA scale.

The smolagents production checklist (T12.13)

This is the module’s shareable asset — a copy-pasteable list that distills the whole course into checkable items, each tied to the module that owns it, so a failing item tells you exactly what to re-read. It ships in the repo as module-15/PRODUCTION-CHECKLIST.md. Run down it before any release.

Security / sandbox

Multi-agent isolation uses Approach 2 — the whole system inside a hand-made sandbox, never executor_type="docker" + managed_agents (which raises). — M5, M15
local is not a sandbox; use a remote executor (Approach 1) or Approach 2 for any untrusted input/model or a public agent. — M5
additional_authorized_imports is minimal — ["pandas","numpy","matplotlib.*","json","statistics"], never "*". — M5
Docker hardening flags set: mem_limit="512m", cpu_quota=50000, pids_limit=100, security_opt=["no-new-privileges"], cap_drop=["ALL"], run as USER nobody. — M5, M15
Secrets in env, never in source; Approach 2 passes HF_TOKEN as a container env var. — M4, M5
trust_remote_code audited (a stdio MCP server runs local code; a Hub from_hub deserializes code); pin structured_output=False explicitly. — M9, M13
No executor_type="wasm" / WasmExecutor — removed in 1.26.0. Valid: {"local","docker","e2b","modal","blaxel"}. — M5

Reliability

final_answer_checks enforce content (no chart; a web claim with no source) — a rejection loops the agent, it does not crash. — M8
max_steps bounded on the manager and every sub-agent. — M8, M10, M15
Retries capped at 2, with a bounded helper that re-raises the last error. — M15
Idempotent outputs: a re-run overwrites outputs/quill-<sig>.png, dir created with exist_ok=True. — M15
Deterministic cleanup: with build_quill() as agent: (or agent.cleanup()) for a remote executor; close the Approach-2 sandbox in a try/finally. — M5, M15

Cost / performance

An explicit model_id via make_model — never rely on the library default. — M4
planning_interval on longer jobs; memory pruned by step_callbacks. — M6, M7
Cost/run visible via Monitor.get_total_token_counts(); vision only when it pays. — M4, M11, M14

Observability / quality

SmolagentsInstrumentor().instrument() runs before the agent is built; import from openinference.instrumentation.smolagents. — M14
In-process inspection via agent.memory.steps / replay() / visualize() — never agent.logs. — M6, M15
An eval golden set + a separate LLM-as-judge; a regression gate (“green or no ship”). — M14, M15

Deploy

GradioUI wraps the hardened agent (reset_agent_memory=False, file_upload_folder). — M13
push_to_hub exports the right artefacts; uv.lock committed; pins exact. — M9, M13, M1
Tag the release (git tag v1.0) only once the gate is green and the sandboxed run works. — M15

Build it: assemble, sandbox, harden, ship

The full, tested code is at smolagents-course-labs/module-15; the offline test suite passes with no token, no network, and no Docker. The capstone adds one new file — quill/runtime.py, the Approach-2 orchestrator and the hardening helpers — and hardens a few existing ones by addition. build_quill stays the construction owner in agent.py; runtime.py only calls it.

The assembled entry point

build_quill_app is the single in-process assembly. It does the production wiring in the order a real run needs — telemetry first, planning on, then build_quill:

def build_quill_app(*, model=None, planning_interval: int | None = None, telemetry: bool = True):
    from .agent import DEFAULT_PLANNING_INTERVAL, build_quill
    from .telemetry import instrument

    if telemetry:
        instrument()   # 06 §2 ORDERING: instrument BEFORE building (no-op if QUILL_TELEMETRY=none)

    cadence = DEFAULT_PLANNING_INTERVAL if planning_interval is None else planning_interval
    return build_quill(model=model, planning_interval=cadence)  # build_quill is the frozen owner

The ordering is the point (M14): instrument() patches smolagents’ classes, so it must run before the agent is built or the first steps’ spans are lost. With QUILL_TELEMETRY=none (the default) it is a clean no-op, so a run with no backend is never broken.

Approach 2: run the whole team inside a hardened container

The headline. We do not use executor_type="docker" (Approach 1 — it raises with a team). We start a hardened container, copy the code and data in, pass HF_TOKEN as an env var, run the entrypoint as nobody, and tear the container down in a finally:

def run_quill_in_docker_sandbox(question, dataset="data/sales.csv", *, image="python:3.11-slim", ...):
    import docker
    client = docker.from_env()
    container = client.containers.run(
        image, command=["sleep", str(timeout)], detach=True,
        environment={
            "HF_TOKEN": os.environ.get("HF_TOKEN", ""),  # secrets cross as ENV, never baked in
            "QUILL_EXECUTOR": "local",                   # the container IS the boundary
        },
        **build_hardened_container_kwargs(),             # mem/cpu/pids caps, cap_drop, user=nobody
    )
    try:
        # copy quill/ + data/ in, write the entrypoint, pip install, then run the WHOLE team:
        container.exec_run(["python", "/quill-app/_run.py", question, dataset],
                           workdir="/quill-app", user=SANDBOX_USER)
        ...
    finally:
        container.remove(force=True)  # resource management taught explicitly — NOT a silent except

The entrypoint that runs inside the box builds the whole team and runs a report task locally within the container — the manager and its sub-agents all execute behind the boundary:

SANDBOX_ENTRYPOINT = '''
import os, sys
os.environ.setdefault("QUILL_EXECUTOR", "local")  # the container is the boundary; no nested remote
from quill.agent import build_quill, build_report_task
from quill.report import QuillReport
...
with build_quill() as agent:                     # the FULL team inside the sandbox (Approach 2)
    output = agent.run(build_report_task(dataset, question))
print(output.to_markdown() if isinstance(output, QuillReport) else output)
'''

run_quill_sandboxed resolves the backend (Docker by default, E2B as the option — neither is an executor_type; “approach2” is not a thing) and wraps the run in run_with_bounded_retries, so a transient sandbox blip is retried at most twice.

The `--sandboxed` CLI and the release gate

The capstone command runs the fully isolated team:

QUILL_EXECUTOR=docker uv run python -m quill --sandboxed \
  "Analyze data/sales.csv vs data/customers.csv and tell me which segment is churning fastest, with a chart and sources."

===== QUILL REPORT (inside the sandbox) =====
# Which segment is churning fastest?
- Enterprise churn is lowest (3.1%); SMB churn leads at 11.4%, ~2x the SaaS benchmark [1].
...
Sources:
[1] [SaaS Churn Benchmarks 2025](https://example.com/saas-churn-2025)

Quill spins up a hardened Docker container, runs the manager + web_researcher inside it (Approach 2), delegates the benchmark lookup, saves a chart with save_chart, and returns a Markdown report with [n] citations mapped to QuillReport.sources. With telemetry on, the trace shows the manager span nesting the sub-agent span.

Then the gate — “green or no ship”:

uv run python -m quill.eval.run_evals --out eval/results/run-v1.0.json

Golden set: 5 tasks · model=Qwen/Qwen2.5-Coder-32B-Instruct
TSR: 0.80 (4/5)  ·  avg report_quality: 4.6/6  ·  avg steps: 6.2  ·  cost/run: ~12.4k tokens
Regression gate: PASS (TSR>=0.70, cost/run<=budget)

run_evals.py exits non-zero when TSR drops below QUILL_EVAL_MIN_TSR (default 0.70) or cost/run exceeds the budget. A failing gate fails the build — you do not ship a Quill you cannot defend with numbers. When the gate is green and the sandboxed run works, you tag the release:

git tag v1.0

Run the tests offline, and the Approach-2 path against a real container when you have Docker:

uv run pytest module-15/tests/                                          # offline: no token, no network, no Docker
QUILL_LIVE_TESTS=1 uv run pytest module-15/tests/ -m "sandbox and live" # the team inside a real container

The offline suite asserts the exact managed_agents exception, idempotent chart paths, bounded retries that re-raise, bounded max_steps on manager and sub-agent, the gate’s non-zero exit below threshold, the exact Docker hardening flags, and that no agent.logs appears anywhere.

Try it yourself. (1) Swap the Approach-2 backend from Docker to E2B in runtime.py (QUILL_SANDBOX_BACKEND=e2b) and compare startup, cost, and isolation. (2) Add a line to PRODUCTION-CHECKLIST.md (e.g. “rate-limit per user”) and write the smoke_test.py assertion that verifies it — closing the loop between the checklist and the tests.

In production

What changes when Quill v1.0 leaves the laptop:

Cost first. A multi-agent + vision + RAG run can be dozens of LLM calls; the HF free tier is ~$0.10/month (as of smolagents 1.26.0, subject to change) — a wall you hit fast. That is why max_steps is bounded, memory is pruned, and cost/run is visible via Monitor (M14). A VLM call costs far more than a text call, so chart self-review runs once at the end, not every step.

Security. Approach 2 isolates the system, but web content is still a prompt-injection vector (M5). Keep additional_authorized_imports minimal and the Docker container hardened. “No solution will be 100% safe” — defense in depth, not a single wall.

Reliability. Every failure mode has a guard-rail: a sub-agent that returns empty (retries ≤ 2, then graceful fallback to the text web_researcher), a Docker container that won’t start (bounded retry, then a loud error), a JS page that times out (the 30s execution cap). Cleanup is deterministic — with ... as agent: or a try/finally around the sandbox, so no container or microVM is left dangling, even on an exception.

Observability. In production you read the traces (M14), not the console — but keep agent.replay() / agent.memory.steps for debugging without a backend.

Quality. The eval gate (M14) blocks a regression before deploy: green or no ship. And the honest part — this system is defensible because it is measured, not because it “looks like it works.”

Concept check

Why this matters. This is the finale, so the quiz is transverse: ten scenarios across the whole course, not just Module 15. Each answer is tagged with the module to re-read if you miss it — a quick self-diagnostic for the parts of smolagents that did not stick.

1. A teammate asks why Quill’s CodeAgent writes Python instead of emitting JSON tool calls. Which is a real advantage of code-as-action?

A. It is always cheaper, regardless of task.
B. Actions compose — Quill can chain operations, store intermediate objects, and loop in one block — and LLMs have seen vast amounts of code.
C. It removes the need for a model.
D. JSON tool calls cannot return data.

2. You write a Quill tool that hits an external API. The call sometimes fails. How should the tool report the failure to the agent?

A. Raise and let the process crash.
B. Return None silently.
C. print() a clear error message and return a string the agent can read, so it can self-correct on the next step.
D. Log to a file the agent never sees.

3. A reader has no GPU and a tiny budget but wants Quill to run. What is the right model setup?

A. Hard-code an InferenceClientModel in agent.py.
B. Use make_model with an explicit model_id; default backend is HF Inference Providers (provider="auto"), and swap to LiteLLM or local Ollama via env — no agent-code edit.
C. Always use a paid OpenAI key.
D. Rely on the library’s default model_id.

4. A developer deploys a public CodeAgent with executor_type="local" and untrusted user input. What is wrong?

A. Nothing — local is a sandbox.
B. local is an AST allow-list, not a security sandbox; untrusted input needs a remote executor (Approach 1) or Approach 2.
C. local is too slow.
D. local can’t run pandas.

5. You try executor_type="docker" plus managed_agents=[web_researcher] and it raises at startup. Why, and what do you do?

A. A Docker bug; retry.
B. Approach 1 ships no secrets into the sandbox, so a sub-agent can’t authenticate its LLM — keep executor_type="local", or use Approach 2 (the whole team inside a hand-made sandbox).
C. Rename web_researcher.
D. managed_agents must be a dict.

6. A run went sideways and you have no telemetry backend running. How do you inspect what happened?

A. agent.logs.
B. agent.memory.steps / agent.replay() / agent.visualize() — in-process inspection.
C. Re-run with verbose=True only.
D. There is no way without a backend.

7. You want Quill to refuse any web-backed claim that lacks a source. What is the mechanism?

A. A custom executor_type.
B. A 3-arg (final_answer, memory, agent) -> bool final_answer_check that returns False when sources are empty but a web tool was used — the agent loops and self-corrects.
C. A regex over the output.
D. response_format.

8. A tutorial shows from smolagents import ManagedAgent. What do you do?

A. Use it — it’s the standard.
B. Recognize it’s removed in 1.26.0; a sub-agent is a normal agent with name + description, registered via managed_agents=[...].
C. pip install managed-agent.
D. Wrap it in a try/except.

9. Quill “looks like it works” on a few questions. Can you declare it good?

A. Yes — eyeballing a few runs is enough.
B. No — you need an eval gate: a golden set scored by a separate LLM-as-judge plus deterministic checks, with a regression gate on TSR before you ship.
C. Yes, if the UI looks nice.
D. Only QA can decide.

10. A sub-agent loops and starts draining your free tier. Which guard-rails stop it?

A. Hope it converges.
B. Bounded max_steps (manager + sub-agent), retries capped at 2 with a re-raising helper, and idempotent outputs so re-runs don’t double-charge.
C. A bigger model.
D. Disable the sub-agent permanently.

Answers

B. Composability (and objects, generality, and the model’s exposure to real code) is the code-as-action advantage; it is not unconditionally cheaper. Review M1.
C. A tool’s errors must be readable text the agent can act on — print() and return a message, so the agent self-corrects rather than crashing. Review M3.
B. make_model is the single env-driven factory: HF Inference Providers by default (free with an HF_TOKEN, provider="auto"), swappable to LiteLLM or local Ollama — and always with an explicit model_id, never the library default. Review M4.
B. LocalPythonExecutor is an allow-list interpreter, “not a security sandbox”; untrusted input on a public agent needs a remote executor or Approach 2. Review M5.
B. The guard in create_python_executor refuses a remote executor with managed_agents because Approach 1 keeps secrets out of the box. Stay local, or run the whole team inside a hand-made sandbox (Approach 2). Review M15 (and M10).
B. In-process inspection is agent.memory.steps / replay() / visualize(). agent.logs was removed in 1.21.0. Review M6.
B. A 3-arg final_answer_check returning False loops the agent (the AgentError lands in ActionStep.error); it self-corrects, it does not crash. Review M8.
B. ManagedAgent is gone in 1.26.0; the current mechanism is name + description + managed_agents=[...]. Review M10.
B. “Looks like it works” is not a quality signal — a golden set + a separate judge + a regression gate is. Review M14.
B. Bounded steps, capped retries, and idempotence are the hardening levers that stop a runaway loop. Review M15.

Common pitfalls

Believing executor_type="docker" + managed_agents “isolates the multi-agent.” It raises. Isolated multi-agent is Approach 2 — the whole system inside a hand-made sandbox.
Confusing “it runs” with “it’s in production.” Without Approach 2, guard-rails, telemetry, and an eval gate, a UI is just a demo exposed to the internet (the Common misconception above).
Writing agent.logs (removed in 1.21.0) instead of agent.memory.steps / replay() — and forgetting idempotence, so re-runs double-write and double-count.

Key takeaways

Multi-agent + sandbox ⇒ Approach 2: run the whole system inside a hand-made sandbox. A remote executor_type + managed_agents raises — Approach 1 can’t do multi-agent.
local is not a sandbox — it’s an AST allow-list interpreter; the library says so itself.
Hardening = bounded max_steps + execution timeouts + retries capped at 2 + idempotent outputs.
In-process inspection is agent.memory.steps / agent.replay() / agent.visualize() — never agent.logs (removed in 1.21.0).
Telemetry on + an eval gate makes a system defensible, not just “working” — green or no ship.
Open Deep Research proves orchestrator-workers + code-action + a sandbox scales to GAIA — the architecture you built, validated.
And always: reach for an agent only when a deterministic workflow won’t cover the cases.

Where to go next

You shipped Quill v1.0. Four honest next steps:

Open Deep Research. Fork examples/open_deep_research and adapt it — it is Quill pushed further, the same orchestrator-workers + code-action shape at GAIA scale.
The Hugging Face AI Agents Course. Free and certifying — a good way to validate your agent skills beyond smolagents.
The rest of the catalog. Course 1 — LangGraph — for fine-grained control of stateful workflows; course 2 — Strands / Bedrock — for managed AWS agents. smolagents is the lightest on-ramp to both.
The other frameworks. When a different shape fits your problem: CrewAI (role/crew orchestration), PydanticAI (type-safe), the OpenAI Agents SDK, Google ADK, LlamaIndex (RAG-first). Pick the one whose abstractions match the job — and, always, don’t reach for an agent when a plain workflow would do.

Want the full smolagents concept-check bank and news of future courses in your inbox? Subscribe here — it’s free, like everything in this series.

Links: Module 15 lab code · previous: Module 14 — Observability and Evaluation · start over at Module 1 · course index

References

smolagents — Secure code execution (Approach 1 vs 2; the remote + multi-agent constraint; hardening): https://huggingface.co/docs/smolagents/tutorials/secure_code_execution
smolagents — Orchestrate a multi-agent system (managed_agents): https://huggingface.co/docs/smolagents/examples/multiagents
Hugging Face blog — Open Deep Research (reproduced on smolagents; GAIA 55.15%): https://huggingface.co/blog/open-deep-research
smolagents — Inspecting runs with OpenTelemetry (telemetry + eval plumbing): https://huggingface.co/docs/smolagents/tutorials/inspect_runs
Anthropic — Building Effective Agents (when to use/avoid an agent; the simplest system): https://www.anthropic.com/engineering/building-effective-agents
Hugging Face — AI Agents Course (the certificate, a complementary resource): https://huggingface.co/learn/agents-course