Models and Providers: What Powers Your Agent (smolagents, Module 4)

This is Module 4 of smolagents Mastery, a free 15-module course that takes you from your first code agent to a production-grade one. Start at Module 1 or browse the full syllabus.

In Module 3 you handed Quill a toolbox: load_dataset, profile_dataframe, save_chart, plus web access. Quill can now load a CSV, profile it, write pandas, and draw a chart. But there is a line of code in agent.py you have been quietly ignoring — the one that builds the model:

model = InferenceClientModel(...)   # right where the agent is created

Every future module would inherit that coupling. Want to try Claude instead of the default? Edit agent.py. Want to run a local model on your laptop for privacy? Edit agent.py. Want to compare two providers on the same question? Edit agent.py twice. The model — the single most swappable part of any agent — is welded to the agent’s construction.

And there is a second, quieter problem. You ran Quill one too many times and saw it: insufficient credits. A multi-step CodeAgent burns tokens fast, and nobody told you what a run costs. By the end of this module Quill has one place to choose what powers it — and the cost of every run in plain sight.

In this module

You’ll learn:

How an agent actually calls a model in smolagents — the Model base class, why subclasses implement generate() (not __call__), and how your tools reach the model.
How to choose the right backend under a real constraint — free, reliable, or local — across InferenceClientModel, LiteLLMModel, the OpenAI/Azure/Bedrock classes, and the local models.
How to route requests through HF Inference Providers — and the honest truth about cost (the real free tier, and when to switch).
How to swap models without touching the agent, via an env-driven make_model() factory.
How to measure the tokens and cost of a run with TokenUsage and Monitor.

You’ll build: Quill gets quill/config.py with a single make_model(role) factory — InferenceClientModel by default (explicit coder model_id), one env var to swap to LiteLLM or a local model — and prints the token cost of every run.

Theory areas covered: T4 — Models & providers — 9% of the course (Quill owns all of T4 this module).

Prerequisites: Module 1 (setup, HF_TOKEN, your first CodeAgent), Module 2 (build_quill, the ReAct loop), Module 3 (Quill’s data tools). Python 3.11/3.12, uv, and a fine-grained Hugging Face token with the “Make calls to Inference Providers” scope. Keep that token in a .env file — never commit it.

Where we are

✅ M1  Setup + your first CodeAgent
✅ M2  Quill v0: a CodeAgent over a CSV
✅ M3  Quill's data toolbox
👉 M4  Models & providers — make_model(), cost in plain sight
⬜ M5  Sandboxing the Python Quill writes
⬜ M6  Memory, steps, callbacks
⬜ M7 … M15

Before this module: the model is hard-wired in agent.py — one backend, no cost visibility. After it: make_model() is the one place, three backends, cost visible. This module freezes the model contract — make_model(role="analyst", **overrides) — that every later module reuses. When Module 10 spins up a sub-agent and Module 14 measures cost per evaluation, they both call this factory.

How an agent calls a model: the `Model` abstraction

Here is the invariant that ties the whole module together: every model in smolagents is callable. The agent does model(messages, ...). That call hits the __call__ dunder on the base Model class, which routes to generate() — and generate() is the method each concrete subclass actually implements. You will see plenty of tutorials override __call__. They are wrong for 1.26.0: you subclass Model and implement generate, full stop. The base class lives in src/smolagents/models.py.

Why does this matter before you write a line of agent code? Because it is the seam. The agent layer knows nothing about Together, Anthropic, or your local GPU. It knows one thing: I can call model(messages) and get a message back. Everything in this module is about what you plug into that seam.

The base `Model` and its uniform kwargs pattern

The base constructor, verified against models.py @ v1.26.0:

def __init__(
    self,
    flatten_messages_as_text: bool = False,
    tool_name_key: str = "name",
    tool_arguments_key: str = "arguments",
    model_id: str | None = None,
    **kwargs,
)

The single most useful thing to internalize here is the **kwargs line, and it is uniform across every model class. Completion parameters — temperature, max_tokens, top_p — are passed at init, stored, and forwarded on every call. You do not pass them per-request:

from smolagents import InferenceClientModel

# temperature/max_tokens are set once at construction and forwarded to every generate() call.
model = InferenceClientModel(
    model_id="Qwen/Qwen2.5-Coder-32B-Instruct",
    temperature=0.2,
    max_tokens=2048,
)

This is why Quill’s factory can take **overrides and forward them blindly — the smolagents convention does the rest. The other two args (tool_name_key, tool_arguments_key) tell the model how to find a tool name and its arguments when a provider returns a tool call as a JSON blob in text rather than as a structured object; flatten_messages_as_text collapses rich message content into plain text for models that can’t handle structured content.

`generate()` — and how your tools reach the model

The method every subclass implements:

def generate(
    self,
    messages: list,
    stop_sequences: list[str] | None = None,
    response_format: dict[str, str] | None = None,
    tools_to_call_from: list[Tool] | None = None,
    **kwargs,
) -> ChatMessage

Walk the arguments, because this signature is the contract between the agent and the LLM:

messages — the conversation, a list of role/content dicts (or ChatMessage objects).
stop_sequences — strings that halt generation.
response_format — a JSON-schema structured-output spec. It exists here; we do not drive it in this module. Structured outputs are Module 8 — for now, know it’s a parameter, not a feature you wire.
tools_to_call_from — this is how your tools reach the model. The agent hands generate() the list of tools to choose from. Module 3 gave Quill its toolbox; this argument is the bridge that puts those tools in front of the LLM each step.

generate() returns a ChatMessage, always. That is the second invariant: whatever backend you plug in, the agent gets back the same shape.

One internal worth knowing: when a provider returns a tool call inline as text (a JSON blob in the content) instead of as a structured object, smolagents calls parse_tool_calls() to recover it, using tool_name_key/tool_arguments_key to find the name and arguments and building a ChatMessageToolCall (with a generated UUID id). You will not call this yourself, but it explains why agents work even against providers with weak native tool-calling — smolagents parses the text.

`ApiModel` — where rate limiting and retries live

Every API-backed model shares an intermediate base, ApiModel: InferenceClientModel, LiteLLMModel, LiteLLMRouterModel, OpenAIModel, AzureOpenAIModel, and AmazonBedrockModel all inherit from it. The local models (TransformersModel, MLXModel, VLLMModel) inherit Model directly — they have no remote API to manage. The ApiModel constructor:

def __init__(
    self,
    model_id: str,
    custom_role_conversions: dict[str, str] | None = None,
    client: Any | None = None,
    requests_per_minute: float | None = None,
    retry: bool = True,
    **kwargs,
)

Why does this base exist? Three production reasons, shared by every API backend so they don’t each reimplement them:

requests_per_minute — a client-side rate-limit guard. A multi-step CodeAgent can fire 5–15 LLM calls per run; on a small free tier or a strict provider, that saturates your quota in seconds. This is the native throttle. (You will see it surface in Quill’s factory as a pass-through override.)
retry (default True) — automatic retries on rate-limit errors, up to RETRY_MAX_ATTEMPTS.
custom_role_conversions — remap message roles for models that don’t accept, say, a "system" role. create_client() is the hook each subclass overrides to build its actual HTTP client.

The takeaway: if your backend talks to an API, you get throttling and retries for free, inherited. That is a production argument, not a footnote.

The data types: `ChatMessage`, roles, `TokenUsage`

A ChatMessage carries:

role: MessageRole
content: str | list[dict[str, Any]] | None = None
tool_calls: list[ChatMessageToolCall] | None = None
raw: Any | None = None                # raw provider response
token_usage: TokenUsage | None = None

Two details that trip people up. First, the MessageRole enum uses hyphenated string values for tool roles — "tool-call" and "tool-response", not underscores. Tutorials that hand-build messages with "tool_call" silently mismatch. Second, raw holds the untouched provider response (useful for debugging), and token_usage holds a TokenUsage for that single message — the per-message half of the cost story we finish at the end of this module. (How these messages get assembled from agent memory is Module 6’s territory.)

Here is the full class hierarchy you are choosing from:

graph TD
    M[Model<br/>base · generate → ChatMessage] --> A[ApiModel<br/>requests_per_minute · retry]
    M --> T[TransformersModel<br/>local · GPU]
    M --> X[MLXModel<br/>local · Apple Silicon]
    M --> V[VLLMModel<br/>local · served]
    A --> IC[InferenceClientModel<br/>default · HF router]
    A --> L[LiteLLMModel<br/>100+ providers]
    A --> LR[LiteLLMRouterModel<br/>load-balancing]
    A --> O[OpenAIModel<br/>OpenAI-compatible]
    A --> AZ[AzureOpenAIModel]
    A --> B[AmazonBedrockModel]

Every shipped backend (and when to reach for each)

Now the concrete classes. Each one is generate() over a different transport. Quill only needs three (InferenceClientModel, LiteLLMModel, and an Ollama path), but you should know the whole roster — a production system reaches for several.

`InferenceClientModel` — the default

This is the default backend and the one Quill ships with. (It is the class formerly known as HfApiModel — that name is gone in 1.26.0; more on that in Common pitfalls.) Its constructor:

def __init__(
    self,
    model_id: str = "Qwen/Qwen3-Next-80B-A3B-Thinking",
    provider: str | None = None,
    token: str | None = None,
    timeout: int = 120,
    client_kwargs: dict[str, Any] | None = None,
    custom_role_conversions: dict[str, str] | None = None,
    api_key: str | None = None,
    bill_to: str | None = None,
    base_url: str | None = None,
    **kwargs,
)

What to teach correctly:

The default model_id is "Qwen/Qwen3-Next-80B-A3B-Thinking" as of smolagents 1.26.0, and the docs say it may change. So Quill always passes an explicit model_id — never relies on the default. A model that silently changes under you is a production incident waiting to happen.
provider routes to a specific HF Inference Provider; it defaults to "auto". (More in the next section — this is the heart of the cost story.)
token and api_key are mutually exclusive (api_key mirrors the openai.OpenAI argument style); both fall back to the HF_TOKEN environment variable. You never hard-code the token.
base_url points at a deployed Inference Endpoint or a local URL — mutually exclusive with model_id.
bill_to charges requests to an Enterprise Hub org.

Under the hood, InferenceClientModel wraps huggingface_hub.InferenceClient. It is a client, not a server — remember that; it is the misconception this module busts later.

`LiteLLMModel` — 100+ providers behind one class

When you want a frontier closed model (Claude, GPT-4o, Gemini) or any of a hundred others, this is the swap:

def __init__(
    self,
    model_id: str | None = None,
    api_base: str | None = None,
    api_key: str | None = None,
    custom_role_conversions: dict[str, str] | None = None,
    flatten_messages_as_text: bool | None = None,
    **kwargs,
)

The model_id format is "<provider>/<model>": "anthropic/claude-3-5-sonnet-latest", "gpt-4o", "gemini/gemini-1.5-pro", "ollama_chat/llama3.2", "bedrock/anthropic.claude-3-sonnet-...", "azure/<deployment>". LiteLLM reads the relevant provider key from the environment itself (ANTHROPIC_API_KEY, OPENAI_API_KEY, …). You install it with the [litellm] extra (litellm>=1.60.2).

from smolagents import LiteLLMModel

model = LiteLLMModel(
    model_id="anthropic/claude-3-5-sonnet-latest",
    temperature=0.2,
    requests_per_minute=60,   # inherited ApiModel guard-rail
)

One internal worth knowing: flatten_messages_as_text defaults (when None) to True if and only if model_id.startswith(("ollama", "groq", "cerebras")), otherwise False. Those providers prefer flattened text content; LiteLLM picks the right default for you based on the id prefix.

And the trap that bites everyone running Ollama through LiteLLM: Ollama’s default num_ctx is 2048, which the smolagents docs say “fails horribly” for an agent — the prompt plus the agent’s growing memory overflow that window almost immediately. Raise it (Quill uses num_ctx=8192):

# Ollama via LiteLLM — the easy laptop path. num_ctx MUST be raised above the 2048 default.
model = LiteLLMModel(
    model_id="ollama_chat/llama3.2",
    api_base="http://localhost:11434",
    num_ctx=8192,
)

`LiteLLMRouterModel` — load-balancing and fallbacks

For reliability under load, LiteLLMRouterModel wraps the LiteLLM Router: its model_id is a model-group name present in model_list, and it gives you fallbacks, cooldowns, retries across deployments, and routing strategies (client_kwargs={"routing_strategy": "simple-shuffle"}). It is the option you reach for when a single provider’s flakiness starts costing you runs. Quill doesn’t need it in the lab — but it’s the right answer to “my agent keeps dying on 429s in production.”

`OpenAIModel`, `AzureOpenAIModel`, `AmazonBedrockModel`

Three enterprise/compat classes. Teach the short names; the *ServerModel variants (OpenAIServerModel, AzureOpenAIServerModel, AmazonBedrockServerModel) are still-present aliases, but the canonical names are the short ones.

OpenAIModel connects to any OpenAI-compatible server — point api_base at vLLM, LM Studio, OpenRouter, whatever speaks the OpenAI protocol. Needs the [openai] extra.
AzureOpenAIModel reads env fallbacks AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY, and — note the inconsistency — OPENAI_API_VERSION (no AZURE_ prefix on the version var).
AmazonBedrockModel talks to boto3 bedrock-runtime. Its custom_role_conversions defaults to converting all roles to "user" for maximum Bedrock model compatibility. Auth comes from the standard AWS credential chain or an API key via AWS_BEARER_TOKEN_BEDROCK (needs boto3>=1.39.0).

from smolagents import OpenAIModel

# Any OpenAI-compatible endpoint: just redirect api_base. Key from the environment.
model = OpenAIModel(model_id="gpt-4o", api_base="https://my-vllm-host/v1")

Local models: `TransformersModel`, `VLLMModel`, `MLXModel`

When you need privacy or zero API cost:

TransformersModel runs a local transformers pipeline (device_map, torch_dtype, max_new_tokens=4096). Note max_tokens is an alias for max_new_tokens and wins if both are set. Extra [transformers] (it pulls in [torch]).
VLLMModel serves via vLLM; model_kwargs forward to vLLM’s LLM(...) (e.g. max_model_len).
MLXModel runs on Apple Silicon. Note the extra is [mlx-lm] (with a hyphen), not [mlx].

The trade-off for a code agent specifically: a CodeAgent writes and executes Python every step, so it needs a code-capable model with a large context (memory grows each step). Small local models (≤7B) frequently produce malformed code, loop, or mis-call tools — more steps, more failures. Local removes API cost and keeps data private, but demands real GPU/RAM and a num_ctx/max_model_len large enough for the growing memory. For teaching and demos, hosted is frictionless; treat local as an opt-in advanced path.

Custom models and `REMOVE_PARAMETER`

You can also write your own backend: subclass Model, implement generate(messages, stop_sequences=..., **kwargs), return an object with a .content attribute (ideally a ChatMessage). That is exactly how this course’s test suite builds its offline FakeModel — a deterministic Model whose generate() returns scripted messages, so the agent loop runs with no network and no token:

from smolagents import ChatMessage, MessageRole, Model

class FakeModel(Model):
    def __init__(self, scripted: list[str], model_id: str = "fake/deterministic"):
        super().__init__(model_id=model_id)
        self.scripted = list(scripted)
        self.calls = 0

    def generate(self, messages, stop_sequences=None, response_format=None,
                 tools_to_call_from=None, **kwargs) -> ChatMessage:
        text = self.scripted[min(self.calls, len(self.scripted) - 1)]
        self.calls += 1
        return ChatMessage(role=MessageRole.ASSISTANT, content=text)

That is the full custom-model contract — match generate(), return a ChatMessage. It is why every offline test in this lab runs in milliseconds without spending a cent.

One edge case: some models reject parameters smolagents auto-sets. The REMOVE_PARAMETER sentinel (exported from smolagents) force-excludes one. For example, gpt-5 rejects stop, so:

from smolagents import OpenAIModel, REMOVE_PARAMETER

model = OpenAIModel(model_id="gpt-5", stop=REMOVE_PARAMETER, temperature=0.7)

Choosing a backend

Goal	Recommendation	Why / notes
Default “just works”, free	`InferenceClientModel(model_id="Qwen/Qwen2.5-Coder-32B-Instruct")`	One `HF_TOKEN`, `provider="auto"`, included credits. Good code model.
Fast/cheap named provider	`InferenceClientModel(provider="together"\|"sambanova"\|"groq"\|"cerebras")`	Explicit partner; some have their own free tiers.
Frontier closed model	`LiteLLMModel("anthropic/claude-3-5-sonnet-latest")` or `"gpt-4o"`	`[litellm]` + provider key. Best agentic reliability.
Fully local, GPU	`TransformersModel(...)` / `VLLMModel(...)`	No API cost, private; needs hardware; use a coder/instruct model.
Local, easy (Mac/laptop)	Ollama via `LiteLLMModel(...)` (`num_ctx≥8192`) or `MLXModel(...)`	Raise `num_ctx`; small models are weak for code agents.
Enterprise	`AzureOpenAIModel(...)` / `AmazonBedrockModel(...)`	Env-var auth; Bedrock can use `AWS_BEARER_TOKEN_BEDROCK`.
Reliability / load-balancing	`LiteLLMRouterModel(model_id="group", model_list=[...])`	Fallbacks, cooldowns, cross-deployment retries.

HF Inference Providers and the cost truth

InferenceClientModel is the default, so it is worth understanding exactly what happens when Quill calls it — because what happens is not what most people assume.

HF Inference Providers is a unified proxy layer in front of many third-party inference providers. One HF token, one API; Hugging Face routes your request to a partner provider and bills you at the same rates as the provider (no markup). That routing layer is precisely the mechanism behind InferenceClientModel.

The provider= argument controls the routing. The default is "auto" — the first available provider for your model, in the order of your account settings (the :fastest policy). You can also name a partner explicitly: "novita", "together", "sambanova", "cerebras", "fireworks-ai", "groq", "hyperbolic", and more. The router’s base URL is https://router.huggingface.co/v1 (OpenAI-compatible). The partner list changes — check the live Inference Providers index rather than memorizing it.

graph LR
    Q[Quill] --> ICM["InferenceClientModel<br/>provider='auto'"]
    ICM --> R[HF router<br/>router.huggingface.co/v1]
    R --> P1[Novita]
    R --> P2[Together]
    R --> P3[SambaNova]
    R --> P4[Cerebras]
    R --> P5[…]

One token in, compute distributed across partners. Now the part nobody puts in the quickstart:

The cost truth

Included credits, as of smolagents 1.26.0 (June 2026, “subject to change”):

Account	Monthly included credits
Free	$0.10
PRO	$2.00
Team / Enterprise	$2.00 per seat (shared pool)

Read that free row again: ten cents a month. Credits only apply to HF-routed requests — if you bring your own provider key, that provider bills you directly, outside HF credits. And hf-inference (the original first-party path) is now mostly CPU/small-model focused, so for big LLMs you route to a partner anyway.

So the honest persona stance for a hands-on course: the free tier is tiny, and a multi-step CodeAgent burns through it fast. For real practice, you want one of — a PRO subscription, a free provider key (Groq and Cerebras run their own free tiers), or a local model. This is exactly why the lab makes cost visible: you cannot manage what you cannot see.

To get the token: create a fine-grained token at huggingface.co/settings/tokens with the “Make calls to Inference Providers” permission, and set HF_TOKEN in your .env (covered in Module 1’s setup).

One place to swap, tokens in plain sight

Time to build. Everything above motivates two pieces of Quill: a factory so the model lives in one place, and a cost line so a run never surprises you.

The `make_model()` factory

The model is a cross-cutting dependency. Hard-wiring it in agent.py couples every module to one backend; centralizing it in a factory makes “change the model in all of Quill” a one-line, one-file change. This is the frozen model contract (make_model(role="analyst", **overrides) -> Model) that all later modules reuse. Here is Quill’s actual config.py, abridged:

import os
from smolagents import InferenceClientModel, LiteLLMModel, Model

# EXPLICIT coder pin — never the library default (Qwen3-Next-80B-A3B-Thinking, "subject to change").
DEFAULT_MODEL_ID = "Qwen/Qwen2.5-Coder-32B-Instruct"

class Settings:
    DEFAULT_BACKEND: str = "hf"
    DEFAULT_MODEL_ID: str = DEFAULT_MODEL_ID
    # MODEL_BACKEND / MODEL_ID are env-backed properties that re-read os.environ on every access,
    # so QUILL_MODEL_BACKEND=litellm takes effect even if config was imported first.
    MODEL_BACKEND = _EnvProperty("QUILL_MODEL_BACKEND", "hf", lower=True)
    MODEL_ID = _EnvProperty("QUILL_MODEL_ID", DEFAULT_MODEL_ID)

The default MODEL_ID is explicit and chosen on purpose — a code-capable model with a large context, because a CodeAgent writes and runs Python every step. We never lean on InferenceClientModel’s own default. The factory dispatches on the backend:

def make_model(role: str = "analyst", **overrides) -> Model:
    """The single entry point for every LLM call in Quill."""
    backend = Settings.MODEL_BACKEND

    if backend == "hf":
        # token comes from HF_TOKEN in the env; provider defaults to "auto" (HF router).
        return InferenceClientModel(model_id=Settings.MODEL_ID, **overrides)

    if backend == "litellm":
        # id format "<provider>/<model>": "gpt-4o", "anthropic/claude-3-5-sonnet-latest", …
        return LiteLLMModel(model_id=Settings.MODEL_ID, **overrides)

    if backend == "local":
        model_id = Settings.MODEL_ID
        if model_id == DEFAULT_MODEL_ID:
            model_id = "ollama_chat/llama3.2"      # sensible local default
        kwargs = {"api_base": "http://localhost:11434", "num_ctx": 8192}
        kwargs.update(overrides)                    # explicit caller overrides win
        return LiteLLMModel(model_id=model_id, **kwargs)

    raise ValueError(f"Unknown QUILL_MODEL_BACKEND {backend!r}. Supported: hf, litellm, local.")

Design choices worth calling out:

role is part of the frozen signature now, even though Module 4 uses the default for every role. A later module can branch on it (a cheaper model for a researcher sub-agent vs the analyst manager) without touching a single caller. Freezing the signature early is what keeps the contract stable.
**overrides is the uniform smolagents pattern in action: make_model("analyst", temperature=0.1) or make_model("analyst", provider="together") or make_model("analyst", requests_per_minute=60) all flow straight to the constructor.
We fail loud on an unknown backend — a typo should never silently route you to the wrong model.
The token is never in the code: InferenceClientModel reads HF_TOKEN, LiteLLMModel reads the provider’s key, both from the environment.

Now agent.py stops knowing about providers entirely. The Module 3 line that built a model is gone; build_quill calls the factory:

from .config import make_model

def build_quill(model: Model | None = None) -> CodeAgent:
    return CodeAgent(
        tools=[load_dataset, profile_dataframe, save_chart(),
               WebSearchTool(), VisitWebpageTool()],
        model=model or make_model(role="analyst"),   # ONE place to choose what powers Quill
        additional_authorized_imports=QUILL_IMPORTS,
        max_steps=8,
    )

The optional model= parameter is still how tests inject the offline FakeModel; otherwise make_model() owns the default. No file in the agent layer imports a model class anymore.

`TokenUsage` and `Monitor` — cost in plain sight

The cost half. A TokenUsage is a small dataclass:

input_tokens: int
output_tokens: int
total_tokens: int = field(init=False)   # computed in __post_init__ = input + output

You never set total_tokens — it’s computed:

from smolagents import TokenUsage
usage = TokenUsage(input_tokens=8142, output_tokens=1203)
assert usage.total_tokens == 9345

Every agent has a Monitor that accumulates per-step token counts (you see them in the run logs, e.g. Input tokens: 2,069 | Output tokens: 60). agent.monitor.get_total_token_counts() returns a TokenUsage aggregated across the whole run. That single accessor is the cost recipe — and the same one Module 14’s evaluation harness reuses to report cost per run. Quill’s run.py prints it:

from .agent import build_quill, build_task
from .config import Settings

def main(argv=None):
    args = sys.argv[1:] if argv is None else argv
    csv_path = args[0] if args else "data/sales.csv"
    question = args[1] if len(args) > 1 else "Which category grew fastest last quarter?"

    agent = build_quill()
    print(f"[Quill] Backend: {Settings.MODEL_BACKEND} | Model: {Settings.MODEL_ID}")
    print(agent.run(build_task(csv_path, question)))

    usage = agent.monitor.get_total_token_counts()   # TokenUsage for the whole run
    print(f"[Quill] Run cost — input tokens: {usage.input_tokens:,} | "
          f"output tokens: {usage.output_tokens:,} | total: {usage.total_tokens:,}")
    return 0

We read tokens through the Monitor — never the removed agent.logs or legacy token attributes (gone since 1.21.0). No silent try/except: if a run errors, you see it.

⚠️ Common misconception: “InferenceClientModel is Hugging Face’s own model server.” No. InferenceClientModel is a client and router, not a GPU farm with an HF logo on it. With provider="auto", your request is served by a third-party partner — Together, SambaNova, Cerebras, Novita, and so on — through the HF router. The original first-party hf-inference path is now mostly CPU/small-model focused. The corollary matters in production: changing model_id can change which provider actually answers, and therefore the latency, the cost, and even the availability. The model name is not just a model name; it’s a route.

Build it

Goal: give Quill quill/config.py with the frozen make_model(role) factory — InferenceClientModel by default (explicit coder model_id), one env var to swap to LiteLLM or a local model — and print the token cost of every run.

Observable result: Quill answers a question on data/sales.csv, then prints which backend/model powered it and the run’s token cost — and the same command swaps to gpt-4o with no edit to the agent.

Setup. Carry the Module 3 state forward (the cumulative rule: M4 code must still pass the M1–M4 smoke tests), then sync the pins. Module 4 adds the [litellm] extra (the swap backend) and the optional [openai] extra:
```
uv venv --python 3.11
uv pip install "smolagents[toolkit,litellm,openai]==1.26.0" \
  "huggingface_hub>=1.0,<2" "pandas>=2.2.3" matplotlib
cp module-04/.env.example module-04/.env   # then add your HF token; never commit .env
```
Create a fine-grained HF token with the “Make calls to Inference Providers” scope. Provider keys (OPENAI_API_KEY, ANTHROPIC_API_KEY, …) are only needed for the litellm backend.
Settings (quill/config.py). A tiny config object that reads the environment, so exactly one place knows which backend and model Quill uses. Pin DEFAULT_MODEL_ID = "Qwen/Qwen2.5-Coder-32B-Instruct" explicitly — never the library default.
make_model(role="analyst", **overrides) (quill/config.py). The frozen factory shown above: dispatch on Settings.MODEL_BACKEND, fail loud on an unknown backend, forward **overrides, read tokens from the environment.
build_quill calls make_model() (quill/agent.py). Delete the hard-wired model; agent.py no longer imports any model class. The injected-model path (model=None) stays for tests.
quill/run.py — the cost-aware CLI. Run Quill, then print the backend, model, and the run’s token cost from agent.monitor.get_total_token_counts().

Run it, then swap with no agent edit:

uv run python -m quill.run data/sales.csv "Which product category grew fastest last quarter?"

Expected tail of the output:

...
[Quill] Backend: hf | Model: Qwen/Qwen2.5-Coder-32B-Instruct
[Quill] Run cost — input tokens: 8,142 | output tokens: 1,203 | total: 9,345

Now swap to a hosted frontier model — no agent code changes at all:

QUILL_MODEL_BACKEND=litellm QUILL_MODEL_ID="gpt-4o" \
  uv run python -m quill.run data/sales.csv "Which product category grew fastest last quarter?"
# [Quill] Backend: litellm | Model: gpt-4o

Tests. The offline suite runs with no network and no token: factory tests construct the model objects and assert only class + model_id (these classes build lazily — the first HTTP call is on .generate(), which the offline tests never call), the swap is monkeypatched through, and the Monitor/TokenUsage accessor is exercised. The live tests (one real run that asserts total_tokens > 0, plus an optional LiteLLM swap) are @pytest.mark.live, skipped unless QUILL_LIVE_TESTS=1, and skip cleanly without HF_TOKEN.
```
uv run pytest module-04/tests/                       # offline (no token needed)
QUILL_LIVE_TESTS=1 uv run pytest module-04/tests/    # + the real LLM runs
```
The offline run is green out of the box:
```
41 passed, 2 skipped in 2.56s
```

The full, tested code lives in the lab repo at smolagents-course-labs/module-04 — config.py, the modified agent.py, run.py, and the smoke tests all run offline.

Try it yourself:

Set QUILL_MODEL_BACKEND=litellm QUILL_MODEL_ID="anthropic/claude-3-5-sonnet-latest" (needs ANTHROPIC_API_KEY) and compare the number of steps and the token cost against the default on the same question.
Add requests_per_minute= to a make_model() override and watch the behavior when Quill chains calls — it’s the client-side rate-limit guard inherited from ApiModel.

We wire the model layer now; the executor that runs Quill’s Python safely is Module 5. Quill still runs in the local executor for now — no executor_type, no "*" wildcard.

In production

What changes when this stops being a lab:

Cost. A single Quill run is 5–15 LLM calls. At $0.10/month of free credits (as of smolagents 1.26.0), you are out of credits in a handful of runs. In production you budget per run and cap it — the Monitor line from the lab graduates from a print statement to a real guard-rail. My rule of thumb: a CodeAgent that loops is also a CodeAgent that bills, so I watch get_total_token_counts() and max_steps together.

Rate limits and reliability. The inherited ApiModel defaults — requests_per_minute (client-side throttle) and retry=True — are the minimum. I cap requests_per_minute at the provider’s documented limit so a busy agent never trips a 429. Under sustained load, reach for LiteLLMRouterModel (fallbacks, cooldowns, cross-deployment retries) or a dedicated provider.

Model pinning. A hosted model can change under you — InferenceClientModel’s own default is documented as “may change”. The explicit model_id in config.py is a production decision, not a detail. Pin it; review it on purpose.

Data residency and privacy. Sensitive data should not leave your boundary. That is when a local backend (VLLMModel/TransformersModel) or a private cloud (AzureOpenAIModel/AmazonBedrockModel) earns its operational cost — and why the factory pattern pays off: switching is one env var, not a code review.

One thing this layer deliberately leaves for later: making the model return validated, structured output per backend (response_format) is Module 8 — here it’s only a parameter that exists on generate().

Concept check

Why this matters: Quill — and every agent you build after this — only runs if you can wire it to a model and afford to run it. Backend choice and cost control aren’t side topics; they gate everything downstream. Get the model layer right and the rest of the course is “what does Quill do”; get it wrong and you’re editing agent.py and bleeding credits forever.

Five scenarios. Answers and explanations are grouped after all five.

1. A small team wants to start building agents for free, with a single token and no extra installs. Which backend?

A. LiteLLMModel("gpt-4o")
B. InferenceClientModel(model_id="Qwen/Qwen2.5-Coder-32B-Instruct") with provider="auto"
C. TransformersModel(...) on a local GPU
D. OpenAIModel(...) pointed at OpenAI

2. You need the most reliable agentic behavior and want to use Claude. What’s the correct swap?

A. InferenceClientModel(model_id="anthropic/claude-3-5-sonnet-latest")
B. A local TransformersModel with a 3B model
C. LiteLLMModel("anthropic/claude-3-5-sonnet-latest") with the [litellm] extra and ANTHROPIC_API_KEY
D. AmazonBedrockModel with no credentials

3. What does provider="auto" do in InferenceClientModel?

A. Runs the model on a dedicated Hugging Face GPU
B. Routes the request to the first available partner provider via the HF router
C. Automatically picks the best model for your question
D. Disables billing

4. A developer sets a small 3B model locally and Quill starts looping and writing broken code. Most likely cause?

A. A bug in smolagents’ agent loop
B. The model isn’t code-capable enough and/or the context window is too small (num_ctx)
C. make_model() returned the wrong class
D. The HF free tier ran out

5. How do you read the token cost of a finished run?

A. agent.logs
B. A total_tokens attribute on the agent
C. agent.monitor.get_total_token_counts()
D. Re-running the agent with verbose=True

Answers:

B. InferenceClientModel with provider="auto" needs only one HF_TOKEN and ships with smolagents — no extra install, free-tier credits included. A and D require provider keys/installs; C needs hardware.
C. Frontier closed models come through LiteLLMModel with the [litellm] extra and the provider’s key. InferenceClientModel (A) is the HF-routed path, not the way you’d target Claude directly; B trades away reliability; D fails without credentials.
B. "auto" routes to the first available partner provider through the HF router — it does not run on a first-party HF GPU and does not choose your model.
B. A CodeAgent needs a code-capable model with a large context; small local models produce malformed code, loop, or mis-call tools. The fix is a stronger coder model and a larger num_ctx/max_model_len (or a hosted backend) — not a smolagents bug.
C. agent.monitor.get_total_token_counts() returns a TokenUsage. agent.logs and legacy token attributes were removed in 1.21.0 — any tutorial using them is stale.

Common pitfalls:

HfApiModel is dead. Any snippet doing from smolagents import HfApiModel is an out-of-date tutorial — the class was renamed and is not importable in 1.26.0 (ImportError). Use InferenceClientModel. (The same goes for agent.logs → agent.memory.steps / agent.monitor.)
“The free tier is enough to run everything.” It isn’t — $0.10/month (as of smolagents 1.26.0) disappears in a few CodeAgent runs. Plan for a PRO subscription, a free provider key (Groq/Cerebras), or local.
Confusing InferenceClientModel (a multi-provider router) with a single HF server. With provider="auto", the compute comes from a third-party partner — so latency, cost, and availability vary, and changing model_id can change who answers.

Key takeaways

Every model is callable; subclasses implement generate() (not __call__), which always returns a ChatMessage.
ApiModel gives every API backend client-side rate limiting (requests_per_minute) and retries (retry=True) — inherited, not reimplemented.
InferenceClientModel is a router/client, not Hugging Face’s own model server; provider="auto" serves you from a third-party partner.
The default model_id is documented as “may change” (as of smolagents 1.26.0) — Quill always passes an explicit one.
The HF free tier is $0.10/month (as of smolagents 1.26.0, subject to change) — tiny; a multi-step CodeAgent burns it fast.
LiteLLMModel unlocks 100+ providers via the "<provider>/<model>" id format; raise Ollama’s num_ctx above its 2048 default.
make_model() is the one place to choose what powers Quill — swap with QUILL_MODEL_BACKEND/QUILL_MODEL_ID, no agent-code edit.
Monitor.get_total_token_counts() returns a TokenUsage for the whole run — your cost in plain sight.

What’s next

Want the full smolagents concept-check bank and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

Now that Quill can talk to any model, Module 5 makes sure the Python it writes can’t hurt you — sandboxing untrusted agent code.

Previous: Module 3 — Tools: Giving Quill New Powers
Next: Module 5 — Running Untrusted Code Safely: Sandboxing smolagents
Course index: smolagents Mastery
Lab code: smolagents-course-labs/module-04 (tests pass offline)

References

smolagents — Models API reference: https://huggingface.co/docs/smolagents/reference/models
smolagents — Guided tour: https://huggingface.co/docs/smolagents/guided_tour
HF Inference Providers — index (routing, partners): https://huggingface.co/docs/inference-providers/index
HF Inference Providers — pricing & free tier: https://huggingface.co/docs/inference-providers/pricing
smolagents source models.py @ v1.26.0: https://github.com/huggingface/smolagents/blob/v1.26.0/src/smolagents/models.py
LiteLLM — supported providers: https://docs.litellm.ai/docs/providers

In this module

Where we are

How an agent calls a model: the Model abstraction

The base Model and its uniform kwargs pattern

generate() — and how your tools reach the model

ApiModel — where rate limiting and retries live

The data types: ChatMessage, roles, TokenUsage