aifromzero.dev

When (Not) to Fine-Tune: The Post-Training Landscape (Fine-Tuning in Production, Module 1)

Module 1 of 15 16 min read Lab code ↗

This is Module 1 of Fine-Tuning in Production, a free 15-module course that takes one open model from raw data to a deployed, published tool-calling specialist on the Hugging Face Hub. Browse the full syllabus.

”Just fine-tune it” is the most expensive sentence in ML

A model gets something wrong once, and somebody on the team says it: “Let’s just fine-tune it.” That sentence has launched more wasted GPU hours than any other in this field. Most of the time it’s the wrong move — you’re about to burn a training run on a problem a better prompt or a retrieval step would have closed in an afternoon, with no weights touched and no GPU rented.

And then there’s the opposite mistake, which is just as expensive: the problems that only fine-tuning can solve, where you sit there re-wording a prompt for two weeks because nobody told you that small base models will not reliably emit a strict output format no matter how you ask. There, prompting forever is the waste.

Here’s the trap for anyone starting out: half the tutorials on the internet teach idioms that have been dead for a year — PPO-RLHF as the default alignment method, tokenizer= on a trainer, transformers==4.x — and almost none of them tell you clearly whether you should fine-tune at all, or when. This module fixes the “when.” By the end you’ll be able to decide whether to fine-tune for a given problem, place every modern post-training method on one map — and you’ll have watched a real base model fail the exact task this course spends the next 14 modules teaching it to do.

In this module

You’ll learn how to:

  1. Climb the escalation ladder — prompting → RAG → fine-tuning — and decide which rung to stop on (the “RAG for facts, fine-tuning for behavior” rule).
  2. Recognize when not to fine-tune: the cases where a better prompt, retrieval, or a bigger context window wins — and why fine-tuning is not how you inject fresh facts.
  3. Map the modern post-training landscape — supervised fine-tuning (SFT), parameter-efficient fine-tuning (PEFT), preference optimization (DPO and family), and reinforcement learning (GRPO/RLVR) — and see how Anvil walks through every stage.
  4. Make the small-specialist bet: argue when a 4B open model fine-tuned for one task can beat a frontier API — the SLM that could power Quill or Forge instead of a hosted model.

You’ll build: your dev environment plus Anvil’s baseline — the pinned post-training stack installed on a free Colab/Kaggle T4, your HF_TOKEN wired up, and a measured run of Qwen3-4B-Base on a handful of tool-call prompts, where you’ll watch it fail the format. No training yet: that starts in Module 3, on data from Module 2.

Theory areas covered:

  • T1 — When/why to fine-tune & the post-training landscape — 7% of the course.

T1 is the lightest area by weight, but it is the conceptual foundation everything else stands on, and this module owns it in full — it’s covered completely here, then reinforced only at the capstone (Module 15). Get this map right and the other eleven areas have somewhere to live.

Prerequisites: solid Python, comfort with an LLM API (any provider) and the command line. No cloud experience and no deep-learning-from-scratch required. The only asset you need is a Hugging Face account and an HF_TOKEN (you’ll create it in the lab), plus free access to Colab or Kaggle (a 16 GB T4).

The series, and the model you’ll forge

The promise of this course is one sentence:

Take ONE open base model and ONE dataset all the way through the modern post-training pipeline — SFT, LoRA/QLoRA, DPO, GRPO, task-grounded evaluation, quantization, and serving — and ship a tool-calling specialist to the Hugging Face Hub, on a free Colab GPU.

The thing you forge is Anvil:

Anvil is a small open base model (Qwen3-4B) hammered into a production-grade tool-calling specialist: you curate the data, supervise-fine-tune it, make it fit a free 16 GB GPU with QLoRA, align it with DPO, push its reasoning with GRPO, prove it beats the base model with task-grounded evals, quantize it to GGUF, serve it with vLLM, and publish it to the Hub with a complete model card — the kind of small specialist that could power Quill or Forge instead of a frontier API.

That last clause is the whole point of this course’s place in the catalog. The catalog’s other courses — Quill, the code-first data analyst (course 3), and Forge, the LangGraph agent (course 4) — consume a model behind an API. Here you build the model they could run instead. This is the model layer, not the app layer. You’re forging the SLM (small language model) that a high-volume, single-task system would deploy in place of a frontier API: cheaper to serve, deployable anywhere, no vendor lock-in, under a license you control.

Here’s Anvil’s target architecture, annotated with the module where each piece arrives. You build this over 15 modules. Don’t try to understand every box yet — this is the map, not the territory. From now on, every module marks where you are with ✅ (done), 👉 (you’re here), and ⬜ (ahead):

flowchart TB
    base["Qwen3-4B-Base<br/>(Apache-2.0)"] --> data["data + chat template<br/>(M2) ⬜"]
    data --> sft["SFT<br/>(M3) ⬜"]
    sft --> lora["LoRA (M4) ⬜<br/>QLoRA (M5) ⬜"]
    lora --> accel["accelerators (optional):<br/>Unsloth / Axolotl (M6) ⬜"]
    accel --> eval["task-grounded eval (M7) ⬜<br/>(re-run after each stage)"]
    eval --> dpo["DPO (M8) ⬜<br/>ORPO·KTO·SimPO (M9) ⬜"]
    dpo --> grpo["GRPO + RLVR<br/>(M10) ⬜"]
    grpo --> merge["merge<br/>(M11) ⬜"]
    merge --> quant["quantize<br/>GGUF/AWQ/GPTQ (M12) ⬜"]
    quant --> serve["serve<br/>vLLM/Ollama/TGI (M13) ⬜"]
    serve --> publish["publish to the Hub<br/>+ model card (M14) ⬜"]
    publish --> capstone["capstone<br/>(M15) ⬜"]

    start["👉 M1: setup + baseline"] -.-> base

Running through every stage are four things that stay constant: a single model/tokenizer factory, get_model_and_tokenizer(tier), defined in one place from Module 3 on; the GPU path (a free T4 by default, a priced rented path for the heavy labs); reproducibility (fixed seeds, a committed uv.lock, an offline smoke test); and a task-grounded eval that gets re-run after every training stage. Here’s the full road map:

  • 👉 M1 — When (Not) to Fine-Tune. Setup + a measured base-model baseline.
  • ⬜ M2 — Data Is the Model: Curation and Chat Templates. Anvil’s data, formatted right.
  • ⬜ M3 — Your First Fine-Tune: SFT from Scratch. Anvil v0 learns the tool-call format.
  • ⬜ M4 — LoRA: Train 1% of the Weights. Swappable, lightweight adapters.
  • ⬜ M5 — Make It Fit a Free GPU: QLoRA from Scratch. Anvil fits a 16 GB T4.
  • ⬜ M6 — Go Faster: Unsloth and Config-as-Code (Axolotl). Optional accelerators.
  • ⬜ M7 — Is It Any Better? Task-Grounded Evaluation. Base vs fine-tuned, on Anvil’s task.
  • ⬜ M8 — Align It: Preference Tuning with DPO. Anvil prefers the right tool call.
  • ⬜ M9 — The DPO Family: ORPO, KTO, SimPO. Pick the method for your data.
  • ⬜ M10 — Teach It to Reason: GRPO and Verifiable Rewards. Reward = a checker.
  • ⬜ M11 — Free Capability Boosts: Model Merging with mergekit. Fuse variants.
  • ⬜ M12 — Shrink It for Serving: GGUF, AWQ, GPTQ. Inference quantization.
  • ⬜ M13 — Serve It: vLLM, Ollama, and TGI. Real endpoints, measured latency.
  • ⬜ M14 — Ship It: Publishing, Model Cards, and Reproducibility. Anvil on the Hub.
  • ⬜ M15 — Capstone: Anvil, End to End. The whole pipeline, replayed.

Each module shows the theory areas (TCM) it covers up top, and ships one increment of Anvil plus the theory behind it.

One honest note on cost before you read further, because the GPU is the structuring constraint of this course, not a footnote. The default path is free to learn: every lab is sized to fit a free Colab or Kaggle T4 (16 GB) using QLoRA, with Unsloth as an optional accelerator. A few heavy labs — GRPO in Module 10, full fine-tuning — document a rented-GPU path with a real measured price (“this lab cost me ~$X.XX”). I won’t pretend the free-tier numbers are permanent: as of June 2026 Colab gives you roughly 15–30 GPU-hours a week and Kaggle guarantees around 30 hours, but those are community figures, subject to change, and you should treat them as such. For Module 1 — a model load and a few generations, no training — the free tier is plenty, and the bill is $0.00.

The escalation ladder: prompting → RAG → fine-tuning [T1: escalation ladder, place in the stack]

Treat “how do I make this model better at my task?” as an engineering decision with a cost ladder, not a default. The escalation ladder is a discipline: you climb the rungs in order, and you stop at the first one that solves the problem — because each rung costs more than the one below it, in complexity, in data, and in GPU.

  • Prompting / in-context learning. Re-word the instruction, add a few-shot example, structure the output. Cost is near zero; you iterate in seconds. This is the correct first reflex, every time.
  • RAG (retrieval-augmented generation) — wiring a knowledge base or search index to the model so it gets the facts it needs at query time. Moderate cost; it updates the model’s knowledge without touching a single weight.
  • Fine-tuning — changing the model’s weights to durably change its behavior: its output format, its style, the way it reasons about a task. Highest cost (you need data, GPU, and an evaluation), but it’s the only lever on intrinsic behavior.

As a decision tree, the ladder looks like this. It’s the thread of the whole module:

flowchart TD
    start([Your model isn't good enough]) --> q1{Is it missing<br/>facts / knowledge?}
    q1 -->|Yes| rag["RAG or a bigger context<br/><i>cheap–moderate · facts</i>"]
    q1 -->|No| q2{Just under-specified,<br/>vague prompt?}
    q2 -->|Yes| prompt["Better prompt / few-shot /<br/>structured output<br/><i>cheapest · behavior nudge</i>"]
    q2 -->|No| q3{Wrong behavior / format / style,<br/>even with a good prompt?}
    q3 -->|Yes| ft["Fine-tuning<br/><i>most expensive · behavior</i>"]
    q3 -->|No| prompt

Read the costs on the leaves. Prompting is the cheapest behavior nudge; RAG is cheap-to-moderate and the answer when facts are missing; fine-tuning sits at the bottom, the most expensive, and it’s the answer only when the behavior is wrong despite a good prompt. Here is the same idea as a table you can keep:

RungWhat it changesCostStop here when…
PromptingThe model’s behavior for this request, transientlyNear zero, secondsA clearer instruction, a few-shot example, or a structured-output ask fixes it
RAGThe facts available at query timeModerate (index + retrieval)The model is right when handed the facts — it just didn’t have them
Fine-tuningThe model’s weights → durable behaviorHighest (data + GPU + eval)Behavior/format/style is wrong even with a good prompt and the right facts in context

The rule to hammer into memory: “RAG for facts, fine-tuning for behavior.” I present this as practitioner consensus, not a hard number — the often-cited statistics (something like “60% of real systems end up hybrid,” or “fine-tune past a ~200k-token context budget”) come from vendor blogs, not peer-reviewed work, so don’t quote them as fact. What is solid: real production systems are frequently hybrid — RAG feeding fresh facts into a model that was fine-tuned for behavior. The two levers are orthogonal, which is exactly why you can pull both.

Why is fine-tuning a bad vehicle for facts? This is the internals-level reason that justifies the ladder’s order, and it’s worth being precise about. Training shifts a distribution over behavior — it nudges the model toward producing outputs that look like your demonstrations. It does not write down retrievable, updatable facts the way a database row does. A fact you fine-tune in today is frozen into the weights; when it changes tomorrow, the model is silently stale and the only fix is another training run. RAG serves the fact at query time, so the moment your knowledge base updates, the answer updates — no retraining. That asymmetry is the whole argument for climbing the cheaper rungs first.

When NOT to fine-tune [T1: when not to fine-tune]

Most tutorials are a sales funnel for fine-tuning. Here’s the honest list of cases where it’s the wrong choice — practitioner consensus, one example each:

  1. You need fresh or proprietary facts. Your catalog prices change weekly, your internal docs change daily — fine-tune those in and the model re-goes-stale the instant a fact changes. Use RAG.
  2. You haven’t seriously worked the prompt yet. A sharper instruction, a couple of few-shot examples, or a structured-output request often closes the gap in seconds, with no GPU. Climb the cheap rung first.
  3. You don’t have — and can’t get — quality data. Garbage in, garbage out: a fine-tune on mediocre data degrades the model. (Curating Anvil’s data is the entire subject of Module 2 — it’s that important.)
  4. A bigger model or a wider context would do. Sometimes the answer is “use a stronger model” or “put more in the context window,” not “train.” Cheaper to try, faster to ship.
  5. You can’t measure success. With no task-grounded evaluation, you will never know whether the fine-tune helped or hurt. (Evaluation is Module 7 — and it’s the single biggest gap in competing material, which is why we treat it early.)

⚠️ Common misconception: “Fine-tuning is how you teach the model new facts / your latest data.”

No. Fine-tuning changes behavior — the output format, the style, the way the model performs a task. It does not give the model a retrievable, up-to-date base of facts; for facts, that’s RAG. Confusing the two is the single error that wastes the most GPU in this field: someone fine-tunes a model on last quarter’s data, ships it, and is mystified when it can’t answer about this quarter — because the fact was baked into a distribution, not stored in a lookup. Come back to the rule from the last section: RAG for facts, fine-tuning for behavior.

Now the flip side, and the bridge to the rest of this course. There is a case where fine-tuning is exactly the right tool: getting a small model to produce a strict output format, reliably, on every single call — a valid JSON tool call, with the right tool name and the right arguments, every time. That is a behavior, not a fact. And no amount of prompting guarantees it on a small base model — you’ll get it 80% of the time and the other 20% will be malformed JSON or prose, which is useless in a pipeline that has to parse the output. That 100%-reliable structured-output behavior is precisely Anvil’s task, and it’s what the lab below shows the base model failing at.

The post-training landscape: SFT, PEFT, preferences, RL [T1: the post-training landscape]

Post-training is the umbrella term for everything you do to a model after pretraining to specialize or align it. This course walks through four families of it. Here’s each one at two levels — (a) what it is, and (b) why it exists and what it acts on, with the founding paper cited by its verified arXiv ID.

  • Supervised fine-tuning (SFT) — next-token loss on demonstrations: pairs of (input → the desired output). This is how you teach a model a format. It’s the workhorse of the whole pipeline, and the modern instruction-tuning paradigm traces to InstructGPT (arXiv 2203.02155), which showed that supervising a base model on human demonstrations makes it dramatically more useful than raw scale alone. → owned by Module 3.

  • Parameter-efficient fine-tuning (PEFT) — train only a tiny fraction of the weights instead of all of them. LoRA (arXiv 2106.09685) freezes the base model and injects small low-rank matrices into the attention/MLP projections, so you train millions of parameters instead of billions and ship a few-MB adapter. QLoRA (arXiv 2305.14314) goes further: it quantizes the frozen base to 4-bit so the whole thing fits in consumer VRAM, then trains the LoRA adapter on top. QLoRA is exactly what makes Anvil trainable on a free 16 GB T4. → owned by Modules 4 and 5.

  • Preference optimization — align the model to preferences expressed as (prompt, chosen, rejected) triples: “given this prompt, this output is better than that one.” DPO (Direct Preference Optimization, arXiv 2305.18290) is the modern default. Its insight is that you can optimize directly against the preference data with a simple classification-style loss — no separate reward model to train, no online RL loop to babysit — which makes it far more stable and cheaper than the historical RLHF-PPO recipe. The family around it (ORPO, KTO, SimPO) trades off what kind of data you need and whether you run an SFT stage first. → owned by Modules 8 and 9.

  • Reinforcement learning / reasoning — optimize the model against a reward signal. GRPO (Group Relative Policy Optimization, from DeepSeekMath, arXiv 2402.03300) drops PPO’s learned value network and instead scores a group of sampled completions relative to each other — cheaper and more stable for this regime. Pair it with RLVR (reinforcement learning from verifiable rewards, formalized in Tülu 3, arXiv 2411.15124) and the reward becomes a deterministic verifier — “is this tool call valid and correct?” — rather than a learned, gameable reward model. That’s why RL is so attractive for tool-calling: correctness is checkable. (This is the wave DeepSeek-R1, arXiv 2501.12948, rode to fame.) → owned by Module 10.

Here is the landscape as one table. It doubles as the table of contents for the series:

FamilyWhat it changesData it needsFounding paper (arXiv)Anvil module
SFTBehavior / output format, via next-token loss on demosDemonstrations (input → desired output)InstructGPT 2203.02155M3
PEFT (LoRA · QLoRA)Which weights you train (a tiny low-rank slice)Same as SFTLoRA 2106.09685, QLoRA 2305.14314M4, M5
Preferences (DPO · family)Preference between two outputs(prompt, chosen, rejected) pairsDPO 2305.18290M8, M9
RL (GRPO · RLVR)Reward-driven behaviorPrompts + a verifiable reward functionGRPO 2402.03300, RLVR 2411.15124M10

There’s a canonical order to all this: data → SFT → PEFT → preferences → RL → evaluation → quantization → serving → publish. That is exactly the order of this course’s modules, and Anvil walks the whole thing on one model and one dataset — which is the property the competing material doesn’t have. Most resources teach SFT on one toy model, DPO on another, GRPO on a third, and never connect them; you’ll carry a single artifact through every stage and re-evaluate it each time.

One freshness marker that distinguishes this course from the dated tutorials, stated plainly: the default alignment method in 2026 is DPO (with GRPO for the RL/reasoning regime), not RLHF-PPO. PPO is historical context — in the current stable TRL (verified at trl==1.6.0, June 2026) it has been moved into trl.experimental.ppo, while DPO and GRPO are first-class, stable trainers. If a tutorial presents PPO as the standard way to align an LLM, it’s teaching you 2023.

The small-specialist bet: why a 4B model can win [T1: the small-specialist bet, place in the stack]

The bet this whole course rests on: on one well-defined, measurable task, a small open model that you fine-tune can beat a frontier API. Not in general — a 4B model will lose a broad benchmark to a 200B+ frontier model all day. But on a single narrow task, the fine-tuned specialist is cheaper to serve, deployable wherever you want (including on-prem or at the edge), free of vendor lock-in, and under a license you control.

What makes the bet defensible rather than wishful is measurability. Anvil’s task — tool-calling / structured output — is exactly the kind you can put a number on: is the output valid JSON? Is the tool name right? Do the arguments match? You can score the base model and the fine-tuned model on a held-out set and see the gap close. A “make the model sound nicer” task has no such yardstick; a tool-calling task does. (Building those exact metrics is Module 7.)

Why Qwen3-4B-Base? Three reasons. It fits in QLoRA on a free 16 GB T4. It ships with a tool-calling-aware chat template already, so the format we’re teaching has somewhere to render (more on that in Module 2). And — this is the one production engineers care about — it’s Apache 2.0, so you can republish your derivative with no naming clause and no monthly-active-user threshold. If you need an alternative: SmolLM3-3B (Apache 2.0) is the documented fallback, and Phi-4-mini (MIT) is a third option. I deliberately do not default to Llama 3.2 (its license carries a “Llama”/“Built with Llama” naming clause and a 700M-MAU threshold) or Gemma 3 (custom terms) — those clauses propagate to your derivative, and the point of this course is that you walk away owning what you built.

Base vs Instruct — a distinction to fix now. We start from a -Base model, not a -Instruct one. A base model has been pretrained on raw text but has not yet been instruction-tuned or taught to follow a chat/tool-call format. That’s the point: we’re going to teach it that behavior ourselves, which is the entire course. It’s also exactly why the baseline lab below will fail the format — a base model has never been shown how to answer with a structured tool call, so it won’t, and that’s not a bug, it’s the starting line.

What you walk away with is the portfolio artifact. There’s no certification for this layer of the stack, so the “badge” is your model, published on the Hub with a complete model card, a base_model: Qwen/Qwen3-4B-Base lineage, and base-vs-fine-tuned eval curves — public, reproducible, and showable in an interview. (That’s Module 14.)

Finally, where this sits in the stack. The course’s core is TRL + PEFT + bitsandbytes + datasets + accelerate, with Unsloth as an optional accelerator and llama.cpp/vLLM at serving time — all open, all vendor-neutral. And the freshness moat is real: we teach TRL v1 idioms only (processing_class rather than tokenizer=, max_length rather than max_seq_length, the stable DPO/GRPO trainers), verified against the installed library, against the grain of the dated tutorials that still dominate search results. We’re not detailing those APIs here — that starts in Module 3 — just framing where everything lives.

Build it: Anvil’s baseline (and your environment)

Goal: set up the post-training stack on a free GPU, wire up your HF_TOKEN, and run Qwen/Qwen3-4B-Base on a few tool-call prompts to watch it fail the format — the gap the rest of the course closes.

What you’ll see: for each prompt, the model’s raw output (free-form prose, broken JSON, or no usable tool call) and a per-example verdict, then a final line: base model produced 0/3 valid tool calls — this is the gap the course closes.

The full, tested code lives in finetune-prod-labs/module-01. The smoke test runs offline — no GPU, no network, no token — so you can verify the harness before you load a single weight.

Step 1 — Get a Hugging Face token. Create a free account, then a fine-grained token at https://huggingface.co/settings/tokens with “Read access to public gated repos” and “Write access to your repos” (you’ll need write access to push Anvil in Module 14). Copy .env.example to .env and paste it in. Never commit .env — the repo’s .gitignore already excludes it, and the token is read from the environment, never hard-coded:

# .env  (copied from .env.example — never committed)
HF_TOKEN=hf_xxx

Step 2 — Clone the labs repo and install with uv (Python 3.12). The repo uses one shared root pyproject.toml and a committed uv.lock for the whole course — every module-NN/ ships the full cumulative anvil/ tree, but they share the locked core stack. GPU-only and serving/eval tooling (bitsandbytes, vLLM, lighteval, …) is installed per runtime later, because each brings its own conflicting torch/trl pins — that’s documented in the pyproject.toml comments. Module 1 needs only the locked core:

git clone https://github.com/dupuis1212/finetune-prod-labs && cd finetune-prod-labs
uv sync   # installs the pinned core: torch, transformers, trl, peft, datasets, accelerate

Step 3 — Verify your pins. The freshness law of this course starts on day one: print the installed versions and confirm they match the lock. These are verified against PyPI as of June 2026 — trl==1.6.0 (the 1.x line shipped; the old “never trl==1.0” advice is obsolete, you pin the exact version), transformers==5.12.0, peft==0.19.1, datasets==5.0.0, accelerate==1.14.0:

uv run python -c "import trl, transformers, peft; \
print('trl', trl.__version__, '| transformers', transformers.__version__, '| peft', peft.__version__)"
trl 1.6.0 | transformers 5.12.0 | peft 0.19.1

Step 4 — Pick your GPU. The default is the free path: open the module’s notebook.ipynb in Colab or Kaggle (a 16 GB T4) and run it there. A quick sanity check confirms the GPU is visible:

import torch
print("CUDA available:", torch.cuda.is_available())   # True on a Colab/Kaggle T4 runtime

Module 1 doesn’t train, so even CPU works for the offline test — but the live baseline wants the T4. Free-tier hours are community figures, subject to change (see the cost note above), so don’t treat them as guaranteed.

Step 5 — The baseline harness. The whole module lives in anvil/baseline.py. The model ID is inline on purpose — the frozen get_model_and_tokenizer factory isn’t born until Module 3, and Module 1 freezes nothing:

MODEL_ID = "Qwen/Qwen3-4B-Base"                 # Apache-2.0, base (not -Instruct): we teach the behavior
FALLBACK_MODEL_ID = "HuggingFaceTB/SmolLM3-3B"  # Apache-2.0 alternative if Qwen3 is unavailable


def load_base_model(model_id: str = MODEL_ID, *, local_files_only: bool = False):
    """Load (model, tokenizer) for the base model. Reads HF_TOKEN from the environment if set."""
    import torch
    from transformers import AutoModelForCausalLM, AutoTokenizer

    tokenizer = AutoTokenizer.from_pretrained(model_id, local_files_only=local_files_only)
    if tokenizer.pad_token is None:
        tokenizer.pad_token = tokenizer.eos_token
    model = AutoModelForCausalLM.from_pretrained(
        model_id, dtype=torch.float32, local_files_only=local_files_only)
    model.eval()
    return model, tokenizer

Step 6 — Build the prompt by hand (deliberately). We construct the prompt with plain string concatenation, not apply_chat_template. Chat templates are Module 2’s whole subject; here we want to feel the naive approach fall over. This is build_naive_prompt:

def build_naive_prompt(example: dict) -> str:
    """A hand-rolled prompt — exactly what you'd try before you know about chat templates (M2)."""
    tools_json = json.dumps(example["tools"])
    return (
        "You have access to these tools (JSON schema):\n"
        f"{tools_json}\n\n"
        f"User: {example['query']}\n"
        "Reply with a single tool call as JSON.\n"
        "Assistant:"
    )

Step 7 — Run it and judge qualitatively. run_baseline generates a greedy completion for each of the three BASELINE_EXAMPLES (weather, currency conversion, note search) and runs looks_like_tool_call over the output — a qualitative check (does it contain a parseable JSON object with a name field, or a <tool_call> block?). The real, frozen metrics arrive in Module 7; here we just want to see the failure:

def run_baseline(model, tokenizer, examples: list[dict] | None = None,
                 max_new_tokens: int = 64) -> list[dict]:
    import torch

    examples = examples if examples is not None else BASELINE_EXAMPLES
    results = []
    for ex in examples:
        prompt = build_naive_prompt(ex)
        inputs = tokenizer(prompt, return_tensors="pt")
        with torch.no_grad():
            out = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False,
                                  pad_token_id=tokenizer.pad_token_id)
        generated = out[0][inputs["input_ids"].shape[1]:]
        raw = tokenizer.decode(generated, skip_special_tokens=True)
        results.append({"query": ex["query"], "raw_output": raw,
                        "tool_call_parseable": looks_like_tool_call(raw)})
    return results

Run the whole thing from the repo root:

uv run --env-file .env python -m anvil.baseline

You’ll see something like this (trimmed). The base model rambles, half-completes a schema, or invents a conversation — anything but a clean, parseable tool call:

Q: What's the weather in Paris right now?
  raw: ' Sure! The weather in Paris is currently sunny with a high of 24°C. Let me know if...'
  tool_call_parseable: False

Q: Convert 100 US dollars to euros.
  raw: ' { "tool": "convert", "amount": 100, currency: "USD" -> ...'
  tool_call_parseable: False
...
base model produced 0/3 valid tool calls — this is the gap the course closes

That’s expected — a base model has never been taught this format. Don’t read it as the model being “bad”; read it as the starting line. The next 14 modules close that 0/3.

Step 8 — The smoke test. The lab ships a smoke test that verifies the harness without a GPU, a network, or a token, by running run_baseline against a tiny Qwen3 fixture shipped in the repo (local_files_only=True). It checks the plumbing — that run_baseline returns well-formed {query, raw_output, tool_call_parseable} dicts, that the detector behaves, that summarize reports the score — and does not assert the model succeeds (a random tiny model won’t, just like the real base model won’t):

def test_run_baseline_offline_returns_verdicts(tiny_model, tiny_tokenizer):
    results = run_baseline(tiny_model(), tiny_tokenizer, examples=BASELINE_EXAMPLES[:2],
                           max_new_tokens=8)
    assert len(results) == 2
    for r in results:
        assert set(r) == {"query", "raw_output", "tool_call_parseable"}
    msg = summarize(results)
    assert "gap the course closes" in msg

There’s exactly one live test that loads the real Qwen/Qwen3-4B-Base; it’s skipped unless you opt in. Run the offline suite (no token needed):

uv run pytest module-01/tests/
# 3 passed, 1 skipped

To also reproduce the article’s live baseline (needs HF_TOKEN and a download):

ANVIL_LIVE_TESTS=1 HF_TOKEN=hf_xxx uv run pytest module-01/tests/

Try it yourself:

  1. Add two or three more tool-call examples to BASELINE_EXAMPLES and see whether the base model gets one right by accident. (It might! That’s the point — eyeballing a handful of examples is not measurement, which is exactly why Module 7 builds a real eval.)
  2. Swap MODEL_ID for the fallback HuggingFaceTB/SmolLM3-3B and compare how it fails the format. (Don’t worry about why it differs — that’s the territory of Modules 3 onward.)

What this lab does NOT do (yet), on purpose. No training (SFT is Module 3). No apply_chat_template or data formatting (Module 2). No QLoRA or BitsAndBytesConfig (Module 5). No frozen eval metrics — the baseline is qualitative; anvil/eval.py is born in Module 7. No DPO or GRPO (Modules 8 and 10). No Hub push (Module 14). No training, no chat template, no eval metrics yet — Module 1 just proves the base model can’t do the job, so the next 14 modules have a reason to exist.

In production

Three things change the moment this stops being a bootstrap lab.

The GPU is a real constraint, not a detail. The free Colab/Kaggle tier is enough to learn — a 4B QLoRA fits on a 16 GB T4 — but a serious production run costs GPU-hours, and those add up. A datable order of magnitude: as of June 2026, a 30-minute 8B QLoRA run on a rented A100 is roughly $0.60–$1.25, depending on the provider. That’s a teachable number, not a fixed tariff — providers and spot prices move, so re-check before you budget. Every heavy lab in this course prints its real measured cost (“this lab cost me ~$X.XX”), because cost discipline is part of the skill.

“Should we even fine-tune?” is the first cost question. In a company, the right reflex is not to fine-tune by default — it’s to climb the ladder (prompt → RAG → fine-tune) and stop at the first rung that passes your eval. Fine-tuning without a task-grounded evaluation is spending GPU blind: you can’t tell whether you helped or hurt, and you’ll have a model you can’t defend in review. (That’s why evaluation, Module 7, comes right after the first trained model — not at the end.)

The base model’s license is a production concern. You’re republishing a derivative. Qwen3-4B-Base is Apache 2.0, so your derivative is republishable with no naming clause; some licenses (Llama’s, Gemma 3’s) carry clauses that propagate to whatever you build on top — naming requirements, usage restrictions, MAU thresholds. Reading the license before you pick the base model is not bureaucracy; it’s the difference between owning your artifact and renting it.

Concept check

Why this matters. This module anchors the decisions you’ll make before every training run for the rest of your career: whether and when to fine-tune (the escalation ladder), facts (RAG) vs behavior (fine-tuning), where SFT/PEFT/preferences/RL sit on the map, and why the small-specialist bet can pay off. T1 is covered in full here and only reinforced at the capstone (Module 15). The questions below stay inside what this module actually taught.

  1. A support team wants its assistant to answer with the latest product-catalog numbers, which change every week. What should they reach for first?

    • A. Fine-tune the model weekly on the new catalog.
    • B. RAG — retrieve the current numbers at query time; the facts change, the weights shouldn’t.
    • C. GRPO with a verifiable reward on catalog accuracy.
    • D. A bigger base model.

  2. A developer wants to fine-tune because the model “answers badly,” but they’ve never structured the prompt or gathered any training data. What’s the right next step?

    • A. Launch an SFT run immediately — behavior problems need training.
    • B. Climb the cheaper rungs first: a sharper prompt, few-shot examples, structured-output requests.
    • C. Switch to DPO, since it’s the modern default.
    • D. Quantize the model to GGUF and serve it faster.

  3. You need a small model to emit a valid JSON tool call — right tool name, right arguments — on every call, and no amount of prompting makes it reliable. Is this a fine-tuning case, and why?

    • A. No — use RAG; the model is missing facts.
    • B. No — just keep iterating on the prompt; format is always promptable.
    • C. Yes — a strict, reliable output format is a behavior, and behavior is what fine-tuning changes.
    • D. Yes — but only because bigger models can’t do tool calls.

  4. Someone describes a method as “align the model on (chosen, rejected) pairs, with no reward model and no online RL loop.” Which method is it, and what is it not?

    • A. GRPO — and it is the historical RLHF-PPO recipe.
    • B. SFT — and it’s the same as instruction tuning.
    • C. DPO — and it is not RLHF-PPO; PPO is the older recipe, now experimental in TRL.
    • D. KTO — and it needs paired data, unlike DPO.

  5. A startup pays a lot for a frontier API to handle one high-volume tool-calling task. Why is fine-tuning a small open model a defensible bet here — and what makes it defensible?

    • A. It isn’t — a 4B model always loses to a frontier API.
    • B. It is — the task is measurable (valid JSON + tool name + args), so you can prove the specialist matches or beats the API, and it’s cheaper to serve under your own license.
    • C. It is — because small models are always cheaper, regardless of the task.
    • D. It is — but only if you skip evaluation to save GPU.

Answers.

  1. B. This is a facts problem, and the facts change weekly. RAG serves the current numbers at query time; fine-tuning would bake last week’s catalog into the weights and re-go-stale on every change. “RAG for facts, fine-tuning for behavior.”
  2. B. Climb the ladder. The prompt has never been worked and there’s no data — the two cheapest rungs (better prompt / few-shot, then maybe RAG) likely close the gap in seconds with no GPU. Fine-tuning here is spending blind, and without data it would degrade the model anyway.
  3. C. A strict, reliable output format is a behavior, not a fact — and prompting a small base model doesn’t guarantee it on every call. That’s the textbook fine-tuning case, and it’s exactly Anvil’s task. (A is wrong: no facts are missing. B is wrong: format is not reliably promptable on a small base model.)
  4. C. That’s DPO — direct optimization on (prompt, chosen, rejected) with no learned reward model and no online RL loop. It is explicitly not RLHF-PPO; PPO is the older, less stable recipe, now living in trl.experimental.ppo. DPO/GRPO are the 2026 defaults.
  5. B. The bet is defensible because the task is measurable: valid JSON, correct tool name, matching arguments — you can score base vs fine-tuned on a held-out set and prove the specialist holds up. Add cheaper serving, deployability, and an Apache-2.0 license you control. (A overstates the general case; C ignores that measurability is the crux; D throws away the one thing that makes the bet provable.)

Common pitfalls.

  • “Fine-tuning = injecting my latest facts.” No — fine-tuning changes behavior; RAG serves facts. This is the error that wastes the most GPU. A fact baked into the weights is stale the day it changes; a fact served by retrieval updates the instant your knowledge base does.
  • “Fine-tune first.” No — climb the ladder (prompt → RAG → fine-tune) and stop at the first rung that passes your eval. Fine-tuning without a task-grounded evaluation is spending GPU blind.
  • Copying dead idioms from old tutorials. Three to watch: PPO-RLHF presented as the default alignment method (it’s DPO/GRPO in 2026; PPO is in trl.experimental.ppo); pip install trl==1.0 from a half-read “TRL v1” headline (pin the real version — trl==1.6.0 as of June 2026); and transformers==4.x (the current line is 5.x, at 5.12.0). These are everywhere in search results and all wrong. Verifying against the installed library — which this course does — is the whole point.

Key takeaways

  • Climb the escalation ladder — prompting → RAG → fine-tuning — and stop at the first rung that passes your eval. Each rung costs more than the one below it.
  • “RAG for facts, fine-tuning for behavior” (practitioner consensus, not a hard number). Fine-tuning shifts a behavior distribution; it does not store retrievable, updatable facts.
  • Fine-tuning is not how you inject fresh data. A fact baked into the weights goes stale the moment it changes — that’s the single most expensive misconception in the field.
  • The post-training landscape is four families — SFT, PEFT (LoRA/QLoRA), preferences (DPO and family), RL (GRPO/RLVR) — in a canonical order: data → SFT → PEFT → preferences → RL → eval → quantize → serve → publish.
  • DPO/GRPO are the 2026 defaults, not PPO-RLHF. PPO is historical context, moved into trl.experimental.ppo.
  • The small-specialist bet: a 4B open model fine-tuned for one measurable task can beat a frontier API — cheaper to serve, deployable anywhere, under a license you control.
  • A base model fails the tool-call format on purpose — that’s the gap we’ll close. You just watched it; the rest of the course is the fix.

What’s next

In Module 2 you’ll learn the first hard truth of fine-tuning — data is the model — and you’ll format Anvil’s tool-calling dataset the right way, through tokenizer.apply_chat_template(messages, tools=...), never by hand. That’s where Anvil’s data is born.

Want the full Fine-Tuning in Production concept-check bank and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

Links: Course index · Module 2: Data Is the Model → · Lab code for this module

References