Capstone: Anvil, End to End (Fine-Tuning in Production, Module 15)

This is Module 15 of Fine-Tuning in Production, a free 15-module course that takes one open model from raw data to a deployed, published tool-calling specialist on the Hugging Face Hub. Start at Module 1 or browse the full syllabus.

Fourteen notebooks that work. Zero you can re-run.

Look at what you’ve built. Anvil works: hand it a request and a set of tools and it emits a clean <tool_call>, or it declines when no tool fits. You proved it beats the base model on a held-out split. You merged it, quantized it, served it, and pushed it to the Hub with a model card. By any reasonable measure, you shipped a fine-tuned model.

Now try to reproduce it from scratch. The data prep lives in one notebook. The SFT run is a script you ran on Colab three weeks ago — you think. DPO was a different cell, after you bumped beta twice. The eval ran four separate times and you copied the numbers into a spreadsheet by hand. The merge, the GGUF conversion, the smoke-serve, the push_to_hub — each is its own island. There is no single path from raw data to the published model that you, or anyone reading your model card, can execute in one go. In six months you won’t be able to.

That gap is where the most expensive misconception in fine-tuning hides. People say “I fine-tuned my model” as if it were one step — a verb you do once. It isn’t. It’s a chain: data → SFT → QLoRA → DPO → GRPO → eval → merge → quantize → serve → publish, where each link can quietly regress the one before it, and where the only defense is to re-evaluate after every single stage. This is the module where that chain becomes one thing. We re-play the whole verified pipeline as a single reproducible journey — one make anvil, on a free T4 — re-evaluate at every stage, and release anvil-qwen3-4b-toolcalling v1.0 to the Hub with a model card and a total cost you can read and defend.

In this module

You’ll learn how to:

1. Assemble the stages you built — data, SFT, LoRA/QLoRA, DPO, optional GRPO, task-grounded eval, merge, quantize, serve, publish — into one coherent, reproducible pipeline driven by a single make anvil.
2. Run the minimal end-to-end pipeline on a free T4: train a small but real Anvil, re-evaluate at every stage, merge, quantize to GGUF, smoke-serve it, and push it to the Hub.
3. Re-evaluate at every stage — internalize that fine-tuning is not one step and you never eval once: SFT, DPO and GRPO each get a base-vs-fine-tuned pass before you trust the gain.
4. Release anvil-qwen3-4b-toolcalling v1.0 with a complete model card (base_model: lineage, base-vs-fine-tuned table, honest total cost, limits) and a reproducible run (seeds, uv.lock, pinned datasets, an offline smoke test).
5. Decide where to go next — full fine-tuning, complete RLHF, distillation, multi-task specialists, and the upcoming serving course (Turbine) — and read the author’s real debrief of training and publishing Anvil.

You’ll build: Anvil v1.0 — one make anvil that runs the whole minimal pipeline (data → SFT → QLoRA → DPO → [GRPO] → eval → merge → quantize → serve → publish) on a free Colab/Kaggle T4, releasing anvil-qwen3-4b-toolcalling v1.0 to the Hub with a complete, reproducible model card.

Theory areas covered:

This is the capstone — a synthesis across all 12 theory areas (T1–T12). It owns no new theory; it re-plays the whole verified pipeline as one reproducible journey, with emphasis on T3 (SFT), T7 (task-grounded evaluation), T8 (preferences) and T11 (serving / quantization). Everything below is a reference back to the module that owns it — “covered in Module N” — not a re-teach. The one genuinely new idea is the orchestration: the order of the pipeline, and the discipline of re-evaluating after every stage, neither of which any single earlier module owns.

There is no exam domain here. This is the finale — there’s no certification to prepare for (decision F12). The Concept check at the end is a transverse self-assessment across the whole course.

Prerequisites: all of Modules 1–14. In particular M2 (data + chat template), M3 (SFT), M5 (QLoRA on a T4), M7 (base-vs-fine-tuned eval), M8 (DPO), M10 (GRPO — optional, 💸), M11/M12 (merge → quantize), M13 (serving), M14 (publish + model card + reproducibility). For the lab: a free Colab/Kaggle T4 (16 GB) and HF_TOKEN set in your environment (M1). GRPO can need the 💸 rented-GPU path (decision F5); the minimal make anvil keeps it optional (off by default).

Where you are

This is the last link in the chain. Module 15 doesn’t open any new theory — it connects and re-plays what the previous fourteen modules built.

• ✅ M1–M14 — the whole pipeline, built and verified one stage at a time.
• 👉 M15 — Capstone: Anvil, End to End. One make anvil from raw data to a published v1.0.
• (nothing after — this is the finale)

Anvil before this module: complete but scattered across fourteen notebooks and scripts, with no single reproducible path and no semantic release tag. Anvil after: one make anvil from raw data to a published anvil-qwen3-4b-toolcalling v1.0 on the Hub, on a free T4, with a model card and an honest total cost. That’s the whole job of the capstone — ship the whole thing, reproducibly.

From fourteen stages to one pipeline (a guided recap)

Here’s the assembly map. Each stage slots into the next; each one is owned, with its full theory, by an earlier module and lives in a frozen file in the cumulative anvil/ package. Read this as a table of contents for what you already know, not a re-lecture.

• When and why to fine-tune (T1, M1). Anvil is a good fine-tuning candidate because its job has a measurable behavior — emit the right tool call — not because it needs to memorize facts (that’s RAG). If you can’t measure it, don’t fine-tune for it. (Covered in Module 1.)
• Data + chat template (T2, M2). anvil/data.py turns the raw argilla/Synth-APIGen-v0.1 subsample into {messages, tools} examples and renders them through tokenizer.apply_chat_template(messages, tools=...) — never a hand-rolled template. (Covered in Module 2.)
• SFT (T3, M3). train_sft.py teaches the format: next-token loss on demonstrations, via SFTTrainer/SFTConfig with TRL v1 idioms (processing_class, max_length). The model gets every choice through config.get_model_and_tokenizer(tier). (Covered in Module 3.)
• LoRA / QLoRA (T4/T5, M4/M5). tier="qlora" loads a 4-bit NF4 base and trains a small LoRA adapter on top, so the whole run fits a free 16 GB T4. (Covered in Modules 4–5.)
• Accelerators (T6, M6). Unsloth and Axolotl are optional — the make anvil stays on plain TRL/PEFT (decision F11); tier="unsloth" is the “go faster” variant. (Covered in Module 6.)
• Task-grounded evaluation (T7, M7). eval.py — tool_call_valid_json, function_name_accuracy, argument_match, and compare_base_vs_finetuned. Re-run after every stage — the spine of this module. (Covered in Module 7.)
• DPO (T8, M8/M9). train_dpo.py learns from (prompt, chosen, rejected) triples — prefer the precise call over plausible-looking failures. (Covered in Modules 8–9.)
• GRPO + RLVR (T9, M10). train_grpo.py plus the verifiable reward_fn — optional and 💸. (Covered in Module 10.)
• Model merging (T10, M11). merge.py folds the adapter into the base (merge_and_unload) and combines variants with mergekit. (Covered in Module 11.)
• Quantize + serve (T11, M12/M13). quantize.py (GGUF/AWQ/GPTQ) then serve.py (vLLM/Ollama/TGI). (Covered in Modules 12–13.)
• Publish + reproducibility (T12, M14). hub.py — push_to_hub plus a model card with base_model:, seeds, uv.lock, and a CI smoke test. (Covered in Module 14.)

⚠️ Common misconception: “Fine-tuning is one step — I fine-tune my model once and I’m done. And I eval once, at the end.”

Both halves are wrong, and they’re wrong together. Post-training is a chain — SFT, then preferences, then RL — and each stage can improve one metric while degrading another. DPO can make Anvil terse and break a tool call on an edge case. GRPO can climb the reward while quietly forgetting general ability (catastrophic forgetting) or learning to game the reward (reward hacking). If you only eval at the end, you see a worse final model and have no idea which stage broke it. The capstone’s rule is the cure: re-evaluate base-vs-current-stage after every stage, and keep a stage only if it improves the task metric without collapsing the rest. This is the “aha” of the module — and the whole reason make anvil calls the evaluator more than once.

There’s no diagram here on purpose — the full pipeline flow is the first diagram in the next section, where it belongs with the discussion of order.

The order of the pipeline, and why it’s that order

The one genuinely new concept in this module is the canonical order of the post-training pipeline — and why it has to be that order. No single earlier module owns this, because it only emerges when you wire the stages together:

data → SFT → (LoRA/QLoRA) → DPO → [GRPO] → eval → merge → quantize → serve → publish

Walk it once, asking “why can’t I swap this?” at each step:

• SFT first. You need a model that already produces the right format — a parseable <tool_call> — before you can align it. DPO on a base model that can’t call a tool has nothing to prefer: both chosen and rejected would be garbage. SFT creates the behavior; alignment refines it. (Why SFT teaches format, not facts: Module 3.)
• DPO after SFT. Preference tuning sharpens a behavior that’s already present — chosen is the precise call, rejected is a degraded one. It cannot conjure a behavior from nothing. (Module 8.)
• GRPO last, and optional. RLVR pushes reasoning with a verifiable reward (valid JSON + right name + matching args), but it generates a group of completions every step, so it’s the expensive stage. Run it only after you have a model that’s already good, and only if the eval proves it earned its keep. (Module 10; 💸 rented-GPU path possible.)
• Eval between and after every stage — not only at the end. (The whole next section.)
• Merge → quantize → serve → publish at the very end. These transform the finished model. The non-negotiable rule: merge_and_unload before you quantize for inference — a raw LoRA adapter is not a deployable format, and you can’t convert one straight to GGUF. (Module 12.) And keep the two quantizations straight: training-time quantization (QLoRA / bitsandbytes 4-bit, on the fly, no calibration — Module 5) is not inference-time quantization (GGUF / AWQ / GPTQ, a deployment format — Module 12). Confusing them is the classic trap.

Here is the whole pipeline as the orchestrator sees it, annotated with the module and the anvil/ file that owns each node. Notice the eval loop in red: it hangs off SFT, DPO and GRPO.

flowchart TB
    raw([raw dataset<br/>argilla/Synth-APIGen-v0.1]) --> tmpl["chat template<br/>data.py (M2)"]
    tmpl --> sft["SFT<br/>train_sft.py (M3)"]
    sft --> qlora{{"tier='qlora'<br/>config.py (M5)"}}
    qlora --> dpo["DPO<br/>train_dpo.py (M8)"]
    dpo --> grpo["GRPO [optional 💸]<br/>train_grpo.py (M10)"]
    grpo --> merge["merge_and_unload<br/>merge.py (M11/M12)"]
    merge --> quant["quantize GGUF/AWQ/GPTQ<br/>quantize.py (M12)"]
    quant --> serve["serve vLLM/Ollama<br/>serve.py (M13)"]
    serve --> pub["push_to_hub + model card<br/>hub.py (M14)"]
    pub --> v1([anvil-qwen3-4b-toolcalling v1.0])

    sft -.-> ev
    dpo -.-> ev
    grpo -.-> ev
    quant -.-> ev
    ev["eval.py — base vs fine-tuned (M7)<br/>you never eval once"]:::red

    classDef red fill:#fee,stroke:#c00,stroke-width:2px;

And here is the same pipeline as a table — stage by stage, with where it runs and what it costs. This is the table that feeds the honest total cost later.

Stage	Module	anvil/ file	What runs	GPU / cost
Data	M2	data.py	Render {messages, tools} via the chat template	CPU only
SFT	M3	train_sft.py	SFTTrainer next-token loss on demonstrations	T4 (free)
QLoRA	M5	config.py	4-bit NF4 base + LoRA adapter — fits 16 GB	T4 (free)
DPO	M8	train_dpo.py	DPOTrainer on (prompt, chosen, rejected)	T4 (free)
GRPO [opt]	M10	train_grpo.py	GRPOTrainer + verifiable reward_fn, rollouts	💸 rented A100
Eval ×N	M7	eval.py	compare_base_vs_finetuned after each stage	T4 (free)
Merge	M11	merge.py	merge_and_unload into full-precision weights	CPU-friendly
Quantize	M12	quantize.py	GGUF via llama.cpp (q4_k_m)	CPU-friendly
Serve	M13	serve.py	Local smoke call against the GGUF	CPU/local
Publish	M14	hub.py	push_to_hub + model card	network only

The orchestrator that drives all of this is anvil/pipeline.py. It is the only genuinely new file in this module, and it orchestrates — it imports the frozen functions and calls them in order; it re-implements nothing:

def run_pipeline(*, tier: str = "qlora", stages: tuple[str, ...] = ("sft", "dpo"),
                 data_limit: int | None = None, eval_limit: int = 100,
                 eval_max_new_tokens: int = 128,
                 output_root: str = "outputs/anvil", **train_overrides) -> dict:
    """Run Anvil's pipeline and return a summary (stages run, final model dir, base-vs-tuned eval)."""
    # 1) SFT — teach the format
    train_sft(tier=tier, use_lora=True, data_limit=data_limit, output_dir=sft_dir, **train_overrides)
    adapter = sft_dir
    # 2) Preference / RL refinements (optional)
    if "dpo" in stages:
        train_dpo(tier=tier, use_lora=True, data_limit=data_limit, output_dir=dpo_dir, **train_overrides)
        adapter = dpo_dir
    if "grpo" in stages:
        train_grpo(tier=tier, use_lora=True, ...)
        adapter = grpo_dir
    # 3) Merge the final adapter into standalone weights (deploy-ready)
    merge.merge_adapter(adapter, merged_dir, base_tier=tier)
    # 4) Evaluate base vs fine-tuned on a held-out split (the model-card table)
    ...

You turn on GRPO by adding it to the stages tuple — stages=("sft", "dpo", "grpo") — which is exactly why it’s opt-in: the default ("sft", "dpo") keeps the free-T4 path light, and the heavy rollout stage is a single deliberate keystroke away. Each stage hands its adapter to the next, and the last adapter is what gets merged into deploy-ready weights.

You never eval once: re-evaluating after every stage

This is the heart of the capstone. The principle (owned by Module 7, reached here at the why level): after SFT, after DPO, after GRPO, you re-run compare_base_vs_finetuned on the held-out eval split — data the model never trained on — and read the three task metrics plus an anti-forgetting signal. A stage can win on the task and lose everywhere else; only a per-stage gate catches that.

The numbers themselves come straight from eval.py, which is shared with the GRPO reward so “correct” means one thing in training and grading:

def evaluate(model, tokenizer, rows, *, max_new_tokens=128) -> dict:
    ...
    return {
        "valid_json": vj / len(pos) if pos else None,   # over positives
        "fn_acc": fn / n,
        "arg_match": arg / n,
        "abstention": abst / len(neg) if neg else None,  # over negatives — "when NOT to call"
        "n": n,
    }

And the cheap proxy for “did fine-tuning break general ability” — the honest, GPU-free stand-in for a full lm-eval capability run:

def anti_forgetting_probe(model, tokenizer, prompts=None) -> float:
    """Fraction of short general prompts the model continues without degenerating (empty/one-token)."""
    prompts = prompts or ["The capital of France is", "2 + 2 =", "Water is made of"]
    ...

The regressions to watch for, by name (each owned by its module): DPO too aggressive → empty replies or broken format on edge cases (lever: lower beta). GRPO reward hacking → the model satisfies the verifiable reward but degrades quality outside it; catastrophic forgetting → general ability drops, which the anti-forgetting probe catches (levers: lower learning rate, fewer steps). The decision rule is mechanical: keep a stage only if the task metric rises and anti-forgetting doesn’t collapse — otherwise roll back to the previous checkpoint or tune the knob.

Here is the gate — the re-eval table you fill in as the pipeline runs. It is exactly the table that ships in the final model card. These are numbers from my run; measure your own.

Stage	valid_json	fn_acc	arg_match	anti-forgetting	keep this stage?
base	0.04	0.05	0.02	1.00	— (baseline)
+SFT	0.93	0.89	0.81	0.97	✅ huge gain, no forgetting
+DPO	0.96	0.92	0.86	0.95	✅ sharper, holds
+GRPO (rented-GPU run)	0.97	0.94	0.90	0.88	⚠️ task up, forgetting creeping — watch it
merged	0.96	0.92	0.86	0.95	✅ matches DPO (no GRPO in the free run)
quantized (GGUF q4_k_m)	0.95	0.91	0.85	0.94	✅ ~1 pt PTQ drop, acceptable

Two things to read off it. First, the quantized row exists because you re-evaluate even after inference-time quantization — a PTQ pass can shave a point off valid_json, and you want to know before you publish, not after a user files a bug (Module 12). Second, the +GRPO row is the textbook case for the gate: the task metrics climbed but anti-forgetting slid from 0.95 to 0.88. On my free run I shipped the DPO checkpoint and left GRPO as a documented, opt-in experiment — the gate, not a hunch, made that call.

Ship it for real: the final model card and the honest total cost

The published model is the portfolio artifact (decision F10). The thing that makes it defensible — and that makes the Hub draw the adapter → merge → quantized lineage automatically — is the YAML front matter, with base_model: pointing back to the base. hub.py’s build_model_card assembles it offline:

---
base_model: Qwen/Qwen3-4B-Base
license: apache-2.0
library_name: peft
datasets:
  - argilla/Synth-APIGen-v0.1
pipeline_tag: text-generation
tags:
  - tool-calling
  - function-calling
  - qwen3
  - fine-tuning-in-production
  - anvil-sft
---

base_model: Qwen/Qwen3-4B-Base is the single most important line: drop it and the Hub can’t show the lineage, and your “where did this model come from” story breaks. The body carries the base-vs-fine-tuned table (the gate from the previous section), the usage snippet, the dataset, and — because Qwen3-4B-Base is Apache-2.0 (decision F3) — a license note that says you can republish your derivative freely. The limits matter too: Anvil is a tool-calling specialist, not a general chatbot, and it inherits the base model’s knowledge cutoff and the dataset’s domain. Say so on the card. Once you push it, that repo is the model’s home — the public artifact you put on a résumé and link from your own write-up of this course.

The honest total cost of the whole journey (decision F6). Add up the stages and you can hand the reader a single number they can quote.

Stage	Time (T4)	Cost (free T4)	Cost (rented)
Data prep	~2 min	$0.00	$0.00 (CPU)
SFT (QLoRA)	~30–45 min	$0.00	—
DPO	~15–25 min	$0.00	—
GRPO [optional]	~30–60 min	(doesn’t fit comfortably)	~$1–$3 on an A100
Eval ×N	~5 min each	$0.00	—
Merge → quantize	~5–10 min	$0.00 (CPU-friendly)	$0.00
Serve smoke + publish	~5 min	$0.00	$0.00 (network)
Total (free path)	~1.5–2 h	~$0.00	—
Total (with rented GRPO)	~2.5–3 h	—	~$1–$3

So the opinion, chiffrée: the free path costs $0.00 and a couple of hours. The only stage that strains a free T4 is GRPO; renting an A100 just for that stage buys those hours back for a couple of dollars. Rented-GPU prices change weekly — as of June 2026, ballpark RunPod ~$1.19/h, Lambda ~$1.99/h, Modal ~$2.50/h (billed by the second). Date every provider page you cite; these numbers move in silence.

Reproducibility (Module 14, decision F14). make anvil is re-runnable because seeds are fixed (seed=0 in every config), uv.lock is committed, the dataset is pinned, and a monthly CI re-plays an offline smoke test of the whole pipeline — a nano model, a couple of steps per stage, no GPU and no network — so a stack bump can never break the chain in silence. That offline replay is what’s verified on every change; the full T4 run is the release step you (and the author, per decision F13) execute to produce the published artifact. The pitfalls below are the ones that chain reliably trips on.

Pitfalls this pipeline is built to survive — and the ones that will still bite you

None of these are exotic; all are cheap to fix once you’ve seen them once:

• QLoRA OOM on the T4. A 4B in 4-bit fits, but a long max_length plus a real batch tips 16 GB over. Fix: gradient checkpointing (already on via prepare_model_for_kbit_training), per_device_train_batch_size=1 with gradient_accumulation_steps to keep the effective batch, and trim max_length to what the tool schemas actually need.
• A preference stage that regresses the task metric. DPO can make Anvil terser and lower valid_json if beta is too high — it drifts from the SFT behavior faster than it learns the preference. Re-eval immediately after DPO; if valid_json dropped, lower beta (try 0.05) before blaming the data. This is exactly why make anvil evaluates after every stage.
• Quantizing before merging. The GGUF/AWQ/GPTQ converters need full standalone weights; point them at a raw LoRA adapter and they error or silently quantize the base only. Always merge_and_unload first (Modules 11–12).
• A push_to_hub that drops base_model:. Upload the folder without the model card and the Hub can’t draw the lineage — your “where did this come from” story breaks. Build the card with hub.build_model_card and upload it alongside the weights.
• The T4 dtype trap. A T4 has no bf16 tensor cores; a script that hard-codes bf16=True silently falls back and crawls. On a T4, fp16 is the honest choice — bf16 is for Ampere and up (A100/L4).

The defense against all of them is the thesis of this course: re-evaluate base-vs-current after every stage, and keep a stage only if it improves the task metric without collapsing the rest.

Hands-on lab: build it — one make anvil from data to v1.0

The full write-up is in module-15/lab.md. This is the big piece of the capstone, and it adds no new training method — it’s orchestration and release only. The new files are anvil/pipeline.py, the Makefile, and lab.md; everything under anvil/ that does real training is the frozen code from Modules 2–14, imported, never re-implemented.

The observable result. On a free T4, the default run trains a small but real Anvil and prints the eval:

make anvil                       # = uv run python -m anvil.pipeline (stages: sft, dpo)
make anvil-grpo                  # adds the 💸 GRPO stage — rented-GPU path

Anvil pipeline complete. Stages: ['sft', 'dpo']
Final model: outputs/anvil/merged
Eval (base vs fine-tuned): {'base': {'valid_json': 0.04, ...}, 'finetuned': {'valid_json': 0.96, ...}}

run_pipeline runs SFT, then (because dpo is in stages) DPO, merges the last adapter into deploy-ready full-precision weights, then evaluates base-vs-fine-tuned on the held-out split — the exact table that ships in the card. Turn on the heavy stage by adding "grpo" to stages; the merge always folds the final adapter, then you quantize, smoke-serve and publish.

The merge-before-quantize rule, in code. merge.py folds the adapter into a full-precision base even when training used QLoRA — quantizing a raw adapter, or merging into the 4-bit base, bakes in error:

def merge_adapter(adapter_dir, output_dir, *, base_tier="qlora"):
    # Merge into a FULL-precision base even if the adapter was trained with QLoRA (4-bit).
    load_tier = "full" if base_tier == "qlora" else base_tier
    base, tokenizer = config.get_model_and_tokenizer(load_tier)
    merged = PeftModel.from_pretrained(base, adapter_dir).merge_and_unload()
    ...

Then, and only then, quantize.to_gguf(...) (M12) converts the merged weights to GGUF, serve.smoke-style local call (M13) proves the GGUF still answers, and hub.push_model(...) (M14) uploads with the model card. The Hub push stays optional — it needs HF_TOKEN, and the smoke test never pushes.

Prove it offline, no GPU, no network:

uv run pytest module-15/tests/

The smoke test runs the real run_pipeline on the tiny in-repo Qwen3 fixture (tier="tiny", two steps per stage, fixed seed), asserts every stage runs, that a deploy-ready merged model was written, and that the base-vs-fine-tuned summary is well-formed — green in CI without a GPU (decision F14).

Try it yourself. (1) Turn on GRPO on a rented GPU (stages=("sft", "dpo", "grpo")) and compare the re-eval table with vs without GRPO — was the gain worth the dollars? (2) Build a second merged variant — a SLERP blend of an SFT checkpoint and a DPO checkpoint (merge.write_mergekit_config(..., method="slerp"), M11) — and regenerate the model card with its eval row to see whether the merge recovers a capability.

What this lab does not do, on purpose: no new training stage; no re-teaching of each stage’s theory (M3–M14 own it — we reference); no full fine-tuning, complete PPO/RLHF, or distillation (those are “where to go next,” below); and never merge_and_unload after quantization (always merge first). The Hub push is documented but not required.

In production

What changes when Anvil leaves the notebook? Cost first: on the free path it’s $0 and time, but a complete pipeline in production — especially GRPO — is billed in rented GPU-hours, which is why the cost table exists and why the call is “free T4 to iterate, rented A100 only for the final GRPO run” (decision F6). Reproducibility: a production fine-tune must be re-runnable — seeds, uv.lock, pinned datasets — and the stack churns fast (TRL, PEFT and transformers move often), so a monthly CI re-plays the smoke test (F14). Quality: the eval gate (re-eval after every stage) blocks a regression before the push — green or no ship. License: Qwen3-4B-Base is Apache-2.0, so you republish your derivative freely (repeat it at every Hub push, F3) — and check the dataset’s license before redistributing it too. And honesty (F10): Anvil is defensible because it’s measured and reproducible, not because it “looks like it works.”

Concept check

Why this matters. This quiz consolidates the whole course, not just Module 15. Each answer is tagged with the module to review if it trips you up — use it as a self-diagnostic. The two skills the capstone insists on: ordering the pipeline and knowing where to re-evaluate.

1. A colleague lists the stages out of order — DPO, data, quantize, SFT, eval, GRPO, merge, serve, publish — and asks you to fix it. What’s the canonical order, and why must SFT come before DPO and merge before quantize?

   • A. Order doesn’t matter as long as eval is last.
   • B. data → SFT → QLoRA → DPO → [GRPO] → eval → merge → quantize → serve → publish; SFT first because alignment refines a behavior that must already exist, and merge before quantize because a raw adapter isn’t a deployable format.
   • C. data → DPO → SFT → GRPO → merge → quantize → serve → publish; preferences set the format and SFT then refines it.
   • D. data → SFT → DPO → quantize → merge → GRPO → publish; quantize early to save VRAM during merging.

2. A dev fine-tunes SFT, then DPO, then GRPO, evaluates only at the very end, and finds the final model is worse than after SFT. What went wrong, and what’s the fix?

   • A. They used too few epochs; train longer.
   • B. GRPO is always harmful; never use it.
   • C. They evaluated only once. Re-evaluate base-vs-current-stage (plus anti-forgetting) after each stage to find which stage regressed, then roll it back or tune its knob.
   • D. The base model was wrong; switch base models.

3. You want to “just serve the QLoRA model” you trained, straight to Ollama. Why can’t you, and what’s the right path?

   • A. You can — a QLoRA checkpoint is a GGUF.
   • B. QLoRA is training-time quantization, not a servable format; merge_and_unload the adapter into full-precision weights, then quantize for inference (GGUF/AWQ/GPTQ).
   • C. Re-train with load_in_8bit=True; that produces a deployable model.
   • D. Push the adapter to the Hub; Ollama merges it at load time.

4. A teammate can’t reproduce your published results. Which combination most likely fixes it, and what makes the artifact defensible?

   • A. Send them your final weights; reproducibility isn’t the point.
   • B. Pinned stack (uv.lock) + fixed seeds + pinned dataset + base_model: in the card, plus a CI smoke test that re-plays the pipeline — a published model is only defensible because it’s reproducible.
   • C. Tell them to use a bigger GPU.
   • D. Re-run on latest versions of every library to get the newest fixes.

5. Your reward signal is a subjective human preference (how “helpful” a reply feels), and you reach for GRPO. What’s the problem, and what’s the right method?

   • A. No problem — GRPO works on any reward.
   • B. GRPO/RLVR needs a verifiable reward (a deterministic checker); a subjective preference isn’t verifiable, so use DPO (preference pairs) or, if you must, complete RLHF (a learned reward model + PPO).
   • C. Convert the preference to a number and GRPO will handle it.
   • D. Use SFT — it optimizes any objective.

Answers.

1. B. SFT creates the tool-call format; DPO and GRPO refine a behavior that must already exist, so SFT comes first. A raw LoRA/QLoRA adapter is not a deployable artifact, so you merge_and_unload before quantizing for inference. C and D both break those two rules. → Review Module 15 (orchestration), M3 (SFT), M8 (DPO).
2. C. Evaluating once hides which stage regressed. The fix is the capstone’s core discipline: re-evaluate base-vs-current-stage after each stage, watch the task metrics and anti-forgetting, and keep a stage only if it improves without collapsing the rest — DPO can over-sharpen, GRPO can forget. → Review Module 7 (and Module 15’s gate).
3. B. QLoRA (bitsandbytes 4-bit) is training-time quantization — on the fly, no calibration, CUDA kernels — not a deployment format. Merge the adapter into full-precision weights, then run inference-time quantization (GGUF for Ollama). C is the load_in_8bit-called-QLoRA trap; D is false (Ollama doesn’t merge adapters). → Review M5 (training quant) + M12 (inference quant).
4. B. Reproducibility is the sum of a pinned stack (uv.lock), fixed seeds, a pinned dataset, and base_model: lineage on the card, validated by a CI smoke test. The portfolio artifact (F10) is only worth anything because someone else can re-run it. D is the opposite of reproducible — unpinned latest is how a run silently breaks. → Review Module 14.
5. B. GRPO + RLVR demand a verifiable reward — a function that computes correctness (valid JSON, right tool name, matching args). A subjective “helpfulness” judgment has no deterministic oracle, so it’s the wrong fit; use DPO on preference pairs, or complete RLHF (learned reward model + PPO) if you genuinely need online RL on a non-verifiable signal — that’s a “where to go next.” → Review M8 (preferences) / M10 (GRPO/RLVR).

Common pitfalls.

• “Fine-tuning is one step, and I eval once at the end.” It’s a chain, and you re-evaluate after every stage — a stage can win on the task and break general ability (DPO over-sharpening, GRPO reward hacking / catastrophic forgetting). This is the ⚠️ misconception of the module.
• Serving a raw QLoRA adapter / quantizing before merging. Always merge_and_unload before inference-time quantization, and never confuse training-time quant (QLoRA) with inference-time quant (GGUF/AWQ/GPTQ). As of bitsandbytes 0.49.2 and gptqmodel 7.1.0, June 2026 — and use GPTQModel, not the archived AutoGPTQ.
• push_to_hub without base_model:, seeds, or uv.lock. It breaks the model card’s lineage and the run’s reproducibility — the portfolio artifact is only defensible if it’s reproducible (F10/F14). And the pin is trl==1.6.0 (TRL’s 1.x line shipped on PyPI; the old “never trl==1.0” ban is obsolete) — as of trl 1.6.0, June 2026.

Key takeaways

• Fine-tuning is an ordered chain, not one step: data → SFT → QLoRA → DPO → [GRPO] → eval → merge → quantize → serve → publish. The order isn’t arbitrary — SFT creates the format, alignment refines it, deployment transforms the finished model.
• You never eval once. Re-run base-vs-current-stage (plus an anti-forgetting probe) after every stage, and keep a stage only if the task metric rises and general ability doesn’t collapse — DPO can over-sharpen, GRPO can hack the reward or forget.
• Training-time quant ≠ inference-time quant. QLoRA (bitsandbytes 4-bit) trains; GGUF/AWQ/GPTQ deploy. And always merge_and_unload before quantizing for inference — a raw adapter is not a servable format.
• A reproducible run = fixed seeds + uv.lock + pinned datasets + a CI smoke test. The stack churns; without pins and CI, your run silently breaks on the next upgrade.
• The honest total cost is ~$0.00 on a free T4 and a couple of hours; the only stage that strains it is GRPO, and renting an A100 just for that buys the hours back for ~$1–$3 (as of June 2026 prices).
• The portfolio artifact is Anvil v1.0 published to the Hub — with base_model: Qwen/Qwen3-4B-Base, a base-vs-fine-tuned table, an honest cost, and limits. It’s defensible because it’s measured and reproducible, not because it “looks like it works.”
• GRPO/RLVR needs a verifiable reward. For subjective signals, reach for DPO or complete RLHF — not GRPO.

Where to go next

You’ve taken one open model from raw data to a published, reproducible specialist. Here’s the map of where this path branches — each is something this course deliberately did not go deep on.

flowchart TB
    you([Anvil v1.0 — you are here]) --> a["Full fine-tuning"]
    you --> b["Complete RLHF<br/>(reward model + PPO)"]
    you --> c["Distillation"]
    you --> d["Multi-task specialists"]
    you --> e["Serving in depth<br/>→ Turbine"]
    you --> f["App-layer courses<br/>(Quill, Forge…)"]

1. Full fine-tuning — train all the weights, not just the LoRA adapter. Reach for it when the GPU budget allows and the task is broad enough to justify it; it costs much more VRAM and trades away the swappable-adapter convenience for the last few points of quality.
2. Complete RLHF (reward model + PPO) — the historical pipeline: train a learned reward model, then optimize the policy with PPO (trl.experimental.ppo 🧪). Useful precisely when the reward is not verifiable (subjective human preference) — the opposite of Anvil’s RLVR/GRPO.
3. Distillation — transfer a large model’s behavior into a small one (SFT on the teacher’s outputs, or logit-matching). The other road to a small specialist, complementary to fine-tuning.
4. Multi-task specialists — one model, or a set of swappable adapters, for several tasks: mixed datasets and adapter routing (vLLM/TGI already serve multiple LoRAs).
5. Serving in depth — the Turbine course. The natural sequel to Anvil (it picks up where Module 13 left off): vLLM internals, paged attention / KV-cache, continuous batching, deeper serving quantization, and TTFT/throughput benchmarking. “Train, then serve” as one track.
6. The app-layer courses in the catalog — agents, LangGraph, Bedrock. Anvil is exactly the kind of small specialist that could power Quill or Forge instead of a frontier API. You built the model; those courses build the app on top.

Want the full fine-tuning concept-check bank and news of future courses in your inbox? Subscribe here — it’s free, like everything in this series.

Links: Course index · ← Module 14: Ship It — Publishing, Model Cards, and Reproducibility · Module 1: start the journey over · Lab code for this module

References

• QLoRA — “QLoRA: Efficient Finetuning of Quantized LLMs”: https://arxiv.org/abs/2305.14314
• DPO — “Direct Preference Optimization: Your Language Model is Secretly a Reward Model”: https://arxiv.org/abs/2305.18290
• GRPO / DeepSeekMath — “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models”: https://arxiv.org/abs/2402.03300
• RLVR / Tülu 3 — “Tülu 3: Pushing Frontiers in Open Language Model Post-Training”: https://arxiv.org/abs/2411.15124
• TRL documentation — the post-training pipeline (verified against trl 1.6.0): https://huggingface.co/docs/trl
• Hugging Face Hub — model cards and the base_model metadata field: https://huggingface.co/docs/hub/en/model-cards