aifromzero.dev

Free Capability Boosts: Model Merging with mergekit (Fine-Tuning in Production, Module 11)

Module 11 of 15 13 min read Lab code ↗

This is Module 11 of Fine-Tuning in Production, a free 15-module course that takes one open model from raw data to a deployed, published tool-calling specialist on the Hugging Face Hub. Start at Module 1 or browse the full syllabus.

You have three Anvils, and none of them is best at everything

By now you’ve hammered the same base model — Qwen/Qwen3-4B-Base — into a small stable of specialists. The SFT variant from Module 3 learned the tool-call format. The DPO variant from Module 8 learned to prefer the tight, correct call over a plausible-looking one. The GRPO variant from Module 10 pushed verifiable correctness with a reward, not a label. These are three independently-trained adapters off the same Qwen3-4B-Base — not one cumulative SFT→DPO→GRPO stack — which is precisely what makes them mergeable. Each one is sharper on one axis. None dominates on all three.

The obvious move is to mash their training data together and run one more combined fine-tune. That’s hours of GPU you don’t have to spend, and you might just trade one regression for another. There’s a near-free alternative: take two variants that share a base and combine their weights directly — no gradient steps, no dataset, no expensive GPU. A 4B merge runs in a few minutes on a CPU or a modest card. The tool is mergekit, and the technique is model merging.

The catch is the part most tutorials skip. Merging is not magic. The hard prerequisite is a shared base model, and the real risk is interference: blend the wrong things and the merged model is worse than either parent. So the rule that makes this module honest is simple — you never assume a merge helped, you re-measure it with your Module 7 eval. This module merges two Anvils into one, and then proves whether it was worth keeping.

In this module

You’ll learn how to:

  1. Explain what model merging is — combining the weights of several fine-tunes that share one base, with no training, no gradients, and no expensive GPU — and why that makes it a near-free capability boost.
  2. Distinguish the four merge families — task arithmetic / linear averaging, SLERP (spherical interpolation, two models only), TIES (trim → elect-sign → merge, many models), and DARE (drop & rescale) — by how many models they take and how they handle conflicts.
  3. Reason about why merging works at all and where it breaks: a shared base means a shared weight space, task vectors, and interference — the reason naive averaging degrades, and the problem TIES and DARE were built to fix. (mergekit paper, arXiv 2403.13257.)
  4. Write a mergekit YAML config to merge an SFT and a DPO variant of Anvil with SLERP, run it on CPU or a modest GPU, and produce one merged model directory.
  5. Re-evaluate the merged Anvil with the Module 7 harness (eval.py) against the SFT-only and DPO-only variants — and decide whether to keep the merge, because merging does not always improve.

You’ll build: Anvil gets an anvil/merge.py. You point mergekit at your Module-3 SFT variant and your Module-8 DPO variant — same Qwen3-4B base — write a short SLERP YAML, and merge them into one model with zero gradient steps. Then you re-run your Module-7 eval.py on the merged model and put SFT-only / DPO-only / merged side by side — and you keep the merge only if it actually wins.

Theory areas covered:

  • T10 — Model merging: mergekit (SLERP/TIES/DARE) — 4% of the course. Module 11 owns this area outright; it’s reinforced once more at the Module 15 capstone.

This module also leans on theory you already have: T7 (evaluation) from Module 7 — you re-run that exact harness on the merged model — and the trained variants from Modules 3, 8, and 10, which are the inputs to the merge, not something you retrain here.

Prerequisites: Modules 1–10, and in particular at least two variants of Anvil that share the base — typically the SFT from Module 3 and the DPO from Module 8 — plus eval.py from Module 7 and get_model_and_tokenizer from config.py. For the lab: mergekit installed (pip install mergekit==0.1.4, as of June 2026). The merge runs on CPU or a modest GPU — no big card needed, which is the whole point — and you only need HF_TOKEN if the variants live on the Hub.

Where you are

You’ve come a long way down the pipeline. From now on, every module marks where you are with ✅ (done), 👉 (you’re here), and ⬜ (ahead):

  • ✅ M1–M2 — Setup, baseline, and Anvil’s data layer (chat templates).
  • ✅ M3 — SFT: Anvil v0 learns the tool-call format.
  • ✅ M4–M5 — LoRA, then QLoRA: Anvil fits a free 16 GB T4.
  • ✅ M6 — Unsloth and Axolotl: same run, faster / config-as-code.
  • ✅ M7 — Task-grounded evaluation: eval.py, base vs fine-tuned.
  • ✅ M8–M9 — DPO and the preference family (ORPO, KTO, SimPO).
  • ✅ M10 — GRPO and verifiable rewards (RLVR).
  • 👉 M11 — Model merging with mergekit. Blend two Anvil variants into one — for free — and re-evaluate it.
  • ⬜ M12 — Shrink it for serving: merge_and_unload, GGUF, AWQ, GPTQ.
  • ⬜ M13 — Serve it: vLLM, Ollama, TGI.
  • ⬜ M14 — Ship it: publishing, model cards, reproducibility.
  • ⬜ M15 — Capstone: Anvil, end to end.

Anvil before this module: several separately-trained variants (SFT, DPO, GRPO), each strong on one axis, all sharing the Qwen3-4B-Base weights — and anvil/merge.py doesn’t exist yet.

Anvil after this module: one merged variant that blends two of them — for free, no training — and you’ve re-run the Module-7 eval to confirm it’s worth keeping. anvil/merge.py now exists in the package (Module 12 adds quantize.py, which reuses this merge_and_unload as step zero of the quantize path). Merging is optional in the pipeline ([merge éventuel] in Anvil’s target architecture): whatever you keep — merged or not — flows on to quantize → serve → publish in Modules 12–14, and is replayed at the capstone.

What merging is (and isn’t): combining weights with no training

Start with the definition at the level that matters — the why, not just the what.

Model merging takes the weights of several models that all derive from the same base and combines them into a single set of weights. That’s it. It is arithmetic on weight tensors, not training. There is no gradient, no training forward pass, no dataset, and no expensive GPU. Merging a 4B model fits on a CPU or a modest card and finishes in a few minutes. That is what earns it the phrase free capability boost — and we’ll hold that line carefully, because “free” does not mean “always good” (see the misconception box below).

Why it works at all

Here’s the part the surface-level tutorials never explain. Two fine-tunes of the same base model live in a shared weight space. Fine-tuning doesn’t relocate the weights to some distant region of parameter space; it nudges them a little, in directions tied to the task. Formalize that nudge as a task vector:

task_vector = weights_finetuned − weights_base

A task vector is the delta a stage of training added on top of the base. The SFT task vector encodes “emit the tool-call format.” The DPO task vector encodes “prefer the tight, correct call.” Because both deltas are measured against the same base, they live in the same coordinate system — so you can do arithmetic with them. Add two task vectors and you may add their capabilities. Average them and you interpolate between two behaviors. This is the conceptual engine under every merge method below.

This idea was made practical by mergekit, the toolkit introduced in Arcee’s MergeKit: A Toolkit for Merging Large Language Models (Charles Goddard et al., arXiv 2403.13257). Treat mergekit as the tool that made these methods reproducible and usable — it implements SLERP, TIES, DARE, task arithmetic and more behind one YAML config and a CLI. The individual methods come from their own papers (TIES and DARE each have one); mergekit is the practical home that put them in reach of anyone with a config file. The stable line as of mergekit 0.1.4, June 2026 is what this module is verified against; method names and config keys move between releases, so pin the version and check the config syntax against the repo at that version — never from memory.

The hard prerequisite: the same base

You cannot merge arbitrary models. Merging requires the same base model — same architecture, same layer dimensions, same parameter shapes. You do not merge a Qwen3-4B with a Llama-3; their weight tensors don’t even line up. You merge variants of Anvil that all started from Qwen/Qwen3-4B-Base. This is also why Anvil’s license story stays clean: every variant inherits Qwen3-4B-Base’s Apache-2.0 license, so the merged model is yours to republish without constraint (decision F3).

One sentence of disambiguation you’ll want now and need in the lab: you are merging models (full sets of weights), not adapters. Folding a LoRA adapter back into its base is a different operation — merge_and_unload — and it’s Module 12’s subject. The misconception box returns to this; for now, hold the line that “model merging” (mergekit) and “adapter merging” (merge_and_unload) are two distinct things that share an unfortunate word.

When to merge — and when not to

Three good reasons to reach for a merge:

  1. Combine specializations. You have an SFT that nails the format and a DPO that nails concision. A merge can give you one model that does both — for the cost of a few CPU-minutes instead of a combined retrain.
  2. Recover a lost capability. An alignment stage sometimes degrades a general skill (catastrophic forgetting, a Module 7 theme). Merging the aligned model with an earlier checkpoint can partially restore that skill without redoing the alignment.
  3. Save a run. No combined training job, no GPU hours, no new dataset to curate.

And when not to merge: when the models have different bases (impossible); when a small, targeted retrain would be cleaner and you have the compute; and — most important — when you can’t re-evaluate. A merge you can’t measure is a blind bet. If you don’t have the Module-7 eval pointed at the right held-out split, you have no business shipping a merge.

Here’s the shape of the whole operation:

flowchart LR
    sft["Anvil-SFT<br/>(Qwen3-4B)<br/>same base"]
    dpo["Anvil-DPO<br/>(Qwen3-4B)<br/>same base"]
    mk["mergekit<br/>weight arithmetic"]
    merged["Anvil-merged<br/>(Qwen3-4B)"]
    eval["eval.py (M7)"]
    keep{"beats both<br/>parents?"}
    sft -- "no training · no dataset · CPU/modest GPU" --> mk
    dpo -- "no training · no dataset · CPU/modest GPU" --> mk
    mk --> merged --> eval --> keep
    keep -- yes --> ship["keep it → quantize/serve (M12+)"]
    keep -- no --> discard["discard the merge"]

The merge methods: task arithmetic, SLERP, TIES, DARE

mergekit exposes a family of merge_method values. Four matter for this course, and the right way to think about them is how many models they take and how they handle conflicts.

Task arithmetic / linear averaging — the conceptual baseline

The simplest method is a weighted average of the weights (equivalently, a sum of task vectors added back onto the base). In mergekit this is merge_method: linear or task_arithmetic. It’s the right mental model to start from: every other method is a refinement of “average the deltas.” It works fine for two well-aligned models. Its weakness shows when you stack many models — the task-vector directions start to blur and cancel, which is exactly the interference problem the next section dissects. Linear is the baseline that motivates TIES and DARE.

SLERP — spherical interpolation, two models only

SLERP stands for Spherical Linear intERPolation. It interpolates along an arc between exactly two models — and “exactly two” is the structural fact to file away: SLERP is intrinsically binary. Pass it three models and mergekit errors out.

Why spherical instead of a straight-line average? Walking the arc between the two weight vectors preserves the geometry of the weights better than a flat midpoint does — a plain linear average can shrink the magnitude of the combined update, whereas SLERP keeps it on the “sphere” the two endpoints define. You don’t need the trigonometry; the intuition is enough: spherical interpolation respects the directions the two fine-tunes pulled in, rather than splitting the difference and diluting both. There’s a single knob, the interpolation weight t ∈ [0, 1]: t=0 is all model A, t=1 is all model B, t=0.5 is the even blend. SLERP is the method for the lab, because we’re blending two variants — the SFT and the DPO.

TIES — TrIm, Elect Sign, & merge

When you have three or more models, naive averaging breaks down, and TIES (TrIm, Elect Sign, & merge) is built for it. It runs three steps over the task vectors:

  1. Trim. Keep only the largest-magnitude components of each task vector and zero out the rest. Most of a delta is small, noisy movement; trimming it away reduces interference before you even combine.
  2. Elect sign. For each parameter, two models may disagree on direction — one pushes it positive, another negative. TIES resolves these sign conflicts by electing the dominant sign (summing the signed magnitudes and taking the winner).
  3. Merge. Average only the components that agree with the elected sign. Disagreeing components are dropped rather than allowed to cancel.

The net effect: TIES keeps the parts of each specialist that don’t fight each other, and discards the parts that do. It’s the anti-interference method when you’re combining several specializations. In mergekit it’s merge_method: ties.

DARE — Drop And REscale

DARE (Drop And REscale) is a reduction step you apply on top of another method, not a standalone rival. It randomly drops a fraction of the task-vector components (sets them to zero) and then rescales what survives so the expected magnitude is preserved. The insight: a lot of a delta is redundant, and pruning it stochastically — then rescaling — keeps the signal while shedding the interference-prone bulk. mergekit ships it combined: dare_ties (DARE drop+rescale plus TIES’s sign election) and dare_linear (DARE on top of a linear merge). Reach for dare_ties when you’re merging many models and want both stochastic pruning and conflict resolution.

The decision map

This is the diagram to internalize for the whole module:

flowchart TB
    root{"How many models<br/>to merge?"}
    two["exactly 2"]
    many["3 or more"]
    slerp["SLERP<br/>(merge_method: slerp)<br/>spherical interp, no<br/>conflict resolution"]
    interf{"interference<br/>a concern?"}
    ties["TIES<br/>(merge_method: ties)<br/>trim + elect-sign"]
    dare["DARE / dare_ties<br/>drop & rescale,<br/>often on top of TIES"]
    base["task arithmetic / linear<br/>(simplest baseline)"]
    root --> two --> slerp
    root --> many --> interf
    interf -- "moderate" --> ties
    interf -- "high / many models" --> dare
    root -. "quick baseline" .-> base

And the comparison table — the names are mergekit’s merge_method: values, verified against mergekit 0.1.4 (June 2026):

Method# of modelsConflict handlingmerge_method nameWhen to use
Task arithmetic / linear2 or moreNone — straight (weighted) averagelinear, task_arithmeticA quick baseline; well-aligned models
SLERPExactly 2None — interpolates the twoslerpBlend two fine-tunes (the lab: SFT + DPO)
TIESMany (3+)Sign election + trimtiesMany specializations that conflict
DAREMany (3+)Stochastic drop + rescale, usually on top of TIESdare_ties, dare_linearReduce interference when stacking 3+ models

Why merges fail: interference, and why you must re-evaluate

You now know the methods. This section is the one that keeps you from shipping garbage.

Interference

When you combine several models, their task vectors can contradict each other (sign conflicts — one model pushes a parameter up, another pulls it down) or dilute each other (redundant components that average toward zero). The result is interference: the merged model can be worse than every one of its parents on their respective tasks. You blended two good models and got a mediocre one.

This is not a rare edge case; it’s the default failure mode of naive averaging once you stack more than two models, and it gets worse the more you stack. It is precisely the problem TIES (sign election) and DARE (stochastic drop + rescale) were designed to mitigate — which is why those methods exist at all. So the link between this section and the last one is direct: the more models you merge and the more they conflict, the more you need TIES/DARE over a plain linear average.

The merge weight is a hyperparameter

Even a two-model SLERP at t=0.5 is not automatically optimal. Push t toward the SFT variant and you favor format fidelity; push it toward the DPO variant and you favor concision. The merge weight is a hyperparameter you tune with the eval, not a constant you set once and forget. The same goes for TIES density and weight parameters. There is no universal best setting — there’s the setting that wins on your held-out task.

The rule of the module: never assume, always measure

So here is the discipline that turns merging from a gamble into a decision: you never assume a merge improved — you measure it. You re-run the Module-7 harness (eval.py, with the frozen metrics valid_json, fn_acc, arg_match, abstention, n) and you compare SFT-only vs DPO-only vs merged on the held-out eval split. That comparison is the entire reinforcement of T7 in this module: task-grounded evaluation is what makes a merge decidable. Sloppy evaluation is the single biggest hole in the competition’s fine-tuning content — don’t dig it yourself by trusting a merge you never measured.

⚠️ Common misconception — two linked versions of the same mistake.

(a) “Merging is a kind of training — it needs gradients or a GPU run.” False. Merging is arithmetic on weights: no gradient, no training forward pass, no expensive GPU. It runs on a CPU or a modest card in minutes, and there is no Trainer class anywhere in merge.py’s mergekit path. That is exactly what makes it a “free” capability boost.

(b) “Merging two good models always gives a better model.” Also false. Interference can degrade the result below either parent. A merge you didn’t re-evaluate is unusable — TIES and DARE mitigate interference, but only the eval settles it.

Freshness corollary: do not confuse model merging (mergekit) with merge_and_unload (PEFT, Module 12). mergekit fuses two complete models that share a base; merge_and_unload folds a LoRA adapter back into its base. Two different operations — the next section’s lab touches both, on purpose, so you see the line clearly.

Build it: merge two Anvils and re-evaluate

Goal: merge an SFT variant and a DPO variant of Anvil (same Qwen3-4B base) into one model with mergekit (SLERP) — no gradient steps, no training GPU — then re-run eval.py (Module 7) to compare SFT-only / DPO-only / merged and decide whether to keep the merge.

What you’ll see: a single merged model directory, and a three-column table on the frozen Module-7 metrics that shows clearly whether the merge wins, ties, or regresses (interference).

The full, tested code lives in finetune-prod-labs/module-11. The offline smoke test runs with no GPU and no network, so you can verify the plumbing before you touch a real merge.

Step 1 — Install. The locked core (trl==1.6.0, transformers==5.12.0, peft==0.19.1, …) is all the offline path needs; mergekit is a per-runtime extra (it manages its own pins), so you install it alongside:

uv venv --python 3.12
uv sync                                    # the locked core (offline path)
pip install mergekit==0.1.4                # the merge tool (per pyproject comments)
cp module-11/.env.example module-11/.env   # HF_TOKEN only if the variants are on the Hub

Step 2 — Write the mergekit YAML config. anvil/merge.py generates the config for you — the YAML is the reproducible recipe. For two models, SLERP; the function enforces “exactly two” so you can’t quietly misuse it:

def write_mergekit_config(model_paths: list[str], output_path: str, *, method: str = "slerp",
                          base_model: str | None = None) -> dict:
    import yaml
    if method == "slerp":
        if len(model_paths) != 2:
            raise ValueError("slerp merges exactly two models")
        cfg = {
            "merge_method": "slerp",
            "base_model": model_paths[0],
            "slices": [{"sources": [{"model": m, "layer_range": [0, 36]} for m in model_paths]}],
            "parameters": {"t": 0.5},
            "dtype": "bfloat16",
        }
    elif method in ("ties", "dare_ties"):
        base = base_model or model_paths[0]
        cfg = {
            "merge_method": method,
            "base_model": base,
            "models": [{"model": m, "parameters": {"weight": 1.0 / len(model_paths), "density": 0.5}}
                       for m in model_paths],
            "dtype": "bfloat16",
        }
    else:
        raise ValueError(f"unknown merge method {method!r} (use slerp/ties/dare_ties)")
    pathlib.Path(output_path).write_text(yaml.safe_dump(cfg, sort_keys=False))
    return cfg

Two things to read here. SLERP uses slices with a layer_range and a single t; TIES/DARE use a flat models: list where each model carries a weight and a density. And note the base_model: key in the YAML — this is the shared base for the weight arithmetic (model_paths[0] for SLERP), which is not the same as the base_model: metadata field on a Hub model card (Module 14, lineage). Same word, two jobs. Note too the t=0.5 default — that’s the hyperparameter the eval will let you tune.

Step 3 — Run the merge (no gradients). run_mergekit shells out to mergekit’s CLI. There is no Trainer, no training loop — it’s weight arithmetic:

def run_mergekit(config_path: str, output_dir: str, *, cuda: bool = True) -> str:
    """Run mergekit on a config (needs ``pip install 'mergekit==0.1.4'``). Returns out dir."""
    import subprocess
    cmd = ["mergekit-yaml", config_path, output_dir] + (["--cuda"] if cuda else [])
    subprocess.run(cmd, check=True)
    return output_dir

mergekit-yaml <config> <out_dir> reads the YAML and writes a complete merged model directory — full weights, not an adapter and not a quantized format. The helper defaults to --cuda; on a CPU-only box just call run_mergekit(..., cuda=False) (slower, but mergekit needs no GPU at all — which is the whole point of merging). No source edit required.

Step 4 — Load the merged model. It’s a normal full-precision model directory, so the frozen get_model_and_tokenizer and from_pretrained load it the same way they load any variant. The merge did not quantize anything — that’s Module 12.

Step 5 — Re-evaluate (Module 7, unchanged). This is the step that earns the merge its place. Call the frozen evaluate on each of the three models against the held-out eval split and put them side by side:

from anvil import data, eval as ev
from anvil.config import get_model_and_tokenizer
from transformers import AutoModelForCausalLM

_, tok = get_model_and_tokenizer("full")
rows = data.load_rows("eval")
for name, path in [("SFT-only", "outputs/anvil-sft"),
                   ("DPO-only", "outputs/anvil-dpo"),
                   ("merged",   "outputs/anvil-merged")]:
    model = AutoModelForCausalLM.from_pretrained(path)
    print(name, ev.evaluate(model, tok, rows))

You’ll see something like this (numbers are illustrative — measure your own; a merge may or may not win):

SFT-only  {'valid_json': 0.93, 'fn_acc': 0.88, 'arg_match': 0.74, 'abstention': 0.81, 'n': 200}
DPO-only  {'valid_json': 0.95, 'fn_acc': 0.91, 'arg_match': 0.79, 'abstention': 0.86, 'n': 200}
merged    {'valid_json': 0.96, 'fn_acc': 0.92, 'arg_match': 0.81, 'abstention': 0.85, 'n': 200}

Then make the decision. If merged beats both parents on the metrics you care about, keep it. If it ties, keep the simpler model. If it regresses — interference — say so and discard it. That honesty is the point: a merge that loses is a real and instructive outcome, not a failure of the lab.

Step 6 — The smoke test. The offline suite runs on CPU with no network. It can’t run a real Qwen3 merge (that needs the mergekit extra and full weights), so it verifies two things instead: that the config writers produce correct SLERP-for-two and TIES-for-three configs (including the “exactly two” guard), and — through the sibling merge_adapter path — that the Module-7 re-eval works on a merged model:

def test_slerp_config_two_models(tmp_path):
    out = tmp_path / "slerp.yml"
    cfg = merge.write_mergekit_config(["anvil-sft", "anvil-dpo"], str(out), method="slerp")
    assert cfg["merge_method"] == "slerp" and out.exists()
    assert cfg["parameters"]["t"] == 0.5
    with pytest.raises(ValueError, match="exactly two"):
        merge.write_mergekit_config(["a", "b", "c"], str(out), method="slerp")

Run the offline suite (no GPU, no network):

uv run pytest module-11/tests/

What this lab does NOT do, on purpose. No merge_and_unload and no GGUF/AWQ/GPTQ quantization — that’s Module 12. (The merge.py file ships a merge_adapter function for the next module, but the mergekit path is this module’s subject; we only pose the distinction here.) No retraining — the variants are inputs, and the merge has zero gradient steps. No push_to_hub or model card — that’s Module 14. No big GPU, and never a merge across different bases.

Try it yourself:

  1. Merge three variants with TIES. Add the GRPO variant from Module 10 and call write_mergekit_config([...], method="ties") over SFT + DPO + GRPO. Re-measure: does sign election beat the binary SLERP blend on three models?
  2. Sweep the SLERP weight. Edit parameters.t to 0.3, 0.5, 0.7, merge each, and plot a Module-7 metric against t. You’ll see the optimum is chosen by the eval, not by the 0.5 default.

In production

Merging is a cheap, high-leverage technique — but only when it’s disciplined by the eval. In real work you don’t merge once and ship; you sweep a few weights and methods (a small-t SLERP, a dare_ties over three variants) and keep the one that wins on the held-out task, not the one that “looks reasonable.” My own rule: a merge costs me cents of CPU time versus hours of GPU for a combined retrain — but I only ship a merge that beats both parents on my eval set.

Three habits make it production-grade. Pin mergekit — the config format and method names move between versions (merge_method: keys and parameters drift), so a config written against 0.1.4 isn’t guaranteed to load on the next minor. Version the YAML — merge_config.yaml is the reproducible recipe; it belongs in git next to the eval that justified it. And record the lineage — a merged model you publish must list its source models in its card (a forward reference to Module 14: base_model: plus the merge sources), or nobody, including future-you, can reconstruct it.

The interference guardrail scales with ambition: the more models you stack, the more TIES/DARE earn their keep, and the more non-negotiable the re-eval becomes. One last teaser — merging is not quantizing. Module 12 takes whatever model you keep, merged or not, and shrinks it for serving with merge_and_unload, then GGUF / AWQ / GPTQ.

Concept check

Why this matters. Model merging is the one near-free lever in the whole pipeline: it combines the weights of fine-tunes that share a base, with no training, no gradients, and no expensive GPU. The two skills you’ll use again and again are choosing the method by model count and conflict risk — SLERP for two, TIES/DARE for three or more — and refusing to trust a merge you haven’t re-evaluated. It’s a free boost, conditioned on the Module-7 eval. The questions below stay inside what this module taught.

  1. A teammate says: “I’ll merge my two Anvil variants — I just need to grab a GPU and run a couple of epochs.” What’s wrong, and what does merging actually require?

    • A. Nothing’s wrong; merging is a short fine-tune and needs a GPU and a few epochs.
    • B. Merging is weight arithmetic — no gradients, no training GPU, minutes on CPU/modest GPU; it only requires that the variants share the same base.
    • C. Merging needs a GPU but not epochs; it runs one forward pass over the dataset.
    • D. Merging needs a reward model first, then a short RL loop.

  2. You want to combine two fine-tunes into one model, and separately a colleague wants to combine four specialized models that pull in different directions. Which method fits each?

    • A. SLERP for both — it’s the general-purpose merge.
    • B. SLERP for the two; TIES (or dare_ties) for the four, because SLERP is binary and TIES elects signs to handle conflicts across many models.
    • C. TIES for the two; SLERP for the four.
    • D. Linear averaging for both, then SLERP to clean up.

  3. You merge three strong models with a plain linear average and the result is worse than every one of them on their own tasks. What happened, and what’s the fix?

    • A. The base models were different; nothing can fix it.
    • B. The learning rate was too high during the merge.
    • C. Interference — sign conflicts and dilution across task vectors; TIES/DARE are built to mitigate it, and the re-eval is what catches it.
    • D. The merge needs more epochs to converge.

  4. A merge of two good models came out, and someone proposes shipping it without re-running the eval, “because both inputs were good.” Why is that wrong?

    • A. It’s fine — a merge of two good models is always at least as good.
    • B. A merge can regress via interference, and the merge weight is a hyperparameter; without eval.py you’re making a blind bet.
    • C. You must retrain on the merged weights before shipping.
    • D. You should run MMLU once and ship if it passes.

  5. A dev keeps saying “merge my model” but means folding a LoRA adapter into its base. How does that differ from what mergekit does in this module?

    • A. They’re the same operation with two names.
    • B. mergekit folds adapters; merge_and_unload combines full models.
    • C. mergekit combines two complete models that share a base; merge_and_unload (PEFT, Module 12) folds a LoRA adapter back into its base — two distinct operations.
    • D. Both require a GPU and gradient steps.

Answers.

  1. B. Merging is arithmetic on weight tensors — no gradients, no training forward pass, no expensive GPU; it runs in minutes on CPU or a modest card. The one real requirement is a shared base model. “A couple of epochs” describes training, which merging is not.
  2. B. SLERP is intrinsically binary — exactly two models — so it fits the pair. For four conflicting specialists you want TIES (trim → elect-sign → merge) or dare_ties, which resolve sign conflicts and prune redundant deltas; SLERP simply doesn’t apply to four.
  3. C. This is textbook interference: task vectors contradict (sign conflicts) or dilute (redundant components average toward zero), so the merge underperforms its parents. TIES (sign election) and DARE (drop + rescale) mitigate it — and the Module-7 re-eval is what reveals the regression in the first place.
  4. B. Merging does not always improve; interference can push it below either parent, and even a clean SLERP has a t to tune. Without re-running eval.py on the held-out split you have no evidence — it’s a blind bet. The rule of the module: never assume, always measure.
  5. C. mergekit fuses two complete models sharing a base into a new full model. merge_and_unload folds a LoRA adapter into its base — a different operation, covered in Module 12. They share the word “merge” and nothing else.

Common pitfalls.

  • “Merging is training.” No — there are no gradients and no training GPU; it’s arithmetic on weights, minutes on CPU or a modest card. That’s the “free” in free capability boost.
  • “Merging two good models always gives a better one.” No — interference can degrade it, and a merge you didn’t re-evaluate (Module 7) is unusable. TIES/DARE mitigate; the eval decides.
  • Confusing mergekit with merge_and_unload. mergekit combines two models that share a base; merge_and_unload (PEFT, Module 12) folds a LoRA adapter into its base. Don’t use one term for the other.
  • (Freshness) Treating the YAML as version-stable. merge_method: names and config keys drift between mergekit releases — pin the version (mergekit==0.1.4, June 2026) and verify the config against the repo at that version, never from memory.

Key takeaways

  • Model merging combines the weights of fine-tunes that share the same base — with no training, no gradients, and no expensive GPU. It’s arithmetic on task vectors, runnable in minutes on CPU or a modest card.
  • The hard prerequisite is a shared base model (same architecture and dimensions); all Anvil variants start from Qwen/Qwen3-4B-Base (Apache-2.0), so the merge is yours to republish.
  • SLERP is for exactly two models (spherical interpolation, merge_method: slerp, one knob t); TIES is for three or more (trim → elect-sign → merge); DARE drops & rescales (dare_ties/dare_linear, usually on top); task arithmetic / linear is the baseline that degrades as you stack models.
  • Interference is the real risk — sign conflicts and dilution can make a merge worse than its parents; TIES and DARE exist specifically to mitigate it.
  • Never assume a merge improved — re-evaluate it with the Module-7 harness (SFT-only vs DPO-only vs merged) and keep it only if it wins. The merge weight is a hyperparameter you tune with the eval.
  • mergekit ≠ merge_and_unload. Model merging fuses two complete models; adapter merging folds a LoRA adapter into its base (Module 12).

What’s next

You can blend Anvil’s strengths for free now — but the model you keep, merged or not, is still a full-precision 4B: too big and too slow to serve cheaply. Module 12 shrinks it — merge_and_unload to fold any LoRA adapter into the base, then GGUF, AWQ, and GPTQ for deployment — and draws the bright line between training-time quantization (which you already met with QLoRA) and inference-time quantization.

Want the full fine-tuning concept-check bank and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

Links: Course index · ← Module 10: GRPO and Verifiable Rewards · Module 12: Shrink It for Serving → · Lab code for this module

References