aifromzero.dev

Shrink It for Serving: GGUF, AWQ, GPTQ (Fine-Tuning in Production, Module 12)

Module 12 of 15 15 min read Lab code ↗

This is Module 12 of Fine-Tuning in Production, a free 15-module course that takes one open model from raw data to a deployed, published tool-calling specialist on the Hugging Face Hub. Start at Module 1 or browse the full syllabus.

Anvil works. Now try to actually serve it.

By now Anvil is good. You supervised-fine-tuned it (Module 3), squeezed it onto a free 16 GB T4 with QLoRA (Module 5), aligned it with DPO (Module 8), pushed it with GRPO (Module 10), and maybe merged two variants with mergekit (Module 11). The Module-7 eval proves it: given a request and a set of tools, Anvil emits a valid <tool_call>{...}</tool_call> far more reliably than the base model. So you go to serve it — run it in Ollama on your MacBook for a demo, or behind vLLM on a rented GPU for an endpoint. You point the runtime at your output folder, and nothing works.

Ollama can’t read a PEFT adapter — it has no idea what adapter_config.json is. And even the merged model, the full-precision bf16 weights, is roughly 8 GB on disk for a 4B model and saturates a modest GPU before you’ve served a single request. This is the trap Module 5 warned you about, finally cashed in: a trained model is not a deployment format. A QLoRA run gives you a 4-bit base in memory plus a separate adapter; a merge gives you fat bf16 weights. Neither is a file a serving runtime loads, runs fast, and ships small.

The fix is two steps, and they are the whole module. First, fold the adapter back into a full-precision base (merge_and_unload). Then, quantize for inference — produce a deployment file in a format your target hardware actually wants: GGUF for a CPU or a Mac, AWQ or GPTQ for an NVIDIA GPU. This is not the quantization QLoRA did. By the end you’ll have turned Anvil into a servable artifact — and re-run the eval to prove it still answers correctly.

In this module

You’ll learn how to:

  1. Distinguish training-time quantization (bitsandbytes/QLoRA, Module 5 — on the fly, no calibration, saves VRAM not latency) from inference-time PTQ (GGUF/AWQ/GPTQ), and explain why a QLoRA-trained or merged model is not yet a serving artifact.
  2. Run the deployment pipeline — merge_and_unload() the adapter into a full-precision base, save_pretrained, then quantize — and explain why you can’t usefully GGUF a bare adapter.
  3. Compare GGUF (llama.cpp container, k-quants like q4_k_m, CPU/Apple Silicon), AWQ (activation-aware, the usual best speed/quality on GPU), and GPTQ (layer-wise Hessian PTQ, via GPTQModel, not the archived AutoGPTQ): what each is, its best hardware, its tooling, and whether it needs calibration.
  4. Pick a format by hardware target: GGUF for CPU/Mac/non-NVIDIA and Ollama; AWQ/GPTQ for an NVIDIA GPU served by vLLM (Module 13).
  5. Verify a quantized model still works — re-run the task-grounded eval (Module 7) and read the small accuracy drop. Quantization is lossy, so you measure it; you don’t assume it.

You’ll build: Anvil becomes deployable. merge.py folds your adapter into a full-precision base with merge_and_unload, then quantize.py exports a GGUF (q4_k_m, via llama.cpp’s convert_hf_to_gguf.py) and a GPTQ variant (via GPTQModel, not the archived AutoGPTQ). You’ll re-run the Module-7 eval on the quantized model, confirm the JSON validity / function-name accuracy barely move, and your lab.md prints the real time and cost.

Theory areas covered:

  • T11 — Inference quantization (GGUF/AWQ/GPTQ) & serving — 7% of the course. Module 12 owns the quantization half of T11; the serving half (vLLM/Ollama/TGI) is Module 13. The whole area is reinforced once more at the capstone (Module 15).

This module pays off the distinction posed in Module 5 (T5 — training-time quantization is the contrast), reuses the Module-7 eval (T7) for the re-eval, and builds on the merge_and_unload introduced in Module 11 (T4/T10).

Prerequisites: Modules 1–11 — a trained Anvil (QLoRA adapters from M5, optionally DPO from M8 and a merge from M11), the merge_and_unload of M11, the eval.py/compare_base_vs_finetuned of M7, and config.py. For the lab: GGUF builds on CPU (llama.cpp’s convert_hf_to_gguf.py needs no GPU for the export); the GPTQ variant and the GPU re-eval want an NVIDIA CUDA GPU (a Colab/Kaggle T4 16 GB is plenty for a 4B). HF_TOKEN set in your environment (M1). GGUF and AWQ/GPTQ target different hardware — that’s the subject of this module, not a setup detail.

Where you are

Anvil’s pipeline, with this module marked:

  • ✅ M1–M2 — baseline + data (chat template, tools=).
  • ✅ M3–M6 — SFT, LoRA, QLoRA (fits a free 16 GB T4), Unsloth/Axolotl.
  • ✅ M7 — task-grounded eval (tool_call_valid_json, function_name_accuracy, argument_match).
  • ✅ M8–M9 — preference alignment (DPO default; ORPO/KTO/SimPO variants).
  • ✅ M10 — GRPO + verifiable rewards.
  • ✅ M11 — model merging with mergekit (merge_and_unload introduced here too).
  • 👉 M12 — quantize for inference (GGUF / AWQ / GPTQ). Anvil becomes a servable file.
  • ⬜ M13–M15 — serve (vLLM/Ollama/TGI), publish (the Hub), capstone.

The one-line before/after for Anvil: it goes from “trained adapters / a merged bf16 model — accurate but not servable” to “merged to full precision, then quantized to a GGUF and a GPTQ artifact, re-eval’d, ready to serve.” The quantize.py you freeze here is the foundation of the rest of the pipeline: Module 13 loads the GGUF into Ollama and the GPTQ build into vLLM, Module 14 publishes the anvil-qwen3-4b-gguf repo, and the capstone (M15) replays it.

Two different quantizations: training-time vs inference-time

You have done quantization before. In Module 5 you fit Anvil onto a free T4 with QLoRA, and the engine under it was bitsandbytes 4-bit NF4. It is worth being precise about what that actually was, because this module does something that looks similar and is fundamentally different.

QLoRA is training-time quantization. The base weights are stored in 4 bits (NF4) to save VRAM, but they are dequantized back to bf16 for every matmul — the 4-bit form is a memory-saving storage trick, never the compute format. There is no calibration step: NF4 is a fixed, data-free encoding applied on the fly as the model loads. And critically, the goal was to make the training run fit — not to make inference faster. What came out of a QLoRA run was a 4-bit base held in GPU memory plus a set of separate LoRA adapters. There was no servable file anywhere.

This module does inference-time quantization, also called PTQ (Post-Training Quantization). You take a model that is already trained, and you quantize its weights into a file format that serving runtimes — llama.cpp/Ollama, vLLM — load directly and run fast and small. The weights stay quantized for compute (or are dequantized with kernels built for speed, not just for fitting a training run). Different goal, different moment, different artifact. (There is a third thing, QAT — quantization-aware training, where you simulate quantization during training. It’s out of scope here; we name it once so you don’t confuse it with PTQ.)

Here is the contrast in one table — the bridge from Module 5 to Module 12:

Training-time quantization (M5)Inference-time quantization / PTQ (M12)
Library / formatbitsandbytes 4-bit NF4 (QLoRA)GGUF / AWQ / GPTQ
WhenDuring the fine-tuning runAfter training, to serve
Calibration?No — data-free, on the flyGGUF k-quants: no · AWQ/GPTQ: yes
GoalFit the training run in VRAMServe small and fast (latency + footprint)
Artifact4-bit base in memory + a separate adapterA servable model file
ModuleModule 5Module 12

⚠️ Common misconception: “I trained with load_in_4bit (QLoRA), so my model is already a 4-bit deployment format — and I can just point llama.cpp at my adapter folder.”

Wrong, twice — and these are the two traps this whole module exists to fix.

(1) QLoRA’s 4-bit is training-time storage, dequantized to bf16 to compute. It is neither a servable file nor an inference PTQ. It saved VRAM during the run; it did nothing for serving latency or footprint. (This was the Module 5 misconception — paid off here.) To deploy, you still have to quantize for inference: GGUF/AWQ/GPTQ.

(2) An adapter is not a model. A LoRA adapter is the low-rank matrices B·A (Module 4); llama.cpp’s convert_hf_to_gguf.py and the AWQ/GPTQ quantizers all expect a complete set of weights. So you merge_and_unload first, then quantize the full model. Pointing a quantizer at an adapter folder fails by construction.

The two rules to carry out: train-quant (M5) ≠ inference-quant (M12), and adapter ≠ servable model.

So shipping Anvil is two steps. First fold the adapter back in (next section). Then PTQ-quantize for the target hardware (the section after).

Step one: merge the adapter back in (merge_and_unload)

A LoRA adapter is not a model. It’s the pair of low-rank matrices B·A you trained in Module 4, sitting on top of a frozen base — useless without that base and the PEFT runtime to apply them. Every inference quantizer wants a single, standalone set of weights: convert_hf_to_gguf.py reads a Hugging Face model directory, and GPTQModel.load(...) loads full weights to quantize layer by layer. An adapter folder is not that. This is the first “aha”: you can’t usefully quantize a bare adapter — you have to fold it into the base first, producing a self-contained full-precision model.

That folding is merge_and_unload, the operation you met in Module 11. It adds the B·A deltas (scaled by alpha/r) into the base weights and drops the PEFT wrapper, leaving a plain Transformers model — and, as a bonus, zero adapter latency at inference (the adapter math is now baked in; there’s nothing to apply at runtime). Anvil’s merge.py exposes it as merge_adapter:

def merge_adapter(adapter_dir, output_dir, *, base_tier="qlora"):
    """Fold a LoRA/QLoRA adapter into its base and save standalone weights."""
    from peft import PeftModel
    # Merge into a FULL-precision base even if the adapter was trained with QLoRA (4-bit).
    load_tier = "full" if base_tier == "qlora" else base_tier
    base, tokenizer = config.get_model_and_tokenizer(load_tier)
    merged = PeftModel.from_pretrained(base, adapter_dir).merge_and_unload()
    pathlib.Path(output_dir).mkdir(parents=True, exist_ok=True)
    merged.save_pretrained(output_dir)      # full-precision, self-contained HF model on disk
    tokenizer.save_pretrained(output_dir)
    return merged, tokenizer

Read the critical line — it’s a freshness trap most tutorials get wrong. You merge into the full-precision (bf16) base, never the 4-bit one. If Anvil’s adapter was trained with QLoRA (base_tier="qlora"), merge_adapter deliberately reloads the base at tier="full" (bf16) before merging. Why: merging the adapter into a 4-bit base bakes the quantization error into the merged weights — you’d be quantizing twice, and the second quantization (the inference one) would compound a degradation you never measured. Merge clean, in bf16; quantize once, deliberately, afterward.

save_pretrained then writes a complete Hugging Face directory — config.json, the safetensors, the tokenizer — and that directory is the input to every inference quantizer. It’s the pivot of the whole pipeline. Here is the pipeline end to end:

flowchart TD
    BASE["Qwen3-4B-Base (bf16)"] --> MERGE
    ADAPTER["Anvil LoRA adapter (B·A)"] --> MERGE
    MERGE["merge_and_unload()<br/><i>an adapter is not a model — fold it in first</i>"] --> SAVE["merged full-precision model<br/>(save_pretrained: config + safetensors + tokenizer)"]
    SAVE --> GGUF["convert_hf_to_gguf.py + llama-quantize<br/>→ GGUF (q4_k_m)"]
    SAVE --> GPU["GPTQModel / AWQ<br/>→ quantized GPU artifact"]
    GGUF -.-> SERVE
    GPU -.-> SERVE
    SERVE["serve — Module 13<br/>(Ollama loads the GGUF · vLLM loads the AWQ/GPTQ)"]:::ghost
    classDef ghost fill:#f4f4f4,stroke:#bbb,stroke-dasharray:5 5,color:#888;

The greyed box is the fence: Module 12 produces the artifacts; Module 13 serves them. We don’t start a single server here.

Step two: the three inference formats — GGUF, AWQ, GPTQ

This is the core of the module. Once you have a merged, full-precision model, you choose a quantization format by where it will run. There are three that matter in 2026, and they are not interchangeable — they target different hardware and different runtimes.

GGUF is the container format of llama.cpp — and the first thing to get straight is that GGUF is not a single quantization algorithm. It’s a file format that packs the weights, the tokenizer, and metadata into one .gguf file that llama.cpp and Ollama load directly. The actual bit-reduction is done by k-quants: named levels like q4_k_m, q5_k_m, q8_0 (the _k means k-quant, a block-wise scheme; _m/_s are medium/small variants of the same level). K-quants need no calibration set — they quantize the weights block by block with no data. Best hardware: CPU, Apple Silicon, non-NVIDIA GPUs, and any machine with no GPU at all. GGUF is the “runs anywhere, even on your laptop” format. You build it with two llama.cpp tools: convert_hf_to_gguf.py converts the HF directory to a 16-bit GGUF, then llama-quantize applies the k-quant level (e.g. q4_k_m).

AWQ is Activation-aware Weight Quantization (paper 2306.00978). It’s 4-bit, but the trick is in the name: it identifies the ~1% of weights that are most salient — judged by their effect on the activations, not their own magnitude — and protects those, so the quantization error lands on the weights that matter least. The reported result is roughly 99% of the original quality at 4-bit. AWQ needs a small calibration set (a few hundred samples, just to measure the activation statistics — it doesn’t retrain anything). Best hardware: NVIDIA GPU, served by vLLM, where it’s usually the best speed/quality trade-off of the three. (AWQ is covered conceptually here; you implement it in the stretch exercise — quantize.py ships to_gguf and to_gptq, not a to_awq.)

GPTQ is layer-wise, Hessian-based PTQ (paper 2210.17323). It quantizes the model one layer at a time, minimizing the reconstruction error of that layer’s output using an approximation of the loss curvature (the Hessian) to decide how to round each weight. It’s typically 4-bit (3/8-bit are options) and, like AWQ, needs a calibration set. Best hardware: high-throughput serving on NVIDIA, via vLLM. And here is the headline freshness fact of the module: you build GPTQ with GPTQModel (ModelCloud), not AutoGPTQ. AutoGPTQ was archived in April 2025 — it’s the dead tool a thousand stale tutorials still import auto_gptq. GPTQModel is the maintained successor; it’s what quantize.py uses.

GGUFAWQGPTQ
What it isllama.cpp container w/ k-quants (q4_k_m…)activation-aware 4-bit (protects ~1% salient weights)layer-wise Hessian PTQ (per-layer reconstruction)
Best hardwareCPU / Apple Silicon / non-NVIDIA / no-GPUNVIDIA GPUNVIDIA GPU, high-throughput
Toolingllama.cpp convert_hf_to_gguf.py + llama-quantize → Ollamaautoawq / compressed-tensors → vLLMGPTQModel (≠ AutoGPTQ) → vLLM
CalibrationNo (k-quants)Yes (~10 min, 8B)Yes (~20 min, 8B)
Founding paper(llama.cpp project, no single paper)2306.009782210.17323

(Calibration times are for an 8B on an A100, as of June 2026 — measure your own. The runtimes — Ollama, vLLM — are Module 13. Here you only produce the artifact.)

The choice isn’t taste — it’s the hardware target. Read it as a decision tree:

flowchart TD
    Q{"Where will it run?"}
    Q -->|"CPU / Apple Silicon / non-NVIDIA / no GPU<br/>(laptop demo, Ollama)"| GGUF["**GGUF** (q4_k_m)<br/>llama.cpp / Ollama · no calibration"]
    Q -->|"NVIDIA GPU, served by vLLM"| GPU{"Which trade-off?"}
    GPU -->|"best speed/quality (default)"| AWQ["**AWQ** 4-bit<br/>activation-aware · needs calibration"]
    GPU -->|"high-throughput / awkward AWQ tooling"| GPTQ["**GPTQ** 4-bit<br/>GPTQModel (≠ AutoGPTQ) · needs calibration"]
    classDef fmt fill:#eef6ff,stroke:#5b9bd5,color:#234;
    class GGUF,AWQ,GPTQ fmt;

On calibration cost (a teachable number, to be dated): on an 8B model on an A100, GPTQ runs ~20 minutes and AWQ ~10 minutes; GGUF k-quants do no calibration at all (the conversion is near-instant). On a 4B on a T4 it’s faster still — but measure it on your run and present it as “on my run,” dated as of June 2026, never an invented constant. The calibration set itself is small: Anvil reuses a few hundred formatted tool-call examples from anvil.data (Module 2).

A chosen-with-a-reason opinion: for Anvil on a Mac I ship a GGUF q4_k_m — no calibration, runs in Ollama, good enough. For a GPU service I’d reach for AWQ first: ~10 minutes of calibration buys near-lossless 4-bit that vLLM loves. GPTQ is the alternative when AWQ tooling is awkward or you want GPTQModel’s high-throughput kernels — and it’s the variant the lab implements end-to-end, because it’s the one with a clean Python API (GPTQModel.load(...).quantize(...).save(...)). Pick by deployment, not by taste.

Quantization is lossy: verify it still works

The spirit of this whole course is you measure, you don’t assume, and PTQ is exactly where that bites. Quantization degrades the model — sometimes imperceptibly, sometimes just enough to break a tool call. A model that loses one bit of numerical precision in the wrong place can start emitting {"name": "get_weathr"} or dropping a required argument. So after you quantize, you re-evaluate with the exact same suite from Module 7 — tool_call_valid_json, function_name_accuracy, argument_match — and you read the drop.

Note the comparison has shifted. In Module 7 you compared base vs fine-tuned. Here you compare merged (bf16) vs quantized: same model, two precisions, so you isolate the quantization cost from the fine-tuning gain. You reuse eval.py unchanged — it’s frozen — and just feed it the quantized model:

from anvil import eval as ev

before = ev.evaluate(merged_model, tokenizer, rows)   # bf16, the reference
after  = ev.evaluate(quantized_model, tokenizer, rows) # GPTQ / AWQ
print({k: (before[k], after[k]) for k in ("valid_json", "fn_acc", "arg_match")})

What to expect: a small drop — often under 1–2 points on 4-bit AWQ, a bit more on an aggressive GGUF q4. If the drop is large, that’s a signal, and there’s a hierarchy of remedies: bump the GGUF level (q5_k_m instead of q4_k_m), prefer AWQ over a poorly-calibrated GPTQ, or widen the calibration set. You don’t shrug at a regression; you climb the level until the measured quality clears your floor.

Measuring where you can. The GPTQ/AWQ model reloads in plain Transformers (or vLLM), so you can run eval.py on it directly — that’s what the lab does. The GGUF model loads via llama.cpp / llama-cpp-python (or Ollama, in Module 13). You can re-eval the GGUF through a Python binding if it’s installed; if it isn’t, it’s honest to evaluate only the GPTQ variant here and note “full GGUF eval lands when we serve it in Module 13.” That’s a deliberate fence, not a gap — Module 13 is where the GGUF gets loaded into a runtime anyway.

Build it: shrink Anvil to a GGUF and a GPTQ artifact

Goal: turn a trained Anvil (LoRA adapter / merged model) into servable artifacts — fold the adapter (merge_and_unload → save_pretrained), export a GGUF (q4_k_m, llama.cpp) and a GPTQ variant (via GPTQModel), then re-evaluate to prove the quantized model still answers correctly.

What you’ll see: quantize.prepare_merged(...) writes a standalone HF model; quantize.to_gguf(...) writes the .gguf and you print its size (well under the bf16 merge); the GPU run (quantize.to_gptq(...)) prints the GPTQ calibration time; and a small merged-vs-quantized table shows a tiny, displayed drop on valid_json / fn_acc / arg_match. (quantize.py/merge.py are library modules — you call them from the notebook or a one-liner, there’s no CLI.)

The full, tested code lives in finetune-prod-labs/module-12. Its smoke test runs offline — no GPU, no llama.cpp, no network — so you can verify the pipeline before you build anything heavy.

Step 1 — Setup. The locked core (trl==1.6.0, transformers==5.12.0, peft==0.19.1) is all the offline path needs. For the GPU variant, layer the inference quantizer per the pyproject.toml comments — gptqmodel==7.1.0 (ModelCloud, not auto-gptq), plus a llama.cpp checkout (convert_hf_to_gguf.py + llama-quantize) for GGUF. Re-check the GPTQModel / llama.cpp versions on the day you run it (the stack churns).

Step 2 — Merge the adapter (reuses M11). merge.merge_adapter(adapter_dir, out_dir) reloads MODEL_ID in bf16 (never 4-bit, even for a QLoRA adapter), applies PeftModel.from_pretrained + merge_and_unload(), then save_pretrained. quantize.py wraps this as the mandatory step zero of every format:

def prepare_merged(adapter_dir, merged_dir, *, base_tier="qlora"):
    """Step 0 of every quantization path: fold the adapter into full weights (Module 11)."""
    merge.merge_adapter(adapter_dir, merged_dir, base_tier=base_tier)
    return merged_dir

Step 3 — Export GGUF. quantize.py builds the two llama.cpp commands and runs them: convert the merged directory to a 16-bit GGUF, then k-quantize to q4_k_m. The conversion is the only place the GGUF level is chosen, and the function rejects an unknown level rather than silently producing garbage:

GGUF_QUANT_TYPES = ("f16", "q8_0", "q5_k_m", "q4_k_m", "q4_0")  # common llama.cpp targets

def to_gguf(merged_dir, out_dir, *, quant_type="q4_k_m", llama_cpp_dir="llama.cpp"):
    """Convert then k-quantize to GGUF (needs a llama.cpp checkout). Returns the quantized path."""
    import subprocess
    out = pathlib.Path(out_dir); out.mkdir(parents=True, exist_ok=True)
    f16 = str(out / "anvil-f16.gguf")
    quant = str(out / f"anvil-{quant_type}.gguf")
    subprocess.run(gguf_convert_command(merged_dir, f16, llama_cpp_dir=llama_cpp_dir), check=True)
    subprocess.run(gguf_quantize_command(f16, quant, quant_type, llama_cpp_dir=llama_cpp_dir), check=True)
    return quant

Step 4 — Export GPTQ (the GPU variant, via GPTQModel). This is the one place the GPU quantizer lives. GPTQModel.load(...) takes a QuantizeConfig(bits=4, group_size=128), .quantize(calibration) runs the layer-wise Hessian pass over a small calibration set (here, a handful of formatted Anvil prompts), and .save writes the quantized directory. Note what’s not imported — no auto_gptq:

def to_gptq(merged_dir, out_dir, *, bits=4, group_size=128, calibration=None):
    """Quantize to GPTQ with GPTQModel (NOT AutoGPTQ — archived). Needs [quant] + a GPU."""
    from gptqmodel import GPTQModel, QuantizeConfig

    samples = calibration or ["Hello, what's the weather in Paris?"] * 16
    model = GPTQModel.load(merged_dir, QuantizeConfig(bits=bits, group_size=group_size))
    model.quantize(samples)
    model.save(out_dir)
    return out_dir

In the real run you pass a real calibration set — a few hundred tool-call prompts built from anvil.data (Module 2) — and you time the .quantize call to report the calibration cost.

Step 5 — Re-evaluate (Module 7, unchanged). Reload the GPTQ model in Transformers and run the frozen eval.evaluate on the held-out split, against the bf16 merged model as the reference. Print the merged-vs-quantized table and read the drop (e.g. # GPTQ 4-bit: -0.8 pt fn_acc — acceptable). The GGUF re-eval is via llama-cpp-python if installed, otherwise deferred to Module 13.

Step 6 — Cost/time. Record the wall-clock (GGUF conversion + GPTQ calibration + re-eval) and write it into the cost line at the bottom of lab.md.

Step 7 — The offline smoke test. The suite verifies the pipeline with no GPU and no llama.cpp: it runs a 1-step tiny SFT to produce an adapter, then asserts prepare_merged yields standalone full weights (a config.json, and crucially no adapter_config.json), and checks the llama.cpp command construction — including that an unknown quant type is rejected:

def test_prepare_merged_yields_standalone_weights(tmp_path):
    adapter = tmp_path / "adapter"
    train_sft(tier="tiny", use_lora=True, data_limit=4, output_dir=str(adapter),
              bf16=False, max_steps=1, per_device_train_batch_size=2, save_strategy="no")
    merged = quantize.prepare_merged(str(adapter), str(tmp_path / "merged"), base_tier="tiny")
    assert (pathlib.Path(merged) / "config.json").exists()
    assert not (pathlib.Path(merged) / "adapter_config.json").exists()

Run it from the repo root:

uv run pytest module-12/tests/
# 4 passed

The actual conversions need llama.cpp or a GPU (gptqmodel), so those calls are out of scope for the offline suite — the merge-then-quantize plumbing and the command construction are what’s verified offline (decision F14 — CI stays green without a GPU).

Try it yourself (not graded):

  1. Export a more aggressive GGUF (q5_k_m) and a less aggressive one (q8_0), re-evaluate all three levels, and plot the size/quality trade-off — the “how few bits can I afford” curve.
  2. Build the GPTQ variant and an AWQ variant on the same merged model and compare calibration time and eval drop — test the “AWQ ≈ best trade-off” claim on Anvil yourself.

What this lab does NOT do. No serving (M13): no Ollama, no vLLM, no TGI, no latency/throughput benchmark — the “serve” box in the diagram is greyed out. No push_to_hub / model card (M14): the artifacts stay local; the anvil-qwen3-4b-gguf repo arrives in M14. No AutoGPTQ — GPTQ is GPTQModel, full stop. No quantizing a bare adapter — always merge_and_unload first, on the bf16 base. No re-training (QLoRA/DPO/GRPO are done; we start from a trained Anvil). No QAT — named once to frame PTQ, never implemented.

In production

Outside the lab, the format choice follows the real deployment, not your preference. CPU / edge / a laptop demo → GGUF (Ollama, llama.cpp). A throughput GPU service → AWQ/GPTQ behind vLLM (Module 13). Three habits keep this from biting you.

Pin the quantizer. The GGUF file format and the GPTQModel/vLLM kernels move between versions — a GGUF produced by an old llama.cpp can stop loading in a new one, and a GPTQ build can become incompatible with a newer vLLM. Date everything (as of June 2026) and re-quantize when you bump the stack, rather than trusting an old artifact silently.

Re-evaluate every time. A quality regression from quantization is silent: an over-aggressive q3 passes your unit tests (it still emits some JSON) but starts missing tool calls in production. The Module-7 eval is the only thing that catches it. Run it on every quantized artifact before you ship, and keep a measured quality floor per format.

Budget the calibration (decision F6). AWQ/GPTQ calibration costs real GPU minutes (~10–20 min on an A100 for an 8B, as of June 2026 — measure your own); GGUF k-quants are near-free in compute. My own rule: I keep both artifacts off one merged checkpoint — a GGUF q4_k_m for the laptop demo and an AWQ build for the GPU endpoint — same weights, two targets, two cost profiles.

The artifacts you produced here don’t run themselves. Next module serves Anvil: Ollama gets the GGUF, vLLM serves the LoRA adapter directly (--enable-lora, no merge — your AWQ/GPTQ build is an equally valid vLLM input), and we benchmark single-request latency.

Concept check

Why this matters. This module owns the quantization half of T11. The skills it anchors: a trained model isn’t servable — you merge_and_unload the adapter into a bf16 base, then PTQ-quantize for inference; the format follows the hardware target (GGUF for CPU/Mac/Ollama, AWQ/GPTQ for NVIDIA/vLLM); GPTQ uses GPTQModel, not the archived AutoGPTQ; AWQ/GPTQ need calibration, GGUF k-quants don’t; and because quantization is lossy, you re-evaluate merged-vs-quantized and read the drop. The quantize.py you freeze here feeds the serving in Module 13.

  1. A developer points llama.cpp’s convert_hf_to_gguf.py at their LoRA adapter folder and it fails. Why, and what’s the fix?

    • A. GGUF doesn’t support 4-bit; convert to fp16 first.
    • B. An adapter is not a model — the converter needs full weights; merge_and_unload the adapter into the bf16 base, save_pretrained, then convert.
    • C. Quantize the adapter directly with a smaller group_size.
    • D. Merge the adapter into the 4-bit base, then convert.

  2. You need to serve Anvil on a MacBook (Apple Silicon, no NVIDIA GPU). Which format, and why not the others?

    • A. GPTQ via GPTQModel — it’s the fastest on any hardware.
    • B. AWQ — it’s near-lossless 4-bit.
    • C. GGUF (q4_k_m) via Ollama/llama.cpp — it targets CPU/Apple Silicon; AWQ/GPTQ target NVIDIA + vLLM.
    • D. Keep the bf16 merged model — quantization isn’t worth it on a Mac.

  3. A teammate says: “I trained Anvil with QLoRA load_in_4bit, so it’s already a 4-bit deployable model.” What’s wrong?

    • A. Nothing — QLoRA output is a servable 4-bit file.
    • B. QLoRA’s 4-bit is training-time storage (dequantized to bf16 to compute); it’s not a servable file and not an inference PTQ. You still merge_and_unload + quantize (GGUF/AWQ/GPTQ).
    • C. It’s true but only on NVIDIA GPUs.
    • D. QLoRA produces GGUF directly, so just rename the file.

  4. You want a GPTQ build and a tutorial shows from auto_gptq import AutoGPTQForCausalLM. What’s the issue, and what does GPTQ require?

    • A. Nothing — AutoGPTQ is the current tool.
    • B. AutoGPTQ was archived (April 2025); use GPTQModel (ModelCloud). GPTQ needs a calibration set (unlike GGUF k-quants).
    • C. Rename the import to gptqmodel — same API, no other change, and GPTQ needs no calibration.
    • D. GPTQ is for CPUs; use GGUF instead and skip calibration.

  5. Your ultra-compact GGUF q3 passes the unit tests, but in production Anvil misses tool calls. What’s the reflex, and the remedy?

    • A. Quantization is lossless, so it must be a serving bug — re-deploy.
    • B. Re-evaluate merged-vs-quantized with the Module-7 eval (quantization is lossy); if the drop is large, bump the level (q4_k_m/q5_k_m), prefer AWQ, or widen calibration.
    • C. Lower group_size to 8 and re-ship without measuring.
    • D. Retrain Anvil from scratch with more data.

Answers.

  1. B. A LoRA adapter is B·A on top of a frozen base — not a standalone model. convert_hf_to_gguf.py (and the AWQ/GPTQ quantizers) need full weights, so you merge_and_unload first, into the bf16 base, and save_pretrained. A is false (GGUF k-quants handle the bit-reduction); C quantizes an adapter (impossible); D merges into a 4-bit base, which bakes in quantization error — you always merge in full precision.
  2. C. GGUF is the llama.cpp container, the right format for CPU / Apple Silicon / non-NVIDIA, and it runs in Ollama with no calibration. AWQ and GPTQ target NVIDIA GPUs served by vLLM — wrong hardware for a Mac. D throws away the whole point of the module (the bf16 model is fat and slow to serve).
  3. B. QLoRA’s 4-bit is a training-time memory trick: the weights are dequantized to bf16 for every matmul, and what you get out is a 4-bit base in memory plus a separate adapter — no servable file. To deploy you merge_and_unload then quantize for inference (GGUF/AWQ/GPTQ). This is the train-quant ≠ inference-quant distinction from Module 5, cashed in here.
  4. B. AutoGPTQ is archived (April 2025); GPTQ is built with GPTQModel today (GPTQModel.load(...).quantize(calibration).save(...)). And GPTQ — like AWQ, unlike GGUF k-quants — needs a calibration set to estimate the per-layer quantization. C is wrong because GPTQ does need calibration; D misplaces GPTQ (it’s a GPU format).
  5. B. Quantization is lossy, and a too-aggressive level (q3) can pass unit tests while silently breaking calls. The reflex is to re-evaluate merged-vs-quantized with the frozen Module-7 metrics and read the drop; if it’s too big, climb the level (q4_k_m/q5_k_m), switch to AWQ, or widen the calibration set. A is the misconception (quantization is not lossless); C ships without measuring; D is overkill — the fix is the quantization level, not the training.

Common pitfalls.

  • Quantizing or serving a bare adapter. A LoRA adapter isn’t a model — merge_and_unload first (into the bf16 base, never the 4-bit one), then quantize the full model. Pointing GGUF/AWQ/GPTQ at an adapter folder fails by construction.
  • from auto_gptq import ... as “the” GPTQ tool. AutoGPTQ was archived in April 2025 — use GPTQModel (ModelCloud). Countless old tutorials still show auto_gptq; date everything as of June 2026 and verify the GPTQModel/llama.cpp versions on the day you run.
  • “My QLoRA 4-bit model is already deployable.” No — train-quant (Module 5) ≠ inference-quant (Module 12); QLoRA’s 4-bit is training storage, not a file. And inference quantization is lossy, so you re-evaluate, you don’t assume. This is the conceptual trap of the module.

Key takeaways

  • Train-quant (QLoRA, M5) ≠ inference-quant (PTQ, M12). Different goals (fit the run vs serve small/fast), different moments (during training vs after), different artifacts (4-bit base in memory + adapter vs a servable model file).
  • An adapter is not a model. Before any quantizer, merge_and_unload the adapter into the bf16 base (never the 4-bit one) and save_pretrained — this is also zero adapter latency at inference.
  • GGUF is the llama.cpp container with k-quants (q4_k_m…) — CPU / Apple Silicon / Ollama, no calibration, “runs anywhere.” It’s a format, not a single algorithm.
  • AWQ = activation-aware 4-bit, ~99% reported quality, NVIDIA GPU / vLLM, needs calibration (~10 min, 8B). The usual best speed/quality trade-off on GPU.
  • GPTQ = layer-wise Hessian PTQ, NVIDIA high-throughput, via GPTQModel (not the archived AutoGPTQ), needs calibration (~20 min, 8B).
  • Quantization is lossy → re-evaluate. Run the Module-7 eval merged-vs-quantized, read the drop; if it’s large, bump the level / switch to AWQ / widen calibration. Measure, don’t assume.
  • Serving (Ollama/vLLM/TGI) is Module 13. Module 12 produces the GGUF and GPTQ/AWQ artifacts off one merged checkpoint; it starts no server.

What’s next

Anvil is shrunk and ready — now let’s actually serve it. Module 13 loads your GGUF into Ollama and serves your LoRA adapter directly on vLLM (--enable-lora, no merge — your AWQ/GPTQ build is an equally valid vLLM input, but the module demonstrates adapter-serving), then benchmarks single-request latency (with one security note about runtime-loaded LoRA). The quantization you froze here is what Ollama consumes.

Want the full fine-tuning concept-check bank and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

Links: Course index · ← Module 11: Model Merging with mergekit · Module 13: Serve It — vLLM, Ollama, and TGI → · Lab code for this module

References