Shipping It — Merge, Quantise, Serve

Lesson 10: Shipping It — Merge, Quantise, Serve

You have a trained adapter. It is a few tens of megabytes sitting next to a multi-gigabyte base model, and it is useless until you decide which of three shapes to ship it in.

Shape	What it is	Choose it when
Unmerged adapter	The LoRA files loaded alongside the untouched base model at serving time	You want many fine-tunes sharing one base model, or you expect to keep iterating and swapping.
Merged model	The adapter mathematically folded into the base weights — one self-contained model	You want the simplest possible deployment, and one model per GPU is fine.
Hosted endpoint	A provider runs the deployment for you and bills by the hour or the token	You don't want to run GPUs. Convenient, and usually the most expensive per-request option above moderate traffic.

Merging: the one-line payoff

Merging evaluates W = W₀ + (α/r)·B·A once, for every adapted matrix, and writes the result out as a normal model file. There is no runtime cost and no quality loss — it is exactly the same maths the model was already doing at inference, just precomputed.

# Unsloth / TRL path — after training completes
model.save_pretrained_merged("merged_model", tokenizer, save_method="merged_16bit")

# Need a smaller artefact for local use?
model.save_pretrained_gguf("gguf_model", tokenizer, quantization_method="q4_k_m")

That second call is how a fine-tune ends up as a file Ollama or LM Studio can load. If you have taken the quantization course on this site, this is the moment those GGUF suffixes stop being trivia: q4_k_m is the standard quality/size compromise, q8_0 is nearly lossless, and anything below 3-bit will visibly damage a model this small. Merging first and quantising second is the right order — quantising before merging throws away precision the merge could have used.

Serving options

Runtime	Best for	Notes
vLLM	Production, concurrency, throughput	PagedAttention and continuous batching give by far the best tokens/sec per GPU. OpenAI-compatible API out of the box. Supports multi-LoRA serving.
Ollama / llama.cpp	Local, laptop, single-user, GGUF files	Trivial to set up, CPU-friendly, no Python environment. Poor at concurrency — it is a personal runtime, not a server.
TGI (Text Generation Inference)	Production, Hugging Face ecosystem	Similar niche to vLLM with a different operational feel; strong Docker story.
Managed endpoint	You don't want to run GPUs at all	Zero ops, highest marginal cost, least control. Check the hourly hosting price before committing (Lesson 8).

Running the merged model locally in Ollama

# Modelfile
FROM ./my-finetune-q4_k_m.gguf

PARAMETER temperature 0.3
PARAMETER stop "<|im_end|>"

SYSTEM "Respond only with the required JSON schema."

ollama create my-claims-model -f Modelfile
ollama run my-claims-model

Multi-LoRA: the biggest serving-side cost lever

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora \
  --lora-modules claims=./adapter-claims triage=./adapter-triage tone=./adapter-tone

# One resident base model. Three fine-tunes. Requests select the adapter
# by name via the "model" field, exactly like separate deployments.

This matters enormously for cost. Without it, each fine-tune needs its own GPU-resident model copy, and you pay the full base-model memory cost per variant. With it, you pay for the base model once and the adapters are megabytes. If your roadmap contains "and then we'll fine-tune it for three more teams," design for multi-LoRA from the start.

🪙 Token angle — the prompt you get to delete: a fine-tune bakes instructions into weights, so the 1,500-token system prompt that enforced your format can shrink to almost nothing. Two effects compound: the model is a smaller, cheaper one and every request carries fewer prompt tokens. At high volume this is often a larger saving than the training run cost. Measure it: log prompt tokens per request before and after.

The pre-deployment checklist

Gate	What it means
Eval passed on the untouched test set	Format compliance and task metric both beat the recorded baseline. Numbers on file.
Regression suite passed	Unrelated capabilities — general chat, reasoning, safety refusals — did not degrade beyond an agreed threshold.
Human review completed	A person read 50–100 outputs and signed off. Judges miss tone and subtle wrongness.
Everything versioned together	Base model checkpoint, adapter hash, dataset hash, hyperparameters, eval results. Without this you cannot reproduce or debug the model next quarter.
Rollback path exists	One flag flips traffic back to the previous model. Fine-tunes fail in ways that only show up at scale.
Monitoring wired up	Log format-compliance rate, refusal rate, latency and output length in production. Behaviour drifts as real inputs drift away from your training distribution.
Retraining trigger defined	What production signal means "collect new data and retrain"? If there is no answer, the model quietly rots.

The distribution trap: your fine-tune is excellent — on inputs that look like your training data. Real traffic contains typos, mixed languages, empty fields, adversarial inputs and entirely new request types. Always sample real production inputs for your test set, not synthetic ones. A model that scores 98% on synthetic tests and 71% in production was never actually evaluated.

📺 Watch:

Fine-Tuning Local LLMs with Unsloth & Ollama — NeuralNine. The merge-then-serve path end to end, exactly as this lesson describes it.
How to Fine-Tune any AI Model Locally (FULL Tutorial) — Tech With Tim. Full local pipeline including export and running the result.
Fine Tuning and Distillation with Azure AI Foundry — Microsoft's Marco Casalaina on the managed deployment side.

🧠 Knowledge Check

1. What does merging a LoRA adapter actually do?

2. You need to serve five different fine-tunes of the same 8B base model. What's the most cost-efficient approach?

3. Correct order of operations for a locally-served fine-tune?