The Fine-Tuning Family Tree — Every Term You'll Meet
Lesson 2: The Fine-Tuning Family Tree — Every Term You'll Meet
Fine-tuning vocabulary is genuinely confusing because three different questions get mixed into one word. Every technique you will encounter answers one of three questions:
| Question | The axis | The options |
|---|---|---|
| How much of the model do you change? | Parameter efficiency | Full fine-tuning · PEFT (LoRA, QLoRA, DoRA) |
| What are you optimising toward? | Objective | SFT · DPO/ORPO/KTO · RL (PPO, GRPO, RFT) · distillation |
| What stage of the pipeline are you at? | Curriculum | Pretraining → continued pretraining → SFT → preference tuning → RL |
When someone says "we fine-tuned it with QLoRA," they mean: parameter-efficiency = QLoRA, objective = (usually) SFT. When someone says "we did DPO," they mean: objective = preference optimisation, efficiency = whatever they happened to use. These are not alternatives to each other.
Group A — Structural terms (how much of the model moves)
| Term | What it means |
|---|---|
| parameters / weights | The learned numbers in the model — the matrices doing the computation. A "7B model" has ~7 billion of them. |
| base model | The general-purpose pretrained model you start from (Llama, Qwen, Mistral, Gemma…). "Base" sometimes strictly means not instruction-tuned — the raw next-token-prediction version, which is a poor chatbot and generally the wrong starting point unless you're doing continued pretraining. |
| instruct / chat model | A base model already post-trained on instructions and conversations. This is almost always your starting point for SFT — you're teaching a specialist on top of a generalist that already understands English and follows instructions. |
| full fine-tuning | Updating every weight. Highest quality ceiling, enormous memory cost, and nothing to merge or serve separately. Rarely the right choice below ~70B-scale budgets. |
| PEFT | Parameter-Efficient Fine-Tuning — the umbrella term for methods that train a tiny fraction of parameters instead of all of them. LoRA is the dominant member. |
| LoRA | Low-Rank Adaptation. Freeze the base weights, and beside each targeted weight matrix add a small trainable pair of low-rank matrices whose product is the update. Fewer than 1% of parameters become trainable. |
| QLoRA | LoRA on top of a base model whose frozen weights are stored in 4-bit (NF4) precision. Roughly a 4× memory reduction versus LoRA-on-bf16, at a small quality cost and slower step time. This is what makes fine-tuning a 7B model on a single free cloud T4 GPU possible. |
| DoRA | Weight-Decomposed LoRA — splits the update into magnitude and direction, which tends to improve quality at the same rank. Slightly slower per step, available in both Unsloth and Hugging Face PEFT as an option flag. |
| rsLoRA / LoRA+ | Variants that change the scaling factor (rsLoRA) or use different learning rates for the two adapter matrices (LoRA+). Incremental wins; not where beginners should spend attention. |
| adapter | The small trained artefact: the merged product of your LoRA matrices. Typically tens to hundreds of megabytes, versus the base model's gigabytes. Adapters are portable between copies of the same base model. |
| rank (r) | The inner dimension of the LoRA matrices — your main capacity dial. r=8–16 is light, r=32–64 is common, r=128+ is heavy. Bigger rank means more capacity to learn and more ability to overfit. |
| alpha (lora_alpha) | A scaling constant applied to the update. The effective scale is alpha / r, so alpha and r must be reasoned about together. A very common convention is alpha = 2 × r. |
| target modules | Which matrices get adapters. Older recipes adapted attention only (q_proj, v_proj); modern defaults adapt every linear layer, including the MLP (gate_proj, up_proj, down_proj). More coverage = better quality and more parameters trained. |
| merged vs unmerged | The adapter can be kept separate (a small file loaded alongside the base model — flexible, swappable, enables multi-adapter serving) or merged into the base weights (one self-contained model, simplest to deploy, no runtime overhead). |
| catastrophic forgetting | When training on your narrow task degrades capabilities you never meant to touch — the model becomes better at your task and worse at everything else. The main risk that grows with epochs and learning rate. |
Group B — Objective terms (what the model is being optimised toward)
| Term | What it means |
|---|---|
| pretraining | Original training on a giant text corpus — "predict the next token." Millions of GPU-hours. You will never do this. |
| continued pretraining (CPT) / domain adaptation | More next-token training on a large corpus of your domain's raw text (contracts, clinical notes, code). Teaches vocabulary, jargon and register rather than task behaviour. Needs far more data than SFT — typically hundreds of MB to GB of text, not hundreds of examples — and is the one legitimate way fine-tuning does inject knowledge. |
| SFT / instruction tuning | Supervised Fine-Tuning — train on input→output example pairs, still with next-token loss but only on the assistant's tokens. This is what 90%+ of "fine-tuning" projects mean. Your dataset is a pile of ideal conversations. |
| RLHF | Reinforcement Learning from Human Feedback. The classic three-step recipe: SFT → train a reward model on human preference comparisons → run PPO against it. Powerful, expensive, operationally fiddly (needs two models in memory). |
| reward model | A model trained to score outputs the way humans do, used as the automated grader in RLHF/RL. |
| PPO | Proximal Policy Optimisation — the RL algorithm in classic RLHF. |
| DPO | Direct Preference Optimisation — skips the reward model entirely and trains directly on pairs of (chosen, rejected) responses. Much simpler and cheaper than PPO, and usually the better first choice. Needs preference data, not just good examples. |
| ORPO / KTO | Further simplifications. ORPO combines SFT and preference alignment into one pass. KTO works from a simple binary signal (this output was good / this was bad) instead of paired comparisons — handy because unpaired thumb-up data is far easier to collect. |
| GRPO | Group Relative Policy Optimisation — the RL method behind modern reasoning-model training. Scores several sampled completions per prompt against a reward and pushes toward the better ones. Requires verifiable rewards (a unit test passes, a maths answer is correct). |
| RFT (reinforcement fine-tuning) | The managed productised version of GRPO — notably the name Azure AI Foundry uses for its RL offering, where you supply graders rather than a reward model. |
| distillation | Using a strong teacher model to generate training data for a small student. Say a frontier model produces 20,000 ideal responses to your real inputs; you fine-tune a small open model on those. This is the highest-ROI technique in this entire course for cost reduction, and it is what most "we fine-tuned a small model to match the big one" stories actually describe. |
Group C — Training mechanics (the words in the config)
| Term | What it means |
|---|---|
| epoch | One full pass over your dataset. SFT runs are usually measured in 1–3 epochs. |
| step | One weight update. A step processes one batch of examples. Steps = (examples ÷ effective batch size) × epochs. |
| batch size / grad accumulation | Examples processed per forward pass, and how many passes are summed before updating. effective batch = batch_size × grad_accum × num_gpus. The effective batch is what matters for optimisation stability; the per-device batch is what fits in your VRAM. |
| learning rate | How big each update is. LoRA tolerates far higher rates than full fine-tuning — 1e-4 to 2e-4 is a normal range for LoRA, versus 1e-5-ish for full FT. The single most common cause of a ruined run. |
| scheduler / warmup | How the learning rate changes over the run. Typically a warmup from near-zero, then a cosine decay. Warmup prevents a destructive first step. |
| loss | The training objective's value. It should fall and then flatten. Falling train loss with rising validation loss is overfitting. A flat, barely-moving loss usually means the learning rate is far too low — or the data is unlearnable. |
| packing | Packing several short examples into one fixed-length sequence so the GPU doesn't waste compute on padding tokens. A big speed win on short-example datasets; must respect example boundaries. |
| chat template | The exact formatting the model expects for turns (<|im_start|>user… etc.). Every model family differs. Using the wrong template produces a model that was trained on gibberish-adjacent structure and behaves erratically at inference. This is one of the most common silent failures in the entire discipline. |
| max_seq_length | The maximum token length per example. Directly drives memory and time — doubling it roughly squares attention memory. Truncated examples teach truncation. |
| gradient checkpointing | Recompute activations during the backward pass instead of storing them. Buys memory at ~20–30% slower training. Essential on small GPUs. |
Group D — Ops and serving terms
| Term | What it means |
|---|---|
| JSONL | JSON Lines — one JSON object per line, the universal fine-tuning data format. Every provider expects it. |
| training / validation file | The examples the model learns from, and a held-out slice used to detect overfitting. A third, untouched test split is for honest final reporting. |
| held-out eval set | Data the model never trains on, used to measure whether the fine-tune actually helped. If you don't have one, you don't have a result — only a vibe. |
| deployment / endpoint | A served, invocable copy of your model. On managed platforms, training and serving are billed separately, and serving is usually the larger ongoing cost. |
| provisioned throughput (PTU) | Reserved capacity with a fixed hourly price instead of pay-per-token. Sensible above a certain steady request volume; wasteful below it. |
| multi-LoRA serving | Serving one base model plus many adapters, swapping per request. Lets dozens of fine-tunes share a single GPU. A large cost lever. |
| GGUF | The quantised, single-file format used by llama.cpp / Ollama / LM Studio for local inference. The usual export target for a merged fine-tune you want to run on your own machine. |
What you actually need to know first: r, alpha, target modules, learning rate, epochs, chat template, and which objective (SFT vs DPO). That's seven concepts. Everything else in this lesson is vocabulary you'll absorb on contact — but the seven above are the ones that decide whether your run works.
🪙 Token angle: the jargon map is also a cost map. Full fine-tuning costs 10–50× a LoRA run in GPU-hours. PPO costs several times a DPO run because it needs a reward model in memory. Continued pretraining costs orders of magnitude more than SFT because it ingests raw corpora instead of curated pairs. Knowing the words is how you avoid picking the expensive branch by accident.
📺 Watch:
- LoRA & QLoRA Fine-tuning Explained In-Depth — Mark Hennings. The best visual explanation of the PEFT branch.
- Fine Tuning LLM Explained Simply — codebasics. Good if the vocabulary above needs a second pass at a slower pace.
- Fine Tuning LLM Models — Generative AI Course — freeCodeCamp. Long-form, covers the objective axis (SFT vs preference tuning) properly.
🧠 Knowledge Check
1. Someone says "we trained with QLoRA and then ran DPO." Which two axes do those terms belong to?
2. What is the practical difference between a LoRA adapter and a merged model?
3. Which technique is the highest-leverage way to make a small model behave like an expensive one?