Track A — A Real Fine-Tune with Unsloth

Lesson 5: Track A — A Real Fine-Tune with Unsloth

Unsloth is an open-source framework for training and running open-weight models. Its selling point is efficiency: it rewrites the attention and MLP kernels so that LoRA and QLoRA training runs substantially faster and in dramatically less VRAM than the standard Hugging Face stack, with no change to your data or your model choice. It also handles the fiddly parts — chat templates, gradient checkpointing, quantisation — with sane defaults.

Three ways to use it

Hardware reality check, straight from their docs: training works on NVIDIA, AMD, Intel and Apple Silicon (MLX); CUDA compute capability 7.0+ (V100, T4, RTX 20-series and newer) is the floor, and anything from 2018 onward generally works. CPU-only still works for chat and dataset generation. Python must be 3.11–3.13, and CUDA 12.4+ is recommended (12.8+ for Blackwell).

Install

# Recommended: uv with an explicit Python and automatic torch backend selection
uv venv unsloth_env --python 3.13
source unsloth_env/bin/activate
uv pip install unsloth --torch-backend=auto

# Or plain pip (unsloth pulls in matching torch / transformers / trl versions)
pip install unsloth

# Unsloth and vLLM together (for serving what you train)
uv pip install unsloth vllm --torch-backend=auto

# Linux, from scratch, if you need a venv first
apt install python3.13-venv -y
python -m venv unsloth_env
source unsloth_env/bin/activate
pip install --upgrade pip
pip install uv
uv pip install unsloth --torch-backend=auto

There is also an official Docker image (unsloth/unsloth on Docker Hub) which sidesteps every CUDA/driver version problem in one step.

The six steps of every fine-tune

Everything below is the same shape whether you are on a free Colab T4 or a rented H100. The numbers change; the pipeline does not.

from unsloth import FastLanguageModel
import torch

max_seq_length = 2048   # context length; 2048 is the recommended test value
dtype = None            # None = auto; torch.bfloat16 on newer GPUs
load_in_4bit = True     # QLoRA. 4x less memory, ~1-2% accuracy cost.

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/llama-3.1-8b-unsloth-bnb-4bit",
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
)

Note the model name. Models ending in unsloth-bnb-4bit are Unsloth's dynamic 4-bit quants — they use a little more VRAM than standard BitsAndBytes 4-bit but recover meaningfully more accuracy. Names ending in plain bnb-4bit are the ordinary variety, and names with no suffix are the full 16-bit originals. Use the Unsloth one when it exists. There is a full model catalogue and a "which model should I use" guide.

model = FastLanguageModel.get_peft_model(
    model,
    r = 16,                       # rank: 8, 16, 32, 64 or 128
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj"],
    lora_alpha = 16,              # set equal to r, or 2x r
    lora_dropout = 0,             # 0 is the optimised path
    bias = "none",                # optimised path
    use_gradient_checkpointing = "unsloth",   # extra ~30% memory saving
    random_state = 3407,          # reproducible runs
    use_rslora = False,           # rank-stabilised LoRA (alpha/sqrt(r))
    loftq_config = None,
)

Unsloth's docs are unusually firm about target_modules: adapt all major linear layers, both attention and MLP. Removing modules to save memory costs quality for very little saving — the RoUGE scores in their guide show all-layers winning over attention-only or MLP-only.

from datasets import load_dataset

dataset = load_dataset("json", data_files="train.jsonl", split="train")
# Each row: {"messages": [{"role": "user", ...}, {"role": "assistant", ...}]}

from unsloth.chat_templates import get_chat_template

tokenizer = get_chat_template(tokenizer, chat_template="llama-3.1")

def formatting_prompts_func(examples):
    convos = examples["messages"]
    texts = [tokenizer.apply_chat_template(c, tokenize=False,
                                           add_generation_prompt=False)
             for c in convos]
    return {"text": texts}

dataset = dataset.map(formatting_prompts_func, batched=True)

This is the step that silently ruins runs when it is skipped or mismatched (Lesson 4). get_chat_template loads the correct special tokens for the family you chose — llama-3.1, qwen-2.5, gemma-3, and so on. The chat templates guide lists the supported names.

from trl import SFTTrainer, SFTConfig

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,          # newer TRL builds call this processing_class
    train_dataset = dataset,
    args = SFTConfig(
        dataset_text_field = "text",
        max_seq_length = max_seq_length,
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,   # effective batch size = 8
        warmup_steps = 5,
        max_steps = 60,              # quick smoke test; use num_train_epochs for real runs
        learning_rate = 2e-4,
        logging_steps = 1,
        optim = "adamw_8bit",
        weight_decay = 0.01,
        lr_scheduler_type = "linear",
        seed = 3407,
        output_dir = "outputs",
        report_to = "none",
    ),
)
trainer.train()

FastLanguageModel.for_inference(model)   # Unsloth's 2x faster inference path

inputs = tokenizer([tokenizer.apply_chat_template(
    [{"role": "user", "content": "Claim: rear-ended at a stoplight."}],
    tokenize=False, add_generation_prompt=True)], return_tensors="pt").to("cuda")

out = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.batch_decode(out)[0])

# Save the adapter (~tens to hundreds of MB)
model.save_pretrained("lora_adapter")
tokenizer.save_pretrained("lora_adapter")

# Or merge into a standalone model / export for local runtimes
model.save_pretrained_merged("merged_model", tokenizer, save_method="merged_16bit")
model.save_pretrained_gguf("gguf_model", tokenizer, quantization_method="q4_k_m")

The Unsloth-specific gotchas worth knowing

Further Reading

• Unsloth — Fine-tuning LLMs Guide
• Unsloth — Requirements (GPU, OS, VRAM)
• Unsloth — Finetune Llama-3 and use in Ollama
• unslothai/unsloth on GitHub