The Cost Ledger — What It Really Costs and How to Cut It
Lesson 8: The Cost Ledger — What It Really Costs and How to Cut It
Fine-tuning has a reputation for being expensive. It is, but not for the reason people assume. Almost everyone budgets for training and gets blindsided by serving. This lesson gives you the real numbers, and then the playbook for shrinking them.
The four buckets
| Bucket | What's in it | Typical share of a first project |
|---|---|---|
| Training | GPU time or per-token training charges | One-time. Often the smallest bucket. |
| Serving / hosting | Hourly deployment charges and per-token inference | Recurring. Usually the largest by far. |
| Evaluation & iteration | Judge-model tokens, repeated runs, human review hours | Recurring during development, easy to forget |
| People | Dataset construction, review, and debugging time | Usually the biggest total cost of all — and it doesn't appear on a GPU bill |
Azure's real price list
These are live published prices pulled from Azure's public Retail Prices API for US regions, in USD. Training is billed per 1M training tokens; hosting is per hour, whether or not anyone uses the model.
| Model | Training / 1M tokens | Hosting / hour |
|---|---|---|
| gpt-4.1 | $25.00 global · $27.50 regional | $1.70 |
| gpt-4.1-mini | $5.50 regional | $1.70 |
| gpt-4.1-nano | $1.50 global · $1.81 regional | $1.70 |
| gpt-4o (0806) | $27.50 data zone | $1.70 |
| gpt-4o-mini (0718) | $3.30 data zone | $1.70 |
| Developer-tier training | gpt-4.1 $12.50 · mini $2.50 · nano $0.75 | — |
| o4-mini RFT | $100/hr global · $110/hr regional | $1.70 |
| Open-weight (Qwen-32B, Llama-3.3-70B, OSS-20B) | — | $0.30–$0.33 |
Four observations that matter more than the individual numbers:
- Hosting is flat across the GPT family. Whether you fine-tune nano or full gpt-4.1, the deployment costs $1.70/hour. Choosing a smaller model saves training money but not hosting money — unless you leave the GPT family entirely.
- Open-weight models host at roughly a fifth of the price. $0.30/hour against $1.70/hour is a 5.7× difference, and it compounds every month.
- The Developer tier undercuts Standard on training by roughly half to two-thirds. gpt-4.1 at $12.50 versus $25.00 per 1M tokens, and nano at $0.75 versus $1.50. That discount is paid for in SLA and preemption risk.
- Regional costs more than global. gpt-4.1 is $25.00 global and $27.50 regional. You are buying data residency with a 10% premium.
The number that should change your plan
One Standard deployment, left running for a month:
$1.70 / hour × 730 hours = $1,241 per month
…and that is before a single inference token is billed.
$1.70 / hour × 730 hours = $1,241 per month
…and that is before a single inference token is billed.
Microsoft's own worked example makes the point dramatically. Take a fine-tuned o4-mini chatbot handling 10,000 conversations a month: 20M input tokens and 40M output tokens.
| Line item | Calculation | Cost |
|---|---|---|
| Hosting | $1.70 × 24 × 30 days | $1,224.00 |
| Input tokens | 20 × $1.10 per 1M | $22.00 |
| Output tokens | 40 × $4.40 per 1M | $176.00 |
| Monthly total | $1,422.00 |
Hosting is 86% of that bill. The token usage — the thing everyone thinks they are paying for — is 14%. Fine-tuning's real cost profile is not "training is expensive," it is "an idle deployment is expensive." A model nobody is calling still bills $1,241 a month.
The alternative: what self-hosting costs
| Option | Cost | Reality |
|---|---|---|
| Google Colab / Kaggle free tier | $0 | Genuinely sufficient for 8B-class QLoRA. Not for production serving. |
| Consumer GPU you already own | $0 marginal | 24 GB cards handle up to ~27B QLoRA comfortably. Electricity only. |
| Rented RTX 4090 / L4-class | ~$0.30–$0.60/hr | Fine-tunes 8B models; serves a small model for a small user base. |
| Rented A100 80 GB | ~$1.00–$2.50/hr | The workhorse. 70B QLoRA, or high-throughput serving. |
| Rented H100 | ~$2.00–$4.00/hr | Overkill for most fine-tunes; great for big serving loads. |
Rental rates above are indicative ranges for 2026 marketplace/spot pricing — they move constantly, so check current listings for your region and GPU before committing. The structure of the comparison is what matters and it is durable: a dedicated rented GPU costs roughly what one managed deployment costs, but serves unlimited token volume and can host many models at once.
The crossover, stated plainly
Do this arithmetic before you deploy anything. Managed:
$1,240/month + tokens, cost rising forever with volume. Self-hosted: fixed hourly rate, cost flat no matter how much traffic you serve. For a low-traffic internal tool, managed is obviously right. For anything serving real volume, the managed endpoint's flat $1,240/month is money you could be spending on a GPU that never gets more expensive as you grow.
The cost-mitigation playbook, ranked
| # | Tactic | Impact | Detail |
|---|---|---|---|
| 1 | Distil from a big model to a small one | highest | Collect production traffic, have a frontier model generate ideal outputs, fine-tune a small open model on them. This is the documented Azure use case: "collect production traffic from an o1 deployment and use that data to fine-tune GPT-4o-mini… can reduce cost and latency since smaller models are more efficient." Cheap one-time generation cost, permanent inference saving. |
| 2 | Kill idle deployments | highest | Hosting bills hourly regardless of traffic. Delete, or use the Developer tier's 24-hour auto-expiry for evaluation. One forgotten deployment is $1,240/month. |
| 3 | Prefer open-weight models when quality allows | highest | Qwen/Llama/Phi host at $0.30–$0.33/hour versus $1.70. A 5.7× hosting reduction, every month, forever. |
| 4 | Use the Developer tier for training experiments | high | gpt-4.1-nano at $0.75 versus $1.50 per 1M tokens; gpt-4.1 at $12.50 versus $25.00. Accepting preemption risk on runs you're iterating on is close to free money. |
| 5 | Iterate on a free GPU first | high | Nail the dataset, chat template and hyperparameters on a free Colab T4. Pay only for the run you already believe in. This is the whole hybrid workflow from Lesson 11. |
| 6 | Train for 1–2 epochs, not more | high | Training is billed per token × epochs. Going from 1 to 3 epochs triples the training bill for a gain your eval set will usually score as negative. |
| 7 | Use multi-LoRA serving | high | One base model plus many adapters instead of N separate deployments. If you are about to fine-tune for three teams, this is the difference between one $1,240/month bill and three. |
| 8 | Shrink the prompt your fine-tune replaces | medium | A behaviour baked into weights no longer needs a 1,500-token system prompt. Fewer input tokens on every request, forever. Measure before and after. |
| 9 | Cap RFT hard | medium | Use the reasoning_effort: Low setting, limit validation examples and eval_samples, choose the smallest grader that judges your task adequately, and tune compute_multiplier. Per-job billing caps at $5,000 — treat that as a kill switch, not a budget. |
| 10 | Choose global over regional when residency permits | medium | ~10% off training (gpt-4.1: $25.00 vs $27.50 per 1M tokens) and faster queue times. |
| 11 | Make evaluation cheap and repeatable | low-med | Generate model outputs once into a file, then score that file repeatedly. Use deterministic parsers for format checks instead of a judge model. Cache and reuse. |
| 12 | Quantise and serve yourself | high | Merge, export to a 4-bit GGUF, and run it on a GPU you rent or own. This is what converts a recurring managed bill into a fixed infrastructure cost — the crossover graph above. |
🪙 Token angle — the discipline that saves the most: compute the cost per 1,000 requests before you deploy, and write it down. Managed endpoints hide their economics because the hosting fee is invisible per-request; a $1,240/month deployment serving 50,000 requests costs $0.025 per request in hosting alone, on top of tokens. Once you express it per request you can compare it to the open-weight alternative, to a rented GPU, and to simply using a bigger base model with a longer prompt — and the right answer is usually obvious. The teams that overspend on fine-tuning are the ones that never did this division.
The hidden costs nobody invoices you for
| Cost | Why it sneaks up |
|---|---|
| Dataset construction | Often 40% of project time. Distillation cuts the generation cost to pennies but human review of the sample is not optional and not free. |
| Iteration | Plan for at least three training runs. Run one never ships. |
| Evaluation | Judge-model tokens plus human review of 50–100 outputs per candidate. Recurring through the whole project. |
| Retraining cadence | A frozen model drifts as production inputs change. Decide the trigger before launch, and budget for it being recurring. |
| Monitoring | Format-compliance rate, refusal rate, latency, output length. Someone has to build and watch it. |
| Storage and retention | Training files, checkpoints, evaluation outputs, and log data all accumulate. Small per unit, real in aggregate. |
Three paths, one project
A 1,000-example dataset, an 8B-class model, moderate production traffic. Rough shapes, so you can see the structure:
| Path | One-time | Monthly | Best when |
|---|---|---|---|
| Free tier + self-host on owned GPU | ~$0 | ~$0 + electricity | You have the hardware, the traffic is modest, and you don't need frontier-model behaviour. |
| Rented GPU + vLLM | policy: a few hours of GPU time | ~$250–$1,800 depending on GPU | Real volume. Cost stops rising with traffic. You accept ops responsibility. |
| Managed endpoint (Azure, GPT family) | training tokens × epochs | ≥$1,241 hosting + tokens | Frontier quality, compliance, zero ops, low-to-moderate traffic, or when data residency is a requirement. |
The one-sentence summary of this lesson: training a fine-tune is usually cheap and one-time; hosting it is the expensive part; and the two decisions that dominate everything else are which model family you fine-tune and whether the deployment stays up when idle.
📺 Watch:
- Deep Dive: Fine-Tuning in Microsoft Foundry — SFT, DPO, Tool Calling and Cost — the cost breakdown segment directly complements this lesson.
- Fine-tuning and distillation with Azure AI Foundry — BRK150 — Microsoft's own framing of distillation as the cost play.
- Fine-Tuning Local LLMs with Unsloth & Ollama — NeuralNine. The self-hosted end of the crossover graph.
🧠 Knowledge Check
1. In Microsoft's worked example, a fine-tuned o4-mini chatbot serving 10,000 conversations/month costs $1,422. What dominates that bill?
2. Which change reduces monthly hosting cost the most?
3. Why is distillation usually the highest-leverage cost tactic?