Token Economics — Compaction, Caching & Context Budgets

Lesson 7: Token Economics — Compaction, Caching & Context Budgets

LLMs have finite context windows, and every token you send costs money (or burns subscription quota). Pi's answer is compaction: when the conversation grows too long, older messages are summarized while recent work is preserved. This lesson covers exactly how that works and every knob that shapes your token bill.

When compaction fires

Auto-compaction triggers when contextTokens > contextWindow − reserveTokens. The default reserveTokens is 16,384 — headroom left for the model's response. You can also trigger it manually with /compact [instructions], where optional instructions focus the summary ("focus on decisions made and files touched").

Before compaction: messagesToSummarize kept Walk back from the newest message, accumulating token estimates until keepRecentTokens (default 20,000) is reached → cut point. After compaction (LLM sees): summary messages from firstKeptEntryId onwards summary is iterative: previous summary feeds the next one /compact [instructions] lets you steer what gets preserved

The mechanics: find a cut point by walking backward from the newest message until keepRecentTokens (default 20,000) is accumulated; summarize everything older into a structured CompactionEntry (summary + firstKeptEntryId); rebuild the context as system + summary + kept messages. On repeated compactions, the summarized span starts at the previous compaction's kept boundary — messages that survived one compaction get included in the next. Two related mechanisms share the same structured summary format: branch summarization (when you navigate with /tree, preserving context across branches) and cumulative file tracking (file operations are tracked across compactions so the summary stays accurate).

Prompt caching

Most providers cache the stable prefix of your request. The cost fields cacheRead/cacheWrite make that visible in the footer. Pi is cache-aware: compaction and branch-summary requests use fresh routing session IDs and disable prompt-cache writes, because those one-off prompts are unlikely to be reused — no point paying cache-write prices for a prompt you'll never send again. The showCacheMissNotices setting surfaces significant cache misses in the transcript, and warnings.anthropicExtraUsage warns about subscription providers that bill extra usage on top of a plan.

Custom summarization via extensions

Extensions can hook session_before_compact, session_compact_failed, and session_before_tree to replace or tune the summarization — your domain's important details (API names, decisions, TODOs) can be forced into every summary, which keeps later turns cheaper because the model doesn't re-derive lost context.

The full token knob table

SettingDefaultEffect
compaction.enabledtrueMaster switch for auto-compaction
compaction.reserveTokens16384Context headroom for the model's response
compaction.keepRecentTokens20000Recent tokens exempt from summarization
branchSummary.reserveTokens16384Reserve for branch summarization on /tree
branchSummary.skipPromptfalseSkip the "summarize branch?" prompt (defaults to no summary)
defaultThinkingLevel—off/minimal/low/medium/high/xhigh/max reasoning budget
thinkingBudgets1024…32768Custom token budget per thinking level
hideThinkingBlockfalseHide reasoning text in output (display only)
showCacheMissNoticesfalseSurface significant prompt-cache misses
retry.maxRetries3Agent-level retries — each retry re-sends the context
🪙 Token-saving checklist:
  1. Pick the cheapest model that can do the job (Lesson 3) — biggest lever by far.
  2. Run /compact "keep API names and open decisions" before starting a new phase of work.
  3. Keep sessions short; fork instead of piling unrelated tasks into one context.
  4. Use !!command so verification commands don't dump output into context.
  5. Write skills and prompt templates so instructions are only paid for when used.
  6. Watch showCacheMissNotices — a session that constantly misses cache is paying full input price.
  7. Set thinking level to minimal/low for mechanical tasks.
  8. Trim keepRecentTokens if you trust your summaries (less re-sent context per turn).
  9. Use a custom session_before_compact summarizer for high-stakes repos.
  10. Prefer subscription auth (flat-rate) for heavy daily use; API keys for sporadic use.

Further Reading

🧠 Knowledge Check

1. When does auto-compaction trigger?

2. What does keepRecentTokens (default 20,000) control?

3. Why does pi disable prompt-cache writes for compaction requests?