Observability, Cost Control, and Evals
Lesson 9: Observability, Cost Control, and Evals
Three questions decide whether an agent survives contact with production: what is it doing, what is it costing, and how do I know it still works. The SDK answers all three with real APIs โ and, unusually for an agent framework, it has a genuine answer for the third that is not "ask a model to grade it".
Part 1 โ Telemetry
Opt in by passing a TelemetryConfig on the client. That is all most applications ever need: the CLI exports its own spans to your OTLP endpoint.
import { CopilotClient } from "@github/copilot-sdk";
const client = new CopilotClient({
telemetry: {
otlpEndpoint: "http://localhost:4318",
},
});
| Option | Purpose |
|---|---|
otlpEndpoint | OTLP HTTP endpoint URL |
otlpProtocol | "http/json" or "http/protobuf" for all signals |
exporterType | "otlp-http" or "file" |
filePath | Path for JSON-lines trace output (dev/offline diagnostics) |
sourceName | Instrumentation scope name |
captureContent | Whether to capture message content |
captureContent is a privacy switch, not a verbosity switch. Several attributes require it โ including gen_ai.skill.description, gen_ai.skill.source.uri, and process.executable.path. With content capture on, prompts and tool payloads can land in your tracing backend. Decide deliberately, and treat your trace store with the same sensitivity as your logs. Note also that while managed telemetry settings are unresolved, pending tracking buffers only capture-independent facts โ descriptions, source URIs, and executable paths observed before the policy resolves are omitted even if capture is later enabled. The final policy governs replay, and a disabled or cancelled tracking session discards the buffer.
Trace context propagation โ the advanced path
If your application already creates its own OpenTelemetry spans and wants them in the same distributed trace as the CLI's, Node.js needs an onGetTraceContext callback. The SDK calls it before session.create, session.resume, and session.send:
import { CopilotClient } from "@github/copilot-sdk";
import { propagation, context } from "@opentelemetry/api";
const client = new CopilotClient({
telemetry: { otlpEndpoint: "http://localhost:4318" },
onGetTraceContext: () => {
const carrier: Record<string, string> = {};
propagation.inject(context.active(), carrier);
return carrier; // { traceparent: "00-...", tracestate: "..." }
},
});
Python, Go, and .NET inject automatically when their OpenTelemetry/Activity API is configured โ no callback needed. In Node the SDK has no OpenTelemetry dependency, so the inbound direction is raw strings on the tool invocation, which you restore by hand:
import { defineTool } from "@github/copilot-sdk";
import { propagation, context, trace } from "@opentelemetry/api";
const myTool = defineTool("my-tool", {
description: "Do work",
handler: async (args, invocation) => {
// The CLI's span context arrives as raw traceparent/tracestate strings
const carrier = {
traceparent: invocation.traceparent,
tracestate: invocation.tracestate,
};
const parentCtx = propagation.extract(context.active(), carrier);
const tracer = trace.getTracer("my-app");
return context.with(parentCtx, () =>
tracer.startActiveSpan("my-tool", async (span) => {
try {
return await doWork(args);
} finally {
span.end();
}
})
);
},
});
This is what lets a "handle tool call" span in your app appear nested inside the CLI's "execute tool" span โ so a slow request is attributable to your downstream call rather than looking like model latency.
Skill and process attributes
The runtime follows the OpenTelemetry GenAI skill semantic conventions by refining existing execute_tool spans rather than adding a separate skill operation. A loader is named execute_tool skill {skill}; an unambiguously identified resource or script becomes execute_tool {tool} {skill} {resource} with a skill-relative resource name.
| Attribute | Notes |
|---|---|
gen_ai.skill.name, gen_ai.skill.resource.name | Available independently of content capture |
gen_ai.skill.description, gen_ai.skill.source.uri | Require content capture. Local source values are file URIs, not bare paths. |
process.executable.name | The directly launched executable (e.g. the shell launcher) โ not a guessed command token |
process.executable.path | Absolute known path; requires content capture |
process.exit.code | Integer including zero when a command-executing tool reports an exit; unavailable exits and pre-spawn failures are omitted |
A few semantics worth knowing so you do not misread a dashboard: poll tools do not inherit process attributes from the command they read. Sandboxed executable identities are omitted when the backend does not expose them. And a nonzero exit does not imply the tool itself failed. Preloaded skills produce invocation events without a fabricated tool span; repeated preparation of the same selected-agent skill context emits one receipt, while changed content or provenance emits another, and child sessions report their own preloads independently.
gen_ai.skill.name instead of github.copilot.tool.parameters.skill_name or github.copilot.skill.name, and gen_ai.skill.source.uri instead of github.copilot.skill.path. The github.copilot.skill.invoked event and its content, source-category, trigger, and plugin fields remain Copilot-specific. The skill attributes are at Development stability; the process attributes are Release Candidate.
Part 2 โ Cost: two mechanisms, six signals
The SDK surfaces usage through events (ephemeral, real-time, per-call) and RPCs (request/response, accumulated). Knowing which to call is the whole game:
| Signal | API | Use it for |
|---|---|---|
| Per-call token counts | assistant.usage event | A live cost or token meter as a turn runs |
| Context-window utilization | session.usage_info event | Live context pressure |
| Context-window breakdown | session.metadata.contextInfo (RPC) | Context usage on resume, before any new turn |
| Accumulated totals | session.usage.getMetrics (RPC) | Final cost summary for a turn or session |
| Per-model pricing | models.list (RPC) | Estimating cost before running work |
| Account quota | account.getQuota (RPC) | Warning users before they exhaust their plan |
// Real-time, per model API call โ fires once per call, INCLUDING sub-agent calls
session.on("assistant.usage", (event) => {
const { model, inputTokens, outputTokens, cost } = event.data;
console.log(`${model}: in=${inputTokens ?? 0} out=${outputTokens ?? 0} cost=${cost ?? 0}`);
});
// Accumulated session totals โ the cleanest way to read AI credit cost
const metrics = await session.rpc.usage.getMetrics();
const aiCredits = (metrics.totalNanoAiu ?? 0) / 1e9;
console.log(`AI credits used: ${aiCredits.toFixed(6)}`);
console.log(`Premium requests: ${metrics.totalPremiumRequestCost}`);
for (const [model, m] of Object.entries(metrics.modelMetrics)) {
if (!m) continue;
console.log(
`${model}: in=${m.usage.inputTokens} out=${m.usage.outputTokens} nanoAiu=${m.totalNanoAiu ?? 0}`,
);
}
Fields on assistant.usage are worth a long look โ this single event is the most information-dense thing the runtime emits:
| Field | What it gives you |
|---|---|
inputTokens, outputTokens | Volume |
reasoningTokens | Output tokens spent on chain-of-thought โ a subset of output |
cacheReadTokens, cacheWriteTokens, cacheExpiresAt | Prompt-cache effectiveness and expiry โ the fastest way to explain a cost change |
duration, timeToFirstTokenMs, interTokenLatencyMs | Latency decomposition: queueing/dispatch vs generation throughput |
model, reasoningEffort | Which model and effort actually served the call |
apiEndpoint | "/chat/completions" ยท "/v1/messages" ยท "/responses" ยท "ws:/responses" โ which API shape served the turn |
initiator | What triggered the call โ e.g. "sub-agent"; absent for user-initiated |
apiCallId, serviceRequestId, providerCallId | Provider completion ID, Copilot service request ID, GitHub request ID โ your correlation handles for a support ticket |
finishReason, contentFilterTriggered | Did the response stop naturally, hit length, or get filtered? |
quotaSnapshots, copilotUsage | Per-quota resource usage and an itemised token cost breakdown |
totalNanoAiu). The docs divide by 1e9 as an SI convenience but say explicitly that the conversion to AI credits and the meaning of premium-request accounting are defined by GitHub Copilot billing, not the SDK โ verify against GitHub's billing docs before surfacing anything currency-like to a user. (2) assistant.usage is ephemeral: delivered live, not replayed on resume. If you want totals after the fact, call session.usage.getMetrics; if you want history, you must persist the events yourself.
cacheReadTokens against inputTokens: a healthy agent reuses prompt cache heavily across turns. A session where cache reads collapse is one where something is invalidating the prefix every turn โ usually a hook that rewrites the prompt (modifiedPrompt, modifiedTransformedPrompt) or a tool returning volatile content early in context. And remember turns are API calls: cost spikes are almost always more round-trips, not a pricier model.
Budgets: making runaway agents impossible
Two independent budget layers exist, and you want both.
Session budget
const session = await client.createSession({
onPermissionRequest: approveAll,
sessionLimits: {
maxAiCredits: 30, // soft cap for the current accounting window
},
});
const resumed = await client.resumeSession(session.sessionId, {
onPermissionRequest: approveAll,
sessionLimits: { maxAiCredits: 30 },
});
Four events let you build real budget UX:
| Event | When | Key fields |
|---|---|---|
session.session_limits_changed | Active limits changed (a null value means none are active) | sessionLimits.maxAiCredits? |
session.usage_checkpoint | Runtime records durable aggregate usage for resume and accounting | totalNanoAiu, totalPremiumRequests? |
session_limits_exhausted.requested | Budget exhausted โ needs a user decision before continuing | requestId, maxAiCredits, usedAiCredits |
session_limits_exhausted.completed | The exhausted-limit prompt was resolved | requestId, response.action, response.additionalAiCredits?, response.maxAiCredits? |
session.on((event) => {
if (event.type === "session_limits_exhausted.requested") {
showBudgetDialog({
requestId: event.data.requestId,
maxAiCredits: event.data.maxAiCredits,
usedAiCredits: event.data.usedAiCredits,
});
}
});
Workflow budget โ the multi-agent guardrail
Dynamic workflows carry their own limits, overridable per invocation, and this is the layer that actually contains a fleet:
| Limit | Semantics |
|---|---|
maxConcurrentSubagents | Concurrency cap. Extra subagents wait in a queue โ backpressure, not failure. |
maxTotalSubagents | Cumulative admission cap. Beyond it the attempt fails with kind maxTotalSubagents. |
timeoutSeconds | Accumulated active-execution time across attempts (extension body, subprocess waits, queued-agent waits, sleeps). Time between attempts is excluded. Soft โ running work may take time to stop. |
maxAiCredits | Budget for the whole run's subagent subtree including descendants. Soft and post-paid; accounting is fail-closed โ an accounting failure stops a budgeted run rather than permitting untracked use. |
maxConcurrentSubagents falls back to maxTotalSubagents, so a declared total cap also bounds concurrency โ but omit both and that dimension is unbounded. Conversely, the docs are blunt about guessing: set a ceiling from real knowledge of what the workflow costs or because the user named one. A guessed ceiling does not make a run safer โ it stops a healthy run partway with workflow_limit_reached after that run has already spent credits. SDK-initiated run and resume do not request permission, so a ceiling only exists if you set it.
Limit breaches use reject-and-retry semantics: the attempt ends with run status error and failure.type: workflow_limit_reached, and the run keeps its ID, arguments, journal, and accounting. Resume it with a raised limit when more work is approved โ previously consumed resources still count.
Part 3 โ Evals: six layers, cheapest first
Agent evaluation is hard for one structural reason: non-determinism enters at three layers โ the model's sampling, your tools' side effects, and the loop's control flow (how many turns it takes). Teams that only test at one layer ship regressions at the other two.
The SDK gives you a much stronger foundation than usual, because its own test suite solves the hard part: a local replaying CAPI proxy. E2E runs go against a proxy that replays recorded model exchanges from YAML snapshots, and the harness prints Listening: http://... for tests to parse and configure against. That architecture is directly copyable, and it is the single best idea in this lesson:
| # | Layer | What you assert | Why it ranks here |
|---|---|---|---|
| 1 | Unit โ pure code | Tool handlers, redaction functions, path validators, budget arithmetic | Cheapest, fastest, zero flake. The tool-factory pattern from Lesson 4 exists precisely so handlers are testable without a live session. |
| 2 | Guardrail evals | onPreToolUse denies ../../etc/passwd and /srv/ws-evil; onPostToolUse redacts a planted key; the permission ladder returns the intended kind |
Guardrails are pure functions of the request, so these are deterministic and they test your highest-consequence logic. Encode every past incident as a case. Highest value per line of test code in the whole system. |
| 3 | Contract / schema | Structured output parses against your schema; a tool's declared schema matches what the handler expects | Catches the silent breakage: a model returning plausible prose where a typed object was required. |
| 4 | Trajectory | Tool-call sequence and arguments from tool.execution_start / tool.execution_complete; turn count within budget; forbidden tools never called; no resultType: "failure" streaks |
Tests the process, which is where most agent regressions live. Composes perfectly with snapshot replay โ same input, assert the same trajectory. |
| 5 | Outcome / artifact | The code the agent wrote actually runs; the tests it wrote actually pass; the file it produced validates | The only layer that proves the work is real. Corollary worth carving on the wall: an agent's self-report is not evidence. "Completed successfully" is a claim; a passing test suite is proof. |
| 6 | LLM-judge | Subjective quality โ usefulness, tone, summary faithfulness | Last, and never alone. It is non-deterministic, costs tokens on every CI run, drifts as models change, and needs its own calibration against human labels. |
Running it: the loop that keeps agents honest
- Freeze the world. Pin the SDK and CLI. A version bump can change event shape, default compaction, and prompt assembly โ all of which invalidate prior evals.
- Replay in CI. Reuse the record-and-replay idea so layers 3โ4 are deterministic.
- Treat prompt edits as code changes. A skill file, a system message, a tool description, a
modifiedPrompthook โ each can invalidate behaviour and each should trigger the suite. Tool descriptions are prompt engineering that runs on every call; they deserve review like code. - Grow a regression corpus from production. Every failure becomes a fixture. This is the compounding asset โ replayable incidents, not anecdotes.
- Track operating metrics alongside pass/fail. Turns per task, tokens per task, tool error rate, compaction frequency per session, permission denial rate, and retry counts from
onErrorOccurred/onAgentStop. Most "quality" regressions announce themselves as a metric moving before any test goes red. - Put a budget in CI. Fail the eval run if tokens or AI credits exceed a ceiling for the fixed corpus. Without this, an eval suite is the one place cost grows silently.
Eval anti-patterns
| Anti-pattern | Why it fails |
|---|---|
| Judging with the same model that generated the output | Correlated blind spots read as high quality |
| An LLM judge with no rubric | Unfalsifiable scores that cannot be reproduced or argued with |
| Only testing the happy path | Your guardrails exist for the unhappy path โ that is the part with incidents |
| Trusting the agent's summary of its own work | A self-report. Verify the artifact. |
| One golden transcript for everything | Prompts evolve; snapshots must be regenerated deliberately, not ignored when they differ |
| No cost ceiling on the eval suite | Evals quietly become your largest line item |
๐ง Knowledge Check
1. You want a cost summary after a session completes. Which call, and why not the obvious event?
2. Why is a session's maxAiCredits described as a soft cap?
3. Which eval layer should carry the most weight for an agent that edits code, and why is the cheapest layer so valuable?
Further Reading
- OpenTelemetry instrumentation (config, propagation, skill/process attributes)
- Usage and billing metrics (full API decision guide)
- Session limits and budget events
- Dynamic workflows โ limits, journaling, and observability APIs
- OpenTelemetry GenAI semantic conventions
- GitHub Copilot billing (source of truth for credit conversion)