Observability, Cost Control, and Evals

Lesson 9: Observability, Cost Control, and Evals

Three questions decide whether an agent survives contact with production: what is it doing, what is it costing, and how do I know it still works. The SDK answers all three with real APIs โ€” and, unusually for an agent framework, it has a genuine answer for the third that is not "ask a model to grade it".

Part 1 โ€” Telemetry

Opt in by passing a TelemetryConfig on the client. That is all most applications ever need: the CLI exports its own spans to your OTLP endpoint.

import { CopilotClient } from "@github/copilot-sdk";

const client = new CopilotClient({
  telemetry: {
    otlpEndpoint: "http://localhost:4318",
  },
});
OptionPurpose
otlpEndpointOTLP HTTP endpoint URL
otlpProtocol"http/json" or "http/protobuf" for all signals
exporterType"otlp-http" or "file"
filePathPath for JSON-lines trace output (dev/offline diagnostics)
sourceNameInstrumentation scope name
captureContentWhether to capture message content
captureContent is a privacy switch, not a verbosity switch. Several attributes require it โ€” including gen_ai.skill.description, gen_ai.skill.source.uri, and process.executable.path. With content capture on, prompts and tool payloads can land in your tracing backend. Decide deliberately, and treat your trace store with the same sensitivity as your logs. Note also that while managed telemetry settings are unresolved, pending tracking buffers only capture-independent facts โ€” descriptions, source URIs, and executable paths observed before the policy resolves are omitted even if capture is later enabled. The final policy governs replay, and a disabled or cancelled tracking session discards the buffer.

Trace context propagation โ€” the advanced path

If your application already creates its own OpenTelemetry spans and wants them in the same distributed trace as the CLI's, Node.js needs an onGetTraceContext callback. The SDK calls it before session.create, session.resume, and session.send:

import { CopilotClient } from "@github/copilot-sdk";
import { propagation, context } from "@opentelemetry/api";

const client = new CopilotClient({
  telemetry: { otlpEndpoint: "http://localhost:4318" },
  onGetTraceContext: () => {
    const carrier: Record<string, string> = {};
    propagation.inject(context.active(), carrier);
    return carrier; // { traceparent: "00-...", tracestate: "..." }
  },
});

Python, Go, and .NET inject automatically when their OpenTelemetry/Activity API is configured โ€” no callback needed. In Node the SDK has no OpenTelemetry dependency, so the inbound direction is raw strings on the tool invocation, which you restore by hand:

import { defineTool } from "@github/copilot-sdk";
import { propagation, context, trace } from "@opentelemetry/api";

const myTool = defineTool("my-tool", {
  description: "Do work",
  handler: async (args, invocation) => {
    // The CLI's span context arrives as raw traceparent/tracestate strings
    const carrier = {
      traceparent: invocation.traceparent,
      tracestate: invocation.tracestate,
    };
    const parentCtx = propagation.extract(context.active(), carrier);

    const tracer = trace.getTracer("my-app");
    return context.with(parentCtx, () =>
      tracer.startActiveSpan("my-tool", async (span) => {
        try {
          return await doWork(args);
        } finally {
          span.end();
        }
      })
    );
  },
});

This is what lets a "handle tool call" span in your app appear nested inside the CLI's "execute tool" span โ€” so a slow request is attributable to your downstream call rather than looking like model latency.

Skill and process attributes

The runtime follows the OpenTelemetry GenAI skill semantic conventions by refining existing execute_tool spans rather than adding a separate skill operation. A loader is named execute_tool skill {skill}; an unambiguously identified resource or script becomes execute_tool {tool} {skill} {resource} with a skill-relative resource name.

AttributeNotes
gen_ai.skill.name, gen_ai.skill.resource.nameAvailable independently of content capture
gen_ai.skill.description, gen_ai.skill.source.uriRequire content capture. Local source values are file URIs, not bare paths.
process.executable.nameThe directly launched executable (e.g. the shell launcher) โ€” not a guessed command token
process.executable.pathAbsolute known path; requires content capture
process.exit.codeInteger including zero when a command-executing tool reports an exit; unavailable exits and pre-spawn failures are omitted

A few semantics worth knowing so you do not misread a dashboard: poll tools do not inherit process attributes from the command they read. Sandboxed executable identities are omitted when the backend does not expose them. And a nonzero exit does not imply the tool itself failed. Preloaded skills produce invocation events without a fabricated tool span; repeated preparation of the same selected-agent skill context emits one receipt, while changed content or provenance emits another, and child sessions report their own preloads independently.

These are breaking renames โ€” update your dashboards. Use gen_ai.skill.name instead of github.copilot.tool.parameters.skill_name or github.copilot.skill.name, and gen_ai.skill.source.uri instead of github.copilot.skill.path. The github.copilot.skill.invoked event and its content, source-category, trigger, and plugin fields remain Copilot-specific. The skill attributes are at Development stability; the process attributes are Release Candidate.

Part 2 โ€” Cost: two mechanisms, six signals

The SDK surfaces usage through events (ephemeral, real-time, per-call) and RPCs (request/response, accumulated). Knowing which to call is the whole game:

SignalAPIUse it for
Per-call token countsassistant.usage eventA live cost or token meter as a turn runs
Context-window utilizationsession.usage_info eventLive context pressure
Context-window breakdownsession.metadata.contextInfo (RPC)Context usage on resume, before any new turn
Accumulated totalssession.usage.getMetrics (RPC)Final cost summary for a turn or session
Per-model pricingmodels.list (RPC)Estimating cost before running work
Account quotaaccount.getQuota (RPC)Warning users before they exhaust their plan
// Real-time, per model API call โ€” fires once per call, INCLUDING sub-agent calls
session.on("assistant.usage", (event) => {
  const { model, inputTokens, outputTokens, cost } = event.data;
  console.log(`${model}: in=${inputTokens ?? 0} out=${outputTokens ?? 0} cost=${cost ?? 0}`);
});

// Accumulated session totals โ€” the cleanest way to read AI credit cost
const metrics = await session.rpc.usage.getMetrics();

const aiCredits = (metrics.totalNanoAiu ?? 0) / 1e9;
console.log(`AI credits used: ${aiCredits.toFixed(6)}`);
console.log(`Premium requests: ${metrics.totalPremiumRequestCost}`);

for (const [model, m] of Object.entries(metrics.modelMetrics)) {
  if (!m) continue;
  console.log(
    `${model}: in=${m.usage.inputTokens} out=${m.usage.outputTokens} nanoAiu=${m.totalNanoAiu ?? 0}`,
  );
}

Fields on assistant.usage are worth a long look โ€” this single event is the most information-dense thing the runtime emits:

FieldWhat it gives you
inputTokens, outputTokensVolume
reasoningTokensOutput tokens spent on chain-of-thought โ€” a subset of output
cacheReadTokens, cacheWriteTokens, cacheExpiresAtPrompt-cache effectiveness and expiry โ€” the fastest way to explain a cost change
duration, timeToFirstTokenMs, interTokenLatencyMsLatency decomposition: queueing/dispatch vs generation throughput
model, reasoningEffortWhich model and effort actually served the call
apiEndpoint"/chat/completions" ยท "/v1/messages" ยท "/responses" ยท "ws:/responses" โ€” which API shape served the turn
initiatorWhat triggered the call โ€” e.g. "sub-agent"; absent for user-initiated
apiCallId, serviceRequestId, providerCallIdProvider completion ID, Copilot service request ID, GitHub request ID โ€” your correlation handles for a support ticket
finishReason, contentFilterTriggeredDid the response stop naturally, hit length, or get filtered?
quotaSnapshots, copilotUsagePer-quota resource usage and an itemised token cost breakdown
Two accuracy traps in cost reporting. (1) Cost is in nano-AI units (totalNanoAiu). The docs divide by 1e9 as an SI convenience but say explicitly that the conversion to AI credits and the meaning of premium-request accounting are defined by GitHub Copilot billing, not the SDK โ€” verify against GitHub's billing docs before surfacing anything currency-like to a user. (2) assistant.usage is ephemeral: delivered live, not replayed on resume. If you want totals after the fact, call session.usage.getMetrics; if you want history, you must persist the events yourself.
Token angle โ€” the metric that predicts your bill best is cache behaviour + turn count. Watch cacheReadTokens against inputTokens: a healthy agent reuses prompt cache heavily across turns. A session where cache reads collapse is one where something is invalidating the prefix every turn โ€” usually a hook that rewrites the prompt (modifiedPrompt, modifiedTransformedPrompt) or a tool returning volatile content early in context. And remember turns are API calls: cost spikes are almost always more round-trips, not a pricier model.

Budgets: making runaway agents impossible

Two independent budget layers exist, and you want both.

Session budget

const session = await client.createSession({
    onPermissionRequest: approveAll,
    sessionLimits: {
        maxAiCredits: 30,      // soft cap for the current accounting window
    },
});

const resumed = await client.resumeSession(session.sessionId, {
    onPermissionRequest: approveAll,
    sessionLimits: { maxAiCredits: 30 },
});
It is a soft cap, and the reason matters. Usage is checked after model calls return โ€” so one response can exceed the configured value before the runtime blocks the next model call. Size it with that slack in mind, and handle the exhausted-budget flow rather than treating it as a hard wall.

Four events let you build real budget UX:

EventWhenKey fields
session.session_limits_changedActive limits changed (a null value means none are active)sessionLimits.maxAiCredits?
session.usage_checkpointRuntime records durable aggregate usage for resume and accountingtotalNanoAiu, totalPremiumRequests?
session_limits_exhausted.requestedBudget exhausted โ€” needs a user decision before continuingrequestId, maxAiCredits, usedAiCredits
session_limits_exhausted.completedThe exhausted-limit prompt was resolvedrequestId, response.action, response.additionalAiCredits?, response.maxAiCredits?
session.on((event) => {
    if (event.type === "session_limits_exhausted.requested") {
        showBudgetDialog({
            requestId: event.data.requestId,
            maxAiCredits: event.data.maxAiCredits,
            usedAiCredits: event.data.usedAiCredits,
        });
    }
});

Workflow budget โ€” the multi-agent guardrail

Dynamic workflows carry their own limits, overridable per invocation, and this is the layer that actually contains a fleet:

LimitSemantics
maxConcurrentSubagentsConcurrency cap. Extra subagents wait in a queue โ€” backpressure, not failure.
maxTotalSubagentsCumulative admission cap. Beyond it the attempt fails with kind maxTotalSubagents.
timeoutSecondsAccumulated active-execution time across attempts (extension body, subprocess waits, queued-agent waits, sleeps). Time between attempts is excluded. Soft โ€” running work may take time to stop.
maxAiCreditsBudget for the whole run's subagent subtree including descendants. Soft and post-paid; accounting is fail-closed โ€” an accounting failure stops a budgeted run rather than permitting untracked use.
An omitted constraint is not a safe default here. An omitted maxConcurrentSubagents falls back to maxTotalSubagents, so a declared total cap also bounds concurrency โ€” but omit both and that dimension is unbounded. Conversely, the docs are blunt about guessing: set a ceiling from real knowledge of what the workflow costs or because the user named one. A guessed ceiling does not make a run safer โ€” it stops a healthy run partway with workflow_limit_reached after that run has already spent credits. SDK-initiated run and resume do not request permission, so a ceiling only exists if you set it.

Limit breaches use reject-and-retry semantics: the attempt ends with run status error and failure.type: workflow_limit_reached, and the run keeps its ID, arguments, journal, and accounting. Resume it with a raised limit when more work is approved โ€” previously consumed resources still count.

Part 3 โ€” Evals: six layers, cheapest first

Agent evaluation is hard for one structural reason: non-determinism enters at three layers โ€” the model's sampling, your tools' side effects, and the loop's control flow (how many turns it takes). Teams that only test at one layer ship regressions at the other two.

The SDK gives you a much stronger foundation than usual, because its own test suite solves the hard part: a local replaying CAPI proxy. E2E runs go against a proxy that replays recorded model exchanges from YAML snapshots, and the harness prints Listening: http://... for tests to parse and configure against. That architecture is directly copyable, and it is the single best idea in this lesson:

Record once, replay forever. Capture your real model exchanges (including tool-call turns) and replay them deterministically in CI. Your agent then executes the same trajectory on every run, so assertions about events, tool calls, hook decisions, and budgets become ordinary deterministic tests. Model responses stop being a source of flake and become a versioned fixture โ€” and you can replay a production failure forever.
#LayerWhat you assertWhy it ranks here
1Unit โ€” pure code Tool handlers, redaction functions, path validators, budget arithmetic Cheapest, fastest, zero flake. The tool-factory pattern from Lesson 4 exists precisely so handlers are testable without a live session.
2Guardrail evals onPreToolUse denies ../../etc/passwd and /srv/ws-evil; onPostToolUse redacts a planted key; the permission ladder returns the intended kind Guardrails are pure functions of the request, so these are deterministic and they test your highest-consequence logic. Encode every past incident as a case. Highest value per line of test code in the whole system.
3Contract / schema Structured output parses against your schema; a tool's declared schema matches what the handler expects Catches the silent breakage: a model returning plausible prose where a typed object was required.
4Trajectory Tool-call sequence and arguments from tool.execution_start / tool.execution_complete; turn count within budget; forbidden tools never called; no resultType: "failure" streaks Tests the process, which is where most agent regressions live. Composes perfectly with snapshot replay โ€” same input, assert the same trajectory.
5Outcome / artifact The code the agent wrote actually runs; the tests it wrote actually pass; the file it produced validates The only layer that proves the work is real. Corollary worth carving on the wall: an agent's self-report is not evidence. "Completed successfully" is a claim; a passing test suite is proof.
6LLM-judge Subjective quality โ€” usefulness, tone, summary faithfulness Last, and never alone. It is non-deterministic, costs tokens on every CI run, drifts as models change, and needs its own calibration against human labels.

Running it: the loop that keeps agents honest

  1. Freeze the world. Pin the SDK and CLI. A version bump can change event shape, default compaction, and prompt assembly โ€” all of which invalidate prior evals.
  2. Replay in CI. Reuse the record-and-replay idea so layers 3โ€“4 are deterministic.
  3. Treat prompt edits as code changes. A skill file, a system message, a tool description, a modifiedPrompt hook โ€” each can invalidate behaviour and each should trigger the suite. Tool descriptions are prompt engineering that runs on every call; they deserve review like code.
  4. Grow a regression corpus from production. Every failure becomes a fixture. This is the compounding asset โ€” replayable incidents, not anecdotes.
  5. Track operating metrics alongside pass/fail. Turns per task, tokens per task, tool error rate, compaction frequency per session, permission denial rate, and retry counts from onErrorOccurred / onAgentStop. Most "quality" regressions announce themselves as a metric moving before any test goes red.
  6. Put a budget in CI. Fail the eval run if tokens or AI credits exceed a ceiling for the fixed corpus. Without this, an eval suite is the one place cost grows silently.

Eval anti-patterns

Anti-patternWhy it fails
Judging with the same model that generated the outputCorrelated blind spots read as high quality
An LLM judge with no rubricUnfalsifiable scores that cannot be reproduced or argued with
Only testing the happy pathYour guardrails exist for the unhappy path โ€” that is the part with incidents
Trusting the agent's summary of its own workA self-report. Verify the artifact.
One golden transcript for everythingPrompts evolve; snapshots must be regenerated deliberately, not ignored when they differ
No cost ceiling on the eval suiteEvals quietly become your largest line item

๐Ÿง  Knowledge Check

1. You want a cost summary after a session completes. Which call, and why not the obvious event?

2. Why is a session's maxAiCredits described as a soft cap?

3. Which eval layer should carry the most weight for an agent that edits code, and why is the cheapest layer so valuable?

Further Reading