An AI endpoint can return HTTP 200 while giving an unsupported answer, calling the wrong tool, consuming ten times the usual context, or quietly failing for one language. Conventional service metrics still matter, but they see only the transport shell. Operators also need to reconstruct which evidence, prompt, model, validation, and feedback produced the user-visible outcome.
The thesis of AI observability is: record the model system as a chain of versioned decisions and evidence, then connect operational signals to task outcomes without turning telemetry into a sensitive-data archive. Observability does not prove answer quality and does not replace controlled evaluation. It provides the production evidence needed to detect change, limit incidents, explain cost, and decide what deserves deeper review.
This chapter follows a warranty-policy assistant that retrieves approved documents and answers customer questions. Its operational challenge is not merely keeping a model API available. It must notice stale retrieval, unsupported claims, changing traffic, feedback bias, and spend that rises even when request counts do not.
Begin with Operational Questions and Stable Events
Instrumentation should answer decisions an operator can make. Start with questions such as: Which deployed configuration produced this answer? Did retrieval return authorized and current sources? Where did latency accumulate? Why did token use rise? Which user segment is seeing more abstentions? Can affected requests be found without searching raw prompts?
Define a stable event vocabulary around the workflow, not a provider’s response object. Useful events include request accepted, policy classified, retrieval completed, model attempt completed, output validated, tool proposal accepted or denied, response delivered, feedback received, and delayed outcome attached. Provider-specific fields can live under an extension, while core names survive a model or vendor change.
Every request needs a trace identifier and a deployable configuration identifier. The latter should resolve to model and provider version, prompt template, retriever and index snapshot, tool catalog, validation policy, routing rules, and feature flags. A model name alone is insufficient: the same model with a new chunking strategy is a different operational system.
Use domain identifiers carefully. A case identifier may be necessary for authorized investigation, but email addresses and raw account numbers do not belong in metric labels. High-cardinality identifiers make poor metrics and create privacy risk. Keep aggregate dimensions such as workflow, locale, risk tier, result kind, model route, and coarse tenant class in metrics; retain request-level links in access-controlled traces with limited retention.
Separate observed facts from inferred labels. schema_valid=false is deterministic. answer_may_be_unsupported=true is a classifier result with a version and threshold. customer_satisfied=false may be explicit feedback, an inferred outcome, or a support-agent disposition. The telemetry schema should preserve that distinction so operators do not compare unlike signals.
Trace the Model-Aware Execution Path
A trace should expose stages that have different owners and failure modes. One giant generate span hides queueing, retrieval, prompt construction, provider latency, streaming, validation, and tools. At the other extreme, recording every token creates noise and cost. Instrument boundaries where a diagnosis or policy decision can change.
The retrieval span should record query strategy, authorization filter outcome, index and embedding version, candidate and selected counts, source identifiers, source freshness, and scores where they are comparable. Avoid copying full passages by default. A hash and stable source revision often support replay while keeping document content in its governed store.
Prompt assembly should record template version, message-role counts, included source IDs, truncation decisions, estimated or provider-reported input units, and policy flags. If prompts are retained for a sampled debugging tier, redact or tokenize sensitive fields before storage and apply stricter access and expiration. Hashing a low-entropy secret such as a phone number does not make it anonymous; an attacker can enumerate likely values.
For each model attempt, capture route, model version, region, queue time when available, time to first token, generation duration, input and output units, cache reads or writes, finish reason, retry classification, and provider request identifier. Record attempts separately from the logical request. Otherwise a successful retry conceals the latency and cost of the failed call.
Validation and tools need their own spans. Store schema errors, citation membership results, safety-classifier version, denial reason, canonical tool name, approval state, and idempotency outcome. Do not serialize sensitive tool arguments into ordinary span attributes. Link to a protected audit record when an incident requires exact values.
Attribute Latency and Cost to Useful Outcomes
Token totals are accounting inputs, not product outcomes. A workflow can use fewer tokens per call while spending more per successful case because it retries, escalates, or invokes an expensive fallback. Attribute cost at both attempt and logical-task levels.
For one request with attempts , a basic estimated model cost is
Rates must be versioned by provider, model, region, cache class, and effective date. Mark estimates as estimates until reconciled with billing exports. Add retrieval, reranking, vector storage, moderation, tool, and human-review costs when comparing workflows. Currency and billing units should be explicit.
Useful derived views include cost per delivered answer, cost per schema-valid extraction, cost per resolved case, retry amplification, fallback share, and cost by outcome. Do not optimize cost per request in isolation: routing difficult cases to humans can reduce model spend and increase total handling cost, while a longer grounded answer may prevent a repeat contact.
Latency needs the same decomposition. Track policy time, retrieval, queueing, prefill, time to first token, decode, validation, tool execution, and total user-visible duration. Streaming changes perception but not completion cost. Record when bytes become visible and whether a later validation failure invalidates already-streamed content.
Budgets should be attached to workflow and risk tier. An autocomplete path cannot tolerate the same delay as document analysis; a high-impact policy answer may justify extra validation. Alert on structural changes such as input units per request rising, retries multiplying, or cache-hit behavior changing, even before total spend crosses a monthly threshold. Those ratios often reveal the mechanism earlier than an invoice.
Measure Quality Without Treating Proxies as Truth
Production quality is partly latent. Most users do not rate an answer, and a thumbs-up does not establish factual correctness. Use a portfolio of signals, each labeled by source and limitation.
Deterministic online checks include schema validity, citation source membership, tool argument validity, forbidden-field disclosure, exact extraction fields, and whether an answer abstained when required data was absent. Model-based monitors can estimate groundedness, relevance, policy adherence, or contradiction, but they are themselves models with versions, thresholds, cost, and drift. Calibrate them against reviewed examples and monitor disagreement.
Behavioral signals include reformulation, immediate abandonment, repeated contact, manual correction, tool reversal, escalation, and task completion. Their meaning depends on the product. A user may reformulate because the interface was confusing, not because the answer was false. A completed transaction can still follow poor advice. Join these signals as evidence rather than declaring one a universal reward.
Track numerator and denominator. “Unsupported-answer alerts doubled” means little if traffic grew tenfold or sampling changed. Report eligible requests, sampled requests, monitor coverage, unknown outcomes, and delayed-label maturity. Slice by task, locale, input length, retrieval result, model route, risk tier, and relevant product dimensions. Global averages conceal regressions in small but important populations.
Do not continuously tune against the same online monitor until its score improves. That creates a feedback loop in which the system learns the monitor’s shortcuts. Operational signals identify candidates for investigation. Independent review and controlled evaluation determine whether a proposed change actually repairs the behavior.
Detect Drift by Locating What Changed
“Drift” is not one metric. Separate at least five mechanisms:
| Drift type | What changed | Example signal |
|---|---|---|
| Input drift | User population or request shape | Language, length, intent, attachment mix |
| Corpus drift | Retrieved knowledge or its indexing | Source freshness, missing documents, score distributions |
| Configuration drift | Prompt, model, router, policy, or tool set | Configuration IDs and traffic allocation |
| Behavior drift | Outputs under comparable inputs | Abstention, citation, tool, or monitor distributions |
| Outcome drift | Relationship between behavior and real results | Corrections, escalations, repeat contacts |
Compare like with like. A rise in output length after launching a long-document workflow is not evidence that an existing route drifted. Use stable slices, traffic-weighted and slice-specific baselines, and deployment annotations. Seasonal changes and product launches may be expected input drift while still requiring capacity or quality adjustments.
For categorical distributions, simple proportion changes may be interpretable. For continuous features, quantiles, histograms, population stability measures, or two-sample tests can help. Statistical significance is not operational importance: large traffic makes tiny changes significant, while rare critical failures may never produce a strong aggregate test. Set thresholds from impact, expected variance, sample size, and the action an alert triggers.
Content and embedding drift deserve caution. A distance distribution can change because an embedding model changed, a source collection changed, or users asked new questions. Preserve index and embedding versions and monitor retrieval outcomes such as empty result, stale source, low score separation, and source concentration. Diagnose the stage before blaming the generator.
Use fixed, privacy-approved probes to distinguish system change from traffic change. Periodically run a small set of operational canaries through retrieval and generation, recording outputs in a controlled environment. These probes are not a complete quality gate; they answer whether a stable input behaves differently and help localize an incident.
Work Through a Warranty-Assistant Incident
The warranty assistant answers questions from approved policy documents. One morning, the unsupported-claim monitor rises for appliance questions, input-token use increases, and user reformulations increase. Provider availability and HTTP latency remain normal.
An operator starts from affected trace IDs and groups them by configuration, locale, product family, and retrieval index. All affected requests use the same model and prompt as the prior week, but the retrieval index revision changed overnight. Retrieval spans show that appliance queries now return several near-duplicate installation manuals ahead of the warranty policy. Prompt assembly reaches its context cap and truncates the lower-ranked policy passage.
One trace for “Does replacing this filter void coverage?” contains manual pages describing replacement steps but no warranty terms. The model gives a confident answer and cites the manual. Citation membership passes because the source was retrieved; a sampled support check flags that the passage does not support the warranty claim. The important diagnosis is retrieval composition and truncation, not generic model degradation.
Containment disables the new index revision for the appliance route and forces abstention when no policy-class source survives selection. Operators identify requests processed by the bad revision through configuration and source IDs, then route a privacy-minimized sample for review. They do not search all raw conversations.
The root cause is a metadata migration that assigned installation manuals the same document class and priority as policies. The repair restores type constraints, deduplicates manual revisions, and adds a retrieval invariant: warranty answers require at least one current policy source. Monitoring gains source-class concentration and “required evidence absent” metrics.
This example shows why a single answer-quality alert is insufficient. The cost increase, truncation signal, index lineage, and source mix reveal the mechanism. It also shows why citations are not self-validating: observability must distinguish “cited an allowed source” from “the cited passage supports the claim.”
Build Privacy-Safe Feedback Loops
Feedback should have a defined purpose and provenance. Explicit ratings can include a reason taxonomy such as incorrect, unsupported, irrelevant, unsafe, or too slow. Free-text feedback may contain new personal data and needs validation, retention, and access controls. Do not automatically append user comments to prompts or training corpora.
Implicit feedback requires interpretation. Link reformulation, correction, escalation, and downstream outcome through short-lived or pseudonymous identifiers where possible. Record the observation window and eligibility rules. Users who leave immediately create missing outcomes, not positive outcomes. Delayed outcomes should not be joined until their window matures.
Sampling determines whose experience becomes visible. Uniform random sampling is simple but may miss rare high-risk routes. Combine a random baseline with stratified samples for critical tasks, anomaly-triggered traces, and a small reviewed sample of apparent successes. Weight analyses when sampling probabilities differ. Keep a holdback from feedback-driven tuning so the monitoring process does not erase independent evidence.
Human reviewers need the minimum context required: user request after approved redaction, selected evidence, response, validation results, and relevant outcome. Capture rubric version, reviewer decision, uncertainty, and disagreement. Reviewer identity and access belong in an audit system, not broad metrics. Feedback exported for improvement needs a separate consent, licensing, and deletion analysis.
Close the loop deliberately. Triage signals into product defects, retrieval defects, policy gaps, data gaps, abuse, and monitor errors. A monitor false positive is itself useful calibration data. Promote reviewed incidents into operational queries and controlled test cases, but do not let every thumbs-down immediately alter a prompt or model. Fast uncontrolled learning lets attackers and noisy users steer production behavior.
Sample, Alert, and Respond to Incidents
Collect aggregate counters and histograms broadly; sample detailed traces according to risk and diagnostic value. Tail-based sampling can retain slow, failed, denied, or high-cost requests after their outcome is known. Keep a small random sample of ordinary successes so incidents have a baseline. Sampling rules and probabilities must be versioned because changing them changes observed rates.
Alerts should map to a runbook and likely owner. Examples include schema-valid rate dropping, required evidence missing, tool denials spiking, p95 time to first token exceeding a workflow budget, retry amplification increasing, cost per resolved task changing, or a drift signal crossing a sustained threshold. Combine symptoms when possible to reduce noise, and include configuration and slice dimensions in the alert.
During an incident, first contain impact: disable a tool, roll back an index, pin a model route, require human review, reduce context, or return a deterministic fallback. Preserve trace, configuration, and source lineage. Determine the affected population by versions and decision paths, not keyword searches over sensitive text. Reconcile attempted and completed side effects separately.
After recovery, verify that signals return for the repaired mechanism and that the containment path actually worked. Add missing telemetry only when it answers a concrete future question; recording everything creates expense and risk without guaranteeing insight. Review retention, access logs, sampling bias, monitor calibration, and cost attribution as the system evolves.
AI observability is successful when an operator can move from “quality seems worse” to a bounded, evidence-backed explanation: which population changed, which configuration handled it, what sources and tools participated, what it cost, and which outcomes may be affected. That production discipline complements offline evaluation without collapsing into it. Traces explain operation, drift signals reveal change, and governed feedback connects model behavior to the world it is meant to serve.