LLM Observability: The Complete Guide (2026)

Nishtha chauhan
Nishtha chauhan
|Published on |10 Mins
Cover Image for LLM Observability: The Complete Guide (2026)

Your LLM feature has been live for three weeks. A customer says the assistant quoted the wrong refund policy, your support lead asks which prompt produced the answer, and the token bill has jumped with no obvious cause. You have application logs, but they cannot tell you what the model actually received, which tool it called, or which prompt version was running.

LLM observability is the instrumentation layer that records each model call, tool call and retrieval as a span inside a trace. With it, you can reconstruct what your application did for a specific user, at a specific version, at a specific cost. It records what happened. Evaluation judges whether what happened was good. The two are separate systems, and this guide covers how to build the first one properly, in the order the decisions have to be made.

LLM observability rollout checklist

Work through these in order. Each item has a one-line reason, and the steps below expand on the ones that need a decision.

Instrument

  1. Write down your unit of work: one chat turn, one agent task, or one document processed. Make it the trace boundary. Why: cost per request, success rate and latency are all averages over that unit, and changing it later silently invalidates your history.

  2. Create one span per model call, and one per tool call, retrieval and decision point, nested under an operation span. Why: a single-call app yields one span and an agent yields dozens, and you cannot attribute a failure to a step that has no span.

  3. Set gen_ai.operation.name and gen_ai.provider.name on every inference span at creation time. Why: both are required, and the spec says the attributes that matter for sampling should be present when the span starts.

  4. Add the model name, request parameters, gen_ai.response.id, gen_ai.response.finish_reasons, token usage, and error.type when a call fails. Why: these turn "something happened" into "this model, this stop reason". The response ID and finish reasons are recommended, not required, so treat them as sometimes absent.

  5. Set span kind to CLIENT for calls that leave the process, and INTERNAL only for models running in-process. Why: CLIENT marks a network boundary whose latency you can measure, and mislabelling breaks latency attribution.

  6. Attach a user key, session key and app version to the root span. Why: without a join key you cannot answer "what did this customer see", and copying keys onto every child inflates each payload.

  7. Propagate context across every boundary the request crosses. Where continuity cannot be guaranteed, start a new trace or add a span link. Why: a trace that silently restarts at a queue, a retry or an app background looks like two unrelated incidents.

Decide what to keep

  1. Leave content capture off by default, and enable it deliberately per environment. Why: the spec's default is NO_CONTENT, because instructions, inputs and outputs are sensitive and often large.

  2. Choose one of three content patterns and write it down: attributes on the span, external storage with a reference on the span, or correlated event or log records. Why: attributes are queryable but sit on the hot path, while a reference keeps spans small and allows separate access controls.

  3. Decide field by field which values must be queryable and which are payload. Why: a value that lives only in a blob cannot be filtered on, and a value on the span cannot be deleted separately from it.

  4. Redact before the process exports. Why: redaction after ingestion means the sensitive value has already travelled and now sits inside your retention window.

Sample and retain

  1. Choose the retention window from the questions you will need to answer months out, not from the tier you were given. Why: nothing errors when a trace ages out; the question just becomes unanswerable.

  2. Plan retention tiers before you send the first trace. Why: in LangSmith a tier change applies to new traces only, and ingested traces keep the window they arrived with.

  3. Separate metadata retention from payload retention. Why: metadata can outlive raw content, which is often enough for trend questions after inputs and outputs are gone.

  4. Make the sampling decision from attributes available when the span starts, and handle "keep every failure" as a separate path. Why: a decision made at span creation cannot see an outcome that has not happened yet.

  5. Copy traces you intend to keep as examples into a dataset with its own retention. Why: in LangSmith, datasets are retained indefinitely, which is the surest way to know an example still exists next year.

Measure and operate

  1. Attribute cost on a dimension you can act on: per feature, per customer or per model, not per endpoint. Why: an endpoint total tells you the bill, not what to change.

  2. Report inference spend and observability spend as separate line items. Why: model tokens and telemetry are billed on different meters, so one can grow while the other stays flat.

Guard, not a step: check your backend's limits before you need them, including run caps, ingest rate and attribute size. You usually discover them in production. LangSmith, for example, caps a single trace at 25,000 runs and rejects further runs for that trace once the cap is reached (LangSmith observability concepts).

Ebook Preview

Get the Mobile Testing Playbook Used by 800+ QA Teams

Discover 50+ battle-tested strategies to catch critical bugs before production and ship 5-star apps faster.

100% Free. No spam. Unsubscribe anytime.

What LLM observability records

LLM observability borrows the vocabulary of distributed tracing. A span is a timed unit of work, such as one model call. A trace is the set of spans that belong to one operation. The OpenTelemetry GenAI conventions define the attribute names for those spans. The client inference conventions are still marked Development, so names can change between versions (OpenTelemetry GenAI client inference conventions). Pin the convention version you instrument against, and check it when you upgrade.

Vendors use their own vocabulary. LangSmith's docs say that if you know OpenTelemetry, you can think of a run as a span. The table maps the two models as a translation aid for carrying one data model between tools. Two vendors using similar shapes does not make an industry standard.

OpenTelemetry term

LangSmith term

Practical meaning

Span

Run

One unit of work, such as a model call, tool invocation or retrieval

Trace

Trace

All runs for one operation

None

Thread

Traces from a multi-turn session linked together

None

Trajectory

An ordered message list for the session, with nesting flattened

Other tools name these objects differently, so check each vendor's documentation before you map a schema. The 25,000-run cap mentioned above is one vendor's limit, not a general ceiling on agent depth.

Observability versus evaluation

Observability records what happened: traces, latency, token counts, cost and errors. Evaluation scores whether what happened was good. A team can evaluate entirely offline, using a fixed test set in a harness with no production monitoring. A team can also instrument every request and never evaluate.

Production traces are one useful input to evaluation. They give you real inputs in the real distribution of user traffic, which a hand-written test set rarely matches. If you plan to score production outputs, design the sampling and retention steps below with that in mind. To build the test set and judge outputs, see Quash's guide to testing AI features, which covers eval sets, adversarial cases and CI gating.

Step 1 — Pick the unit of work before you pick a tool

Start with the question "what is one thing my app does for a user?" Your answer is the trace boundary. For a chat feature, it is usually one user turn. For an agent, it is one task from start to finish, which may include several model calls and tool calls.

This comes first because every metric you chart is an average over that unit. Change the boundary after six months of data and the old and new numbers no longer describe the same thing. Write the definition in your team's documentation, and name the owner who can change it.

Step 2 — Choose your instrumentation path

There are three common paths, and each trades control for effort:

  • Auto-instrumentation adds spans with little code, but you get the spans the library chooses to emit. Check which attributes it sets before you rely on them.

  • An SDK gives you control over span boundaries, names and attributes, at the cost of writing and maintaining that code.

  • A gateway or proxy sits between your application and the model provider and can record every call in one place without changing application code. It sees the request and response boundary but not the logic around it, so you still need spans for your own tools and retrievals.

Many teams combine two: the gateway for coverage, the SDK for steps only the application knows about.

Step 3 — Decide what goes on the span and what stays out of it

This is the decision most setups skip. The OpenTelemetry GenAI spec treats model instructions, user messages and model outputs as sensitive and often large. It says instrumentation should not capture them by default, and should offer an opt-in (OpenTelemetry GenAI spans specification).

The spec names one configuration variable, OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT, with four values: NO_CONTENT, SPAN_ONLY, EVENT_ONLY and SPAN_AND_EVENT. The default is NO_CONTENT, and an unknown or invalid value falls back to it. The spec lists three patterns for handling content:

  1. Capture nothing. This is the default.

  2. Record content on span attributes. The spec describes this as suitable where the telemetry store meets your privacy requirements, for example in pre-production.

  3. Store content externally and record a reference on the span. The spec recommends this in production where telemetry volume is a concern or sensitive data needs secure handling, because external storage allows separate access controls.

Tool definitions follow a narrower rule. The non-required properties of gen_ai.tool.definitions may contain sensitive information and can be large, so the spec says not to populate them by default. Tool-definition schemas and message content are separate opt-ins.

The 2024 OpenTelemetry tutorial advised recording payload detail as span events, because backends struggle with large span attributes (OpenTelemetry blog, 2024). In March 2026, OpenTelemetry announced the deprecation of the Span Event API and said new code should write events as logs correlated with the current span (OpenTelemetry blog, 2026). The reason for keeping bulk data off span attributes still holds. The API once recommended for it is what changed.

Write your choice down field by field. A workable production default: queryable identifiers and operational fields on the span, bulk content in external storage behind a reference, and redaction before export.

Step 4 — Set sampling and retention against the questions you will ask later

Retention is a design decision because the window you buy decides which questions remain answerable. Two vendors expose it directly.

Langfuse publishes historical data access as a main difference between plans. Its pricing page, observed 19 November 2026 (Langfuse pricing):

Plan

Monthly price

Historical data access

Hobby

Free

30 days

Core

$29

90 days

Pro

$199

3 years

Enterprise

$2,499

3 years

Core, Pro and Enterprise include 100,000 units per month, and extra usage is billed at $8 per 100,000 units at the published defaults. Enterprise also offers custom usage pricing on a yearly commitment; those negotiated terms are not published.

LangSmith on Cloud keeps base-tier traces for 14 days and extended-tier traces for 180 days by default. Enterprise customers can choose 30, 60, 90, 120, 150 or 180 days. Self-hosted and BYOC deployments default to 14 days base and 400 days extended (LangSmith usage and billing, observed 19 November 2026). Two behaviours change the calculation:

  • Traces that receive feedback through the API, are scored by an online evaluator, or match an automation rule move to extended retention, and are billed as a separate, higher line item.

  • When a base-tier trace expires, it becomes inaccessible in the UI and API, and its inputs and outputs are deleted shortly after. Some metadata may be kept longer, which is why monitoring can keep working after the trace itself is gone.

LangSmith's own guidance is to plan retention tiers before you start sending traces, because a tier change applies to new traces only. Traces already ingested keep the tier they arrived with.

The two bill on different meters, units for Langfuse and traces plus seats for LangSmith (LangSmith pricing), so their prices cannot be compared directly. Read them as two examples of one fact: the access window you choose determines which questions you can answer later, and you choose it before any data exists.

For sampling, the spec says attributes that matter for sampling decisions should be present when the span is created, and names gen_ai.operation.name, gen_ai.provider.name, server.address and server.port. A sampler that decides at span start can only use what is already on the span. To keep every failed call, build a separate path, because the outcome is not known at span start.

Step 5 — Attribute cost on a dimension you can act on

Token spend is easy to see and hard to act on until you can split it by feature, customer or model. An endpoint total tells you the bill has grown. A feature total tells you which change to review.

Keep two budgets separate. Model spend is what your provider bills for tokens. Observability spend is what your telemetry vendor bills for units or traces. Track and alert on each one, because a rise in one does not explain a rise in the other.

Step 6 — Score a sample of production traces

Choose a sample of production traces on a fixed schedule, and score them against a rubric your team agreed on. Record the sampling rate beside every quality number. A score over 5% of traces and a score over 100% of traces make different claims, and a reader who sees only the score cannot tell them apart. If a metric covers only sampled traces, say so wherever the number appears.

Step 7 — Instrument the client, not just the server

Some model calls start from a mobile app or a browser, so the parent span lives on the device. The OS can suspend the app, the network can drop, and export can be interrupted.

OpenTelemetry publishes guidance for client applications, including offline buffering and batched export for unreliable networks (OpenTelemetry client-side apps). Client-side spans are ordinary OpenTelemetry spans, and the GenAI client conventions anticipate callers outside the server.

Two choices need your decision. First, how context crosses the boundary between device and server, so a backend call joins the user's trace. Second, what happens to a trace when the app is backgrounded or killed mid-request. Some teams keep one trace across the gap. Others close the device-side span, start a new trace on the server, and link the two. Pick one, then test it by interrupting a request on purpose and checking that you can still join both halves.

What the tools still do not trace

A November 2024 systematic mapping study by researchers at Data61, CSIRO, examined 17 agent observability tools found through GitHub and Google searches (Dong, Lu and Zhu, arXiv:2411.05285v2). Its principal finding: most existing DevOps tools for agents focus on LLM-specific metrics and prompt management, with limited support for observing agent artifacts such as goals, plans and tools.

Its taxonomy works as a checklist of what to consider instrumenting: the user goal, reasoning, plan, workflow and tasks, knowledge retrieved, tools invoked, evaluation and guardrails, and the LLM calls. The study is a preprint describing 17 selected tools as of late 2024. It says nothing about what any named product traces today.

Five ways an observability setup quietly fails

  1. No written unit of work. Each dashboard averages over a different unit, and the numbers disagree without anyone noticing.

  2. Content capture enabled everywhere. The spec's default is no content, and its production pattern is external storage with references. Turning capture on across all environments reverses both.

  3. A retention window shorter than your incident cycle. Traces expire and nothing errors. The question you need to answer simply stops being answerable.

  4. Traces with no user or session key. Without a join key, the trace that explains a customer ticket cannot be found.

  5. Quality numbers with no sampling rate attached. A score over an unrepresentative sample gets believed. State the rate and coverage every time the number appears.

Conclusion

Start with the unit of work and the retention window, because those two choices are the hardest to change later. Then decide what content leaves the process, add cost attribution by feature and customer, and sample production traces for evaluation.

Your first decision this week is the one with the longest tail: how long you need to be able to answer questions about a single request. Write that number down before you turn on tracing.

FAQ

Is LLM observability the same as AI observability?

The terms overlap heavily and are often used interchangeably. Check the scope of any specific product page, since some use "AI observability" to include evaluation and other functions.

Do I need LLM observability if I make only one LLM call per request?

You need spans, but not many. A single-call app produces one inference span per request. That span still records the model, the stop reason, token usage and any error, which is usually enough to start.

Does OpenTelemetry work for LLM observability?

Yes. The OpenTelemetry GenAI conventions define attributes for model calls and content capture. They are still in Development, so pin a version and check attribute names when you upgrade.

How long should I keep traces?

Long enough to answer the questions you will ask after an incident or a quality review. LangSmith's Cloud default is 14 days base and 180 days extended, and Langfuse's Core plan gives 90 days of historical access. Choose the window from your own questions, then confirm the plan that provides it.

What is the cheapest useful setup?

One span per model call with the operation name, provider, model, token usage and error type. Attach a user key and app version to the root span, keep content capture off, and set a retention window you can afford. Add tool and retrieval spans when you run an agent.

Is observability the same as monitoring?

They overlap. Monitoring usually means alerting on aggregate signals such as error rate or latency. Observability means recording enough per-request detail to explain an unusual result you have not seen before. Monitoring tells you something changed, and traces help you find out what.