AI agent observability: what traces and evals show, and what only a customer sees

AI agent observability records the prompt, each model call and each tool call an AI agent makes, with its cost and speed, so a team can see why an answer went wrong. Langfuse, LangSmith, Arize Phoenix, Datadog, Helicone and Braintrust collect these traces, and most also score the answers. To see what reaches the customer after the run, use Obsession: its AI customers use your live agent as customers do, keep each answer and wait for the email or callback the agent promised.

An AI agent is software built on a large language model (LLM), the type of AI model behind ChatGPT and Claude, that answers people and acts for them. It might look up an order, issue a refund or book a slot, calling the model several times and using several tools along the way. The same question can take a different path from one run to the next, which is why teams record each run.

AI agent observability is that record, kept for every run: the prompt, each model call, each tool call and the answer. Engineering and AI teams use it to debug failures and control cost, and CX teams use it to review what the agent told customers. The tools differ in where they run, how they charge and whether they are open source, and all of them watch the agent from the inside.

What AI agent observability records

A trace records one run. It is made of spans, one for each step: the request that came in, each call to the model, each tool the agent used and the answer it gave. Each span carries its start time, how long it took and details such as the model, the tokens it used and any error. OpenTelemetry, an open source standard for traces, metrics and logs, gives those details the same names in every tool that accepts it.

AI agent observability and evaluation

An eval is a scored test of what an agent produced. An AI model acting as a judge, a rule written in code or a person marks each answer. It might check whether the answer matched the refund policy, called the right tool or stayed on topic.

• Offline evals run a fixed set of test cases before a change ships, such as a new prompt or model, and compare the scores with the last version.
• Online evals score live runs as they arrive, so a drop in quality after a change shows on a dashboard and can raise an alert.
• Human review sends a sample of runs, or the ones that scored low, to a person, whose labels become new test cases.

Most of these tools now do both and attach each score to its trace, so a low score opens the run that earned it. Evals before launch with simulated users are compared in AI agent testing tools, and voice agents in AI voice agent testing.

OpenTelemetry for AI agents

Most of the tools here accept OpenTelemetry traces. OpenTelemetry’s GenAI semantic conventions name the parts of AI work, so a trace from one framework reads the same in every tool that accepts it. They cover model calls, agent runs, tool calls and embeddings, as spans, events and metrics, and are marked as in development.

OpenTelemetry’s blog post on GenAI observability of 14 May 2026 shows how such a trace looks. An agent run is one invoke_agent span, with a chat span for each model call and an execute_tool span for each tool. Attributes such as gen_ai.request.model and the input and output token counts sit on each model call. Prompt text and tool arguments are left out by default, because they can hold sensitive data, and a team opts in to record them.

Datadog, Langfuse, Arize Phoenix and LangSmith all take OpenTelemetry traces, so instrumenting in the standard keeps more than one tool open to a team. Each accepts particular versions of it, so a team checks the docs before switching.

AI agent observability tools

All 6 record what an agent sends to its models, and most score the results. They differ in licence, hosting and how they charge. Each is described from its own pages as they stood on 1 November 2026, with prices in US dollars as listed that day, in alphabetical order.

Arize Phoenix

• What it does well: Phoenix is Arize’s tool to trace, evaluate and improve AI agents. It builds datasets from traces, runs experiments on them and has a prompt editor, and its page lists native OpenTelemetry support.
• Price as listed: Arize calls Phoenix open source. Its licence, the Elastic License 2.0, lets a team host it on its own infrastructure but bars offering it to others as a hosted service. Hosted by Arize, it runs on Arize AX. AX Free holds 25,000 spans a month for 15 days, and AX Pro, at $50 a month, holds 50,000 for 30 days. Enterprise is custom.
• Where it stops: self-hosted, a team runs and scales it on its own servers. Hosted, AX Free keeps data for 15 days.
• Best for: teams that want tracing and evals they host themselves, built on OpenTelemetry, with a hosted platform to move to.

Braintrust

• What it does well: Braintrust calls itself the active observability platform for agents: it shows prompts, responses and tool calls in real time and scores outputs with AI models, code or people. It also runs experiments against real datasets and tracks latency, cost and quality. Its Loop agent suggests how to improve an agent.
• Price as listed: Starter is $0 a month with 1 GB of processed data, 10,000 scores and 14 days of retention, for unlimited users. Pro is $249 a month with 5 GB, 50,000 scores and 30 days. Enterprise is custom, and qualifying startups can get 6 to 12 months of Pro free.
• Where it stops: usage beyond a plan is billed per GB of data and per 1,000 scores, and Starter keeps data for 14 days.
• Best for: teams that run evals and production traces in one place, with an AI agent of its own to suggest fixes.

Datadog LLM Observability

• What it does well: Datadog’s docs now call it Agent Observability, for monitoring and improving LLM applications and AI agents. Each request is a trace, with ready-made dashboards for cost, latency, performance and usage. It groups live traffic by topic, scans and redacts sensitive data, and flags prompt injections, messages written to take over an agent. It accepts traces that follow OpenTelemetry’s GenAI conventions from version 1.37.
• Price as listed: the free tier covers up to 40,000 LLM spans a month, kept 15 days, without payment details. Pro starts at $160 a month on an annual commitment, $200 month to month or $240 on demand, with 100,000 LLM spans included. After that it is $3.50 per 10,000 on annual billing. An LLM span is one call to a model provider such as OpenAI or Anthropic, and Datadog bills on those and never on tool or retrieval spans.
• Where it stops: retention past 15 days is an add-on, from $6.50 per 10,000 LLM spans a month for 30 days to $9 for 90.
• Best for: teams already on Datadog that want agent traces in the same place as the rest of their monitoring.

Helicone

• What it does well: Helicone is an AI gateway and observability platform: an app sends its model requests through Helicone, which logs each one. Its dashboard lists requests, sessions, users, prompts, datasets, a playground, rate limits and alerts. It is open source under the Apache License 2.0, and its docs describe self-hosting with Docker Compose or Helm on Kubernetes.
• Price as listed: Hobby is free with 10,000 requests, 1 GB of storage, 1 seat and 7 days of retention. Pro is $79 a month plus usage, with unlimited seats and 1 month of retention. Team is $799 a month plus usage, with 5 organisations, SOC 2 and HIPAA compliance and 3 months. Both paid plans have a 7-day trial, and Enterprise is priced on request.
• Where it stops: Mintlify announced on 3 March 2026 that it was acquiring Helicone, which would keep running in maintenance mode with security updates, bug fixes and new models. On 1 November 2026, Helicone’s own site says it is no longer part of Mintlify, with details to come.
• Best for: teams that want every model request logged through one gateway, under an open source licence they can host.

Langfuse

• What it does well: Langfuse is an open source platform for agent evaluation and observability. Its traces capture every model call, tool call and document lookup, each scored by an AI model, a rule in code or a person. It manages prompts with one-click deploys and rollbacks, it is OpenTelemetry native, and every product feature is under the MIT licence and free to self-host.
• Price as listed: Hobby is free with 50,000 units a month, 30 days of data and 2 users. Core is $29 a month with 100,000 units and 90 days, and Pro is $199 with 3 years. Enterprise is $2,499 a month. Past a plan’s allowance, units cost $8 per 100,000, lower with volume. A unit is any data point sent: the trace, each step and each score.
• Where it stops: Hobby keeps 30 days of data for 2 users. ClickHouse acquired Langfuse on 16 January 2026, and Langfuse said it stays open source and self-hostable, with no planned licence changes.
• Best for: teams that want open source tracing, evals and prompt management they can host themselves.

LangSmith

• What it does well: LangSmith, from LangChain, traces what an agent does step by step. It works with any framework and plugs into existing OpenTelemetry pipelines. Dashboards track token use, latency, error rates, cost and feedback scores, with alerts to PagerDuty or another system. It runs online evals judged by an AI model or code, and groups traces into usage patterns and failure types.
• Price as listed: Developer is $0 a seat a month with up to 5,000 base traces, and Plus is $39 a seat a month with up to 10,000. Traces past those cost $0.005 each, or $5 per 1,000, and Enterprise is custom.
• Where it stops: base traces are kept 14 days, and keeping them 180 days costs extra. Self-hosting is an Enterprise option.
• Best for: teams building with LangChain or LangGraph, or any framework, that want traces, dashboards and online evals together.

Open source AI agent observability

Three of the tools can run on a team’s own servers from code their makers publish, which keeps prompts and customer data inside the company. Their licences differ, which matters to a team that plans to build on the code or offer it as a service. The Elastic License 2.0 that covers Phoenix, for example, bars offering it to others as a hosted service.

LangSmith offers self-hosted and hybrid deployment on its Enterprise plan. Self-hosting swaps a usage bill for the cost of running the servers and the database behind them.

What AI agent observability costs at volume

The tools charge by different units: seats, traces, data processed, data points or model calls. The same traffic costs different amounts in each, so a team prices its own volume on each pricing page.

Datadog charges by the model call, so its bill is the simplest to work out. Its listed annual rate is $160 a month with 100,000 LLM spans, then $3.50 per 10,000. At that rate, an agent platform making 100 million model calls a month would pay about $421,500 a year.

Langfuse counts every data point, the trace, each step and each score, so a run with many steps uses more units than a run with few. Braintrust charges by data processed and by scores, and LangSmith by seat and by trace.

What only a customer sees

A trace ends when the run ends. What the customer gets next often comes from other systems: the confirmation email from the help desk, the refund from the payment provider, the callback from the phone team. If those systems send nothing to the agent’s trace, a run can score well while the customer gets nothing.

• What arrives after the answer. The email, refund or callback the agent promised, often hours later, from a system with its own logs.
• Every channel at once. The same question on chat, email and phone can get different answers from different systems, each with its own trace or none.
• After every update. A new model, a prompt edit or a new help article can change an answer overnight, and the same question asked every morning shows the day it changed.
• Agents your team doesn’t run. A vendor’s support agent or a partner’s booking line is traced in its maker’s systems, where the company that buys it may never see the trace, and its customers meet it anyway.

Customer-side observability covers that side. Customer-side observability is seeing what a company does to its customers by being one of them: every email, text, price, page, checkout and AI answer it sends, kept over time with evidence anyone can check.

AI customers carry it out. An AI customer is an AI agent that becomes a real customer of a company, with its own inbox, phone number and browser, and keeps what the company sends and shows it as evidence. The term is explained in full in what AI customers are.

Obsession makes AI customers that sign up to any company and keep every email, offer and price it sends, so you can find your revenue leaks, watch your competitors and prospect with proof. The team behind Obsession has mystery shopped 1,500 ecommerce stores as a new customer since September 2026.

For a team running an AI agent, Obsession’s AI customers call, chat and email the agent as customers do, every day and after every update. They ask what customers ask and check each answer against the company’s policy. Then they ask for a person and wait in their own inbox for the email or callback the agent promised. Every AI customer says it’s an AI when it talks to a person and never uses fake payment details, as what every AI customer will and won’t do sets out.

• Support bot check asks your support bot what your customers ask, on every channel, every day, and checks each answer against your policy.
• Drift watch replays your hardest cases on your AI agent every morning and after every update, and flags every answer that changed.
• Resolution check checks every resolution your AI support vendor bills you for against what the customer got.

When a check fails, it shows what the customer was told, and its time and channel let the team find the matching trace to see why. For CX teams, AI agent QA works out what a wrong answer costs in a team’s own numbers.

AI agent observability tools compared

Obsession makes AI customers, so every other option is described from its provider’s own pages as of 1 November 2026.

Which to choose

If your team needs to know what customers were told on its live chat, email and phone, and whether the email or callback the agent promised arrived, use Obsession. Its AI customers check every day and after every update, and keep each answer with its time and channel.

• Open source and self-hosted: Langfuse for MIT-licensed tracing, evals and prompts, or Arize Phoenix on OpenTelemetry.
• A team already on Datadog: Datadog LLM Observability, billed per model call.
• Building with LangChain or LangGraph: LangSmith.
• Evals and production traces in one place: Braintrust.
• Every model request logged through a gateway: Helicone.

A team running an agent at volume needs both views: one of these tools inside the agent, and a check from the customer’s side on the live channels. A failed check’s time and channel then help the team find the trace that explains it.

For developers, the same AI customers are a tool their own agents can call. MCP, the Model Context Protocol, is an open source standard for connecting AI applications to outside systems. Obsession offers its AI customers over MCP, over an API that code calls directly, and through an SDK, a ready-made code library. All three expose 5 verbs: lookup, facts, evidence, become_customer and notify, and API keys go to teams one at a time. AI customers for developers covers building on them, and long running AI agents covers agents that wait for weeks.

How to set up AI agent observability

A team can set it up in 6 steps, each with a pass mark written down before the first run.

1. Trace every run. Instrument each model call and tool call, with OpenTelemetry’s GenAI conventions or a tool’s own SDK. A pass: every run produces one trace that holds every model call and tool call.
2. Group runs into conversations. Pass a conversation ID with each run, which OpenTelemetry names gen_ai.conversation.id. A pass: one customer’s whole chat opens as one thread.
3. Write evals from your policies. Turn the refund window, the handover rule and the tone of voice into test cases, and run them before every prompt or model change. A pass: a change that breaks a passing case doesn’t ship.
4. Score live runs and set alerts. Score a share of live traffic online, with alerts on cost, latency and eval scores. A pass: the owner hears within the hour when a score crosses its line.
5. Check the customer’s side. Every day and after every update, ask the live chat, inbox and phone line what customers ask, then wait for what the agent promised. A pass: each answer matches that morning’s policy, and each promised email arrives within the time the agent gave.
6. Join the two views. Match each failed customer-side check to its trace by time and channel. A pass: every failure points to the run that caused it.

Questions about AI agent observability