The best AI chatbot and AI agent testing tools in 2026

The best AI chatbot testing tools depend on the job: Braintrust and Promptfoo for evals before launch, LangSmith and Langfuse for live monitoring, and Cekura, Coval, Hamming and Roark for voice agents. For checks from the customer’s side, Obsession’s AI customers use your company’s live chat, email and phone as real customers do, every day and after every update. They keep each answer as proof: the transcript or recording, with its time and channel.

AI chatbot testing tools check what an AI chatbot or AI agent tells customers, before release and once it is live. An AI agent here is software that answers customers in writing or by voice using an AI model. It can also act for them, such as starting a refund or booking a slot. Engineers, product managers and CX leads pick a tool by the job they need done, and the tools fall into 5 jobs.

• Evals before launch: scored tests of the agent’s answers before a change goes live.
• Live monitoring: a record and a score for every live conversation, taken from the agent’s own logs.
• Voice agent testing: simulated callers that put a voice agent through its paces and score each call.
• Enterprise chatbot QA: quality assurance tests across many bots, languages and platforms on every release.
• Checks from the customer’s side: questions put to the live chat, inbox or phone line as a customer, with each answer kept.

In the UK, a business is responsible for what its AI agent does in the same way it is responsible for what an employee does, even if someone else provides it. The Competition and Markets Authority set this out in guidance published on 9 March 2026, which also tells businesses to check regularly how well an AI agent is performing.

How we compared the tools

Every tool is described from its own pages, with prices in US dollars as listed on 28 October 2026. 5 questions separate them:

• What it tests: the agent in development, its live conversations, a voice agent, or what a customer gets on the live channels.
• How it reaches the agent: test cases, records your app sends, simulated callers, or the chat page, inbox and phone number customers use.
• What comes back: scores, step by step records, call recordings, transcripts or a written report.
• Price as listed, and whether there is a free plan, free credit or a trial.
• Where it stops, taken from its own pages.

Evals before launch

An eval is a scored test. The agent answers a set of test cases, and another AI model, code or a person marks each answer against the one expected. For engineers testing prompts and models before release, an eval tool is the right pick: it scores every change against the same cases before any customer sees it.

Braintrust

Braintrust offers evals and tracing for a whole team, and scores answers with an AI model, code or a person.

• What it does well: experiments that run test cases and compare prompts and models side by side. It also records each prompt, reply and tool call, with alerts on cost, speed and quality.
• Price as listed: Starter is free, with 1 GB of data and 10,000 scores before usage fees. Pro is $249 a month, and Enterprise is priced on request. Both listed plans include unlimited users.
• Where it stops: it scores the cases and records your team sends it. Its pages describe no simulated customers or callers.
• Best for: engineering teams comparing prompts and models with scored evals.

Promptfoo

Promptfoo is open source software for testing AI apps and agents, and its home page says 156 of the Fortune 500 use it.

• What it does well: evals for prompts, models and RAG pipelines, systems that look up a company’s documents before answering. It adds red teaming, automated attacks that try to make an agent break its rules or leak data, and runs in the build pipeline on GitHub, GitLab or Jenkins.
• Price as listed: Community is free, with every evaluation feature and up to 10,000 red team probes a month. Enterprise and On-Premise are priced on request.
• Where it stops: it tests the app your team points it at. Its home page describes no customer personas.
• Best for: developers who want free, open source evals and red teaming in the build pipeline.

Maxim AI

Maxim AI calls its product a simulation and evaluation platform for agents.

• What it does well: simulated conversations across scenarios and user personas, scored by an AI model, a person or code. It runs before launch and in production, and its dashboards compare runs across versions.
• Price as listed: not listed for the evaluation product, whose page offers a free start. Its pricing page lists only Bifrost, its open source gateway to AI models.
• Where it stops: its pages don’t say how simulations reach the agent or what each plan includes.
• Best for: teams that want persona-led simulations and evals in one place.

Live monitoring and tracing

Monitoring tools keep a trace of each live conversation and score it as it arrives. A trace is a step by step record of the prompt, the model’s reply and each tool the agent called.

LangSmith

LangSmith, from LangChain, traces and monitors agents in production, including apps built without LangChain.

• What it does well: dashboards for cost, speed, errors and feedback scores, with alerts to PagerDuty or another system. It scores live traces with an AI model or code, and groups them into usage patterns and failure types.
• Price as listed: Developer is free for 1 seat and 5,000 traces a month. Plus is $39 per seat a month with 10,000 traces, and its pricing page prices Enterprise on request.
• Where it stops: it reads the runs your own app sends, and its pages describe no simulated users. Hosting it on your own servers is an Enterprise option.
• Best for: teams building on LangChain or LangGraph who want traces and live scoring.

Langfuse

Langfuse is open source software for tracing and scoring AI apps, free to host on your own servers.

• What it does well: traces of model calls, tool calls and document lookups, scored by an AI model, a rule or a person. It also manages prompts, with one-click rollbacks, and alerts on cost, speed and quality.
• Price as listed: Hobby is free, with 50,000 billable units a month and 30 days of data. Core is $29 a month, Pro $199 and Enterprise $2,499.
• Where it stops: it reads the runs your app sends, and its pages describe no simulated users. ClickHouse announced it had acquired Langfuse on 16 January 2026, and said Langfuse stays open source.
• Best for: teams that want open source tracing and evals they can host themselves.

Voice agent testing tools

Voice testing tools put a voice agent through simulated callers, voices played by AI. Some vary accents, background noise and interruptions. They score each call, and all four of these tools also score live calls once the agent is in production.

Cekura

Cekura tests and monitors voice and chat AI agents.

• What it does well: simulated conversations before go-live, such as bookings, refunds and handovers to a person. It adds attacks that try to break the agent’s rules or leak personal data, compliance checks such as recording notices, and monitoring of live calls.
• Price as listed: pay as you go has no base fee, at $0.25 per voice test minute and $0.05 per monitored call. Startup is $500 a month. Every plan starts with 300 free credits, about 60 minutes, without payment details.
• Where it stops: its documentation names Retell and Vapi as voice platforms it connects with. Its pages don’t say how test calls reach the agent.
• Best for: teams shipping a voice or chat agent who want per-minute prices and live call monitoring.

Coval

Coval runs voice AI testing, evals and quality review.

• What it does well: thousands of simulated conversations before launch, with difficult callers, policy edge cases and noisy audio. Live calls are scored too, and high-stakes, failed or uncertain ones go to human reviewers.
• Price as listed: Starter is $100 a month for 100 simulation minutes, and Growth is $500 a month for 1,000. Both come with a 7-day free trial that asks for payment details, and Enterprise starts at $4,500 a month.
• Where it stops: its documentation lists inbound, outbound, chat and voice-to-voice connections. Its pages don’t say whether a test call reaches the agent by phone number or another route.
• Best for: teams that want voice testing on published monthly plans.

Hamming

Hamming describes itself as “the trust platform for voice and chat agents”.

• What it does well: test scenarios written from the agent’s own instructions and scored on 50+ measures. Simulated callers bring accents, interruptions, noise, keypad tones and phone menus. Its home page lists 50,000+ test calls at once in 65+ languages. Failed tests become regression tests, which run on every release, and on live traffic it replays a set of reference calls every few minutes.
• Price as listed: its Agency, Startup and Enterprise plans all say “Contact us” on its pricing page.
• Where it stops: its pages give no prices or call allowances, and its documentation needs an access code.
• Best for: voice agent teams that want very large call simulations on a sales-led plan.

Roark

Roark offers testing and evals for voice AI, before launch and in production.

• What it does well: simulated and adversarial callers with personas, accents and background noise in 45 languages, and regression tests that can block a release. It scores every production call as it arrives on 500+ measures, alerts in Slack and proposes fixes. Its documentation says it reaches an agent through phone numbers or a connected voice platform.
• Price as listed: pay as you go starts with $50 of free credit, then $0.15 per simulation minute plus provider costs. Team starts at $500 a month and Enterprise at $4,000 a month on its pricing page.
• Where it stops: its pages don’t describe how chat simulations are set up. Single sign-on and data residency are Enterprise only.
• Best for: voice agent teams that want usage-based prices and a score on every call.

Enterprise chatbot QA

Cyara Botium

Cyara Botium tests and monitors bots and AI agents on voice and chat without code.

• What it does well: functional, regression, performance and security tests on every build, in 50+ languages across 55+ chatbot platforms. It also checks for prompt injection (instructions hidden in a message to take over the agent), made-up answers, data leaks and bias. Cyara’s Voice Assure tests customer-facing phone numbers in 145+ countries.
• Price as listed: not listed, with a demo on request.
• Where it stops: it connects to the bot platforms your team runs, and its pages give no prices or usage limits.
• Best for: enterprise teams testing many chatbots and voice bots on every release.

Checks from the customer’s side

Customer-side checks talk to a live chatbot or AI agent the way a customer does and report what it said. They serve the people who answer for the agent, such as CX leads, compliance and legal, as well as the engineers who build it.

UndercoverAgent

UndercoverAgent runs customer scenarios against one live chatbot, voice agent or custom connection, and sends a weekly written brief.

• What it does well: 200+ scenarios with confused, angry and adversarial customers. It also checks the disclosures that health, card payment and data protection rules require (HIPAA, PCI and GDPR). The weekly brief rates findings by severity, with example transcripts.
• Price as listed: $500 a week for 1 live agent, cancel any time. A sample report is on its demo page.
• Where it stops: 1 live agent per retainer, connected through Intercom, Zendesk or a custom connection. Its page says its findings are test evidence and carry no certification.
• Best for: teams that want a weekly written brief on one live bot.

TokenSurf

TokenSurf is a founder-run service that tests a company’s public chatbot with synthetic customers, with the company’s permission.

• What it does well: test scenarios written for the chatbot, such as an angry customer, a refund request or someone who needs a person. They need no production integration, and the founder reviews every important finding, with a transcript behind each one.
• Price as listed: a one-off AI Agent Audit is $1,500, sold as a founding offer. Continuous AI Assurance is $1,500 a month for weekly monitoring with 100 synthetic conversations, and hardening and security reviews start at $4,000 and $6,000.
• Where it stops: it needs the chatbot’s address, policies and key questions. Its pages describe chatbot reviews with no phone or email checks.
• Best for: teams with a public chatbot that want a review done for them.

Obsession

Obsession makes the first AI customers that work for you. An AI customer is an AI agent that becomes a real customer of a company, with its own inbox, phone number and browser, and keeps what the company sends and shows it as evidence. A chatbot is an AI customer service agent, which answers customers for a company. An AI customer is one of those customers, as AI customer service agents vs AI customers sets out, and what AI customers are explains the term in full.

For a team running a chatbot or AI agent, Obsession’s AI customers chat, email and call it on the live channels customers use. They run every morning and within the hour of each update. Each one says it is an AI, asks what customers ask and checks the answer against the company’s policy page. It then asks for a person, and watches its own inbox for the email or callback the agent promised.

The same AI customers sign up to any company your team names and keep every email, offer and price it sends, so your company can also find its revenue leaks, watch its rivals and prospect with proof. The team behind Obsession has mystery shopped 1,500 ecommerce stores as a new customer since September 2026.

• What it does well: answers checked from outside on chat, email and phone, every day and after every update. Each is kept as the transcript or recording, beside the policy page as it read that minute. Its checks cover a support bot, a voice receptionist, answers that change after an update, the notice that an agent is AI, resolutions an AI vendor bills for, and shortlisted vendors.
• Price as listed: running now. The first AI agent check is free, and plans are credit-based, like Claude’s.
• Where it stops: it covers the live chat, inbox and phone line your company names, and scoring prompts and models inside the build is an eval tool’s job.
• Best for: knowing what customers are told on the live chat, email and phone, every day and after every update, with each answer kept.

AI chatbot testing tools compared

Obsession makes AI customers, so every other option is described from its provider’s own pages as of 28 October 2026.

Which tool to choose, by job

Use Obsession when your team needs to know what customers are told on the live chat, email and phone your company runs, every day and after every update. Each answer is kept beside the policy it should match. An eval tool is enough while the agent is in development and the question is whether a change passes its test cases. A monitoring tool is enough when the question is cost, speed or errors inside your own app.

• Prompts and models before release: Promptfoo, free and open source, runs in the build pipeline. Braintrust, free to start, runs scored experiments across a team, and Maxim AI adds simulated user personas.
• Live conversations from the agent’s own logs: LangSmith for teams on LangChain or LangGraph, and Langfuse for open source a team hosts itself.
• A voice agent: Cekura or Roark for per-minute prices with free credit to start, and Coval for fixed monthly plans. Hamming suits very large call simulations on a sales-led plan.
• Many bots across an enterprise: Cyara Botium.
• A written review of one public chatbot, done for your team: TokenSurf for a one-off audit, and UndercoverAgent for a weekly brief.

A team running an agent at volume often uses 2: an eval or monitoring tool inside the build, and a check from the customer’s side on the live channels. For CX teams, AI customer service quality assurance works out what a wrong answer costs in a team’s own numbers. For a store’s chat, how to test a chatbot gives 8 questions and a scoring sheet to run by hand.

How to automate chatbot testing

Automated chatbot testing runs the same checks on every change and every day, so a broken answer shows up before customers meet it. A team can set it up in 5 steps, each with a pass mark written before the first run.

1. Write test cases from real questions. Take the questions customers ask most from the help desk’s topic report, and write each expected answer from the policy page. A pass: every case has one right answer a person can check.
2. Run them on every change, before release. Promptfoo and Cyara Botium run in the build pipeline, and Roark’s regression tests can block a release. A pass: a change that breaks a passing case doesn’t ship.
3. Score live conversations as they happen. LangSmith, Langfuse and Braintrust score live traces with an AI model or code, and alert on cost, speed and quality. A pass: an alert reaches its owner when a score crosses the line your team set.
4. For a voice agent, run simulated callers against it. Cekura, Coval, Hamming and Roark score each simulated call, and Hamming and Roark list accents and background noise. A pass: bookings, prices and handovers are right on every simulated call.
5. Check the live channels as a customer, every day and after every update. Ask the live chat, inbox and phone line the same questions, ask for a person, and keep each answer with its time. A pass: each answer matches that morning’s policy page, a person answers within your target, and the agent says it is an AI.

Questions about AI chatbot testing tools