AI voice agent testing: how teams test a voice agent as a caller, platforms compared

AI voice agent testing checks what a voice agent says and does on a call: whether it picks up, understands the caller, gives the right answer and hands over to a person. Coval, Hamming and Cekura put the agent through simulated callers before launch and score its live calls once it launches, and Vapi and Retell test agents built on their own platforms. To call the live number as a real caller every morning, use Obsession: its AI customers ring your company’s line, say they’re AI and check each answer against your price list.

AI voice agent testing checks a voice agent before customers meet it and after every change. A voice agent is software that answers or makes phone calls with AI. It turns the caller’s speech into text, writes a reply with a large language model (LLM) and speaks it. Along the way it may book a slot, quote a price or transfer the call.

Teams launching AI receptionists, booking lines and sales callers test it at three layers, and so do the agencies that build those agents for clients.

• The conversation: what the agent says, which tools it calls and whether it follows the policy.
• The voice: whether it hears accents, background noise and interruptions, and how fast it replies.
• The phone line: whether the number rings through, the keypad works and a transfer reaches a person.

What to test on an AI voice agent

Write each check with its pass mark before the first call, so every result is a pass or a fail and never an opinion.

Simulated callers and real calls

Testing platforms put the agent through simulated callers: AI voices that follow a scenario, such as a patient booking a check-up or a customer asking for a refund. They can run thousands of calls before launch, vary accents, noise and interruptions, and score every call.

A simulated call reaches the agent through a connection the team sets up. That might be an agent imported from Vapi or Retell, a SIP line (a phone line carried over the internet) or a direct audio link. A real call rings the public number through a phone carrier, the way a customer’s call does. Retell’s test guide names a real phone call as the way to check carrier audio, keypad tones and transfers.

The two answer different questions. Simulated callers give breadth before each release, and real calls show what a caller meets on the live number that day.

Automated AI voice agent testing

Automated AI voice agent testing runs the same calls without a person dialling, on a schedule and on every release. A scenario that fails becomes a regression test, which runs again on every change so the same fault can’t ship twice.

Coval offers a quality gate for every release, and Retell runs saved scenarios as a batch, including from CI, the build pipeline that tests each change. Hamming turns a production failure into a permanent regression test in one click.

After launch, the same platforms score live calls. Cekura checks every live call for drift, answers that move away from what passed before, and Coval routes the calls that matter to human reviewers. Hamming replays a fixed set of reference calls every few minutes to catch drift or an outage. AI agent observability is the view from inside the agent: each prompt, model call and tool call behind a call.

AI voice agent testing platforms

Cekura, Coval and Hamming are built to test voice agents, before launch and once they are live. Each is described from its own pages as they stood on 1 November 2026, with prices in US dollars as listed that day.

Cekura

• What it does well: Cekura runs thousands of synthetic conversations with a voice or chat agent before go-live. It probes for jailbreaks (attempts to talk an agent out of its rules), data leaks and off-script answers, and checks compliance items such as a recording notice. Once the agent is live, it detects drift across every call. It names Vapi, Retell, ElevenLabs, LiveKit and Pipecat among the tools it works with.
• Price as listed: pay as you go has no base fee, at $0.25 per voice testing minute and $0.05 per monitored call, with 10 calls at once. The Startup plan is $500 a month with 10,000 credits and 50 calls at once, and Enterprise is custom. Every plan starts with 300 free credits, about 60 minutes, and pay as you go starts without payment details.
• Where it stops: test calls reach the agent by a route the team sets up first. The routes are SIP, the standard for internet calls, a link to its voice platform, or the team’s own Twilio numbers.
• Best for: teams that want per-minute prices, free credits to start and monitoring of live calls.

Coval

Norwest announced its investment in Coval’s $28M Series A on 24 June 2026.

• What it does well: Coval runs thousands of realistic conversations before launch, with hard policies, noisy audio and difficult callers who interrupt or talk slowly. It runs evals on live conversations to catch failures as they happen, and routes the calls that matter to human reviewers. Its documentation lists inbound, outbound, chat and voice-to-voice connections.
• Price as listed: Starter is $100 a month for 100 simulation minutes and 1,000 monitored calls. Growth is $500 a month for 1,000 minutes and 10,000 calls. Extra minutes cost $0.40 on Starter and $0.25 on Growth, annual billing takes 20% off, and Enterprise starts at $4,500 a month.
• Where it stops: the 7-day trial on Starter and Growth asks for payment details, and Enterprise terms are negotiated.
• Best for: teams that want simulations and live call scoring on published monthly plans.

Hamming

• What it does well: paste an agent’s system prompt into Hamming and it writes hundreds of test scenarios, scored on 50+ built-in measures. Simulated callers bring accents, interruptions, background noise, keypad tones and phone menus. Its page lists 50K+ concurrent calls in 65+ languages and accents. It imports agents from Vapi, Retell or ElevenLabs, dials a SIP number, or connects to LiveKit or Pipecat directly.
• Price as listed: its Agency, Startup and Enterprise plans all say “Contact us”. The Agency plan adds management of several clients, and Enterprise adds SOC 2 and HIPAA compliance.
• Where it stops: its pages publish no prices, minutes or call allowances, and list no free trial.
• Best for: voice agent teams and agencies that want very large call simulations on a sales-led plan.

Testing inside Vapi and Retell

Vapi and Retell are platforms for building voice agents, and both include their own tests. They suit agents built on that platform, tested in the same place they are built.

Vapi

• What it does well: Vapi’s Simulations run an AI tester through a live conversation with an assistant built on Vapi. A personality sets the tester’s model, its speech-to-text transcriber and its voice, and a scenario sets its goal and the criteria that decide the result. Each run happens in voice or chat, and voice runs are checked against latency limits too. Evals check the agent’s next decision at set points in a mock conversation, such as calling the right tool with the right arguments.
• Price as listed: testing and simulation come with every plan, and Vapi hosting is $0.05 a minute, pre-paid.
• Where it stops: the AI tester talks to assistants inside Vapi, and the overview doesn’t mention testing a phone number or an agent built elsewhere.
• Best for: teams building on Vapi that want tests in the same place as the agent.

Retell AI

• What it does well: Retell offers 4 ways to test. A playground chats with the agent in text, and simulation tests save scenarios as graded test cases and run them alone or as a batch. Web calls let a team hear real audio, latency and interruptions in the browser, and phone calls check carrier audio, keypad tones and transfers. Simulation tests can mock functions, so a mocked step makes no real booking. A step left unmocked still calls its live system.
• Price as listed: text tests are billed per message at the chat rate of the model used. Voice tests are billed per minute at the same rate as a production call, with telephony on top for phone tests. AI QA, which scores real calls, is free for the first 100 minutes, then $0.10 a minute. New accounts get $10 of credit.
• Where it stops: simulation tests run as text, without placing a call, and agents using a custom LLM aren’t supported. Retell has no separate test tier or free testing sandbox.
• Best for: teams building on Retell that want graded scenarios and real call tests in one place.

Testing the live line as a real caller

The testing platforms reach a team’s own agent through a connection the team sets up. A customer meets the public number: the greeting, the hold, the transfer and the answer after last night’s update.

AI customers ring that number as callers. An AI customer is an AI agent that becomes a real customer of a company, with its own inbox, phone number and browser, and keeps what the company sends and shows it as evidence. The term is explained in full in what AI customers are.

Obsession makes AI customers that sign up to any company and keep every email, offer and price it sends, so you can find your revenue leaks, watch your competitors and prospect with proof. The team behind Obsession has mystery shopped 1,500 ecommerce stores as a new customer since September 2026.

Obsession’s Voice agent check calls your AI receptionist every morning as new customers, books a slot, asks a price and a person, and checks every answer. Each AI customer rings from its own phone number and says at the start that it’s an AI customer, who it works for and that the call is recorded, in a voice used with consent, with different voices and accents across the calls.

Every call comes back with a verdict, check by check, and the recording and transcript timed to the second beside the price list as it read that morning. The right answer comes drafted wherever the receptionist went wrong. A real slot is booked in the calendar the business connects and cancelled its usual way, and 3 calls a morning leave the line free for real callers.

The same check can test an AI receptionist before going live, on the number it will answer. AI disclosure check checks your AI agents say they’re AI on every channel, every day. Resolution check checks every resolution your AI support vendor bills you for against what the customer got.

Every AI customer says it’s an AI when it talks to a person and never uses fake payment details, as what every AI customer will and won’t do sets out.

AI voice agent testing platforms compared

Obsession makes AI customers, so every other option is described from its provider’s own pages as of 1 November 2026.

Which to choose

If your team needs to know what callers hear on the live number every day, with every call recorded and checked against the price list, use Obsession. For testing before launch, pick by how your team buys and builds:

• Published monthly plans: Coval, from $100 a month.
• Per-minute prices and free credits: Cekura, at $0.25 a test minute, with 300 free credits.
• Very large simulations, or many clients: Hamming, on a sales-led plan.
• An agent built on Vapi or Retell: the platform’s own tests, next to the agent.

A team launching a voice agent at volume needs both: simulated callers on every release, and real calls to the live number every day. An agency can send each client a morning report on its receptionist. Chat agents and evals are compared in AI agent testing tools, and AI agent QA works out what a wrong answer costs a CX team.

Questions about AI voice agent testing