AI chatbot fails: real customer service failures, and how CX teams catch them first

AI chatbot fails are public cases where a business’s chatbot or voice agent misbehaved or gave a false policy, price or promise, like the refund rule Air Canada had to honour. A customer or reporter found each one first, and some followed an update, so CX teams catch them by asking the live chat what customers ask and checking each answer against policy. To catch them before customers do, use Obsession: its AI customers ask your company’s chat and phone line what customers ask, every day and after every update, and keep each answer as proof.

An AI chatbot fail is an answer from a business’s AI agent that a customer can hold the business to, or shares in public. Typical ones are a refund rule that doesn’t exist, a price nobody approved and a legal right misstated. Customer experience (CX) teams, which run the email, chat and phone lines customers contact, can use the public cases to test their own channels before a customer does. So can ecommerce teams with a store chat and anyone about to launch an AI agent.

Each case comes from a tribunal’s decision or a named news outlet, with its date, what the bot said and what it cost.

AI chatbot fails, 2023 to 2026

Chevrolet of Watsonville and the $1 car, December 2023

Chevrolet of Watsonville, a car dealership, ran a sales chatbot from Fullpath, a software company for car dealers. Visitors saw that it was powered by ChatGPT. In December 2023 people online got it to write computer code and to agree to sell a car for $1, Business Insider reported. Business Insider noted that the deal was not legally binding. Fullpath shut the bot down on that dealer’s site, and its chief executive said the logs showed people trying for hours, with the bot often refusing.

DPD’s chat swears after an update, January 2024

DPD, the parcel delivery company, runs an AI element in its online support chat beside human operators. In January 2024 a customer got the chat to swear at him, write a poem criticising the company and call DPD “the worst delivery firm in the world”, the BBC reported. DPD said that “An error occurred after a system update yesterday”, and that it had disabled the AI element and was updating it. One post about the exchange was viewed 800,000 times in 24 hours.

Air Canada and the refund rule a tribunal made it honour, February 2024

In November 2022 Jake Moffatt used Air Canada’s website chatbot to ask about bereavement fares after a grandparent died. The chatbot said a traveller who had already flown could submit a ticket for a reduced bereavement rate “within 90 days of the date your ticket was issued”. Air Canada’s own bereavement page said the policy did not apply once travel was completed, and the airline would not give the partial refund.

In its decision of 14 February 2024, the British Columbia Civil Resolution Tribunal found Air Canada liable for negligent misrepresentation. It ordered the airline to pay CA$812.02: CA$650.88 in damages, CA$36.14 in interest and CA$125 in tribunal fees. Air Canada had argued that it could not be held liable for what its chatbot said. The tribunal called that “a remarkable submission” and asked why customers should have to check one part of a website against another.

New York City’s MyCity chatbot and the law, 2024 to 2026

New York City released an AI chatbot in autumn 2023 to answer business owners’ questions about city rules, built on Microsoft’s Azure AI services. In March 2024 The Markup reported that it told landlords they did not have to accept tenants with housing vouchers, and told employers they could take a cut of workers’ tips. It also said “Yes, you can make your restaurant cash-free”. The Markup found each of those answers wrong under the law. When 10 Markup staffers asked the voucher question, the bot told every one of them no.

In January 2026 Mayor Zohran Mamdani called the bot “functionally unusable” and said it was costing the city around half a million dollars, The Markup reported. Building its foundations had reportedly cost nearly $600,000. The city took it down in February 2026, with a note on its page that the “beta test has ended”.

Cursor’s support bot invents a policy, April 2025

Cursor sells an AI code editor. In April 2025 a user kept being logged out when switching computers and emailed support. A reply from “Sam” said Cursor was designed to work with “one device per subscription as a core security feature”. No such policy existed, and Sam was an AI support bot, Ars Technica reported. The user believed Sam was human, and several users posted on Reddit that they had cancelled.

A Cursor representative called it “an incorrect response from a front-line AI support bot”. A Cursor co-creator apologised, the user was refunded, and Cursor began labelling AI responses in email support.

Taco Bell’s voice AI at the drive-through, August 2025

AI bot fails reach the phone and the drive-through too. Taco Bell has used voice AI to take orders at more than 500 US drive-throughs since 2023. In August 2025 the BBC reported that videos of its mistakes had been viewed millions of times. In one, a customer seemingly crashed the system by ordering 18,000 water cups. In another, viewed over 21.5 million times, the AI kept asking a customer who had ordered a large Mountain Dew what he would drink with that.

Taco Bell’s chief digital and technology officer told The Wall Street Journal that the company would think carefully about where to use AI, drive-throughs included. Its teams would be coached on when to step in. Taco Bell says the voice AI has processed 2 million orders successfully.

Why AI chatbots fail in public

• A wrong answer on policy, price or rights. Air Canada’s refund rule, Cursor’s device rule, the $1 car and New York City’s answers on tips and vouchers each contradicted the business’s own policy or the law.
• A customer found it first. Each first public report came from a customer, a member of the public or a reporter.
• A change came first. DPD’s chat broke after a system update. Cursor’s bot invented its rule to explain logouts that a change to how logins worked had caused, as a Cursor co-creator explained.
• People set out to break it. The $1 car and DPD’s poem came from users trying to make the bot misbehave, and in both cases the AI was taken offline.
• A bot taken for a person. Cursor’s users believed Sam was human. Chatbot disclosure laws set out when a business’s AI agent must say it’s AI.

Who is responsible for what a chatbot says

The business is, wherever a tribunal, regulator or legislature has said so. In British Columbia, Canada, the tribunal held Air Canada to its chatbot’s answer. In the UK, the consumer regulator is the Competition and Markets Authority (CMA). It tells businesses that “you are responsible for what an AI agent does in the same way you are responsible for what an employee does”. That holds even when someone else provides the agent, and breaking consumer law can bring a fine of up to 10% of worldwide turnover.

In Utah, the law says it is no defence to a consumer protection violation that generative AI “made the violative statement”. General information, not legal advice.

AI chatbot problems a CX team can test for

Each failure has a question that exposes it, and a pass mark a CX team can write down before asking.

How CX teams catch them first

Most of these cases are a chatbot giving wrong answers, and a team finds one by asking what customers ask, on the live channels, before customers do.

1. List the questions with money or rights attached. Refund and returns windows, prices and codes, delivery dates, cancellations and anything a regulator covers in your sector. Write the right answer from the policy page beside each one before asking.
2. Ask on the live channels, as a customer. Use a phone, logged out, on every channel the agent answers: site chat, email, messaging apps and the phone line. A vendor’s test panel shows the agent as it was set up, and the live channel shows what a customer gets.
3. Push where customers push. Ask for an exception, a discount and a price under list, and ask it to criticise the company or promise what it can’t give. The $1 car and DPD’s poem both came from people trying to make the bot misbehave.
4. Ask for a person, then check one arrives. Note when the agent offers a person and when a person replies, in staffed hours and outside them.
5. Ask again after every change. A vendor update, a new prompt, an edited help article or a policy change can alter an answer overnight, as DPD’s update did. Keep each day’s answers as the baseline for the next.
6. Keep every answer with its time. A transcript or recording with the time it was given settles what the agent said when a customer disputes it.

The by-hand version, with 8 questions and a scoring sheet, is in how to test a store chatbot on returns, delivery dates and codes. AI chatbot testing tools compares the software that tests an agent before launch and monitors it after. AI agent QA works out what a wrong answer costs a CX team in its own numbers.

How AI customers catch failures first

AI customers run the same checks as a customer would. An AI customer is an AI agent that becomes a real customer of a company, with its own inbox, phone number and browser, and keeps what the company sends and shows it as evidence. Obsession makes AI customers that sign up to any company and keep every email, offer and price it sends, so you can find your revenue leaks, watch your competitors and prospect with proof.

For a CX team, Obsession’s AI customers chat, email and call the agents your company runs, every morning and after every update. Each one asks what customers ask and checks the answer against the policy page as it read that minute. Then it asks for a person and watches its own inbox for the refund or email the agent promised.

Each AI customer says it’s an AI when it talks to a person, never uses fake payment details and never gets round a CAPTCHA or a bot block. The Obsession agents page sets out what each one will and won’t do. The team behind Obsession has mystery shopped 1,500 ecommerce stores as a new customer since September 2026.

Use Obsession when your CX team needs to know what customers are told on the live chat, email and phone line every day. Each answer is kept beside the policy it should match. For one chatbot that rarely changes, the checks by hand are enough.

Questions about AI chatbot fails