Is your store’s chat telling customers the right returns policy? How to test it

To test a store’s chatbot, ask it the questions customers ask this season, as a stranger on your live site and on every channel it answers: the last order date for Christmas, the returns window on sale items, whether codes stack, an order that hasn’t arrived and a request for a person. Hold each answer against your own policy page and score it pass, wrong, made up or no handoff, then ask again after every policy or bot change.

On 9 March 2026 the UK’s Competition and Markets Authority wrote it down: a business is responsible for what its AI agent does in the same way it is responsible for what an employee does, even if someone else designed or provides the agent. For a store, the employee is the chat in the corner of the page.

Most store chats were set up in a quiet month, from the policy pages as they stood that week. November changes the questions: the last day to order for Christmas, how long a gift can come back, whether the Black Friday code works on sale prices. A chat that was right in September can quote last year’s answer now. The only comparison that counts is the one you make, with the answer on one side and your page on the other.

Black Friday is Friday 27 November and Cyber Monday is Monday 30 November. More shoppers means more questions put to the chat, all answered from the same pages. The test takes a phone, a sheet and about 40 minutes, best finished before your freeze, about Friday 20 November. If your chat is only a form that emails your team and no AI writes a reply, skip it.

A wrong answer from your chat is your answer

The CMA’s guidance goes past blame: it calls testing a crucial part of training an AI agent, and tells businesses to check on one regularly, with a human in the loop.

The case that made the point is older and smaller. In February 2024 a British Columbia tribunal ordered Air Canada to pay a customer CA$812 in all: CA$650.88 in damages, CA$36.14 in interest and CA$125 in tribunal fees, as Dentons itemises it. The airline’s chatbot had told him he could claim a partial bereavement refund within 90 days; staff later said the airline did not allow retroactive claims. Air Canada argued the chatbot was a separate entity. The tribunal rejected that, as McCarthy Tétrault reports.

The sum was small. What carries over is the reasoning, that it makes no difference to the customer whether a page or a chatbot said it.

Put a figure on any answer you score wrong. Take last November’s chats on that topic (your helpdesk’s report has the count), multiply by what one wrong answer costs you, the average refund or code you honour or the average order you lose, then halve it, because not every shopper will have been given the wrong answer. That is a deliberately low cost of one stale line in one month. The CMA’s 10% is a ceiling it names for any business, not a figure for yours.

How to test a chatbot: be a stranger on the channel your customer uses

Start as a stranger, on every channel the bot answers

Use your phone, in a private window, logged out of your store and its admin. Your own login gets the answers a known customer gets. Then repeat each question on every channel the bot is switched on for. Gorgias, for one, lets you choose which channels AI Agent is enabled on, from email and chat to SMS, Instagram DMs, Messenger and WhatsApp, and adapts its replies to each. A good answer on chat says nothing about WhatsApp.

Copy the policy line before you ask

Open the page or setting that governs the answer and paste the sentence into your sheet with the date and time. Then ask. A line copied afterwards drifts towards what the bot said. For codes, the source isn’t a policy page: Shopify decides whether two discounts stack from the Combinations setting on each one, and discounts don’t combine unless both are set to.

Ask in a customer’s words, twice

Ask once in policy words, such as “What is your returns window for sale items?”, and once the way customers type, such as “can i send back a jumper from the sale”. Customers don’t write in policy words, so the second is the realistic test. Start a new conversation for each, so the first answer can’t colour the second.

Score every answer with one of four marks

• Pass. Every fact matches your page or setting, for the market and channel you asked on.
• Wrong. The bot had a source and it is stale or misread: last year’s cut off date, another market’s returns window, an expired code. Update the source it reads, then ask again.
• Made up. The answer sits on none of your pages: a status for an order number that doesn’t exist, a price match you never offered, a refund it has no way to issue. Put the true answer on a page the bot reads, or send that topic to a person, then ask again.
• No handoff. You asked for a person, or for something outside the policy, and got a loop, a dead end or a promise nobody received. Fix the rule, then check the conversation reached your helpdesk.

The split matters because the fixes differ: a wrong answer points to a source to correct, a made up answer to a gap. Keep each conversation with its time. A screenshot showing the time is the minimum; a helpdesk export is better.

Follow the promise

If the chat says a refund is started, a code is on its way or a person will email, write down the time and watch for it. Give it the window the bot named, or 1 working day if it named none. A promise that never arrives is the one that returns as a complaint.

Test my Shopify chatbot: the questions customers ask this season

These eight fill a store’s inbox between now and January. Ask each one twice, on each channel, and change the details to fit your store.

Two of these need setup. For question 5, ask about a real order of your own and about an order number that doesn’t exist: a status for the false one is made up. For question 6, ask once in staffed hours and once outside them, since Tidio says Lyro creates a ticket when no agent is online.

AI customer service agent testing: what the vendor’s test panel can’t show

Gorgias, Tidio and Fin each document a way to test before customers do, and each is the right place to check a draft before it goes live.

Gorgias’s test conversations let an owner, admin or lead try AI Agent as a new or existing customer on any channel it supports, and preview a draft of guidance or a help centre article before publishing it. They are free and stay out of reporting. Gorgias says they never reach shoppers and create no tickets or real messages. With actions switched on a test can touch real order data, so it advises a fake customer profile.

Tidio’s own page, updated October 2026, describes a testing environment built into Lyro. It says Lyro hands a conversation to a person when it can’t answer or a customer asks for one, and creates a ticket if no agent is online.

Fin’s testing guide recommends 50 to 100 test cases before launch, at least 90% accuracy on core workflows and no policy violations, then a retest on every change to content and weekly runs, using Fin’s Previews and Simulations. Its performance figures are the vendor’s own and aren’t repeated here.

Shopify’s Inbox listing describes an AI sales associate that learns from your catalogue, policies and store content. Its help page on instant answers lets you write or edit each answer and says you are responsible for the accuracy of what you publish, even text the AI suggested. It doesn’t say whether a suggestion follows a later change to your policies.

These tools answer one question well: does the bot do what I set it up to do. Four things sit outside them.

• The live widget. Your theme, your phone, each live channel. A sandbox that never reaches shoppers can’t show what a shopper sees.
• The handoff. A test that sends no real message and creates no ticket can’t show a person receiving the conversation, in hours or out of them.
• The promise. The refund, code or email the bot says is on its way.
• Change you didn’t make. A vendor update, a help article a colleague edited, a source the bot read again.

A test panel shows what your bot is set up to say. A customer shows what it said. Use the panel before a change goes live and the customer test after.

If the chat is an AI, it should say so

Open a new conversation and read the first message, then type “Am I talking to a person?” A pass: it says it is an AI or an automated assistant, in the first message or in a label on its replies, and it answers truthfully when asked. Gorgias, for one, labels AI replies in chat “Automated”.

The rules are moving, and this isn’t legal advice. The CMA tells businesses to label an AI agent so customers aren’t misled into thinking a real person is serving them. Article 50(1) of the EU AI Act requires providers of AI systems that talk to people to design them so people are told, unless it is obvious to a reasonably well informed person, and the Act applies from 2 August 2026. That duty sits with the provider, and who carries what when a store runs a vendor’s bot is a question for a lawyer. US states and other countries have their own rules, so check where you sell.

When to ask again

• Before your freeze, about Friday 20 November: all eight questions, on every channel.
• After every change to a policy page, a help article, a discount, the bot’s guidance or a channel. Fin’s guide also says to retest on every change.
• After a vendor update, which you may not hear about.
• Black Friday morning and Cyber Monday morning, when codes and cut off dates move.
• The first week of January, when gifts start coming back.

If you freeze policies and bot settings until the new year, one run before the freeze and one on Black Friday morning are enough. A pass is dated: it says what the chat said at that time, and nothing about next week.

Where testing by hand stops

By hand you are one customer, in one session, on one day. The method stops where the checking has to be daily, where a vendor changes something you didn’t see, and where a client wants a dated record. For one store with one bot, a sheet, a phone and a repeat after each change is a sound method. For twenty client stores it isn’t.

That is where a daily check takes over. The wider idea is on Check your AI agents.

For agencies that set up the bot

Agencies build the Gorgias, Tidio, Fin and Shopify chats and then move on to the next client. The same questions, asked after each policy or bot change with the client’s OK, turn “the bot is fine” into a sheet the client can check: the question, the policy line, the answer, the mark and the time.

A client who has a dated sheet from Black Friday morning has a record to hold up if a customer disputes what the chat said. Pick one client whose bot you built, ask the eight questions this week and send the sheet before their freeze.

Questions