Shopify SimGym review: what its AI shoppers test on your store, and what they can’t

Shopify SimGym is an app by Shopify that sends AI shoppers through a store’s live theme, a draft theme or both, and in a comparison predicts which theme earns more add-to-carts. Shopify’s pages describe its shoppers browsing, adding to cart and giving feedback, and say nothing of signing up, receiving emails or trying a discount code. For what a customer receives after the click, use Obsession: its AI customers become real customers of your store or any rival, try the code at checkout and keep every email with its time.

Shopify merchants and the agencies that build and optimise their themes need 3 answers before relying on SimGym: what it tests, how far to trust it, and what it leaves unchecked. Every price, rating and feature here is as Shopify listed it on 29 October 2026.

What Shopify SimGym is and how it works

SimGym went on the Shopify App Store on 8 December 2025, and Shopify opened it to every eligible merchant, with no waitlist, on 11 March 2026. Shopify labels it an AI Research Preview. Its AI shoppers are software agents that open the store in a real browser and choose what to click from what they see on the page, the way a person would.

Each AI shopper is given a persona, a profile that sets how price-sensitive it is, how far it browses and what it values. In Shopify’s research, the personas are built from each store’s own records of how real shoppers moved through it. In Shopify’s help centre, the shoppers browse collections, view products, add items to the cart and describe what they found.

A simulation runs on the published theme, on a draft theme (an unpublished copy in the store’s theme library), or on both at once. With both, SimGym compares their add-to-cart rate, the share of shoppers who put at least one product in the cart, and reports whether one theme did better or both did the same. A simulation usually takes 1 to 5 minutes, with a choice of 4 test focuses:

• Entire shopping journey: the store from start to finish, to find the biggest points of friction.
• Product discovery: whether the home page and collections lead a shopper to a relevant product.
• Add-to-cart confidence: whether the product pages give a shopper the confidence to buy.
• Theme comparison: the same AI shoppers on the live theme and a draft.

The results open on a key finding, then storefront screenshots with numbered markers and recommendations written by Sidekick, Shopify’s AI assistant, from the shoppers’ feedback. Behind each recommendation, View findings lists the shoppers who met the issue, with a replay of each session and a timestamp on each of their thoughts.

How to run a simulation in the SimGym app

1. Install the SimGym app from the Shopify App Store.
2. In the Shopify admin, go to Apps > SimGym and click Create simulation.
3. Choose a test focus, then the theme to test. For a comparison, the second theme must be a draft.
4. Click Run simulation, then open the result from the notification in the admin when it completes.

Who can use SimGym, and SimGym pricing

Shopify’s help centre sets 5 conditions:

• A Liquid storefront. The store runs on a standard Shopify theme, written in Shopify’s template language, Liquid. Hydrogen and other headless storefronts (a custom front end built on top of Shopify) aren’t supported.
• Shopify Network Intelligence turned on. It is a setting in Settings > Customer Privacy that lets Shopify combine a store’s customer data with other merchants’ data to power some of its apps. Shopify’s terms FAQ says the data isn’t shared with any other merchant.
• No permanent password page, as some development stores have. A store in private mode, including one on a free trial, can still run simulations.
• An active Shopify plan.
• At least 1 product published to the Online Store, with a price over zero, that isn’t a gift card.

The SimGym app is free to install. Each month includes an allowance of simulations that renews on the 1st and doesn’t carry over, and a theme comparison normally uses twice as much of it as a single-theme run. Past the allowance, the help centre lists $0.75 for a single-theme simulation and $1.50 for a theme comparison. SimGym shows the price before a paid run starts, and bills it only when the run completes successfully.

Neither the help centre nor the App Store listing, which says “Charges per simulation run”, states how many simulations the allowance holds.

What SimGym does well, and when to use it

A live A/B test splits real visitors between 2 versions of a store and measures what each group does. Shopify’s researchers note it can take weeks to reach a clear result, and some real shoppers see the weaker version while it runs. SimGym’s strengths sit on both sides of that trade:

• A draft tested before launch. No real shopper meets the new theme until it is published.
• A result in minutes. A run usually takes 1 to 5 minutes.
• Shoppers modelled on the store’s own visitors. In Shopify’s research, its predictions got much worse when the shoppers lost those store-specific profiles.
• Its working shown. Each recommendation links to the shoppers behind it and a replay of their sessions.

Shopify names theme changes as SimGym’s primary use. It also fits holiday or campaign layouts, a new menu or collection structure, and a check before and after a single edit. Shopify’s own tip is to test one change at a time, so that any difference in the result has one cause.

How accurate is SimGym

Shopify’s researchers tested SimGym on 50 real stores in 16 countries and 11 product categories. Each store had already run a live A/B test of a visual theme change, so the researchers knew how real buyers’ add-to-cart rate had moved. Their paper was published on arXiv on 19 May 2026. It reports that SimGym predicted the right direction, up or down, at 77% of the stores, with a 95% range of 66% to 87%. The figures are for the shoppers the paper calls skimmers, the largest group that didn’t leave at once, almost two thirds of engaged visits.

For the size of the change, the correlation was 0.55 (95% range 0.32 to 0.72), where 1 would be perfect. In its talk at ICLR 2026 on 25 April, Shopify’s team presented SimGym as a way to screen changes in under an hour instead of weeks, without exposing real buyers. The paper sets out the scope of that result:

• One measure. It predicts add-to-cart. The authors list other shopper behaviour as future work.
• Visual theme changes. Tests swayed by promotions, pricing or changes to the range were left out.
• One platform. The authors say results beyond it remain to be tested.
• A crowd of 600. The study ran 600 AI shoppers per store, in about 5 minutes. The help centre doesn’t say how many the app sends.
• Goals without offers. The product each shopper hunts is described without bundles, sizes or discounts.

The help centre adds that results might differ from how a store’s actual customers behave. On 29 October 2026 the App Store listing showed a rating of 2.8 from 40 reviews, with as many 1-star reviews as 5-star. Merchants who rated it well found its recommendations useful. Others reported problems it flagged that weren’t there, such as a Checkout button that worked or an available variant read as sold out. Some found the advice generic or saw runs stuck on “scheduled”. Shopify’s replies say it is working to reduce false findings.

Independent reviewers reach the same view. EcommerceFastlane’s review of 26 February 2026 advises treating SimGym as a hypothesis generator, a source of ideas to test with real shoppers. Moving Primates’ guide of 1 April 2026, updated 16 April calls its results a first indication to confirm with real traffic. It names mega menus and drawer carts as layouts the shoppers may misread, and says SimGym doesn’t suit new products without data or stores with very little traffic.

What SimGym’s AI shoppers don’t test

Shopify’s pages describe browsing, adding to cart and feedback, and don’t describe sign-ups, emails or discount codes. Its research measures add-to-cart, and its paper shows an example shopper going on to checkout within set limits on which cart and checkout steps it may take. Moving Primates puts checkout, prices, discounts and campaigns outside SimGym’s scope. EcommerceFastlane says SimGym’s feedback covers the checkout experience, and that it doesn’t test the checkout flow beyond the add-to-cart. Each step that follows comes after the add-to-cart, and each can lose a sale the theme has already won.

Obsession makes AI customers that sign up to any company and keep every email, offer and price it sends, so you can find your revenue leaks, watch your rivals and prospect with proof. An AI customer is an AI agent that becomes a real customer of a company, with its own inbox, phone number and browser, and keeps what the company sends and shows it as evidence (what AI customers are). In October 2026 Obsession ran a pilot for 4 agencies on the stores they pitch and serve. Between 21 and 24 October 2026, its AI customers signed up, browsed, left a basket and started a checkout at each store. Each of these journeys had its own inbox, watched for 48 hours. Every store in the pilot had at least one gap. On 26 October 2026, an AI agent working for Obsession walked the client stores that Shopify agencies and freelance experts chose to show on their own websites.

• A sign-up with a real inbox. Neither Shopify’s help centre nor its research describes a shopper typing in an email address, and every email a store sends a new customer starts there.
• The welcome email and its code. Whether it arrives within minutes, from a sender the shopper recognises, with the code the pop up promised. If it never comes, why a welcome or basket email never arrives works through the causes.
• Codes and offers at checkout. In Shopify’s study, the product each shopper hunted was described without discounts. The 26 October walk found offers that broke between the page and the cart. A banner promised free shipping while the cart set a spend threshold. Carts dropped a subscription price, never added a promised free gift, or added a protection charge the shopper hadn’t chosen.
• The checkout reminder. In the pilot, nearly half sent nothing in the 48 hours after a shopper left a checkout with their email typed in.
• What arrives over 48 hours and weeks. A simulation lasts minutes. In the pilot, most first reminders came within 4 hours, the fastest after 30 minutes, and some about 10 hours later. Over weeks, offers go stale: the walk found a sale banner still up the day after the sale ended.
• Rivals’ stores. Shopify’s pages describe tests of the store that installed it. A rival’s welcome offer, prices and reminders need a customer at the rival.
• The page behind an ad. Shopify’s pages describe shoppers who browse the storefront, and don’t describe starting from an ad’s link. A shopper from an ad lands on the page the ad links to, expecting the price or offer the ad showed.

Both kinds of AI shopper meet bot checks. The help centre says bot protection or traffic limits can block SimGym’s shoppers and leave their visits unfinished. In Obsession’s pilot, most stores put a CAPTCHA or bot check in front of at least one step. A CAPTCHA is a puzzle that blocks automated visitors. Each Obsession store check ends in a report on every journey. Where a store puts a CAPTCHA in front of a step, the report names the step it couldn’t test.

Simulated shoppers and AI customers, side by side

The two answer different questions. SimGym asks how a crowd of the store’s shoppers would browse this theme. An AI customer asks what a customer receives. That second question is where sales are lost after the click, through a refused code, a reminder that never came or a free shipping promise the cart dropped.

On a Shopify store, Obsession’s AI customers open the page behind your ads from their links and sign up through the pop up. They take the welcome code to checkout, leave a basket or a checkout, and keep every email and page with the time it arrived. Before a launch, they take a draft theme’s preview link and run every journey on the new theme first. The store’s team chooses when to read the results, at 4, 12, 24 or 48 hours, or keeps the AI customers watching for as long as it wants. The same platform checks every rival’s prices, offers and delivery each morning, and asks AI assistants what they tell buyers about the store. The team behind Obsession has mystery shopped 1,500 ecommerce stores as a new customer since September 2026. Obsession runs these checks at 300, 1,000 or 5,000 brands a month, and it’s built for millions.

SimGym alternatives compared

Obsession makes AI customers, so every other option is described from its provider’s own pages as of 29 October 2026. CustomerEye and Insper, 2 other AI shopper apps for Shopify, showed as not currently available on the Shopify App Store that day.

SimGym, an A/B test or AI customers: which to use when

• Everything after the click, every week: use Obsession. Its AI customers sign up, take the welcome code to checkout, wait for the reminders and keep every email and page, at your own store and at the rivals you name. Before a launch, the same AI customers run every journey on the draft theme from its preview link.
• A theme, layout or menu decision before launch: SimGym. Test one change at a time, as Shopify advises, and read the result as a prediction.
• A big redesign on a store with steady traffic: confirm it with a live A/B test once SimGym has narrowed the options.
• A free second simulation with session replays: Avada Shop Sim.
• One small change on one store: check it by hand with a test order, as set out in how to test your Shopify store as a customer.

For tools that call staff, test chatbots or walk a store, the AI mystery shopping tools guide sorts 20 of them by what they test.

Agencies that redesign themes for clients can run the same checks on the brands they pitch.

Questions about Shopify SimGym