The blank text box
A bandit only finds the best of the versions you give it. Once the test settles, you need new versions to beat the leader, and coming up with them is the slow part.
A general chat assistant can write ten headlines in a second. But it hasn't seen your page, it doesn't know which of your current versions is winning, and it doesn't know that the step before this one promised free shipping. So you paste all of that in, every time, and the drafts still read like they came from somewhere else.
Bandit's variant chat starts with that context already loaded.
What the chat sees
When you open a chat on an experiment, Bandit assembles a briefing and fixes it as the first message. It has up to four parts.
The experiment. Its name, description, content type, what it is scored on, such as conversion or margin, and the algorithm.
Each existing version, with real results. Its name, description, content up to 500 characters, and any strategy and tone labels. Then the numbers from the bandit's own counters: impressions, conversions, and conversion rate with a credible interval. For a test scored on margin, the average margin as well. A version with fewer than 30 impressions is marked "insufficient data", so the model doesn't read meaning into noise.
The funnel, if there is one. One line per step, in order. Each line gives the step name, what that step tests, its current leading version, and the drop-off to the next step. The current step is marked. That lets a draft keep the promise made on the step before.
The page itself. Not a summary of it. More on that below.
The model is told to judge versions by conversion rate and its interval, not by how much traffic they get. Traffic share only tells you what the bandit is currently exploring.
How Bandit learns your page
You don't upload anything. In a browser, the SDK sends the page address, without the query string or fragment, along with each assignment request. Bandit keeps up to 5 addresses per experiment and counts hits on each. Recording the address adds no fetch to the assignment request, and prefetch requests are ignored.
When a chat starts, Bandit fetches the most visited address once and caches it for 24 hours. The page has to be on your company's domain, or your connected Shopify store. The fetch gives up after 5 seconds, reads at most 2 MB, and accepts only HTML.
Visitor loads page
SDK sends the address
Chat starts
Most visited address fetched
Page converted
Text kept, scripts dropped
Spot marked
Where the experiment renders
The HTML becomes plain structured text. Headings, paragraphs, lists, links, buttons, form labels, image alt text, and prices stay, in page order. Scripts, styles, tracking pixels, hidden elements, and cookie banners go. Navigation and footer shrink to one line each. Bandit finds your current version's text on the page and marks that spot, so the model knows exactly where its draft will appear.
The page gets up to about 8,000 tokens. On a longer page, Bandit drops sections furthest from the marked spot first, so the copy around your experiment always stays intact. If the page yields fewer than about 200 words, as a page rendered by JavaScript often does, Bandit tries again with a headless browser. If the page can't be fetched at all, the chat goes ahead without it.
What you get back
Each reply has a short message and zero to three drafts. You can ask for drafts, ask why the leader is winning, or ask for a critique of a version. Opening a chat without typing anything asks for one strong new variant.
Each draft has a name, a short description, the content, and labels for its strategy, tone, call-to-action style, and emotional hook, plus a sentence on its reasoning. The content matches your experiment's content type, so a plain text test gets plain text and an HTML test gets HTML.
Choose Add as treatment and the draft joins the experiment, marked as generated. From there the bandit treats it like any other version. It earns traffic by converting.
Guardrails
The chat is meant for one job, drafting versions for this experiment, and its limits are fixed.
1,200 tokens
Maximum length of each reply
2,000 characters
Maximum length of each message you send
20 turns
Per chat, then start a new one
10 turns a minute
Per person, plus at most 5 new chats a minute
Off-topic requests get a one-sentence refusal and no drafts. The briefing stays around 12,000 tokens. When the experiment's details run long, Bandit trims funnel detail before it trims version content. Each turn sends the briefing plus the last 6 messages. Older drafts collapse to one line each.
What it costs
Each chat turn counts as one AI draft against your plan:
| Plan | Included each month | After that |
|---|---|---|
| Growth | 25 | $2 each |
| Scale | 100 | $1.50 each |
| Enterprise | none | $1 each |
A turn that returns three drafts still counts as one. Bandit records the tokens each turn used.
Limits
- The page has to be public and on your domain. Pages behind a login, or on a domain that isn't yours, aren't fetched.
- Shopify theme tests don't capture the page yet. The storefront picks versions without calling the assignment endpoint, so it never sends a page address. In the Shopify app, the chat works from your versions and results.
- One page per chat. If an experiment runs on several pages, the chat sees the most visited one.
- Results are only as good as the counters. A new experiment has little data, and the model is told to treat thin data as thin.
- Chat runs on an OpenAI model. The briefing, including your page text and results, goes to OpenAI to produce each reply.
- Drafts are not checked for claims. Read each draft before you add it. The model can write a promise your store doesn't make.
How to turn it on
- Make sure your pages use the browser SDK, so Bandit has a page address to fetch. Nothing else is needed for page capture.
- In the dashboard, open an experiment and choose Chat with AI. In the Shopify app, the Variant chat panel sits under the versions in the test editor.
- Ask for what you want, or send nothing for one strong draft.
- Choose Add as treatment on the drafts you like.
The developers page covers the SDK setup. For drafts on Shopify placements, see the Shopify app post.
The bandit picks among the versions you give it, and the chat writes the next ones from what the bandit has already measured.
Ready to try algorithmic testing?
Stop wasting traffic on losing variants. Bandit's multi-armed bandit algorithms automatically shift traffic to your best-performing treatments in real time.
Start free trial