How Bandit works

In a classic A/B test you split traffic 50/50, wait weeks, then pick a winner by hand. Bandit does the picking for you, continuously: as soon as one version starts converting better, it shows that version to more visitors, while still testing the others a little in case things change.

A funnel is one path through your site that ends in one goal: a purchase, a signup, a booked demo. The goal is what Bandit optimizes for. Every experiment in the funnel is scored by whether the visitors who saw it went on to reach that goal, not just whether they clicked the thing being tested.

That matters because a headline that gets more clicks is not always the one that sells more. Scoring everything against the funnel's goal keeps Bandit pointed at the result you actually care about.

An experiment is one thing on one page you want to test: a headline, a button label, a hero image, an offer. Experiments always live inside a funnel, and a funnel can hold several at once, for example the homepage headline and the checkout button. Each visitor gets one version of every experiment in the funnel, and Bandit learns which combination reaches the goal most often.

You can create an experiment by pointing at an element on your site with the picker, from the CLI, or in code with the SDK.

A treatment is one version of the thing being tested. Some tools call these variants; the app shows them under Variants on an experiment page. Every experiment has a control, which is what your site shows today, plus one or more alternatives. Two to four treatments that say genuinely different things work best: “Free returns for 60 days” versus “Rated 4.8 by 2,000 hikers” teaches Bandit more than two near-identical phrasings.

Each time a visitor is shown a treatment and later reaches the funnel's goal (or doesn't), Bandit updates its estimate of how good that treatment is. The next visitor is more likely to see the treatments that look best, but uncertain ones still get some traffic so a slow starter can catch up. You don't need to stop an experiment or declare a winner. When you want to try a new idea, add a treatment and Bandit starts testing it alongside the rest.

The algorithms page has the details for the curious.

Most sites start with one funnel, and many never need more. Create another one when:

• You have a different goal. A store might have a purchase funnel and a newsletter signup funnel. Experiments aimed at signups should not be judged by purchases.
• You have a different path. A B2B site might have a self-serve trial funnel and a “book a demo” funnel with different pages and different visitors.
• You run more than one site. Each site gets its own funnel, so results from one never mix into the other.

If two experiments are trying to get people to the same goal, put them in the same funnel, even if they are on different pages. That lets Bandit see how they work together.

Highlights

A summary of what's winning right now across your funnels, campaigns and audiences. Start here to see what changed since you last looked.

Audiences

Groups of visitors Bandit discovered that behave differently, for example mobile visitors from paid search, and which treatments win for each group. You do not create these; Bandit finds them once there is enough traffic.

People

Look up one visitor and see everything Bandit has observed about them: which treatments they were shown, where they came from, and what they did. Useful for checking your setup or answering a customer question.

Campaigns

Every ad, email and social campaign that sent visitors to your site, found from the UTM tags on their links, with how each one converts. Bandit fills this in from your traffic.

Funnels

Where your experiments live. Open a funnel to add experiments, watch which treatments are winning, and see where its visitors came from. This is the tab you will use most.

Docs and Support

These docs, and a form that sends a question or bug report straight to us. You can replay the app tour from the sidebar at any time.