Whole-page experiments: one call, every slot

Four tests on one page

Your landing page has a headline, a subhead, a hero image, and a button. You want to test all four. Run them as four separate tests and you make four requests per page view, the page can render a headline from one decision and a button from another, and each test reports its own conversion rate as if the other three weren't there.

A bundle treats the page as one unit. You write a few versions for each slot. Bandit assembles a page for each visitor at request time and learns from one payout.

Four slots with four versions each is 16 pieces of copy to write and 256 possible pages. You never build any of those pages by hand.

What a bundle is

A bundle is a group of experiments served together. Each experiment is a slot. Bandit picks the content for each slot. Your code decides where each slot goes on the page.

• One call returns every slot. getBundle returns a version for each member experiment, keyed by experiment name. A slot can't be requested on its own. Asking /api/assignments for a member experiment returns an error that points you to its bundle.
• The page stays the same for a returning visitor. Within the bundle's assignment window, 24 hours by default, a visitor who comes back gets the same combination.
• Every combination is logged. Each page view records exactly which version of each slot the visitor saw. That record is what lets the second model, below, learn from your full history the day you turn it on.

In the dashboard, a bundle is the set of slots in a campaign. Each slot's table shows its versions with an estimated reward, the probability each is best, trial counts, and a Leading badge.

How it learns by default

By default each slot learns on its own. Bandit keeps a posterior for every version in every slot and, for each request, draws one version per slot.

When a visitor pays, you report one payout for the page. Bandit credits the full amount to every version that visitor saw. A $40 order teaches the headline, the subhead, the image, and the button that were on screen.

Visitors who never pay still count. If a visitor shows no activity for 3 days, Bandit records a $0 for them, once. If they pay within 30 days of that $0, the real payout replaces it. A payout counts when it arrives within 7 days of the page being served, and a second payout from the same visitor inside that window is ignored.

This per-slot model converges fast, because it estimates a handful of small posteriors instead of 256 page-level ones. Its weakness is that it assumes the slots don't affect each other.

When slots interact

Say a store tests two headlines and two buttons, and each combination gets equal traffic. The payout per visitor for each pair works out like this:

Per slot, headline A averages $2.50 against B's $2.15, so A leads. Button Y averages $2.40 against X's $2.25, so Y leads. The per-slot model serves A with Y, which earns $2.00. The best page is A with X at $3.00. Button X only works with headline A.

The second model exists for this case.

Fair credit and slot pairs

Alongside the per-slot model, Bandit always trains a second model in the background. It is a Bayesian regression of each visitor's payout on which version of each slot they saw. It credits each version only with the payout difference it causes, rather than the whole payout. In the dashboard this is called Fair credit.

It becomes available in two stages, and the server rechecks each one:

Both steps need an admin. Bandit tracks up to 6 candidate pairs, ranked by how many visitors saw both slots. With a pair turned on, the model can learn that button X works with headline A and serve that combination, even one few visitors have seen. Up to 4,096 combinations are checked directly. Larger pages use a faster search.

Reverting is always allowed and loses nothing. The per-slot model kept learning the whole time.

Limits

• Email campaigns stay on per-slot credit. Bandit refuses Fair credit for a bundle that sends email, because the email send path only supports the per-slot model. Every bundle the dashboard creates comes with a campaign, so today the Funnel interactions panel on a funnel is where you turn on Fair credit.
• No forgetting. Bundles use a model that keeps every observation. The memory setting doesn't change how they serve.
• No margin credit. The Shopify margin score applies to single placement tests, not bundles. Send the payout you want optimized, such as order value, as the bundle reward.
• Content only. Bandit returns the content for each slot. Layout stays in your code.
• A slot belongs to one bundle. To reuse a slot in another bundle, copy its versions. Learned results are not shared between bundles.
• Changing the slots means a rebuild. When you add or remove a slot or version, the second model rebuilds itself from your payout history. The per-slot model keeps serving throughout.

How to turn it on

1. In the dashboard, open Campaigns and create one for the page.
2. Choose Add experiment once per slot, and give each experiment the name your code will use as the slot key.
3. Ask for the whole page in one call, using the bundle id that belongs to the campaign:

const page = await bandit.getBundle(bundleId, visitorId, { device: 'mobile' })

headline.textContent = page.slots['Hero headline'].config.content
cta.textContent = page.slots['CTA'].config.content

4. When the visitor pays, report one payout for the page:

await bandit.trackBundleReward(bundleId, visitorId, 40)

The docs list both bundle endpoints. The growth marketers page shows how page tests fit with funnels.

One request renders the page, and one payout teaches every slot on it.