Always-on tests: forgetting, half-lives, and sliding windows

Last season's winner is still winning

You start a headline test in March and leave it running, because a bandit is supposed to keep optimizing. "Spring sale ends soon" wins and gets most of the traffic. In June the sale is over and the line reads oddly, but it still gets most of the traffic. It has three months of conversions behind it. A few weeks of weaker results barely move its average.

That is how a standard bandit is meant to behave. It assumes each version's true conversion rate never changes, so a conversion from March counts as much as one from this morning. After enough traffic, no plausible run of new results can move a posterior built on that much old data. The test has stopped learning while it looks like it is still running.

Real tests change. Creative wears out, seasons turn, your traffic mix shifts, a competitor changes their price. Bandit's memory setting tells an experiment how much of its past to keep believing.

Three memory settings

Standard is right for a test you intend to finish. You want all the evidence, and you'll stop it when you have an answer.

Discounted fades evidence smoothly. A result's weight is 0.5 ^ (age / half-life). With the default 7-day half-life, a result from a week ago counts half as much as one from today, two weeks ago a quarter, and four weeks ago about 6%.

Sliding window keeps results inside the window at full weight and drops everything older. With the default 14 days, a result exactly 14 days old is already out.

How discounting changes the bandit

Discounting shrinks a version's trial count as well as its conversion count. Older evidence fades on both sides, so the posterior gets wider. A wider posterior means Bandit is less sure, and it starts showing the other versions again. When the world changes, the bandit goes back to exploring instead of quietly nudging its estimate.

It also caps how much history the bandit can hold. Under steady traffic, a 7-day half-life adds up to about 10 days' worth of results at full weight, since 7 divided by the natural log of 2 is about 10.1. Three months of results carry barely more weight than the last four weeks, about 10.1 days' worth against 9.5. New results can overturn an old winner.

A few details keep the numbers honest:

• Age is calendar time. Age is measured in real time, not in how many visitors came after. A quiet experiment forgets at the same rate as a busy one, because seasons and campaigns change on a calendar.
• A late conversion is aged from the impression. If a visitor converts two weeks after seeing a version, the conversion gets the weight of the original impression. Conversions can never outweigh the impressions that produced them.
• A clock in the future doesn't help. A result stamped in the future counts at full weight, never more.

How the sliding window differs

A running total can be discounted in place, because exponential decay composes. Decaying for a week and then another week is the same as decaying for two weeks at once. So a discounted experiment stores its aged totals as three numbers per version and updates them on each result.

A hard cutoff doesn't compose. Under a sliding window, Bandit rebuilds each version's counts from the impressions inside the window every time it picks a version. That reacts faster to an abrupt change, such as a redesign, a price change, or the day a sale ends. The cost is a smaller sample, since everything older than the window counts for nothing.

Your history isn't lost

The dashboard's lifetime counts never decay. Impressions and conversions stay the honest totals. Memory only changes what the bandit believes when it picks a version.

Bandit also keeps the discounted totals up to date for every experiment, whatever its setting. If you move a standard experiment to discounted later, it starts from its real aged history instead of from zero.

The contextual bandit supports both memory settings too. It refits from the most recent 2,000 impressions on each request, with older ones weighted or dropped the same way.

Limits

• Shopify theme tests support standard and discounted. The storefront picks versions from a published snapshot, which can't carry a sliding window, so a theme test set to a sliding window is not published to your store. The Shopify app's editor has no memory control yet.
• A shorter memory means less evidence. With a short half-life or window and little traffic, the bandit may never settle. Match the half-life to how fast things change for you, and to how much traffic arrives in that time.
• Sliding windows cost a read per request. Each assignment scans the impressions inside the window. That is fine for most tests, but discounted is cheaper at high volume.

How to turn it on

The memory setting lives on the experiment, in two places:

• Dashboard. The experiment form has a Memory field with Standard, Discounted, and Sliding window. Discounted asks for a half-life in days, and Sliding window for a window in days.
• CLI. bandit experiments create takes --forgetting none, discounted, or sliding_window, with the 7-day or 14-day default. To change an existing experiment, run bandit experiments update <id> and choose Memory. The prompts ask for the number of days.

bandit experiments update <experiment-id>

As the note above says, the setting changes serving only for an experiment that runs on its own. The CLI reference lists every flag, and the algorithms docs cover the math. The developers page shows where memory fits in an integration.

Pick standard for a test you'll finish. Pick a half-life for one you won't.