What Is Multi-Armed Bandit Testing?

The Gambler's Dilemma

Imagine you're standing in front of five slot machines in a casino. Each machine has a different, unknown payout rate. You have a limited number of coins. How do you maximize your winnings?

You could play each machine an equal number of times and then pick the best one. That's essentially what traditional A/B testing does. But there's a problem: you're wasting coins on machines you already suspect are bad, just to be statistically rigorous.

This is the multi-armed bandit problem, and it's one of the most studied problems in probability theory and machine learning. The "arms" are the slot machine levers; "multi-armed" means you have multiple options to choose from.

Try It Yourself

Before diving into the theory, get an intuition for the problem. Click the arms below to pull them. Each has a hidden reward rate, and your job is to figure out which one is best while maximizing your total reward.

Notice what happened? If you played manually, you probably started by exploring all the arms, then gradually shifted to the ones that seemed to pay out more. That instinct, explore then exploit, is exactly what bandit algorithms formalize.

The Explore-Exploit Tradeoff

Every bandit algorithm must balance two competing goals:

• Exploration: trying different options to learn which is best
• Exploitation: choosing the option you currently believe is best

Too much exploration wastes resources on underperforming variants. Too much exploitation risks getting stuck on a locally optimal choice because you stopped looking too early.

The sweet spot is adaptive allocation: shifting traffic toward winners as evidence accumulates while maintaining enough exploration to detect changes or correct early mistakes.

Three Algorithms You Should Know

Epsilon-Greedy

The simplest approach. With probability epsilon (typically 0.1 or 10%), choose a random arm. Otherwise, choose the arm with the highest observed reward rate.

Pros: Dead simple to implement and understand. Cons: Explores uniformly and doesn't favor promising arms over bad ones. The epsilon parameter is fixed, so it doesn't reduce exploration as confidence grows.

UCB1 (Upper Confidence Bound)

Instead of exploring randomly, UCB1 is optimistic in the face of uncertainty. It calculates an upper confidence bound for each arm's reward rate, then picks the arm with the highest bound.

Arms that haven't been tried much have wide confidence intervals, so they get selected naturally. As an arm accumulates data, its bound tightens and its selection depends more on actual performance.

Pros: No tuning parameters. Exploration naturally decreases over time. Cons: Can be slower to converge in practice than Thompson Sampling.

Thompson Sampling

The Bayesian approach. For each arm, maintain a probability distribution over its true reward rate (a Beta distribution for binary outcomes). At each step, sample from each distribution and pick the arm with the highest sample.

Arms with uncertain posteriors will occasionally produce high samples and get explored. Arms with well-estimated low rates will almost never produce the highest sample.

Pros: State-of-the-art empirical performance. Naturally adapts exploration based on uncertainty. Cons: Slightly more complex to implement. Requires understanding of Bayesian inference.

When to Use Bandits Over A/B Tests

Bandit algorithms shine when:

• Traffic is limited: you can't afford to waste half your visitors on a losing variant for weeks
• Speed matters: you need to converge on the winning variant quickly
• Regret is costly: every conversion lost to a bad variant has real business impact
• Variants change: you're continuously adding new options (e.g., ad creatives, headlines)

Traditional A/B tests are still appropriate when:

• You need a statistically rigorous estimate of the exact effect size
• You're making a one-time, high-stakes decision where precision matters more than speed
• You have ample traffic and can afford to wait for full statistical power

The Math Behind It

For a binary outcome (convert or not), each arm's true conversion rate is some unknown value p. After n trials with s successes:

• Observed rate: s / n
• UCB1 bound: s/n + sqrt(2 * ln(N) / n) where N is total trials across all arms
• Thompson Sampling posterior: Beta(s + 1, n - s + 1)

The key insight is that all three approaches converge on the optimal arm, but they differ in how much regret they accumulate along the way. Thompson Sampling typically achieves the lowest regret in practice.

Getting Started with Bandit

Bandit makes this easy. Instead of manually implementing algorithms, you configure an experiment in the dashboard, choose an algorithm, and the platform handles everything:

1. Create an experiment with your treatment variants
2. Choose an algorithm (Epsilon-Greedy, UCB1, or Thompson Sampling)
3. Request assignments via the SDK: the algorithm selects the best variant
4. Track events: conversions feed back into the algorithm
5. Watch it optimize: traffic automatically shifts to winning variants

The platform maintains algorithm state, handles caching for fast assignments, and provides real-time analytics so you can see the optimization happening live.

Key Takeaways

• Multi-armed bandits solve the explore-exploit tradeoff algorithmically
• They minimize regret by shifting traffic to winning variants as evidence accumulates
• Thompson Sampling is the recommended default for most use cases
• Bandits complement (not replace) traditional A/B testing
• Use bandits when traffic is limited, speed matters, or you're continuously iterating

The next article in this series compares A/B testing and bandit approaches head-to-head with a live simulation so you can see the difference in real time.