Why Bandit Runs Thompson Sampling and Not Epsilon-Greedy or UCB1

Three Algorithms, One Goal

Every multi-armed bandit algorithm tries to solve the same problem: maximize total reward by balancing exploration and exploitation. But they approach this trade-off with fundamentally different strategies.

Epsilon-Greedy uses randomness. UCB1 uses optimism. Thompson Sampling uses probability. Each philosophy leads to different behavior in practice.

Epsilon-Greedy: The Simple Baseline

Epsilon-Greedy is the most intuitive bandit algorithm. With probability 1 - epsilon, it picks the arm with the highest observed reward (exploit). With probability epsilon, it picks a random arm (explore).

How it works:

1. Generate a random number between 0 and 1
2. If the number is less than epsilon (default 0.1), pick a random arm
3. Otherwise, pick the arm with the highest average reward

Strengths:

• Dead simple to implement and explain
• Predictable exploration rate
• Works with any reward distribution

Weaknesses:

• Explores uniformly, wasting pulls on clearly bad arms
• The epsilon parameter needs tuning (too high = too much exploration, too low = gets stuck)
• Never stops exploring, even when the best arm is obvious

UCB1: Optimism Under Uncertainty

UCB1 (Upper Confidence Bound) takes a different approach. For each arm, it computes an upper confidence bound on the true reward and picks the arm with the highest bound. Arms with less data get wider bounds, so they get explored. Arms with lots of data have tight bounds that reflect their true performance.

The UCB1 formula:

For each arm, the score is: average_reward + sqrt(2 * ln(total_pulls) / arm_pulls)

The first term is exploitation (observed performance). The second term is an exploration bonus that shrinks as you gather more data for that arm.

Strengths:

• Zero tuning parameters
• Deterministic: same data always produces the same choice
• Strong theoretical regret guarantees (logarithmic regret)

Weaknesses:

• Deterministic behavior can be a liability in non-stationary environments
• Explores more aggressively than necessary in practice
• Confidence bounds assume sub-Gaussian rewards
• Determinism also means every visitor in a burst gets the same arm until the counts move, which concentrates risk in exactly the moments traffic spikes

Thompson Sampling: Bayesian Probability Matching

Thompson Sampling maintains a probability distribution over each arm's true reward rate. At each step, it samples from each distribution and picks the arm with the highest sample. Arms with more uncertainty produce more variable samples, so they get explored naturally.

How it works (binary outcomes):

1. For each arm, maintain a Beta distribution: Beta(successes + 1, failures + 1)
2. Sample a value from each arm's distribution
3. Pick the arm with the highest sampled value
4. Update the selected arm's distribution with the outcome

Strengths:

• No tuning parameters
• Naturally concentrates exploration on uncertain arms
• Lowest empirical regret across most scenarios
• Adapts well to non-stationary environments (due to stochastic selection)

Weaknesses:

• Requires choosing a prior distribution (Beta for binary, Normal for continuous)
• Slightly more complex to implement than Epsilon-Greedy
• Stochastic behavior makes exact reproducibility harder

Head-to-Head Comparison

Try It Yourself

The simulator below is a teaching sandbox, not a preview of the product: it runs all three algorithms side by side against the same set of arms so you can watch the difference yourself. Only Thompson Sampling is available on Bandit.

Things to experiment with:

• Close conversion rates (e.g., 48% vs 52%): Thompson Sampling's advantage is most visible here
• One dominant arm (e.g., 80% vs 20% vs 15%). All algorithms converge quickly, but Epsilon-Greedy wastes the most traffic
• Many arms (5+), where UCB1's aggressive exploration becomes costly and Thompson Sampling scales better

What We Concluded

Read that table as a product decision rather than a menu, and the answer is not close.

Epsilon-Greedy's selling point is simplicity, and simplicity is worth nothing once the algorithm is somebody else's problem. You are not implementing it; we are. What you inherit is a tuning parameter you have to defend and a fixed tax on your traffic that never goes away.

UCB1's selling point is determinism, and determinism turns out to be the wrong thing to want here. Reproducibility matters when you are debugging an algorithm. It actively hurts when your winner is drifting, because a deterministic rule commits hard to a stale estimate and needs a large run of contrary evidence to be talked out of it.

Thompson Sampling gives up nothing to either. No tuning, logarithmic regret, and the lowest empirical regret in the benchmarks. Its stochasticity, the thing that makes it non-reproducible, is precisely what keeps it honest when the world moves.

What You Configure Instead

Dropping the algorithm picker did not leave you with fewer decisions, just a better one. Rather than choosing between exploration strategies, you choose the algorithm's memory: how much of the past its posterior still believes.

This is the setting that actually changes outcomes on a live experiment. An always-on test on standard memory eventually stops learning: after enough traffic, no plausible run of new results can move a posterior built from millions of old ones. That is a real failure mode, and none of the three algorithms above fixes it. A half-life does.

The Practical Answer

For teams optimizing web experiences, Thompson Sampling is the best default, which is why it is the only default. It requires no tuning, produces the lowest regret, and handles the messy realities of production environments better than the alternatives.

The question worth your attention is not which algorithm to run. It is whether your winner is going to stay your winner, and how quickly you want the algorithm to notice if it doesn't.