The Real Question Isn't "Which Is Best"
Every optimization technique on this ladder is strictly more powerful than the one below it. A trained model can, in theory, do everything a simple A/B test can and more. So why doesn't everyone just start there?
Because each rung up the ladder trades a fixed cost (more engineering effort, more moving parts, more ways to get it wrong) for a data requirement that grows roughly an order of magnitude at a time. Skip a rung before your traffic supports it, and the fancier technique doesn't outperform the simpler one. It just runs longer on noise before admitting it has no idea what's going on.
The right question is never "which algorithm is best." It's "which algorithm can my traffic actually make work, and is the next rung up worth the effort yet."
The Five Rungs
1. A/B Testing
ELI5: Flip a coin to show each visitor version A or B. Wait until you've collected enough people to be statistically sure which one wins, then permanently ship the winner.
Data needed to be worth it: A fixed sample size, calculated upfront: typically thousands of visitors per variant to detect a realistic lift with standard significance and power thresholds. This is a threshold, not a curve. Below it, the test is inconclusive; above it, it's conclusive. More traffic doesn't make an A/B test "better," it just gets you to the answer faster.
Cost: Low. No algorithm to run, just a fixed split and a statistical test at the end.
2. Multi-Armed Bandit
ELI5: Instead of splitting traffic evenly for the whole test, keep shifting more visitors toward whichever variant is winning so far, while still checking the others occasionally in case the early leader was a fluke.
Data needed to be worth it: Far less than a full A/B test needs to reach significance. A bandit starts producing value almost immediately (real gains from a few hundred conversions) because it stops wasting traffic on a losing variant instead of waiting for a fixed sample size.
Cost: Low-to-medium. Needs an algorithm (Epsilon-Greedy, UCB1, or Thompson Sampling) and assignment bookkeeping, but no visitor-level features.
3. Linear Contextual Bandit
ELI5: Same idea as a multi-armed bandit, but it also looks at who the visitor is (device, location, referral source) and learns that different variants win for different kinds of people, using a simple straight-line relationship between those traits and the outcome.
Data needed to be worth it: Enough conversions within each context slice, not just in aggregate. If you're segmenting by a handful of dimensions, that's roughly the multi-armed bandit's requirement multiplied by your number of meaningful segments, often tens of thousands of events before the per-segment estimates stop being noisy.
Cost: Medium. Requires feature extraction from context and a linear model (Linear UCB or contextual Thompson Sampling) instead of one static estimate per arm.
Note
This is the deepest rung the Bandit platform automates today: Epsilon-Greedy, UCB1, Thompson Sampling, and Linear Contextual Bandits are all available out of the box. Rungs 4 and 5 below are typically where teams outgrow an off-the-shelf platform. Most never need to, since the first three rungs cover the large majority of real-world optimization problems.
4. Non-Linear Contextual Bandit
ELI5: Same as the linear contextual bandit, but instead of assuming a straight-line relationship between visitor traits and outcome, it can learn bent, interacting relationships. "Mobile + evening + returning visitor" might behave nothing like any one of those traits alone.
Data needed to be worth it: Hundreds of thousands of events. A non-linear model (gradient-boosted trees, or a neural net with a Bayesian last layer, sometimes called a "bandit forest" or "regression-oracle bandit") has far more parameters to fit than a linear one, and needs enough data to separate a real interaction effect from noise.
Cost: High. A fit/retrain loop instead of closed-form updates, and you still need to keep calibrated uncertainty estimates so the exploration behavior stays statistically sound.
5. Custom Trained Model / Offline RL Policy
ELI5: Stop running a lightweight algorithm in real time. Take your full history of events, train a model (often a neural network) offline, and deploy it as a fixed policy that gets refreshed on a schedule. This is, functionally, building your own recommendation engine.
Data needed to be worth it: Millions of events. At this scale you're not fitting a handful of parameters anymore. You're training a model with enough capacity to find patterns a bandit's online updates would never converge on in a reasonable timeframe.
Cost: Very high. A dedicated ML pipeline: offline training, offline evaluation before deployment, a retraining cadence, and monitoring for drift. This is a standing engineering commitment, not a one-time setup.
Try It: How Much Does Your Traffic Support?
Adjust your monthly event volume and number of variants below. Each row shows roughly how much data that approach needs before it's reliable, and how your current traffic stacks up.
Readiness Explorer
How well each approach works at your traffic level
A few things worth noticing as you drag the slider:
- At low traffic, everything below "Multi-Armed Bandit" is red. There's no way around this. No algorithm creates statistical power out of nothing.
- Adding variants pushes every bar backward. More arms means the same traffic gets split thinner, which is why "just add more variants to test more ideas at once" quietly makes every technique on this list harder to run well.
- The gap between rungs 3 and 4 is the biggest jump on the ladder. Segmenting by a few dimensions is cheap; letting a model discover its own segments is not.
Effort vs. Reward
More data-hungry doesn't automatically mean better ROI. Each rung has a real engineering cost, and the reward it unlocks only shows up once your traffic clears that rung's data floor.
| Rung | Engineering effort | Reward ceiling | Realized only when |
|---|---|---|---|
| A/B Testing | Low | Low: single fixed winner, no personalization | You just need a conclusive answer |
| Multi-Armed Bandit | Low-medium | Medium-high: less wasted traffic, faster convergence | You have a few hundred+ conversions/month |
| Linear Contextual Bandit | Medium | Medium-high: personalizes across obvious segments | Segments individually clear the bandit's data floor |
| Non-Linear Contextual Bandit | High | High: captures interactions linear models miss | You have hundreds of thousands of events/month |
| Custom Trained Model | Very high | Highest, but only if the data supports it | You have millions of events and a standing ML team |
The Decision Framework
Start at the highest rung your traffic clears today, not the highest rung that exists. A non-linear contextual bandit running on 5,000 events a month isn't more sophisticated than a plain multi-armed bandit. It's a multi-armed bandit with worse convergence and more code to maintain.
Don't skip rungs. Each one validates an assumption the next one depends on. If a linear contextual bandit isn't beating a context-free bandit on your traffic, a non-linear one won't either. It'll just take longer to tell you that.
Revisit the ladder as you grow, not as a one-time decision. The rung that made sense at 10,000 monthly events won't be the ceiling at 500,000. Re-run the numbers above periodically rather than assuming today's setup is permanent.
Most teams should stop at rung 2 or 3. The jump to non-linear contextual bandits or a fully custom trained model is a real engineering commitment, and it only pays for itself once your traffic (and your patience for building and maintaining an ML pipeline) clears a very high bar. If you're not sure whether you're there yet, you're probably not.
Ready to try algorithmic testing?
Stop wasting traffic on losing variants. Bandit's multi-armed bandit algorithms automatically shift traffic to your best-performing treatments in real time.
Get Started Free