A/B Testing with Low Traffic: Why Bandits Are the Better Choice

The Low-Traffic Problem

Here's a scenario that most growth teams face: you have an idea for a better headline, a new CTA, or a redesigned pricing page. You set up an A/B test. And then you wait. And wait. And wait.

With 500 daily visitors and a 5% baseline conversion rate, detecting a 2 percentage point improvement (5% → 7%) with 95% confidence requires approximately 3,600 total visitors, over a week of traffic split 50/50 between two variants.

During that week, roughly 900 visitors see the inferior variant. At a $50 average order value, that's $2,250 in lost revenue just to prove what you suspected after the first few days.

Calculate It Yourself

Use the calculator below to see how your specific numbers look. Plug in your baseline conversion rate, the minimum effect you want to detect, and your daily traffic.

The results are often sobering. Low-traffic sites face a harsh tradeoff: either run tests for weeks (or months) or accept that you can't detect small effects with confidence.

Why Traditional A/B Tests Need So Many Visitors

Statistical significance testing was designed for contexts where data collection is expensive: clinical trials, agricultural experiments, manufacturing quality control. The framework is powerful but rigid:

• Fixed sample sizes are calculated before the test starts
• Equal allocation between treatment and control (50/50)
• No peeking: you can't act on interim results without inflating false positive rates
• Binary outcome: you either reject or fail to reject the null hypothesis

These constraints are scientifically sound but operationally expensive when every visitor has value.

The Multiple Comparisons Problem

Low-traffic sites face another issue: they often need to test multiple variants. If you're testing 4 headline options, a traditional approach requires even more traffic because you need to correct for multiple comparisons (Bonferroni, Holm-Bonferroni, etc.), which raises the required sample size per variant.

Bandits handle multiple variants naturally, since they're designed for exactly this scenario.

How Bandits Help Low-Traffic Sites

Multi-armed bandits address low-traffic constraints directly:

1. Adaptive Allocation

Instead of a fixed 50/50 split, bandits shift traffic toward the winning variant as evidence accumulates. After 200 visitors, if Variant B is clearly winning, a bandit might allocate 80% of traffic to B and 20% to exploration.

This means fewer visitors see the losing variant, reducing lost conversions during the test period.

2. No Fixed Duration

Bandits don't have a "test duration." They continuously optimize. You don't need to calculate a sample size upfront, start the test, wait, then analyze. The algorithm is always running, always learning, always optimizing.

3. Regret Minimization

The objective function is different. A/B tests optimize for measurement precision. Bandits optimize for cumulative reward. For a business, cumulative reward (total conversions, total revenue) is usually what matters.

4. Anytime Results

You can check bandit results at any time without statistical penalty. There's no p-hacking risk because the algorithm isn't designed around hypothesis testing. It's designed around decision-making.

The Realistic Scenario

Here's a realistic example for a SaaS marketing site:

Setup:

• 400 daily visitors to the pricing page
• 3.5% baseline conversion rate (trial signups)
• Testing 3 headline variants + control
• Goal: improve conversion rate by at least 1.5 percentage points

Traditional A/B Test:

• Required sample: ~12,000 visitors (with Bonferroni correction for 4 variants)
• Duration: 30 days at 400/day
• Visitors sent to losing variants: ~9,000
• Lost conversions: ~270 (at control rate)

Bandit Approach:

• Starts optimizing from day 1
• Within 5 days (~2,000 visitors), traffic is 60%+ on the leading variant
• Within 15 days, traffic is 85%+ on the winner
• Lost conversions: ~80 (70% fewer than A/B)

The bandit doesn't give you a p-value after 30 days. It gives you more conversions during those 30 days while still identifying the best variant.

When You Still Need A/B Tests

Bandits aren't always the answer, even for low-traffic sites:

• You need a precise effect size for a business case or board presentation
• Regulatory requirements mandate specific testing methodologies
• You're making an irreversible decision and need high confidence before committing
• The cost of a wrong decision far exceeds the cost of delayed implementation

In these cases, accept the longer timeline and use proper A/B testing. The statistical rigor is worth the wait.

Practical Recommendations

For low-traffic sites (under 1,000 daily visitors):

1. Default to bandits for conversion optimization, CTA testing, and content experiments
2. Use Thompson Sampling, which performs best with limited data
3. Focus on large effects: don't try to detect 0.5% differences with 200 daily visitors
4. Run bandits continuously. Set up experiments and let them run indefinitely
5. Reserve A/B tests for major decisions that require measurement precision

The Bottom Line

Low-traffic sites don't have the luxury of running textbook A/B tests. The sample sizes are too large, the timelines are too long, and the opportunity cost is too high.

Multi-armed bandits offer a pragmatic alternative: they start optimizing immediately, waste less traffic on losing variants, and naturally handle multiple variants. You trade statistical precision for speed and cumulative performance (a tradeoff that makes sense for most optimization use cases).

Stop waiting for significance. Start optimizing.