Your last signup test came back +5%. Nobody made a slide about it.
Your onboarding test came back +6%. "Directionally positive, not significant." Archived.
Your pricing page test came back +7%. Same story.
Three tests, three shrugs. Put them end to end and they are a 19% lift in paid conversion. That is where most teams leave money on the table: not in the one big win they keep hunting for, but in the small ones they keep throwing away.
The math nobody puts on a slide
Conversion through a funnel is multiplication, not addition. Here is a simple three-step path. The numbers are illustrative, not customer data.
| Step | Before | After | Relative lift |
|---|---|---|---|
| Visit → sign up | 4.0% | 4.2% | +5% |
| Sign up → activate | 40% | 42.4% | +6% |
| Activate → pay | 25% | 26.75% | +7% |
| Visit → pay | 0.40% | 0.48% | +19% |
1.05 × 1.06 × 1.07 = 1.19.
On 20,000 visitors a month, that is 80 paying customers becoming about 95. On a $100-a-month plan, roughly $1,500 more MRR, every month, from three changes nobody called a win.
No single step moved more than 7%. The end result moved 19%. Add a fourth step, say upgrade or renewal, at +5% and it is 25%.
Why the small lifts get thrown away
Nobody decides to waste them. The default way of testing does it for you.
A small lift is expensive to prove. With a fixed 50/50 test, detecting a 5% relative lift on a 4% signup rate takes about 150,000 visitors per version at the usual settings (95% confidence, 80% power). At 20,000 visitors a month that is over a year. So the test gets called flat and stopped, and the lift goes with it.
While you wait, half your traffic sees the worse version. A fixed split doesn't care that you already suspect B is better. Every visitor still on A is money you chose not to collect so the test could finish.
Each step gets graded on its own number. A louder landing page pumps signups and sends weaker accounts downstream. The signup test wins. Paid conversion loses. Step-level metrics celebrate the wrong thing, and the compound runs backwards.
The average hides who is buying. A test that is flat overall can be a real lift for mobile visitors from search and a drop for everyone else. Average them and you get zero, and another archived ticket.
None of this is an argument against testing. It is an argument against a setup that only pays out when one step produces a number big enough to prove fast.
What captures the compound instead
Score every step on the conversion that pays you. If signup is graded on accounts created, it will learn to create low-intent accounts. If it is graded on eventual purchase, it learns to send people who buy. The funnel experiments guide walks through linking steps so the final conversion feeds back to each one.
Stop making traffic wait for a verdict. A bandit won't make a 5% lift easier to prove. It changes what happens while you can't prove it yet: traffic keeps leaning toward the version that is earning, with some exploration left so the ranking can still move. You collect most of the lift during the test instead of after a verdict that may never come.
Test the page as a page. A headline and a button that each look fine alone can be the worst pair on screen together. Whole-page experiments test every slot at once and credit one payout to everything the visitor saw.
Look for the group the average is hiding. Before you archive a flat test, check whether it is flat for everyone. Audiences surfaces the visitor groups that respond differently, with the counts behind each one.
What to do Monday
- Write down the path that makes money, step by step, with today's rate at each step.
- Multiply it out. Then put +5% on every step and multiply again. That gap is what is on the table.
- Reopen the last three tests you called flat. If they were positive and underpowered, they weren't failures. They were unclaimed money.
- Pick the two leakiest steps and grade them on the final conversion, not their own click.
- Keep allocation moving instead of waiting for a winner to freeze.
One big win is a slide. Five small ones stacked is the quarter.
Bandit runs Thompson sampling on each step and can feed your final conversion back to every step before it. Start a 30-day trial; no charge until day 31.
Ready to try algorithmic testing?
Stop wasting traffic on losing variants. Bandit's multi-armed bandit algorithms automatically shift traffic to your best-performing treatments in real time.
Try it on my site