More Orders Isn't Always More Money

You ship a 20% off banner on the product page. Orders jump. The test dashboard calls it a winner. You roll it out.

Then you look at the month.

Units are up. Revenue is flat. Contribution margin is down. The "winning" version was selling more of the product that barely clears cost, or at a price that barely clears cost. Your bandit did exactly what you asked. You asked it to count purchases.

That is the quiet failure mode of conversion optimization: the number you train on is not always the number that pays you.

Three different wins

When you set up a funnel, you pick what counts as a win. Most marketers pick one of three, even if they don't say it out loud.

A conversion event. Did the visitor do the thing? Purchase, signup, booked demo, add to cart. Every success is worth 1. A $12 order and a $240 order look the same. A full-price sale and a loss-leader sale look the same.

Revenue. Did they buy, and for how much? A $240 order teaches the algorithm more than a $12 one. Discounts that cut the ticket size start to look worse, even when they convert more often.

Profit (revenue minus cost of goods). Did they buy, and what did you keep after unit cost? A high-converting discount that sells below cost is a loss. A quieter full-price pitch that sells less often but clears more margin per visitor can win.

Those three goals will often rank the same variants in different orders. The algorithm is not confused. It is obedient.

An illustrative example

Made-up numbers, real pattern. Same product page, three banners, 1,000 visitors each:

Train on conversions, and 40% off wins. Traffic moves to the version that loses $2.40 every time someone buys.

Train on revenue, and 40% off still wins: more orders at a lower price still beats the full-price banner on top line. You grow sales while shrinking what you keep.

Train on margin, and no discount wins. Twenty percent off is second. Forty percent off is a loss, and the bandit should treat it like one.

Same page. Same visitors. Three different "winners," depending on the scoreboard.

Why marketers usually pick the wrong one

Conversion is the default because it is easy. Your analytics already count purchases. Agencies report conversion rate. Your last A/B tool optimized for it. And for some goals, it is the right answer: a booked demo, a free trial start, a waitlist signup. There is no dollar value on the event yet, so counting the event is honest.

The trouble starts on storefronts, checkout flows, and any page where the thing you are testing changes price, mix, or which SKU people reach for. A louder "Sale" badge, a cheaper plan in the comparison table, a free-shipping threshold that fills the cart with low-margin add-ons. Conversion goes up. The P&L does not clap.

Revenue is the middle ground. It catches ticket-size problems. It still ignores cost. If your catalog has fat-margin and thin-margin products, or if COGS moved when you changed suppliers, revenue will cheer for volume in the wrong aisle.

Margin is closest to the business. It is also the pickiest. You need unit cost on the products you sell. Miss the cost, and you cannot score the order honestly. Guessing zero makes every discount look free.

What changes for the algorithm

A bandit learns from the reward you give it after each visitor. Change the reward, and you change what it explores toward.

Event rewards learn fastest. Every conversion is a clean 1 or 0. You get the most signal per visitor. On a low-traffic page, that matters. The tradeoff is blindness to value and cost.

Revenue is noisier. One whale order can make a weak variant look brilliant for a day. You need more traffic before the ranking settles. The tradeoff is worth it when ticket size is the thing your variants actually move.

Margin is closest to the real goal, and the noisiest of the three when costs are incomplete. An order without a known cost should not pretend to be profitable. Negative margin has to stay negative, or a loss-leader will keep collecting traffic it did not earn.

None of this replaces funnel thinking. Small lifts still compound across steps. But compounding the wrong metric just accelerates the wrong outcome. Five steps each optimized for "more orders at any price" will happily compound you into a busy, unprofitable month.

How I think about the choice

Pick the scoreboard from the business, not from the tool's default.

• Demo / signup / lead form: train on the event. There is no honest dollar yet. If lead quality varies a lot, grade on a later event (qualified opportunity, closed-won) once you can attribute it.
• Checkout or pricing page where price or plan mix moves: train on revenue at minimum. Prefer margin if you have unit cost.
• Shopify or any catalog with real COGS: train on margin when you are testing offers, badges, or banners that change willingness to pay. Otherwise you will crown the steepest discount.
• Agency reporting to a client: ask what the client banks. Conversion rate makes a pretty slide. Contribution margin makes payroll.

And be honest about data gaps. If half your SKUs have no cost set, a margin-trained test will under-credit those orders. Fix the costs, or stay on revenue until you can.

What Bandit does with this

Bandit lets you set the payout on a funnel or campaign to one of three units: count, revenue, or margin.

Count credits 1 per conversion and ignores any dollar you send. That is the right default for a booked demo or a signup.

Revenue credits the order value you send with the reward.

Margin credits contribution margin: revenue minus cost of goods (and it can go negative when you sell below cost). You send cost with the order, either as a total or as unit cost on each line. Orders without a known cost wait until the cost shows up, instead of guessing.

On Shopify, placement tests can score the same way from the Cost per item field on your products, so a discount banner that wins on orders and loses on margin gets treated as a loss.

The point is not that margin is always better. The point is that the bandit will climb whatever hill you point it at. Point it at purchases, and it will find more purchases. Point it at the money you keep, and it will protect that instead.

The short version

Your test did not lie. You asked "which version gets more of the event?" and it answered. If what you actually needed was "which version makes more money after cost?", that was a different question.

Conversion for leads. Revenue when ticket size moves. Margin when cost matters. Then let the small lifts compound on a scoreboard your P&L would recognize.