The Shopify app: test theme copy, scored on margin

The discount that sells more and earns less

You run a 20% off banner and orders go up. The test dashboard calls it a winner. Then you look at the month and the money is flat. The extra orders came in at a price that left little after the cost of the goods.

Most testing tools count conversions. A conversion counts the same whether it earned you $16 or lost you $2. If the thing you are testing changes the price, a conversion count can pick the version that loses money.

Bandit's Shopify app tests the copy in your theme and can score each version by contribution margin instead of order count.

What the app does

The app adds a Bandit Text block to the theme editor. You put the block where the copy goes and choose its placement. There are five:

• Hero headline
• Hero subhead
• Primary CTA
• Banner text
• Badge text

Each placement holds one active test at a time. In the app, you write the versions for that placement, or draft them with the AI variant chat, and start the test. Bandit then decides which version each shopper sees and moves traffic toward the version that is doing best.

The block always renders the default text you typed in the theme editor first. If no test is running for that placement, or Bandit has not synced to your store yet, the default text stays. A test can't leave a blank space on your page.

How it picks a version without slowing the page

The storefront never waits on Bandit. Every 5 minutes, Bandit publishes a snapshot of each active test's traffic split into a metafield on your shop. The theme reads that snapshot when the page loads, picks a version in the browser, and swaps the text in place. The impression is reported afterwards. Nothing runs before first paint.

The script swaps text, never HTML. A version can't inject markup into your theme, even one an AI drafted.

How margin is counted

When a test's reward basis is set to margin, Bandit listens for Shopify's order and refund webhooks and works out each order's contribution margin:

margin = net revenue - cost of goods + (shipping charged - shipping cost)

Net revenue is each line's price minus that line's own discounts. Cost of goods is the variant's unit cost times the quantity, read from the Cost per item you set on the product in Shopify. Shopify's order webhook does not include what you paid the carrier, so shipping usually counts as zero. Bandit does not book the shipping you charged as profit.

The result can be negative, and Bandit keeps it negative. A discount that sells below cost counts as a loss. It is not rounded up to zero.

Say a store sells a $48 item with a $31.20 unit cost and tests three banners on 1,000 visitors each:

The bandit's score for each version is margin per visitor. That is the margin from orders tied to the version, spread over every shopper who saw it, including the ones who didn't buy.

What happens when the numbers are messy

Real order data has gaps. Here is how Bandit handles each one.

A variant with no unit cost. If any line in an order has no cost recorded in Shopify, Bandit can't know the order's margin. It stores the order and credits nothing. Guessing a cost, even zero, would make a full-price version look far more profitable than it is. An hourly job rechecks uncosted orders. Once you add the cost in Shopify, the order is credited.

A refund. Bandit never edits the original credit. It posts a second, negative result for the share of the order that came back. Refund one of two identical units and half the order's margin is reversed. Refunded shipping comes off in full. The total reversed for an order can never exceed the margin it earned, and Shopify retrying the same refund webhook does not debit twice.

An order edit. Shopify sends both orders/create and orders/updated for the same order. Bandit credits each order once per shop.

The margin floor

A margin test can also carry a floor, a minimum margin per visitor in cents. Once a version has been served at least 30 times, and its average margin is below the floor, Bandit stops showing it. Below 30 trials every version stays in play, so one bad early order can't knock a version out.

If the floor would exclude every version, Bandit falls back to the oldest version in the test, the same one every time. A shopper always gets a page. The storefront snapshot applies the same floor, so an excluded version drops out of the published split.

Limits

• Margin scoring is a setting on the test. A test without it counts conversions, and orders don't change its score. The editor's "What counts as a win?" menu offers Click on the tested element today. Add to cart and purchase are listed as coming soon. The editor has no switch for the margin setting or the floor yet.
• One attributed placement per page. A cart holds one Bandit attribute. If two margin tests run on the same page, the order is credited to the last one decided.
• Funnel experiments don't take margin credit. Margin applies to a placement test on its own. Experiments inside a funnel or campaign learn from the payout you send to the funnel.
• Shipping cost is usually absent. Shopify doesn't send carrier cost, so shipping nets to zero unless a real cost is supplied.
• Results show margin, not alerts. A margin test's results table adds a Margin / visitor column and says which version earns the most margin per visitor. When the version that converts most often earns less, the results say how much less per visitor. There is no in-app badge yet for a version paused by the floor, and no warning yet for a product with no cost set.
• Currency. Each order keeps its own currency. The app displays amounts in US dollars.

How to turn it on

1. Install Bandit from the Shopify admin. The app asks for order and inventory access, which it needs to read orders and unit costs.
2. In the theme editor, turn on the Bandit Storefront app embed.
3. Add a Bandit Text block where the copy goes. Choose its placement and type the default text.
4. In the app, choose New test. Pick the same placement and choose what counts as a win.
5. Write two or more versions, or open the variant chat to draft some, and choose Start test.
6. Set Cost per item on the products you sell, so orders can be scored by margin when the test's reward basis is margin.

Within 5 minutes the snapshot reaches your store and shoppers start seeing versions. The Shopify page has a short tour.

A discount that wins on orders can still lose on margin, and Bandit counts it as a loss.