Price test: profit per visitor and significance
Compares two price levels by profit per visitor and not by the number of orders: the expensive variant almost always gets fewer orders but may earn more. Counts the conversion, profit per visitor and the significance of the difference both by conversion and by profit, and suggests the decision. The test should be run on your own site: platforms restrict A/B tests of price.
At a glance
- What question does it answer?
- Which price brings more profit per visitor and whether the difference is significant and not just whether there are more orders.
- What data is needed?
- For each variant: the price, visitors and orders.
- How to read the result?
- Decide by profit per visitor and its z; if conversion falls while profit rises, that is normal for a higher price. Example on the default values: Conversion A: 3.2%; Conversion B: 2.8%; z for conversion (B − A): −2.34; Profit per visitor A: RUB 47.06. The figures are illustrative: enter your own to see your result.
Why and how to reason
The expensive variant almost always gets fewer orders but may earn more, so profit per visitor and the significance of the difference are compared. In the example conversion is 3.20% and 2.80% (z = −2.34), profit per visitor 47.06 and RUB 52.65, z for profit 1.96: at the significance bound.
When to use it
Apply when checking a new price on your own site, a test of at least a week.
How to read the result
Decide by profit per visitor and its z and not by the number of orders.
Common mistakes
They choose a variant by the number of orders and stop the test earlier than a week.
What data is needed?
- For each variant: the price, visitors and orders.
- The cost excl. VAT and the VAT rate.
- The test duration of at least a week.
- An example: A RUB 3,990, 20,000 visitors, 640 orders; B RUB 4,490, 20,000, 560; cost RUB 1,800: conversion 3.20% and 2.80% (z = −2.34), profit per visitor RUB 47.06 and RUB 52.65 (z = 1.96, at the significance bound) (illustrative numbers).
Inputs
| Field | Unit | Value in the example | Meaning |
|---|---|---|---|
| Price A incl. VAT | ₽ | 3,990 | The price on the site for the buyer. The example value is illustrative. |
| Visitors A | people | 20,000 | How many unique visitors saw the variant. The example value is illustrative. |
| Orders A | orders | 640 | Placed orders. The example value is illustrative. |
| Price B incl. VAT | ₽ | 4,490 | The price of the new variant. The example value is illustrative. |
| Visitors B | people | 20,000 | How many unique visitors saw the variant. The example value is illustrative. |
| Orders B | orders | 560 | Placed orders. The example value is illustrative. |
| Unit cost excl. VAT | ₽ | 1,800 | With redemption and unit costs if they are noticeable. The example value is illustrative. |
| VAT rate in the price | % | 22 | A guide on a date; the rate depends on the goods. Check against the act in force. The example value is illustrative. |
| Days of the test (optional) | days | 14 | For a warning about a short test. Empty means do not check. The example value is illustrative. |
How it is calculated
| Step | Formula |
|---|---|
| Profit per visitor | conversion × (price ÷ (1 + VAT) − cost) |
| z for conversion | (CR_B − CR_A) ÷ √(p(1 − p)(1/n_A + 1/n_B)), p is the pooled share of orders |
| z for profit | difference of profits ÷ √(SE_A² + SE_B²), SE = profit per order × √(CR(1 − CR) ÷ n) |
Worked numbers
Default values from the app; in the interactive version you enter your own.
- Conversion A
- 3.2 %
- Conversion B
- 2.8 %
- z for conversion (B − A)
- −2.34p-value 1.9 %.
- Profit per visitor A
- 47.06 ₽
- Profit per visitor B
- 52.65 ₽
- Difference in profit per visitor (B − A)
- +5.59 ₽
- z for profit (B − A)
- 1.958p-value 5.02 %; the 5% bound: |z| ≥ 1.96.
- Profit per 1,000 visitors: difference
- +5,593 ₽
Data as a table
| Measure | Profit per visitor |
|---|---|
| A | 47.1 |
| B | 52.7 |
The difference in profit per visitor is not significant at the 5% level: continue the test until data accumulates or consider the variants equal in profit.
Conversion and profit per visitor point in different directions: decide by profit and not by the number of orders.
What to do with the result: switch the price only at a significant difference in profit and watch returns and repeat purchases after the test.
A typical mistake: choosing a variant by the number of orders and stopping the test earlier than a week.
Nuances and limits
- Choosing a variant by the number of orders.
- Stopping the test earlier than a week.
- Testing price on a platform where A/B testing of price is restricted.