Business Decisions (A/B)

The question

An online store had nine growth hypotheses and no way to choose between them. Two questions had to be answered in order: which hypothesis deserves the first test, and once that test has run, is the observed difference real or noise?

The data

Nine scored hypotheses, each with Reach, Impact, Confidence and Effort. Plus order and visit logs covering 1–31 August 2019 — 31 consecutive days with no missing day in either group, which is more than four complete weekly cycles and dilutes day-of-week seasonality. Order values run from US$5.00 to US$19,920.40.

Method, and why this method

Prioritisation was scored twice — once with ICE, once with RICE — deliberately, because the two frameworks disagree and the disagreement is the finding. RICE adds Reach, which ICE ignores entirely.

Before testing anything, 58 visitors turned out to be present in both group A and group B — 5.6% of the 1,031 unique visitors, but 181 orders, 15.1% of the total. They were removed: a user exposed to both versions gives an order that cannot be attributed to either treatment, and keeping them would violate independence between groups, pushing A and B artificially toward each other and masking any real difference. Orders fell from 1,197 to 1,016, and the split stayed balanced (89 removed from A, 92 from B, leaving A=468 and B=548).

That fix has a known limitation worth stating rather than hiding. visits_us.csv is aggregated by date and group with no visitorId, so those same 58 users cannot be removed from the visit side: the numerator of the conversion rate loses 181 orders while the denominator still counts all their sessions. Conversion is therefore understated in both groups and should be read as a floor, not as the true level. Because the contamination split almost symmetrically between A and B, the comparison between groups stays valid — it is the absolute level that is displaced.

For the test itself the revenue distribution is heavily skewed by a small number of very large orders, so a mean-based test would report whatever those few orders happened to do. The analysis therefore uses non-parametric testing on both raw and filtered data, and compares the two results rather than trusting either alone. The filter comes from the data: the 99th percentile sits at 2 orders per user (only 7 users are above it) and at US$830.30 per order (9 orders are above it).

Findings

Swapping ICE for RICE reorders the ranking almost completely. Under ICE the leader is a birthday-discount promotion (16.2), followed by adding two traffic channels (13.3) and adding a subscription form to every page (11.2). Under RICE the subscription form jumps to 112.0 and takes the lead, precisely because its reach is high — while the birthday promotion, the ICE winner, drops to fifth place. Its score is unchanged at 16.2; it simply lacks the reach to stay ahead once RICE accounts for it.

On the test itself, the cumulative curves showed two different behaviours. Revenue and average order value looked like a wide win for group B — +27.8% on average order value in the raw data — but that entire advantage was born from a single order of US$19,920.40; without it, B's average order value drops below A's. Conversion behaved nothing like that: a smooth climb that crossed A's line on 7 August and stabilised through the second half of the month, averaging +15.7% with a standard deviation of only 2.6 percentage points, and closing at +16.0% for B with no one-off event holding the number up.

The Mann-Whitney tests confirmed exactly that division. Conversion rejects H0 on raw data (p = 0.0110) and on filtered data (p = 0.0070), and the relative difference does not shrink when the anomalies come out — it grows, from +16.0% to +18.9%. That is the opposite of how an outlier artefact behaves. Average order value never rejects H0 (p = 0.8622 raw, p = 0.8220 filtered), and its relative difference collapses from +27.8% to −3.2%, changing sign, once the US$19,920.40 order and the other anomalies leave the calculation.

The decision it enables

Prioritise by RICE when reach varies widely between hypotheses, which is exactly when ICE is most misleading. The subscription form ships first.

Stop the A/B test and declare group B the winner — on conversion alone. It is the only metric that holds up in both scenarios, and it strengthens rather than weakens when the anomalies are removed. Revenue is deliberately not used as an argument for B, because it survives neither the significance test nor the removal of the outlier. Continuing the test would add nothing: conversion was not a trend still developing, it had already settled by the second half of the month, so more days would not change the conclusion.

Links

Repository