Statistical Significance

The probability threshold used in online controlled experiments (A/B tests) to judge whether an observed difference between a control and a variant is unlikely to have arisen by chance. In ecommerce it is the primary gate for deciding whether to ship a product change, and also one of the most widely misunderstood metrics in the discipline.

What it actually measures

Statistical significance in A/B testing is a frequentist null-hypothesis significance test (NHST) output. When a testing dashboard declares "95% chance of beating original," it is asking: how often would we see a difference this large by chance if there were actually no difference between variants? (Optimizely glossary, 2026-06-18)

The 95% threshold — equivalent to p ≤ 0.05 — is the most common convention in CRO. High-stakes tests may use 99%; fast-iteration programs sometimes lower to 90%. This threshold is an industry convention, not a statistical law. (CXL, updated 2022-12-20)

What statistical significance does NOT tell you:

• Whether the effect is commercially meaningful (see Minimum Detectable Effect)
• Whether the change will produce the same lift after deployment
• Whether the uplift is worth the cost of implementation
• Whether the result is even real — see False Discovery Rate below

(RevLifter, 2025-07-15)

The four interdependent levers

Four parameters define any hypothesis test; changing one changes all the others:

Lever	Typical setting	What it controls
α (significance level)	0.05 (5%)	Maximum acceptable Type I error rate (false positive)
Power (1 − β)	0.80 (80%)	Probability of detecting a real effect of the specified size
MDE	varies	Minimum difference worth detecting commercially
Sample size	calculated	Volume of visitors/conversions required

Jacob Cohen established 80% power as a reasonable balance between Type I and Type II risk; it is still the industry default as of 2026. (CXL Statistical Power, 2023-08-18)

Sample size is the "800-pound gorilla of power" — lowering MDE from 10% to 5% requires approximately 4× more traffic for the same confidence and power. (SplitMetrics, 2024-03-28)

Ecommerce-specific sample size reality

With a 2% baseline conversion rate and a goal of detecting a 10% relative improvement, approximately 39,000 visitors per variation (78,000 total) are required at 95% confidence and 80% power. Most ecommerce teams run underpowered tests. (RevLifter, 2025-07-15)

Statistical frameworks

Three methodological approaches are used in production experimentation platforms as of 2026:

1. Fixed-horizon frequentist (classical)

The original A/B testing method, derived from agricultural experiment design. Requires:

• Pre-specified sample size
• A single end-point analysis (no peeking during the test)
• Pre-declared primary metric and hypothesis

Pros: widely understood; regulator-accepted; simple to audit. Cons: peeking severely inflates false positives; requires traffic pre-commitment. (Optimizely Support, 2026-05-09)

2. Sequential testing (mSPRT / Stats Engine)

Developed at Optimizely (2015) using the mixture Sequential Probability Ratio Test from David Siegmund (Stanford, 1985). Key properties:

• Always-valid p-values at any stopping point — peeking is mathematically permitted
• False Discovery Rate (FDR) control, not just false-positive rate
• No MDE pre-specification required
• If true lift is 7.5pp above the pre-specified MDE, Stats Engine is ~75% faster than fixed-horizon; for experiments >50,000 visitors, can be up to 2.5× faster

In 48,000 historical Optimizely experiments: fixed-horizon changed its winner/loser declaration in 44% of tests post-launch; Stats Engine changed in only 6% — indicating far more stable conclusions. Sequential testing remains Optimizely's default method as of 2026. (Optimizely Stats Engine blog, 2015, updated 2025-12-30; Optimizely Support, 2026-05-09)

3. Bayesian

Asks a different question: "What is the probability that variant B is better than variant A?" — directly interpretable without the contortions of p-value logic. (Dynamic Yield, 2024-01-31)

Key Bayesian properties:

• No pre-specified sample size required
• Output is intuitive probability and "expected loss" (risk of being wrong)
• Requires a prior distribution; non-informative priors (all rates equally likely) are used by VWO SmartStats to avoid prior-specification bias
• VWO SmartStats claims 50% reduction in testing time vs fixed-horizon (as-of 2024-12-03)

(VWO SmartStats, 2024-12-03)

As of 2026-05-09, Optimizely has added a Bayesian engine as a beta option alongside its default sequential (Stats Engine) method. (Optimizely Support, 2026-05-09)

Common pitfalls

Peeking

Checking results mid-test and stopping at first significance inflates the false positive rate far beyond the nominal 5%:

Checking frequency	Actual false positive rate
Every visitor	~57%
Every 500 visitors	~26%
Every 1,000 visitors	~20%
Sequential test (mSPRT)	~3%

(as-of 2026, Optimizely simulation of A/A tests to 5,000 visitors; [Optimizely Stats Engine blog](https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/))

Practical ecommerce impact: 1 in 4 "winners" stopped at first significance are pure noise under typical daily-check behaviour. Peeking also inflates the measured effect size — the estimate at the moment of early stopping overshoots the true lift. (Towards Data Science, 2026-03-11)

P-value misinterpretation

The most common error: treating (1 − p) as "the probability that variant B is truly better than A." That requires Bayes' theorem and a prior on the historical test success rate, which is not what the p-value supplies. (Ronny Kohavi / Lenny's Podcast, 2023-07-27; GrowthBook docs, current)

Multiple comparisons

Testing 20 metrics simultaneously at α=0.05 produces a ~64% probability of at least one spurious significant result by chance. (Towards Data Science, 2026-03-11; GrowthBook docs)

Corrections: Bonferroni (conservative; controls family-wise error rate); Benjamini-Hochberg (recommended for ecommerce; controls False Discovery Rate at higher power). (Statsig blog)

False Discovery Rate in practice

In Ron Berman's peer-reviewed empirical study (Management Science, 2022) of real ecommerce A/B tests: roughly 18–25% of tests declared significant at α=0.05 are actually null. The root cause: approximately 70% of ecommerce experiments test truly null effects. Even a correctly-calibrated 5% per-test false positive rate, applied to a programme where most tests have no real effect, produces a material proportion of false "winners." (Berman, Management Science 2022)

Sample Ratio Mismatch (SRM)

Microsoft found ~8% of experiments had Sample Ratio Mismatch (SRM)|SRMs — a mismatch between the intended and actual traffic split — that caused invalid results. Third-party companies adopting SRM checks found 6–10% rates. Most results run before SRM checks were introduced were invalid. (Ronny Kohavi / CXL LIVE 2016)

Other documented pitfalls

• Winner's curse: tests stopped early show inflated effect estimates that shrink post-launch
• Simpson's Paradox: a trend in aggregate data reverses when the data is split by segment (e.g., desktop vs mobile) — critical for ecommerce with heterogeneous user populations
• Novelty bias: new UI elements attract disproportionate attention from existing users; the effect decays after launch
• Seasonal confounds: Black Friday traffic behaves materially differently from January; promotional tests must account for weekly and seasonal variation
• Client-side flickering: when JavaScript loads slowly, users briefly see control before variant loads — can inflate apparent lift
• Goodhart's Law: optimising a proxy metric (e.g., add-to-cart) may break its correlation with the actual goal (purchase) — "when a measure becomes a target, it ceases to be a good measure"

(GrowthBook docs; RevLifter, 2025-07-15)

Variance reduction methods

Variance reduction techniques reduce the required sample size for a given power level without collecting more data, by removing extraneous noise from the metric estimator.

CUPED (Controlled-experiment Using Pre-Existing Data)

Introduced by Microsoft Research (Deng, Xu, Kohavi, Walker) in 2013. Mechanism: adjusts each user's observed metric by their pre-experiment behaviour, removing variance unrelated to the treatment. Mathematically equivalent to ANCOVA / OLS regression; the adjusted metric is Ŷ = Y − (cov(X,Y)/var(X)) × X. (Towards Data Science, 2021, updated 2025-01-24; Optimizely Field Notes, 2025-04-17)

Results from Kameleoon analysis of 200+ experiments (Q1 2023 as-of): for experiments with >5% returning visitors, CUPED delivers 5–40% variance reduction, translating to 10–60% reduction in required sample size. (as-of 2023-Q1, [Kameleoon, 2023-04-21](https://www.kameleoon.com/blog/accelerate-experiment-results-cuped))

Variance reduction is proportional to R²: Var(δ_VR) = Var(δ_DiM) × (1 − R²). If the pre-experiment covariate correlates at 0.7 with the metric, variance falls by ~51%. (Microsoft Research ExP, 2022-11-15)

Limitations: fails for new users with no pre-experiment history; best for high-variance continuous metrics (revenue, AOV, session duration) rather than binary conversion; requires committing before launch.

CUPAC (ML covariate predictions)

Uses machine-learning predictions of the metric as covariates rather than raw historical values. At Vinted (18 historical experiments, 20M+ user-metric observations, 2022–2024): CUPAC achieved 38.5% confidence interval width reduction — the highest of four methods tested — but is 21× more computationally expensive than outlier capping. (KDD-UMC'25, August 2025)

Outlier capping

Capping extreme values (e.g., at 5 standard deviations from mean) achieved 36.7% CI width reduction in the Vinted study — nearly as effective as CUPAC but the fastest method computationally. Recommended as the first-step variance reduction approach for teams with limited compute or no pre-experiment history. (KDD-UMC'25, August 2025)

Summary (Vinted, KDD 2025, as-of 2025-08)

Method	CI width reduction	Relative compute cost
CUPAC	38.5%	21.2×
Outlier capping	36.7%	1× (baseline)
CUPED	21.2%	5.6×
Doubly Robust	21.1%	31.2×

All four methods produced unbiased estimates. CUPED and CUPAC improve more with larger samples; outlier capping is more robust to noise. (as-of 2025-08, [KDD-UMC'25](https://kdd.org/kdd2025/wp-content/uploads/2025/07/CameraReady-05.pdf))

Scale of ecommerce experimentation

• Google, Amazon, Facebook, Booking.com each run 10,000+ controlled experiments annually. Bing runs 1,000+ A/B tests monthly. Google, LinkedIn, and Microsoft collectively launch more than 20,000 A/B tests per year. (as-of 2023, [Kohavi et al. 2020, cited KDD 2025](https://kdd.org/kdd2025/wp-content/uploads/2025/07/CameraReady-05.pdf))
• Failure rates at scale: Microsoft ~70%, Bing ~85%, Google Ads ~90%, Netflix ~90%, Airbnb ~90% of experiments show no positive effect — described by Kohavi as normal and expected, not a culture failure. (as-of 2023, [Kohavi / Lenny's Podcast](https://www.youtube.com/watch?v=hEzpiDuYFoE))
• At Microsoft broadly, one-third of experiments actively hurt the metrics they intended to improve. (Towards Data Science, 2026-03-11)
• Retail companies allocate more than 90% of their traffic to testing — the highest traffic allocation of any sector studied. (as-of 2024, [VWO, updated 2025-05-02](https://vwo.com/blog/ab-testing-statistics/))
• 84% of marketers run A/B tests at least monthly; 38% test weekly (as-of 2025, Ascend2 2025 A/B Testing Report, cited [Kameleoon, 2026-04-30](https://www.kameleoon.com/blog/a-b-testing-experimentation-stats-you-need-to-know))
• 52.8% of CRO professionals lack a standardised stopping point for A/B testing (as-of 2024, Enterprise Apps Today, cited [VWO, updated 2025-05-02](https://vwo.com/blog/ab-testing-statistics/))

• In a meta-analysis of 1,001 A/B tests, only 33.5% produced a statistically significant positive result (as-of analysis date, [Analytics Toolkit meta-analysis, cited Kameleoon 2026-04-30](https://www.kameleoon.com/blog/a-b-testing-experimentation-stats-you-need-to-know))

What practitioners report

• "If your team is running frequentist tests and checking results every day, your false positive rate is significantly higher than your stated confidence level." Most ecommerce teams are in this state. (Atticus Li, 2026-03-31)
• The ACM Interactions account of an ecommerce department switching from frequentist to Bayesian: stakeholders misread confidence as "certainty of uplift." Switching to revenue-framed Bayesian output ("with 95% probability we will make at least €10,000/month extra") improved decision quality. (ACM Interactions, February 2022)
• Minimum user base for reliable A/B testing: "tens of thousands" to start; 200,000 users is the threshold where 5% effects become reliably detectable, per Kohavi. (Kohavi / Lenny's Podcast, 2023-07-27)
• Ecommerce companies sometimes implement at 94% confidence when the business case is compelling enough — business impact overriding the statistical threshold. (RevLifter, 2025-07-15)
• Most Optimizely users are actually running sequential frequentist (Stats Engine), not Bayesian — describing it as "Bayesian" to stakeholders is methodologically incorrect and causes confusion. (Atticus Li, 2026-03-31)
• "Getting numbers is easy; getting numbers you can trust is hard." — Kohavi's formulation of the central challenge of ecommerce experimentation. (Kohavi / CXL LIVE 2016)
• For low-traffic ecommerce sites unable to reach sample size within 4 weeks: practitioner workarounds include lowering confidence threshold to 80–70%, testing more aggressive changes (larger MDEs), using micro-conversions as proxy primary metrics, or switching to Bayesian or sequential methods. Counter-position: lowering threshold and using micro-conversions changes what is being measured, not how confidently — a test at 80% confidence on add-to-cart has not established that purchase revenue will follow. ([multiple practitioner sources])

Key terms

Term	Meaning
α (alpha)	Significance level; maximum acceptable Type I error rate (false positive probability)
β (beta)	Type II error rate; probability of missing a real effect; power = 1 − β
MDE	Minimum Detectable Effect; smallest lift worth designing the test to detect
p-value	Probability of observing results this extreme if the null hypothesis (no difference) is true
FDR	False Discovery Rate; proportion of declared significant results that are actually null
SRM	Sample Ratio Mismatch (SRM); mismatch between intended and actual traffic split
CUPED	CUPED; variance reduction via pre-experiment covariate
mSPRT	Mixture Sequential Probability Ratio Test; mathematical basis of Optimizely's Stats Engine
OEC	Overall Evaluation Criterion; Kohavi's term for the single primary metric causally predictive of long-term user value
Peeking	Stopping a test early based on interim results; inflates false positive rate under fixed-horizon methods
Winner's curse	Inflated effect estimate produced by early stopping; causes revenue projections to overshoot
Twyman's Law	"Any figure that looks interesting is usually wrong" — check implementation before celebrating a large observed lift