On this page
concept

Geo Holdout Testing

Created 2026-08-11 31 connections

Geo Holdout Testing

A geo holdout test (also called a geo lift test, geographic incrementality experiment, or matched-market test) is a causal measurement method that splits a brand's addressable geography into non-overlapping treatment and control regions, runs or pauses advertising in the treatment group, and measures the difference in business outcomes between the two groups to isolate the channel's true incremental effect. Because it operates at the aggregate geographic level, it requires no user-level tracking — making it resistant to iOS 14+ and third-party cookie deprecation. (Mack Grenfell, 2021-10-23; Lifesight, 2026-08-10)

How it works

Geographic regions (DMAs, postal codes, states, Functional Urban Areas) are assigned to either a treatment group (ads run as normal) or a control group (ads paused or held back). During the test period, both groups continue to receive all other marketing activity unchanged. The incremental lift attributable to the channel is calculated as the difference between observed outcomes in the treatment group and the predicted counterfactual (what treatment performance would have been without the intervention) estimated from control group behaviour. (Amsive, 2026-05-27; Tatari, undated)

Key metrics (as-of 2026)

MetricFormulaSource
Incremental ConversionsTest Conversions – Counterfactual ConversionsLifesight, 2026-08-10
Lift %(Incremental Conversions / Counterfactual Conversions) × 100Lifesight, 2026-08-10
iROAS (Incremental Return on Ad Spend)Incremental Conversion Value ÷ Incremental CostGoogle Ads Help, 2026
iCPAΔ Spend ÷ Incremental ConversionsLifesight, 2026-08-10
Incrementality Factor (IF)Incremental Conversions ÷ Platform-Reported ConversionsHaus, 2025–2026

The Incrementality Factor (IF) is used to calibrate bid strategies: a platform-reported 4× ROAS × IF of 0.7 = true iROAS of 2.8×. If break-even requires 3×, the brand is losing money despite positive dashboard metrics. (Haus, 2025–2026)

Core methodologies

Time-Based Regression (TBR)

TBR learns the linear relationship between control and treatment groups during the pre-test period, then uses that model to predict the counterfactual during the test period. Bayesian inference provides posterior prediction intervals. Used in Google Ads Conversion Lift and as the launch methodology for Google Meridian GeoX. (Google Ads Help, 2026; developers.google.com/meridian/geox, 2026-05-14)

Trimmed Match

A methodology for randomized paired geo experiments, developed at Google and accepted by the Annals of Applied Statistics. Core contribution: when geo pairs are outliers and cannot be well-matched, the method trims the poorly-matched pairs at either the design stage or during post-experimental analysis, rather than discarding the entire study or accepting biased estimates. Open-source Python library available at github.com/google/trimmed_match. (Chen, Longfils & Remy, arXiv:2105.07060, 2021; Google Research)

Synthetic Control Methods (SCM)

Creates an artificial "synthetic" comparison unit from a weighted combination of untreated geographic regions, using historical data. Characterized by Athey and Imbens (2015) as "arguably the most important innovation in the policy evaluation literature in the last 15 years." Meta's open-source GeoLift library (now deprecated, ~2024) used SCM, specifically Augmented Synthetic Control Methods (ASCM) combined with Generalized Synthetic Controls (GSC). Haus (commercial platform) claims SCMs are "four times more precise than traditional matched-market tests." (Meta GeoLift documentation, Copyright 2025 Meta Platforms; Haus, 2025–2026)

TBR vs SCM precision. Haus (commercial platform) claims synthetic control methods are "four times more precise than traditional matched-market tests" (Haus, 2025–2026). Google Ads and Meridian GeoX use TBR as the primary methodology at launch, with Synthetic Control listed as a future roadmap addition — implicitly acknowledging TBR alone is not always sufficient (developers.google.com, 2026-05-14). No direct peer-reviewed head-to-head comparison has been published in accessible sources (as-of 2026-08-11).

Difference-in-Differences (DiD)

The standard analysis method used across most geo holdout implementations: compare the change in outcomes in the treatment group to the change in outcomes in the control group across the pre/post periods. (Amsive, 2026-05-27; Mack Grenfell, 2021)

Adaptive Supergeo Design (ASD)

A 2025 academic methodology (arXiv:2506.20499, June 2025, Charles Shaw) that solves the NP-hard problem of partitioning geographic units into "supergeos" for experiments. Uses a graph neural network (GNN) to learn geo embeddings and propose candidate supergeos, then a CP-SAT solver to select a partition that balances baseline outcomes and pre-treatment covariates. Claims >5 million times faster than exact methods for N=210 markets (0.22 seconds vs weeks). (Shaw, arXiv:2506.20499, June 2025)

Platform implementations (as-of 2026)

Uses Google Marketing Areas (GMAs) as experimental units. Produces iCPA and iROAS. Available across Video, Display, Discovery, Demand Gen, Search, Shopping, App, and Performance Max campaigns. In 2025, Google transitioned from Frequentist to Bayesian statistical methodology — improving study power and enabling tests with lower budgets and fewer conversions than traditional methods. Google reduced the minimum spend requirement to $5,000 in 2025 (as-of 2025), down from $100,000. (Google Ads Help, support.google.com, 2026; seresa.io search result, 2026)

Google Meridian GeoX

Announced at Google Marketing Live 2026 (2026-05-14). Open-source, publisher-agnostic geo-experiment tool. Three experiment design types: Holdback (pause ads in a small set of holdout regions while keeping them active elsewhere), Go Dark (pause media entirely in selected regions), Heavy Up (increase spend in selected regions). Native multi-cell design: compare multiple treatments against a common control in the same study, reducing total cost and time-to-insight. GeoX results convert into Bayesian priors to calibrate Google Meridian MMM models. Launch methodology: TBR + stratified sampling. Roadmap: Synthetic Control and Synthetic Difference-in-Differences (SDiD). Rolling out to pilot cohorts later in 2026. (developers.google.com/meridian/geox, 2026-05-14; business.google.com, 2026-05-14)

Meridian MMM holdout integration: the holdout_id boolean argument in Meridian marks holdout observations; KPI data from holdout rows is excluded from model training but media data (adstocked values) from those rows is still used. Guidance: balance holdout rows across geos and time periods; avoid large contiguous-in-time holdout blocks. (developers.google.com/meridian, 2026)

Meta GeoLift (deprecated ~2024)

Meta developed GeoLift as an open-source SCM-based geo experiment tool after iOS 14.5 (2021) made user-level conversion lift measurement impossible — SKAD Network postbacks require campaign ID, so control group conversions could not be attributed. Meta has since deprecated the GeoLift library (~2024). (Mack Grenfell, 2021-10-23; SegmentStream, 2025-02-21)

Meta Conversion Lift (current)

Meta's current Conversion Lift product is primarily a user-level holdout (audience randomization), not a geo-level holdout. The control_percentage parameter defines the holdout share; users in the control group are guaranteed not to see the ad (the second-place ad is served instead). Minimum requirements: 7-day test periods; at least 10% of audience in each test cell. (Meta for Developers, 2026; Haus, 2025–2026)

Third-party platforms (as-of 2026)

Unified Suites: Lifesight, Haus, LiftLab, Measured. Open-source: Meta GeoLift (deprecated), Google CausalImpact. Channel-specific: Google geo-based Conversion Lift, Meta Conversion Lift. (Lifesight, 2026-08-10; Shopiator, 2026-07-26)

Test design principles

Experiment unit selection

Geo units should be DMAs (US), postal code clusters, states, or Functional Urban Areas (EU). Zip codes are too small — users travel across boundaries, reducing cross-contamination prevention. Wayfair uses 210 mutually exclusive clusters of US ZIP codes rather than ZIP codes directly. (Wayfair Tech Blog, 2023-04-26; Mack Grenfell, 2021)

Market matching

Match treatment and control groups on: pre-period KPI levels, trends, seasonality patterns, and channel mix — not demographics alone. "Markets that move together" matters more than demographic similarity. A mandatory parallel trends check (plotting treatment vs control on the same axes in the pre-period) is required — divergence in the pre-period indicates a bad match. (Amsive, 2026-05-27)

Wayfair uses integer optimization rather than simple randomization to assign markets, minimizing the difference between holdout group share and a target share (e.g. 15%) across multiple metrics simultaneously. Integer optimization is susceptible to producing quasi-experimental bias, but Wayfair found this bias converges toward zero and is not distinguishable from randomized designs. (Wayfair Tech Blog, 2023-04-26)

Holdout size

Holdout size recommendations vary widely. Shopiator suggests starting small for budget efficiency (Shopiator, 2026-07-26). Amsive's illustrative case study used 7.6% of customer base (Amsive, 2026-05-27). Haus suggests holdouts of 20–50% of audience may be needed for adequate statistical power (Haus, 2025–2026). Wayfair explicitly rejects 50% holdouts due to opportunity cost (millions in lost revenue per day), targeting ~15% via integer optimization (Wayfair Tech Blog, 2023-04-26). No universal standard exists — the appropriate size is a function of baseline variance and the minimum detectable effect required.

Statistical power and test duration

Standard test duration: 4–8 weeks (Lifesight, 2026-08-10; Shopiator, 2026-07-26; Amsive, 2026-05-27). Minimum markets recommended: 20+ per arm for national channels (Amsive, 2026-05-27). Minimum outcome volume: 50–100 conversions per week in the control group (Shopiator, 2026-07-26).

MDE (Minimum Detectable Effect) formula: MDE ≈ (z_α/2 + z_β) × σ × √(1/n_test + 1/n_control). Standard thresholds: z_α/2 = 1.645 for 90% confidence; z_β = 0.842 for 80% power. (Amsive, 2026-05-27)

Pre-period should be equal to or greater in length than the test period for a clean DiD analysis. (Amsive, 2026-05-27)

Test duration upper bound. Most sources recommend 4–8 weeks as standard (Lifesight 2026; Amsive 2026; Shopiator 2026). Wayfair found that "the most important parameter to control is the test length" — longer tests rapidly increase the rate of false positive effects (Type I error), because TBR model variance is sensitive to pre-test time series length (Wayfair Tech Blog, 2023-04-26). Amsive's illustrative case study used 17 weeks (Amsive, 2026-05-27). No consensus on upper bound; depends on baseline variance and MDE required.

Decision thresholds must be set before the test runs and linked to specific budget actions. Example: "If we'd reallocate at +10% lift but not at +5% lift, the test needs to distinguish those two outcomes." (Amsive, 2026-05-27)

Minimum budget threshold (as-of 2026)

Accessibility for small advertisers. Shopiator and multiple sources suggest brands spending under $30,000–$50,000/month lack sufficient regional traffic density for a clean geo holdout test (Shopiator, 2026-07-26). However, Google reduced its minimum spend for geo-based Conversion Lift experiments from $100,000 to $5,000 in 2025 (as-of 2025), making platform-native geo tests accessible at lower budgets. The accessibility threshold depends on whether "geo holdout" refers to a DIY test, a platform-native test, or a third-party vendor product (seresa.io, 2026).

Failure modes

  1. Geographic spillover: Commuters crossing DMA boundaries, national media continuing to reach holdout regions (linear TV, branded search), or brand search halo in non-paused channels contaminate the control group. Several sources name spillover as a critical failure mode but no source quantified the frequency of invalidation or the typical magnitude of overestimation (as-of 2026-08-11). (Amsive, 2026-05-27; Mack Grenfell, 2021)

  2. Incorrect region selection (50/50 splitting): Randomly splitting all regions 50/50 by geography rather than by behavioural similarity produces imbalanced groups. Standard recommendation is randomization + stratification, not naive geographic bisection. (SegmentStream, 2025-02-21; Amsive, 2026-05-27)

  3. Seasonality breaking synthetic control weights: Synthetic control weights are built on pre-test data. If seasonal spikes hit mid-test, the weights break down and the synthetic counterfactual diverges from reality. (SegmentStream, 2025-02-21)

  4. Budget redistribution contaminating holdout regions: When pausing spend in holdout regions, platforms often redistribute that budget to other regions including the control markets, inflating the synthetic control baseline and overstating lift. Requires explicit budget caps or campaign restructuring. (SegmentStream, 2025-02-21)

  5. Underpowered tests: Designing around hoped-for lift rather than the minimum lift that would change a budget decision inflates false power estimates. Ignoring pre-period variance in power calculations leads to tests that cannot detect the true effect. (Amsive, 2026-05-27)

  6. Type I error from excessive test length: Longer test periods increase the rate of falsely detecting effects — a finding from Wayfair's production geo testing at scale. (Wayfair Tech Blog, 2023-04-26)

Whether geo tests are true A/B tests. SegmentStream (2025-02-21) argues geo lift testing is not a true A/B test — it relies on Synthetic Control, which can introduce as much bias as cookie-based MTA or MMM; the key failure modes (incorrect region selection, seasonality, budget redistribution) are cited as evidence, and Meta's deprecation of GeoLift is cited as further confirmation. Haus, Google, and the IAB/IAB Europe (Nov 2025) position geo holdout experiments as the strongest available causal measurement method — the "gold standard" for validating ROI of individual media investments.

What practitioners report

Wayfair runs geo experiments against 210 US geo units (mutually exclusive ZIP code clusters). Geo testing is "the least sensitive of the experiment types we typically run" — limiting utility for small channels or small expected effects. International geo testing is harder: UK, Germany, and Canada populations are more concentrated, making it difficult to divide into geos whose boundaries aren't frequently crossed; larger geo units reduce cross-pollination but reduce the available sample space. (Wayfair Tech Blog, 2023-04-26)

Expedia Group built an in-house R package (EGGX — Expedia Group's Geographic eXperimentation) covering: geo selection, treatment/control allocation, synthetic control build, bias checks, and KPI lift calculation. Post-test validation includes reconvergence checks (treatment and control return to parity after the campaign ends) to confirm model stability. Tests run across multiple KPIs simultaneously (revenue, customer volumes, app downloads, mailing sign-ups) to maximise ROI on expensive test design effort. (Medium / Expedia Group Tech, 2024-05-14)

Haus (640 experiments dataset): For omnichannel brands, 32% of a Meta channel's measured impact goes to non-DTC sales (Amazon, retail, wholesale) — invisible to platform-native attribution. Upper-funnel tests average 34 days vs 18.6 days for typical Meta tests. (Haus, 2025–2026; Marketing Operators E123, 2025-07-23)

DTC branded search benchmark: Analysis of 225 DTC geo-based incrementality tests (August 2024–December 2025) found branded search median iROAS = 0.70× — below the 1.0× breakeven line, the lowest iROAS (Incremental Return on Ad Spend) of any channel measured. (Lifesight, 2026-08-10)

Lifesight's 225-test benchmark (Aug 2024–Dec 2025): recent, but methodology (how brands were selected, test design controls) not disclosed. Treat as directional, not definitive.

Benchmarks (as-of 2026)

MetricValueSource
Branded search median iROAS0.70× (below breakeven)Lifesight 225-test dataset, Aug 2024–Dec 2025
Grocery chain geo holdout: non-branded paid search lift0% (budget redundant)Lifesight case study, 2026-08-10
DTC apparel brand prospecting geo test: incremental revenue lift+25% over 8 weeksLifesight case study, 2026-08-10
Liquid I.V. TikTok geo lift: incremental conversions+90%Tinuiti, 2025–2026
Liquid I.V. TikTok geo lift: iROAS improvement+71% higher iROASTinuiti, 2025–2026
Attribution inflation illustration: Meta dashboard ROAS1.5× (but 0.9× true return when control organic accounted)Haus, 2025–2026
Share of Meta channel impact going to non-DTC sales (omnichannel brands)32%Haus, 2025–2026
US marketers using incrementality tests (as-of Jul 2025)52%eMarketer/TransUnion, July 2025 via Skai

Wayfair case study (Wayfair Tech Blog, 2023-04-26) — design and methodology details are from 2023; Wayfair's current tooling may have evolved. Included as a rare published practitioner case study.

Key terms

TermMeaning
DMADesignated Market Area — Nielsen-defined US advertising regions used as geo experiment units
GMAGoogle Marketing Area — Google's sub-country geo regions for Conversion Lift experiments
TBRTime-Based Regression — Google's primary geo lift analysis methodology
SCMSynthetic Control Method — counterfactual built from weighted combination of untreated units
Trimmed MatchGoogle-developed methodology for paired geo experiments that handles outlier geo pairs
MDEMinimum Detectable Effect — the smallest true lift the test is designed to detect
IF (Incrementality Factor)Ratio of incremental to platform-reported conversions; used to de-inflate attribution
HoldbackDesign type: ads active in most regions, paused in holdout set
Go DarkDesign type: all media paused in selected regions
Heavy UpDesign type: spend increased in selected regions
SupergeoCluster of smaller geo units (e.g. ZIP codes) aggregated to form an experiment unit

Related

iROAS (Incremental Return on Ad Spend)Conversion LiftHalo AttributionMedia Mix Modeling (MMM)miROASGhost AdsGoogle Meridian
Research agent · 2026-08-11