On this page
concept

Geo Holdout Experiments

Created 2026-10-13 33 connections

Geo Holdout Experiments

A geo holdout experiment pauses a specific marketing activity in certain geographic regions — the holdout group — and compares outcomes to regions where advertising continues as normal — the control group. The gap between the two groups reveals how much of business performance depended on the paused activity. This page records what sources report; it does not advise.

Definition and core distinction

Per Funnel.io (updated 2025-07-02): a geo holdout test removes something existing to judge its contribution and necessity. This distinguishes it from a geo lift test, which introduces something new to assess its potential. Both are types of Incrementality Testing, but the direction of intervention differs.

Per Darkroom Agency (2026): geo experiments use aggregate market-level data — revenue, orders, new customers, store visits — and require no cookies, device IDs, or user-level consent, making them described as privacy-safe.

How it works

Per Amsive (2026-05-27): the standard analytical method is Difference-in-Differences (DiD), which compares the change in the treatment group to the change in the control group before and after the intervention. DiD results are only meaningful if pre-period trends between test and control groups are parallel; diverging pre-period trends will cause the model to report an existing gap as lift.

Other valid statistical methods per Measured (2025) and Haus include:

  • Synthetic control — used by Haus, claimed to deliver ~4× more precision than matched-market tests (as-of 2026, Haus vendor claim — not independently validated in sourced material)
  • Lasso regression — used by Measured, trained on two years of conversion and market data; Measured targets at least 95% correlation between test and modeling markets (as-of 2025)
  • Bayesian structural time series — cited as valid alternative (Measured, 2025)
  • Linear regression and flat average — compared by Measured in A/A analysis; Lasso described as most commonly used in practice

Per Meta's open-source GeoLift library documentation: GeoLift combines Augmented Synthetic Control Methods (ASCM) with Generalised Synthetic Controls (GSC), using parametric bootstrapping for inference, and provides three power calculators to help choose test markets, market count, and test duration.

Per SegmentStream (blog, 2025-03-18): the A/A validation period is required before running a geo holdout to catch competitor launches, seasonal anomalies, or assignment bias. Results should report confidence intervals; a lift is meaningful only if its interval excludes zero.

Design principles

Per Amsive (2026-05-27), the recommended starting-point design is:

  • Start from the budget decision, not the measurement question. The decision sets the required minimum detectable effect (MDE), which drives all other design choices.
  • MDE formula: MDE ≈ (zα/2 + zβ) × σ × √(1/n_test + 1/n_control), where σ is taken from pre-period data
  • Power target: 80% statistical power at 90% confidence is a common starting point
  • Pre-period: at least as long as the test period; recommended at least 6 months of clean pre-test data (Lifesight, 2026)
  • Duration: 4–8 weeks minimum for national paid media (Amsive, 2026); Darkroom Agency adds 2 weeks pre-test calibration and 1–2 weeks post-test lag (2026)
  • Markets per arm: ≥20 per arm as a starting point (Amsive, 2026)
  • Market matching over random assignment: random draws at typical test scale can produce imbalanced groups; markets should be matched on pre-period KPI levels, trends, seasonality, demographics, and channel mix (Amsive, 2026)
  • Holdout size: ~15–20% of total market spend (Darkroom Agency, 2026)

Per Amsive (2026): most geo test failures are structural — weak pre-periods, vague treatment definitions — rather than analytical.

Per Measured (2025): test markets should represent only 7–40% of national sales — not so large the holdout costs become prohibitive, not so small the signal is too thin.

When to use geo holdout vs alternatives

Per Funnel.io (updated 2025-07-02): geo holdout tests are best suited to evaluating an existing channel's incremental contribution and budget efficiency; geo lift tests are better suited to evaluating a new channel before wider rollout.

Per Measured (2025): geo testing is a near-universal approach for measuring incrementality, especially for prospecting tactics, and is one of the few methods for cross-channel incrementality that avoids data fidelity or privacy concerns.

Geo experiments are preferred over user-level holdouts when: Per LeadEnforce (2026): TV, radio, OOH, programmatic display, CTV — channels where user-level exclusion is technically impossible. Also preferred for large-scale multi-channel budget shift decisions.

User-level holdouts are preferred when: Per LeadEnforce (2026): audience-targeted digital ads, email/CRM, retargeting, and marketing automation — especially when precise segmentation and sufficient traffic volume are available.

Geo holdouts are a poor fit for: Per Measured (2025): long consideration cycles (purchase cycles above one month, where conversion tail extends beyond the observation window). Per Lifesight (2026): granular creative or network optimisation decisions.

MMM integration: Per Darkroom Agency (2026): geo test results can feed and calibrate Media Mix Modeling (MMM) models, replacing correlational estimates with causal ones; this combination is described as best practice.

Key limitations and failure modes

Per Amsive (2026): an underpowered test will return a null result even when a real effect exists. Worked example: a YouTube holdout with 10 control cities, 7.6% customer base held out, at 90% confidence / 80% power, achieves MDE = 15.0% — "if YouTube's true incremental lift is 10%, this test will not detect it."

Per Darkroom Agency (2026): statistical significance does not guarantee business significance; a small but statistically significant lift on a costly channel can still be unprofitable.

Per Amsive (2026): spillover — where pausing ads in holdout regions causes demand to be absorbed by adjacent regions or non-paused campaigns — is a structural contamination risk; geographic buffers are one mitigation.

Per SegmentStream (Constantine Yurevich, 2025-03-18): unused budget withheld from holdout regions can leak back into control regions through automated bidding, inflating control-region exposure (in-channel contamination). Other channels may compensate by increasing spend in control regions (cross-channel contamination).

Per Measured (2025): local CPMs in test markets can rise when spend is concentrated, making marginal cost estimates less reliable (as-of 2025).

Per INCRMNTAL (2026, vendor critical of geo holdouts): claims limitations include location data imprecision, spillover from ad pausing, operational freezes on creatives and bids, and measurement lags of weeks; proposes always-on "micro-experiment" modelling as an alternative. Note: INCRMNTAL sells a competing product.

Budget thresholds:

Per Darkroom Agency (2026): major seasonal events (Black Friday, Prime Day) should be avoided as test windows; they introduce uncontrolled variables that can distort results.

Benchmarks (as-of 2025)

Per Haus (190 geo-based YouTube incrementality tests, 74 DTC brands, 16 months, average test duration 21 days + 13-day post-treatment window, sourced via haus.io/blog, 2025-03-06):

  • YouTube drove 3.4× more incremental DTC lift than Google Ads platform reporting indicated (as-of 2025)
  • YouTube drove 85% higher lift on new-customer KPIs than on repeat-customer KPIs; new customers accounted for 76% of YouTube's DTC sales impact (as-of 2025)
  • For brands selling beyond DTC: average additional halo-effect sales lift of +99% from YouTube across retail/Amazon channels, vs +46% for other tested channels (as-of 2025)

Vendor landscape (as-of 2026-09-23)

Per Sellforte vendor comparison (self-authored, AI-scored out of 32 points, September 2026 — note: Sellforte evaluated itself; methodology: LLM review of public materials):

VendorScore /32Notes
Sellforte25.0Self-assessed; highest overall
Lifesight18.0Forrester Q3 2026 Landscape inclusion (vendor claim)
Measured15.0MCP access; launches geo tests from platform; 0/6 on conversion lift analysis
Haus13.35.5/6 in Experiment Recommendations & Insights; AI-powered summaries
Recast12.3—
LiftLab12.0—
Analytic Partners8.5—
Meridian GeoX5.8Open-source; no hosted interface or enterprise support

Open-source tools: Per Lifesight (2026): Meta's open-source GeoLift library and Google's CausalImpact library are the primary open-source options; Meta's GeoLift is described as an accessible starting point for practitioners.

Per NMS Editorial (2026-05-05): Google announced Meridian GeoX at Google Marketing Live on 5 May 2026 as free open-source geographic incrementality testing code, packaging existing GitHub repos (trimmed_match, matched_markets) with documentation and an upgrade path into Meridian MMM. GeoX output feeds Meridian, Google's open-source MMM. Enterprise commercial access (Meridian Studio on Google Cloud) had not yet been priced or made generally available as of May 2026 (as-of 2026-05-05).

Update (2026-11-02): Meridian GeoX reached general availability globally on September 9, 2026; Meridian 2.0.0 was released September 2, 2026. GeoX uses a JAX backend and Time-Based Regression (TBR) as its primary estimator; requires Python 3.11+ and daily time-series data. Minimum 10 geo units (50+ recommended). (ppc.land, 2026-09; developers.google.com, 2026-09-09)

Per NMS Editorial (2026-05-05): a structural concern with GeoX — Google built a tool that may grade Google's own advertising lift; while the code is auditable, routing through Meridian, Studio, and Google Cloud centralises measurement with one vendor. Amazon's MMM API reached general availability earlier in 2026, meaning both Google and Amazon are simultaneously vendors of MMM tooling that each evaluates its own ads (as-of 2026-05-05; Amazon GA claim not independently verified in this run).

Vendor-specific design features: Per Haus: uses Commuting Zones derived from open-source mobile GPS mobility data to define geo boundaries, reducing spillover between test and control markets; GeoFences allow excluding irrelevant markets. Haus launched Fixed Geo Tests for channels with fixed geographic footprints (OOH, billboards, linear TV), using a synthetic control group from lookalike regions.

Per Measured (2025): reserves markets generating ~50% of conversions for modeling (they receive no test conditions); runs hundreds of candidate market combinations through simulations comparing predicted vs. actual orders; refreshes selection quarterly.

What practitioners report

Per Zach Epstein (Haus, echai.ventures interview, date unknown): platform-reported numbers tend to overstate top-of-funnel spend, where overlap with organic demand is greatest. A clean geo split over a few weeks can produce an actionable answer without perfect infrastructure.

Per Recast (cited in Web — Incrementality 2026-10-13): Recast lists 15+ geo experiment failure modes, including contamination from cross-geo digital ad spillover, market selection bias, and underpowered geo clusters — concluding geo holdouts are not automatically reliable; model quality matters.

Key terms

TermMeaning
Geo holdoutA geographic region deliberately excluded from advertising to act as a natural control group
Geo lift testA geographic region where new advertising is introduced; opposite direction from a holdout
Holdout groupThe region(s) where the marketing activity is paused
Control groupThe region(s) where advertising continues normally; comparison baseline
DiDDifference-in-Differences; compares the pre/post change in test vs control to isolate causal effect
MDEMinimum Detectable Effect; the smallest true effect size a test at given power/confidence can reliably detect
iROASIncremental Return on Ad Spend; revenue attributable to the ad beyond what would have happened organically
Synthetic controlStatistical method that builds a weighted combination of control units to approximate the counterfactual
ASCMAugmented Synthetic Control Methods; used by Meta's GeoLift; adds bias correction to standard synthetic control
PowerProbability that a test detects a true effect when one exists; 80% is a common target
SpilloverDemand shifts from paused-ad regions into adjacent test or control regions, contaminating results
DMADesignated Market Area; US TV market geography commonly used as a geo test unit
Commuting ZoneAlternative geo unit used by Haus; derived from mobility data; designed to reduce spillover
A/A testValidation run before the experiment; both groups treated identically to confirm pre-period trend parity
Research agent · 2026-10-13