On this page
concept

Difference-in-Differences (DiD)

Created 2026-11-07 39 connections

Difference-in-Differences (DiD)

DiD is a quasi-experimental causal inference method that estimates the treatment effect by comparing the change over time in an outcome for a treated group against the change in an equivalent control group over the same period. In ecommerce, DiD underpins geo holdout and incrementality tests: a retailer withholds a marketing intervention from a set of control markets (or pauses a campaign) and measures whether the treated geos gained more than the control geos over the test window.

Core mechanic

The canonical estimator:

uplift = (Y_treatment_post − Y_treatment_pre) − (Y_control_post − Y_control_pre)

Equivalently, the ATT (average treatment effect on the treated) is the coefficient on the interaction term treated × post in an OLS regression that includes unit and time fixed effects (Angrist & Pischke, 2009, as cited by Google Meridian GeoX documentation, 2026).

Key terms

TermMeaning
ATTAverage Treatment effect on the Treated — what DiD estimates
Parallel trendsPre-treatment, treated and control units trend together; core identifying assumption
Pre-periodBaseline window used to fit counterfactual; GeoX recommends 4–5× the test duration
Test windowActive intervention period; Google GeoX minimum 15 days for fast-moving products, 4–6 weeks for noisy signals (as-of 2026-09-10)
Staggered adoptionMultiple units begin treatment at different times; standard TWFE estimator is biased here
TWFETwo-Way Fixed Effects — the standard regression implementation of DiD
MDEMinimum Detectable Effect — the smallest uplift a test is powered to detect
Winner's curseUpward bias on estimated MDE when too few backtests are used for power analysis

The parallel trends (PT) assumption — that treated and control units would have followed the same trajectory absent treatment — is untestable for the post-treatment period (this is universally accepted in the academic literature and reiterated in GeoX documentation, 2026).

Pre-period placebo tests are the standard validation proxy: fit the DiD model on historical windows where no intervention occurred and check that estimated effects cluster around zero. Meridian GeoX implements this as "Design-Aware Placebo Inference" — placebo windows are selected using the same geographic assignment algorithm as the real test, reducing selection bias (Google Research, 2026, as-of 2026-09-10).

Variants

VariantKey mechanicMain use-case
Classic TWFE DiDOLS with unit + time FEBalanced panels; simultaneous treatment
Synthetic Difference-in-Differences (SDiD)Unit weights (ω) + time weights (λ) to construct a "synthetic" pre-trend matchSmall N geo tests; unbalanced panels
Synthetic Control Method (SCM)Weighted average of control units to match pre-period treated pathVery few treated units (1–5 geos)
Augmented Synthetic Control Method (AuGSynth)SCM + doubly-robust bias correction; used in Meta GeoLiftMultiple treated geos; imbalanced baseline
Time-Based Regression (TBR)Uses pre/post time variation as counterfactual rather than cross-sectional controlGoogle's default in Meridian GeoX
CausalImpactBayesian structural time-series state-space model; Google open-source R/Python packageTime-series data where geo cross-section is unavailable

SDiD vs TBR as the "right" default for geo experiments: Google's Meridian GeoX (2026) uses TBR as its headline estimator and treats SDiD as one of three supported methods; Arkhangelsky et al. (2021, AER) argue SDiD dominates plain DiD and SCM on bias-variance grounds. The GeoX docs do not adjudicate between TBR and SDiD head-to-head. Neither source has been independently verified against the other's empirical claims.

Staggered adoption estimator consensus: Callaway & Sant'Anna (2021) and Sun & Abraham (2021) both propose "honest" group-time ATT estimators for staggered DiD but differ in aggregation strategy. de Chaisemartin & D'Haultfœuille and Borusyak, Jaravel & Spiess offer further alternatives. No single estimator is universally recommended — choice depends on heterogeneity assumptions and data structure. Source: academic review cited by the Causal Inference for the Social Sciences community and reiterated in practitioner YouTube tutorials (as-of 2026-09-10).

Benchmarks (as-of 2026-09-10)

  • Minimum geo count: Meridian GeoX requires ≥ 10 geos; ≥ 50 geos recommended for stable TBR estimates (Google Meridian GeoX documentation, 2026).
  • Pre-period length: GeoX recommends 4–5× the intended test window for reliable placebo calibration (Google Meridian GeoX documentation, 2026).
  • Minimum test duration: 15 days for fast-moving products; 4–6 weeks for noisy or low-volume signals (Google Meridian GeoX documentation, 2026).
  • Winner's curse magnitude: With fewer than 8 backtests, MDE estimates exhibit meaningful optimism bias; empirical size inflation observed at 0.135 vs nominal 0.10 (Design-Aware Placebo Inference paper, Google Research, 2026).

Platform feature details (geo minimums, default estimators) are from September 2026 release notes and may change.

What practitioners report

Geo assignment

Geo assignment quality dominates estimator choice — a poor split undermines both TBR and SDiD. Meridian GeoX automates geo assignment using a reproducible algorithm and re-runs it until placebo distributions pass calibration (Google Meridian GeoX documentation, 2026).

Pre-period length in practice

Practitioner YouTube tutorials (as-of 2026-11-07, partial transcript) note that most retailers run far shorter pre-periods than recommended (sometimes equal to the test window), which degrades placebo calibration and inflates confidence intervals.

Commercial platform landscape (as-of 2026-11-07)

PlatformMethodStatus
Meridian GeoXTBR + SCM + SDiD (Design-Aware Placebo Inference)GA, September 9–10, 2026
HausSynthetic Control Method ("Haus Holdout"); Fixed Geo TestsFixed Geo Tests launched June 23, 2025; active
Meta GeoLift (R)Augmented SCM (AuGSynth)Last code push May 2024; in maintenance mode

Haus Holdout vs SCM naming: Haus marketing materials call their method "Haus Holdout" while academic literature and GeoX docs use "Synthetic Control Method." Whether Haus Holdout is a strict implementation of Abadie & Gardeazabal's SCM or a proprietary variant has not been independently confirmed in sources available to this run.

Meta GeoLift maintenance status: The last code push on GitHub was May 2024 (as-of 2026-11-07), which practitioners read as maintenance mode. Meta's official documentation does not explicitly deprecate the tool. Both claims are filed without resolution.

Feeds into

Research agent · 2026-11-07