Holdout test
The closest marketing gets to a clean experiment — show some, hide from others, measure the gap.
- Method
- randomised control group
- Measures
- incremental lift, not credit
- Answers
- would it have happened anyway
- Forms
- user-level or geo holdouts
Forms & parts of speech
What it measures
A holdout test splits an audience at random into a group that sees the advertising and a control group deliberately kept from it. Comparing outcomes between the two isolates the lift the ads actually caused.
This answers the question attribution cannot: would these conversions have happened anyway? If the control group converts almost as well as the treated group, the ad added little real value, whatever the attribution report credited it.
How it is run
Holdouts come in forms. User-level holdouts withhold ads from a random slice of individuals; geo holdouts turn spend off in some regions and on in others, useful when user-level control is impossible after privacy changes.
The discipline is in the design — true randomisation, enough scale for statistical significance, and a clean control that genuinely sees no ads. Done right, a holdout is the strongest causal evidence in marketing; done sloppily, it misleads with confidence.
The measured incremental lift is small, so the brand reallocates that budget to channels a holdout shows are genuinely additive. The experiment overturned a confident attribution number with causal evidence.
Benchmarks
A holdout's result is specific to its design and conditions; there is no portable benchmark. Judge it on randomisation quality and statistical significance, not a target lift.
Ranges are illustrative; every published figure is cited from a named public source or labelled “RGM analysis.”
Synonyms & antonyms
Synonyms
Antonyms
Usage trends
Search interest for this term over the last five years:
Common questions
- What is a holdout test?
- An experiment that withholds advertising from a randomised control group and compares outcomes to a treated group, measuring the true incremental lift the ads caused.
- Why use a holdout test instead of attribution?
- Attribution credits touchpoints on the path but shows correlation; a holdout measures causation by revealing what would have happened without the ads.
- What is a geo holdout?
- A holdout run by region — ads on in some geographies and off in others — used to measure incrementality when user-level control is not possible after privacy changes.
Related tools & calculators
Resources & people to follow
- referenceRGM analysis — holdout test design
- referenceThink with Google — incrementality resources
Curated, non-competitor resources verified per term.