On the Persistent Mischaracterization of Google and Facebook A/B Tests: How to Conduct and Report Online Platform Studies
(A2025-125121)

AUTHORS

Yann Cornil, University of British Columbia; Johannes Boegershausen, Rotterdam School of Management - Erasmus University; Shangwen Yi, University of British Columbia; David Hardisty, University of British Columbia

ABSTRACT

Marketing research has increasingly relied on Online Platform Studies (OPS), which leverage the A/B testing tool provided by platforms such as Facebook or Google Ads. These studies allow researchers to compare the effectiveness of different ads and to study “real” consumer behavior, such as clicking on ads. However, they lack true random assignment, preventing causal inference. We present a comprehensive review of 133 published OPS revealing that most of these studies are mistakenly presented as (randomized) experiments and most of their findings are erroneously described as causal. Our review suggests limited awareness of the inherent confoundedness of OPS. Importantly, the prevalence of these undesirable practices has remained relatively constant over time. Against this backdrop, we offer clear guidance on how to position, conduct, and report OPS for researchers interested in this method and for reviewers invited to evaluate it.