A cautionary tale about p < 0.05
(A2025-125831)

AUTHORS

Pranadharthiharan Narayanan, Nova School of Business and Economics; Sumit Malik, University of Liverpool Management School (ULMS); Michael Haenlein, ESCP Business School; Lucas Franieck, Nova SBE

ABSTRACT

We demonstrate that p-values are highly sensitive to the selective exclusion of small samples of data, even when outliers are absent. To highlight this issue, we introduce a simple, context-free robustness metric called the "Significance Fadeaway Score" (SFS). This measure quantifies the extent to which statistical significance (i.e., p<0.05) persists as the most extreme data points are removed iteratively (one-at-a time and without replacement). In a pre-registered study, we evaluate SFS of 52 experimental studies published in leading marketing journals over the past five years. Our results reveal that nearly half of these studies lose statistical significance when less than 5% of the sample is selectively removed. These findings challenge the widespread reliance on dichotomous notions of statistical significance and promote the use of complementary metrics like SFS to enhance the transparency and rigor of experimental method in consumer behavior.