The Filter Curve: Uncovering p-Hacking from filtering
(A2021-94598)

AUTHORS

Florian Dost, Brandenburg University of Technology; Lennard Schmidt, Brandenburg University of Technology; Erik Maier, HHL Leipzig Graduate School of Management

ABSTRACT

Many empirical studies filter participants (e.g., for incorrect attention checks or quick re-sponses), especially when using participant pools such as Amazon MTurk. Yet, there is no consensus on whether and how to filter. This might originate from different perspectives on filtering participants: it may be evaluated positively (e.g., as it might be necessary to prevent inattentive participants from biasing results) or negatively (e.g., as it may enable p-hacking). This research aims to bridge these opposites: first, we empirically compare the effects of different filters and filter levels on validity, reliability, power and effects sizes of the results. Second, we introduce the Filter Curve and our R-package “FiltR” as a means to recognize filtering which might be used to p-hack results. We suggest that filtering is not per se bad – although some filters decrease reliability and validity – but that researchers should be trans-parent in how sensitive results are for different filter combinations.