Do p values lose their meaning in exploratory analyses? It depends how you define the familywise error rate

Review of General Psychology 21:269-275 (2017)
Download Edit this record How to cite View on PhilPapers
Abstract
Several researchers have recently argued that p values lose their meaning in exploratory analyses due to an unknown inflation of the alpha level (e.g., Nosek & Lakens, 2014; Wagenmakers, 2016). For this argument to be tenable, the familywise error rate must be defined in relation to the number of hypotheses that are tested in the same study or article. Under this conceptualization, the familywise error rate is usually unknowable in exploratory analyses because it is usually unclear how many hypotheses have been tested on a spontaneous basis and then omitted from the final research report. In the present article, I argue that it is inappropriate to conceptualize the familywise error rate in relation to the number of hypotheses that are tested. Instead, it is more appropriate to conceptualize familywise error in relation to the number of different tests that are conducted on the same null hypothesis in the same study. Under this conceptualization, alpha level adjustments in exploratory analyses are (a) less necessary and (b) objectively verifiable. As a result, p values do not lose their meaning in exploratory analyses.
Keywords
No keywords specified (fix it)
Categories
(categorize this paper)
PhilPapers/Archive ID
RUBDPV
Upload history
Archival date: 2019-12-10
View other versions
Added to PP index
2019-12-10

Total views
273 ( #22,343 of 2,439,667 )

Recent downloads (6 months)
97 ( #6,239 of 2,439,667 )

How can I increase my downloads?

Downloads since first upload
This graph includes both downloads from PhilArchive and clicks on external links on PhilPapers.