Preregistration has been proposed as a useful method for making a publicly verifiable distinction between confirmatory hypothesis tests, which involve planned tests of ante hoc hypotheses, and exploratory hypothesis tests, which involve unplanned tests of post hoc hypotheses. This distinction is thought to be important because it has been proposed that confirmatory hypothesis tests provide more compelling results (less uncertain, less tentative, less open to bias) than exploratory hypothesis tests. In this article, we challenge this proposition and argue that there (...) are several advantages of exploratory hypothesis tests that can make their results more compelling than those of confirmatory hypothesis tests. We also consider some potential disadvantages of exploratory hypothesis tests and conclude that their advantages can outweigh the disadvantages. We conclude that exploratory hypothesis tests avoid researcher commitment and researcher prophecy biases, reduce the probability of data fraud, are more appropriate in the context of unplanned deviations, facilitate inference to the best explanation, and allow peer reviewers to make additional contributions at the data analysis stage. In contrast, confirmatory hypothesis tests may lead to an inappropriate level of confidence in research conclusions, less appropriate analyses in the context of unplanned deviations, and greater bias and errors in theoretical inferences. (shrink)

The current strategy of searching for an effective treatment for COVID-19 relies mainly on repurposing existing therapies developed to target other diseases. Conflicting results have emerged in regard to the efficacy of several tested compounds but later results were negative. The number of conducted and ongoing trials and the urgent need for a treatment pose the risk that false-positive results will be incorrectly interpreted as evidence for treatments’ efficacy and a ground for drug approval. Our purpose is twofold. First, we (...) show that the number of drug-repurposing trials can explain the false-positive results. Second, we assess the evidence for treatments’ efficacy from the perspective of evidential pluralism and argue that considering mechanistic evidence is particularly needed in cases when the evidence from clinical trials is conflicting or of low quality. Our analysis is an application of the program of Evidence Based Medicine Plus to the drug repurposing trials for COVID. Our study shows that if decision-makers applied EBM+, authorizing the use of ineffective treatments would be less likely. We analyze the example of trials assessing the efficacy of hydroxychloroquine as a treatment for COVID-19 and mechanistic evidence in favor of and against its therapeutic power to draw a lesson for decision-makers and drug agencies on how excessive hypothesis testing can lead to spurious findings and how studying negative mechanistic evidence can be helpful in discriminating genuine from spurious results. (shrink)

Metascientists have studied questionable research practices in science. The present article considers the parallel concept of questionable metascience practices (QMPs). A QMP is a research practice, assumption, or perspective that has been questioned by several commentators as being potentially problematic for metascience and/or the science reform movement. The present article reviews ten QMPs that relate to criticism, replication, bias, generalization, and the characterization of science. Specifically, the following QMPs are considered: (1) rejecting or ignoring self-criticism; (2) a fast ‘n’ bropen (...) scientific criticism style; (3) overplaying the role of replication in science; (4) assuming a replication rate is “too low” without specifying an “acceptable” rate; (5) an unacknowledged metabias towards explaining the replication crisis in terms of researcher bias; (6) assuming that researcher bias can be reduced; (7) devaluing exploratory results as being more “tentative” than confirmatory results; (8) presuming that questionable research practices are problematic research practices; (9) focusing on knowledge accumulation; and (10) focusing on specific scientific methods. It is stressed that only some metascientists engage in some QMPs some of the time, and that these QMPs may not always be problematic. Research is required to estimate the prevalence and impact of QMPs. In the meantime, QMPs should be viewed as invitations to ask questions about how we go about doing better metascience. (shrink)