What Do the Data Tell Us? Justification of scientific theories is a three-place relation between data, theories, and background knowledge. Though this should be a commonplace, many methodologies in science neglect it. The article will elucidate the significance and function of our background knowledge in epistemic justification and their consequences for different scientific methodologies. It is argued that there is no simple and at the same time acceptable statistical algorithm that justifies a given theory merely on the basis of certain (...) data. And even if we think to know the probability of a theory, that does not decide whether we should accept it or not. (shrink)

The inflation of Type I error rates is thought to be one of the causes of the replication crisis. Questionable research practices such as p-hacking are thought to inflate Type I error rates above their nominal level, leading to unexpectedly high levels of false positives in the literature and, consequently, unexpectedly low replication rates. In this article, I offer an alternative view. I argue that questionable and other research practices do not usually inflate relevant Type I error rates. I begin (...) by introducing the concept of Type I error rates and distinguishing between statistical errors and theoretical errors. I then illustrate my argument with respect to model misspecification, multiple testing, selective inference, forking paths, exploratory analyses, p-hacking, optional stopping, double dipping, and HARKing. In each case, I demonstrate that relevant Type I error rates are not usually inflated above their nominal level, and in the rare cases that they are, the inflation is easily identified and resolved. I conclude that the replication crisis may be explained, at least in part, by researchers’ misinterpretation of statistical errors and their underestimation of theoretical errors. (shrink)

Preregistration has been proposed as a useful method for making a publicly verifiable distinction between confirmatory hypothesis tests, which involve planned tests of ante hoc hypotheses, and exploratory hypothesis tests, which involve unplanned tests of post hoc hypotheses. This distinction is thought to be important because it has been proposed that confirmatory hypothesis tests provide more compelling results (less uncertain, less tentative, less open to bias) than exploratory hypothesis tests. In this article, we challenge this proposition and argue that there (...) are several advantages of exploratory hypothesis tests that can make their results more compelling than those of confirmatory hypothesis tests. We also consider some potential disadvantages of exploratory hypothesis tests and conclude that their advantages can outweigh the disadvantages. We conclude that exploratory hypothesis tests avoid researcher commitment and researcher prophecy biases, reduce the probability of data fraud, are more appropriate in the context of unplanned deviations, facilitate inference to the best explanation, and allow peer reviewers to make additional contributions at the data analysis stage. In contrast, confirmatory hypothesis tests may lead to an inappropriate level of confidence in research conclusions, less appropriate analyses in the context of unplanned deviations, and greater bias and errors in theoretical inferences. (shrink)

The replication crisis has caused researchers to distinguish between exact replications, which duplicate all aspects of a study that could potentially affect the results, and direct replications, which duplicate only those aspects of the study that are thought to be theoretically essential to reproduce the original effect. The replication crisis has also prompted researchers to think more carefully about the possibility of making Type I errors when rejecting null hypotheses. In this context, the present article considers the utility of two (...) types of Type I error probability: the Neyman–Pearson long run Type I error rate and the Fisherian sample-specific Type I error probability. It is argued that the Neyman–Pearson Type I error rate is inapplicable in social science because it refers to a long run of exact replications, and social science deals with irreversible units that make exact replications impossible. Instead, the Fisherian sample-specific Type I error probability is recommended as a more meaningful way to conceptualize false positive results in social science because it can be applied to each sample-specific decision about rejecting the same substantive null hypothesis in a series of direct replications. It is concluded that the replication crisis may be partly due to researchers’ unrealistic expectations about replicability based on their consideration of the Neyman–Pearson Type I error rate across a long run of exact replications. (shrink)

How one may use descriptive statistics to give a succinct description of research data is first discussed. The probability basis of inferential statistics, namely, the random sampling distribution of the test statistic, is then introduced. The said sampling distribution is used to introduced the null-hypothesis significance-testing procedure (NHSTP). The emphasis on 'procedure' serves to highlight the fact that significance tests are about data, not about the substantive hypothesis. The distinction is made between (a) the statistical alternative hypothesis (H1) and the (...) substantive hypothesis, (b) using NHSTP to test whether or not chance is responsible for the data and using the embeddding conditional propositions to corroborate the theory. NHSTP serves to supply the minor premise for the theory-corroboration procedure. Some conceptual difficulties with effect-size, statistical power and meta-analysis are also discusse. (shrink)

The paper proposes a practical answer to Duhem's problem within the framework of experimental psychology. First, this problem is briefly discussed; second, two studies in psychology are presented illustrating how theories are tested. Thirdly, based on the foregoing, an approach called the “Empirical Reasoning” is developed and justified. It is shown that the ER approach can successfully cope with Duhem's problem. Finally, the ER approach and the Error Statistics approach of Mayo are critically compared with regard to Duhem's problem.