The replication crisis has caused researchers to distinguish between exact replications, which duplicate all aspects of a study that could potentially affect the results, and direct replications, which duplicate only those aspects of the study that are thought to be theoretically essential to reproduce the original effect. The replication crisis has also prompted researchers to think more carefully about the possibility of making Type I errors when rejecting null hypotheses. In this context, the present article considers the utility of two (...) types of Type I error probability: the Neyman–Pearson long run Type I error rate and the Fisherian sample-specific Type I error probability. It is argued that the Neyman–Pearson Type I error rate is inapplicable in social science because it refers to a long run of exact replications, and social science deals with irreversible units that make exact replications impossible. Instead, the Fisherian sample-specific Type I error probability is recommended as a more meaningful way to conceptualize false positive results in social science because it can be applied to each sample-specific decision about rejecting the same substantive null hypothesis in a series of direct replications. It is concluded that the replication crisis may be partly due to researchers’ unrealistic expectations about replicability based on their consideration of the Neyman–Pearson Type I error rate across a long run of exact replications. (shrink)

Preregistration has been proposed as a useful method for making a publicly verifiable distinction between confirmatory hypothesis tests, which involve planned tests of ante hoc hypotheses, and exploratory hypothesis tests, which involve unplanned tests of post hoc hypotheses. This distinction is thought to be important because it has been proposed that confirmatory hypothesis tests provide more compelling results (less uncertain, less tentative, less open to bias) than exploratory hypothesis tests. In this article, we challenge this proposition and argue that there (...) are several advantages of exploratory hypothesis tests that can make their results more compelling than those of confirmatory hypothesis tests. We also consider some potential disadvantages of exploratory hypothesis tests and conclude that their advantages can outweigh the disadvantages. We conclude that exploratory hypothesis tests avoid researcher commitment and researcher prophecy biases, reduce the probability of data fraud, are more appropriate in the context of unplanned deviations, facilitate inference to the best explanation, and allow peer reviewers to make additional contributions at the data analysis stage. In contrast, confirmatory hypothesis tests may lead to an inappropriate level of confidence in research conclusions, less appropriate analyses in the context of unplanned deviations, and greater bias and errors in theoretical inferences. (shrink)

The paper proposes a practical answer to Duhem's problem within the framework of experimental psychology. First, this problem is briefly discussed; second, two studies in psychology are presented illustrating how theories are tested. Thirdly, based on the foregoing, an approach called the “Empirical Reasoning” is developed and justified. It is shown that the ER approach can successfully cope with Duhem's problem. Finally, the ER approach and the Error Statistics approach of Mayo are critically compared with regard to Duhem's problem.

What Do the Data Tell Us? Justification of scientific theories is a three-place relation between data, theories, and background knowledge. Though this should be a commonplace, many methodologies in science neglect it. The article will elucidate the significance and function of our background knowledge in epistemic justification and their consequences for different scientific methodologies. It is argued that there is no simple and at the same time acceptable statistical algorithm that justifies a given theory merely on the basis of certain (...) data. And even if we think to know the probability of a theory, that does not decide whether we should accept it or not. (shrink)

Research in experimental philosophy has increasingly been turning to corpus methods to produce evidence for empirical claims, as they open up new possibilities for testing linguistic claims or studying concepts across time and cultures. The present article reviews the quasi-experimental studies that have been done using textual data from corpora in philosophy, with an eye for the modeling and experimental design that enable statistical inference. I find that most studies forego comparisons that could control for confounds, and that only a (...) little less than half employ statistical testing methods to control for chance results. Furthermore, at least some researchers make modeling decisions that either do not take into account the nature of corpora and of the word-concept relationship, or undermine the experiment's capacity to answer research questions. I suggest that corpus methods could both provide more powerful evidence and gain more mainstream acceptance by improving their modeling practices. (shrink)

How one may use descriptive statistics to give a succinct description of research data is first discussed. The probability basis of inferential statistics, namely, the random sampling distribution of the test statistic, is then introduced. The said sampling distribution is used to introduced the null-hypothesis significance-testing procedure (NHSTP). The emphasis on 'procedure' serves to highlight the fact that significance tests are about data, not about the substantive hypothesis. The distinction is made between (a) the statistical alternative hypothesis (H1) and the (...) substantive hypothesis, (b) using NHSTP to test whether or not chance is responsible for the data and using the embeddding conditional propositions to corroborate the theory. NHSTP serves to supply the minor premise for the theory-corroboration procedure. Some conceptual difficulties with effect-size, statistical power and meta-analysis are also discusse. (shrink)