The inflation of Type I error rates is thought to be one of the causes of the replication crisis. Questionable research practices such as p-hacking are thought to inflate Type I error rates above their nominal level, leading to unexpectedly high levels of false positives in the literature and, consequently, unexpectedly low replication rates. In this article, I offer an alternative view. I argue that questionable and other research practices do not usually inflate relevant Type I error rates. I begin (...) with an introduction to Type I error rates that distinguishes them from theoretical errors. I then illustrate my argument with respect to model misspecification, multiple testing, selective inference, forking paths, exploratory analyses, p-hacking, optional stopping, double dipping, and HARKing. In each case, I demonstrate that relevant Type I error rates are not usually inflated above their nominal level, and in the rare cases that they are, the inflation is easily identified and resolved. I conclude that the replication crisis may be explained, at least in part, by researchers’ misinterpretation of statistical errors and their underestimation of theoretical errors. (shrink)

Preregistration has been proposed as a useful method for making a publicly verifiable distinction between confirmatory hypothesis tests, which involve planned tests of ante hoc hypotheses, and exploratory hypothesis tests, which involve unplanned tests of post hoc hypotheses. This distinction is thought to be important because it has been proposed that confirmatory hypothesis tests provide more compelling results (less uncertain, less tentative, less open to bias) than exploratory hypothesis tests. In this article, we challenge this proposition and argue that there (...) are several advantages of exploratory hypothesis tests that can make their results more compelling than those of confirmatory hypothesis tests. We also consider some potential disadvantages of exploratory hypothesis tests and conclude that their advantages can outweigh the disadvantages. We conclude that exploratory hypothesis tests avoid researcher commitment and researcher prophecy biases, reduce the probability of data fraud, are more appropriate in the context of unplanned deviations, facilitate inference to the best explanation, and allow peer reviewers to make additional contributions at the data analysis stage. In contrast, confirmatory hypothesis tests may lead to an inappropriate level of confidence in research conclusions, less appropriate analyses in the context of unplanned deviations, and greater bias and errors in theoretical inferences. (shrink)

Many examples of calibration in climate science raise no alarms regarding model reliability. We examine one example and show that, in employing Classical Hypothesis-testing, it involves calibrating a base model against data that is also used to confirm the model. This is counter to the "intuitive position". We argue, however, that aspects of the intuitive position are upheld by some methods, in particular, the general Cross-validation method. How Cross-validation relates to other prominent Classical methods such as the Akaike Information Criterion (...) and Bayesian Information Criterion is also discussed. (shrink)

Two of the most influential theories about scientific inference are inference to the best explanation and Bayesianism. How are they related? Bas van Fraassen has claimed that IBE and Bayesianism are incompatible rival theories, as any probabilistic version of IBE would violate Bayesian conditionalization. In response, several authors have defended the view that IBE is compatible with Bayesian updating. They claim that the explanatory considerations in IBE are taken into account by the Bayesian because the Bayesian either does or should (...) make use of them in assigning probabilities to hypotheses. I argue that van Fraassen has not succeeded in establishing that IBE and Bayesianism are incompatible, but that the existing compatibilist response is also not satisfactory. I suggest that a more promising approach to the problem is to investigate whether explanatory considerations are taken into account by a Bayesian who assigns priors and likelihoods on his or her own terms. In this case, IBE would emerge from the Bayesian account, rather than being used to constrain priors and likelihoods. I provide a detailed discussion of the case of how the Copernican and Ptolemaic theories explain retrograde motion, and suggest that one of the key explanatory considerations is the extent to which the explanation a theory provides depends on its core elements rather than on auxiliary hypotheses. I then suggest that this type of consideration is reflected in the Bayesian likelihood, given priors that a Bayesian might be inclined to adopt even without explicit guidance by IBE. The aim is to show that IBE and Bayesianism may be compatible, not because they can be amalgamated, but rather because they capture substantially similar epistemic considerations. 1 Introduction2 Preliminaries3 Inference to the Best Explanation4 Bayesianism5 The Incompatibilist View : Inference to the Best Explanation Contradicts Bayesianism5. 1 Criticism of the incompatibilist view6 Constraint - Based Compatibilism6. 1 Criticism of constraint - based compatibilism7 Emergent Compatibilism7. 1 Analysis of inference to the best explanation7. 1. 1 Inference to the best explanation on specific hypotheses7. 1. 2 Inference to the best explanation on general theories7. 1. 3 Copernicus versus Ptolemy7. 1. 4 Explanatory virtues7. 1. 5 Summary7. 2 Bayesian account8 Conclusion. (shrink)

This is the second of three parts of an introduction to the philosophy of climate science. In this second part about modelling climate change, the topics of climate modelling, confirmation of climate models, the limits of climate projections, uncertainty and finally model ensembles will be discussed.

There has been a lively debate in the philosophy of science over _predictivism_: the thesis that successfully predicting a given body of data provides stronger evidence for a theory than merely accommodating the same body of data. I argue for a very strong version of the thesis using statistical results on the so-called “model selection” problem. This is the problem of finding the optimal model (family of hypotheses) given a body of data. The key idea that I will borrow from (...) the statistical literature is that the level of support a hypothesis, _H,_ receives from a body of data, _D,_ is inversely related to the number of adjustable parameters of the model from which _H_ was constructed. I will argue that when _D_ is not essential to the design of _H_ (i.e., when it is predicted), the model to which _H_ belongs has fewer adjustable parameters than when _D_ is essential to the design of _H_ (when it is accommodated)_._ This, I argue, provides us with an argument for a very strong version of predictivism. (shrink)