Preregistration has been proposed as a useful method for making a publicly verifiable distinction between confirmatory hypothesis tests, which involve planned tests of ante hoc hypotheses, and exploratory hypothesis tests, which involve unplanned tests of post hoc hypotheses. This distinction is thought to be important because it has been proposed that confirmatory hypothesis tests provide more compelling results (less uncertain, less tentative, less open to bias) than exploratory hypothesis tests. In this article, we challenge this proposition and argue that there (...) are several advantages of exploratory hypothesis tests that can make their results more compelling than those of confirmatory hypothesis tests. We also consider some potential disadvantages of exploratory hypothesis tests and conclude that their advantages can outweigh the disadvantages. We conclude that exploratory hypothesis tests avoid researcher commitment and researcher prophecy biases, reduce the probability of data fraud, are more appropriate in the context of unplanned deviations, facilitate inference to the best explanation, and allow peer reviewers to make additional contributions at the data analysis stage. In contrast, confirmatory hypothesis tests may lead to an inappropriate level of confidence in research conclusions, less appropriate analyses in the context of unplanned deviations, and greater bias and errors in theoretical inferences. (shrink)

Theoretical virtues play an important role in the acceptance and belief of theories in science and philosophy. Philosophers have well-developed views on which virtues ought and ought not to influence one’s acceptance and belief. But what do scientists think? This paper presents the results of a quantitative study with scientists from the natural and social sciences and compares their views to those held by philosophers. Some of the more surprising results are: all three groups have a preference order regarding theoretical (...) virtues, making theory choice a much more determinate matter than what has sometimes been suggested; the preference orders are very similar for the three groups; simplicity is viewed as an epistemic virtue particularly by social scientists ; and syntactic parsimony is preferred to ontological parsimony by all three groups. (shrink)

To succeed, political science usually requires either prediction or contextual historical work. Both of these methods favor explanations that are narrow-scope, applying to only one or a few cases. Because of the difficulty of prediction, the main focus of political science should often be contextual historical work. These epistemological conclusions follow from the ubiquity of causal fragility, under-determination, and noise. They tell against several practices that are widespread in the discipline: wide-scope retrospective testing, such as much large-n statistical work; lack (...) of emphasis on prediction; and resources devoted to ‘pure theory’ divorced from frequent empirical application. I illustrate, via Donatella della Porta’s work on political violence, the important role that is still left for theory. I conclude by assessing the scope for political science to offer policy advice. (shrink)

We provide a novel articulation of the epistemic peril of p-hacking using three resources from philosophy: predictivism, Bayesian confirmation theory, and model selection theory. We defend a nuanced position on p-hacking: p-hacking is sometimes, but not always, epistemically pernicious. Our argument requires a novel understanding of Bayesianism, since a standard criticism of Bayesian confirmation theory is that it cannot represent the influence of biased methods. We then turn to pre-analysis plans, a methodological device used to mitigate p-hacking. Some say that (...) pre-analysis plans are epistemically meritorious while others deny this, and in practice pre-analysis plans are often violated. We resolve this debate with a modest defence of pre-analysis plans. Further, we argue that pre-analysis plans can be epistemically relevant even if the plan is not strictly followed—and suggest that allowing for flexible pre-analysis plans may be the best available policy option. (shrink)

Kerr coined the term ‘HARKing’ to refer to the practice of ‘hypothesizing after the results are known’. This questionable research practice has received increased attention in recent years because it is thought to have contributed to low replication rates in science. The present article discusses the concept of HARKing from a philosophical standpoint and then undertakes a critical review of Kerr’s twelve potential costs of HARKing. It is argued that these potential costs are either misconceived, misattributed to HARKing, lacking evidence, (...) or that they do not take into account pre- and post-publication peer review and public availability to research materials and data. It is concluded that it is premature to conclude that HARKing has led to low replication rates. (shrink)

Should we insist on prediction, i.e. on correctly forecasting the future? Or can we rest content with accommodation, i.e. empirical success only with respect to the past? I apply general considerations about this issue to the case of economics. In particular, I examine various ways in which mere accommodation can be sufficient, in order to see whether those ways apply to economics. Two conclusions result. First, an entanglement thesis: the need for prediction is entangled with the methodological role of orthodox (...) economic theory. Second, a conditional predictivism: if we are not committed to orthodox economic theory, then we should demand prediction rather than accommodation – against most current practice. (shrink)

Scientific hypotheses may either predict particular unknown facts or accommodate previously-known data. Although affirmed predictions are intuitively more rewarding than accommodations of established facts, opinions divide whether predictive hypotheses are also epistemically superior to accommodation hypotheses. This paper examines the contribution of predictive hypotheses to discoveries of several bio-molecular systems. Having all the necessary elements of the system known beforehand, an abstract predictive hypothesis of semiconservative mode of DNA replication was successfully affirmed. However, in defining the genetic code whose biochemical (...) basis was unclear, hypotheses were only partially effective and supplementary experimentation was required for its conclusive definition. Markedly, hypotheses were entirely inept in predicting workings of complex systems that included unknown elements. Thus, hypotheses did not predict the existence and function of mRNA, the multiple unidentified components of the protein biosynthesis machinery, or the manifold unknown constituents of the ubiquitin–proteasome system of protein breakdown. Consequently, because of their inability to envision unknown entities, predictive hypotheses did not contribute to the elucidation of cation theories remained the sole instrument to explain complex bio-molecular systems, the philosophical question of alleged advantage of predictive over accommodative hypotheses became inconsequential. (shrink)

This article argues that common intuitions regarding (a) the specialness of ‘use-novel’ data for confirmation and (b) that this specialness implies the ‘no-double-counting rule’, which says that data used in ‘constructing’ (calibrating) a model cannot also play a role in confirming the model’s predictions, are too crude. The intuitions in question are pertinent in all the sciences, but we appeal to a climate science case study to illustrate what is at stake. Our strategy is to analyse the intuitive claims in (...) light of prominent accounts of confirmation of model predictions. We show that on the Bayesian account of confirmation, and also on the standard classical hypothesis-testing account, claims (a) and (b) are not generally true; but for some select cases, it is possible to distinguish data used for calibration from use-novel data, where only the latter confirm. The more specialized classical model-selection methods, on the other hand, uphold a nuanced version of claim (a), but this comes apart from (b), which must be rejected in favour of a more refined account of the relationship between calibration and confirmation. Thus, depending on the framework of confirmation, either the scope or the simplicity of the intuitive position must be revised. (shrink)

Successful instances of interdisciplinary collaboration can eventually enter a process of disciplinarisation. This article analyses one of those instances: agent-based computational social science, an emerging disciplinary field articulated around the use of computational models to study social phenomena. The discussion centres on how, in knowledge transfer dynamics from traditional disciplinary areas, practitioners parsed several epistemic resources to produce new foundational disciplinary shared commitments, and how disagreements operated as a mechanism of differentiation in their production. Two parsing processes are examined to (...) illustrate this claim. The first one is the parsing of the qualitative–quantitative dualism, arguably the most important methodological disagreement in social science. The second one is the parsing of prediction, a key value in contemporary science. The analysis evidences that disagreements have fostered both external and internal dynamics of differentiation in agent-based computational social science. The former have permitted a more efficient use of epistemic resources, whereas the latter have forced practitioners to modify the foundational narrative and the agenda of the field. (shrink)

I offer an explicit account of the underdetermination thesis as well as of the many challenges it poses to scientific realism; a way to answer to these challenges is explored and outlined, by shifting attention to the content of theories. I argue that, even if we have solid grounds (as I contend we do) to support that some varieties of the underdetermination thesis are true, scientific realism can still offer an adequate picture of the aims and achievements of science.

Many climate scientists have made claims that may suggest that evidence used in tuning or calibrating a climate model cannot be used to evaluate the model. By contrast, the philosophers Katie Steele and Charlotte Werndl have argued that, at least within the context of Bayesian confirmation theory, tuning is simply an instance of hypothesis testing. In this paper I argue for a weak predictivism and in support of a nuanced reading of climate scientists’ concerns about tuning: there are cases, model-tuning (...) among them, in which predictive successes are more highly confirmatory of a model than accommodation of evidence. (shrink)

This article argues that common intuitions regarding (a) the specialness of ‘use-novel’ data for confirmation and (b) that this specialness implies the ‘no-double-counting rule’, which says that data used in ‘constructing’ (calibrating) a model cannot also play a role in confirming the model’s predictions, are too crude. The intuitions in question are pertinent in all the sciences, but we appeal to a climate science case study to illustrate what is at stake. Our strategy is to analyse the intuitive claims in (...) light of prominent accounts of confirmation of model predictions. We show that on the Bayesian account of confirmation, and also on the standard classical hypothesis-testing account, claims (a) and (b) are not generally true; but for some select cases, it is possible to distinguish data used for calibration from use-novel data, where only the latter confirm. The more specialized classical model-selection methods, on the other hand, uphold a nuanced version of claim (a), but this comes apart from (b), which must be rejected in favour of a more refined account of the relationship between calibration and confirmation. Thus, depending on the framework of confirmation, either the scope or the simplicity of the intuitive position must be revised. 1 Introduction2 A Climate Case Study3 The Bayesian Method vis-à-vis Intuitions4 Classical Tests vis-à-vis Intuitions5 Classical Model-Selection Methods vis-à-vis Intuitions 5.1 Introducing classical model-selection methods 5.2 Two cases6 Re-examining Our Case Study7 Conclusion. (shrink)

The practice of HARKing---hypothesizing after results are known---is commonly maligned as undermining the reliability of scientific findings. There are several accounts in the literature as to why HARKing undermines the reliability of findings. We argue that none of these is right and that the correct account is a Bayesian one. HARKing can indeed decrease the reliability of scientific findings, but it can also increase it; which effect HARKing produces depends on the difference of the prior odds of hypotheses characteristically selected (...) ex ante and ex post to observing data. Further, we show how misdiagnosis of HARKing can lead to misprescription in the context of the replication crisis. (shrink)

Decision theorists have attempted to accommodate several violations of decision theory’s axiomatic requirements by modifying how agents’ choice options are individuated and formally represented. In recent years, prominent authors have worried that these modifications threaten to trivialize decision theory, make the theory unfalsifiable, impose overdemanding requirements on decision theorists, and hamper decision theory’s internal coherence. In this paper, I draw on leading descriptive and normative works in contemporary decision theory to address these prominent concerns. In doing so, I articulate and (...) assess several different criteria for individuating and formally representing agents’ choice options. (shrink)