It is widely agreed that generics tolerate exceptions. It turns out, however, that exceptions are tolerated only so long as they do not violate homogeneity: when the exceptions are not concentrated in a salient “chunk” of the domain of the generic. The criterion for salience of a chunk is cognitive: it is dependent on the way in which the domain is mentally represented. Findings of psychological experiments about the ways in which different domains are represented, and the actors affecting such (...) representations, account for judgments of generic sentences, facts which cannot be explained by linguistics alone. The reason for the homogeneity requirement itself is, in turn, also dependent on cognitive considerations. Generics express default rules, and psychological findings have shown that, the more homogeneous the domain, the easier it is for subjects to infer rules about it. Thus, cognitive results form a crucial part of a comprehensive account of the meaning of a linguistic expression. (shrink)

The three main approaches in statistical inference—classical statistics, Bayesian and likelihood—are in current use in phylogeny research. The three approaches are discussed and compared, with particular emphasis on theoretical properties illustrated by simple thought-experiments. The methods are problematic on axiomatic grounds, extra-mathematical grounds relating to the use of a prior or practical grounds. This essay aims to increase understanding of these limits among those with an interest in phylogeny.

We introduce a distinction, unnoticed in the literature, between four varieties of objective Bayesianism. What we call ' strong objective Bayesianism' is characterized by two claims, that all scientific inference is 'logical' and that, given the same background information two agents will ascribe a unique probability to their priors. We think that neither of these claims can be sustained; in this sense, they are 'dogmatic'. The first fails to recognize that some scientific inference, in particular that concerning evidential relations, is (...) not (in the appropriate sense) logical, the second fails to provide a non-question-begging account of 'same background information'. We urge that a suitably objective Bayesian account of scientific inference does not require either of the claims. Finally, we argue that Bayesianism needs to be fine-grained in the same way that Bayesians fine-grain their beliefs. (shrink)

This paper is concerned with the role of rational belief change theory in the philosophical understanding of experimental error. Today, philosophers seek insight about error in the investigation of specific experiments, rather than in general theories. Nevertheless, rational belief change theory adds to our understanding of just such cases: R. A. Fisher’s criticism of Mendel’s experiments being a case in point. After an historical introduction, the main part of this paper investigates Fisher’s paper from the point of view of rational (...) belief change theory: what changes of belief about Mendel’s experiment does Fisher go through and with what justification. It leads to surprising insights about what Fisher had done right and wrong, and, more generally, about the limits of statistical methods in detecting error. (shrink)

This paper has the aim of making Johannes von Kries’s masterpiece, Die Principien der Wahrscheinlichkeitsrechnung of 1886, a little more accessible to the modern reader in three modest ways: first, it discusses the historical background to the book ; next, it summarizes the basic elements of von Kries’s approach ; and finally, it examines the so-called “principle of cogent reason” with which von Kries’s name is often identified in the English literature.

Quantitative researchers often discuss research ethics as if specific ethical problems can be reduced to abstract normative logics (e.g., virtue ethics, utilitarianism, deontology). Such approaches overlook how values are embedded in every aspect of quantitative methods, including ‘observations,’ ‘facts,’ and notions of ‘objectivity.’ We describe how quantitative research practices, concepts, discourses, and their objects/subjects of study have always been value-laden, from the invention of statistics and probability in the 1600s to their subsequent adoption as a logic made to appear as (...) if it exists prior to, and separate from, ethics and values. This logic, which was embraced in the Academy of Management from the 1960s, casts management researchers as ethical agents who ought to know about a reality conceptualized as naturally existing in the image of statistics and probability (replete with ‘constructs’), while overlooking that S&P logic and practices, which researchers made for themselves, have an appreciable role in making the world appear this way. We introduce a different way to conceptualize reality and ethics, wherein the process of scientific inquiry itself requires an examination of its own practices and commitments. Instead of resorting to decontextualized notions of ‘rigor’ and its ‘best practices,’ quantitative researchers can adopt more purposeful ways to reason about the ethics and relevance of their methods and their science. We end by considering implications for addressing ‘post truth’ and ‘alternative facts’ problems as collective concerns, wherein it is actually the pluralistic nature of description that makes defending a collectively valuable version of reality so important and urgent. (shrink)

The information age can be dated to the work of Norbert Wiener and Claude Shannon in the 1940s. Their work on cybernetics and information theory, and many subsequent developments, had a profound influence on reshaping the field of psychology from what it was prior to the 1950s. Contemporaneously, advances also occurred in experimental design and inferential statistical testing stemming from the work of Ronald Fisher, Jerzy Neyman, and Egon Pearson. These interdisciplinary advances from outside of psychology provided the conceptual and (...) methodological tools for what is often called the cognitive revolution but is more accurately described as the information-processing revolution. Cybernetics set the stage with the idea that everything ranging from neurophysiological mechanisms to societal activities can be modeled as structured control systems with feedforward and feedback loops. Information theory offered a way to quantify entropy and information, and promoted theorizing in terms of information flow. Statistical theory provided means for making scientific inferences from the results of controlled experiments and for conceptualizing human decision making. With those three pillars, a cognitive psychology adapted to the information age evolved. The growth of technology in the information age has resulted in human lives being increasingly interweaved with the cyber environment, making cognitive psychology an essential part of interdisciplinary research on such interweaving. Continued engagement in interdisciplinary research at the forefront of technology development provides a chance for psychologists not only to refine their theories but also to play a major role in the advent of a new age of science. (shrink)

It is surely obvious that medicine, like any other rational activity, must be based on evidence. The interest is in the details: how exactly are the general principles of the logic of evidence to be applied in medicine? Focussing on the development, and current claims of the ‘Evidence-Based Medicine’ movement, this article raises a number of difficulties with the rationales that have been supplied in particular for the ‘evidence hierarchy’ and for the very special role within that hierarchy of randomized (...) controlled trials (and meta-analyses of the results of randomized controlled trials). The point is not at all to question the application of a scientific approach to evidence in medicine, but, on the contrary, to indicate a number of areas where philosophers of science can contribute to a proper implementation of exactly that scientific-evidential approach. (shrink)

Frequentist inference typically is described in terms of hypothetical repeated sampling but there are advantages to an interpretation that uses a single random sample. Contemporary examples are given that indicate probabilities for random phenomena are interpreted as classical probabilities, and this interpretation of equally likely chance outcomes is applied to statistical inference using urn models. These are used to address Bayesian criticisms of frequentist methods. Recent descriptions of p-values, confidence intervals, and power are viewed through the lens of classical probability (...) based on a single random sample from the population. (shrink)

Chance has been a focus of attention ever since the beginning of population genetics, but neutrality has not, as natural selection once appeared to be the only worthwhile issue. Neutral change became a major source of interest during the neutralist–selectionist debate, 1970–1980. It retained interest beyond this period for two reasons that contributed to its becoming foundational for evolutionary reasoning. On the one hand, neutral evolution was the first mathematical prediction to emerge from Mendelian inheritance: until then evolution by natural (...) selection was considered the alternative to the fixity of species; now it appears to be the alternative to continuous change. Second, neutral change generated a set of clear predictions on standing variation. These could be used as a reference for detecting more elusive alternative mechanisms of evolution including natural selection. In the wake of the transition from Mendelism to genomics, the combination of coalescent theory, DNA sequence variation, and numerical analysis made it possible to integrate contingent aspects of the history of species into a new null model, thus opening a new dimension in the concept of population that the Modern Synthesis formerly considered as a mere gene pool. (shrink)

As health care embraces the tenets of evidence-based medicine it is important to ask questions about how evidence is produced and interpreted. This essay explores normative dimensions of evidence production, particularly around issues of setting the tolerable level of uncertainty of results. Four specific aspects are explored: what health care providers know about statistics, why alpha levels have been set at 0.05, the role of randomization in the generation of sufficient grounds of belief, and the role of observational studies. The (...) essay concludes with recommendations to acknowledge the value permeation of outcome measures and suggests that attention to reasoning and argument analysis can augment traditional evidence-based approaches in providing a robust critical approach to medical knowledge. (shrink)

The principle of maximum entropy is a method for assigning values to probability distributions on the basis of partial information. In usual formulations of this and related methods of inference one assumes that this partial information takes the form of a constraint on allowed probability distributions. In practical applications, however, the information consists of empirical data. A constraint rule is then employed to construct constraints on probability distributions out of these data. Usually one adopts the rule that equates the expectation (...) values of certain functions with their empirical averages. There are, however, various other ways in which one can construct constraints from empirical data, which makes the maximum entropy principle lead to very different probability assignments. This paper shows that an argument by Jaynes to justify the usual constraint rule is unsatisfactory and investigates several alternative choices. The choice of a constraint rule is also shown to be of crucial importance to the debate on the question whether there is a conflict between the methods of inference based on maximum entropy and Bayesian conditionalization. (shrink)

The contrast between the past-future symmetry of mechanical theories and the time-arrow observed in the behaviour of real complex systems doesn’t have nowadays a fully satisfactory explanation. If one confides in the Laplace-dream that everything be exactly and completely describable by the known mechanical differential equations, the whole experimental evidence of the irreversibility of real complex processes can only be interpreted as an illusion due to the limits of human brain and shortness of human history. In this work it is (...) surmised that in the description of real events it would be more reasonable to renounce exactness and completeness of mechanical differential equations, assuming that also further effects exist in nature, governed by different kinds of rules, in spite of being so weak to be directly unobservable in single motions. This surmise can explain not only the time-arrow, but also why, in particular cases, it can happen that approximate and/or statistical models represent an improvement of mechanical theories instead of an approximation: this happened for Boltzmann gas-model and also for the famous work of Max Planck on blackbody-radiation. And it also appears as a more promising “working hypothesis”, stimulating and guiding us to learn more about limits and origin of the basic equations, and also about the nature of chance and the meaning of probability, which is nowadays not clear in spite of the fundamental role it plays in physics. Particularly that kind of probability which gives the connection between quantum–mechanical differential equations and observable events. (shrink)

According to influential accounts of scientific method, such as critical rationalism, scientific knowledge grows by repeatedly testing our best hypotheses. But despite the popularity of hypothesis tests in statistical inference and science in general, their philosophical foundations remain shaky. In particular, the interpretation of non-significant results—those that do not reject the tested hypothesis—poses a major philosophical challenge. To what extent do they corroborate the tested hypothesis, or provide a reason to accept it? Popper sought for measures of corroboration that could (...) adequately answer this question. According to Popper, corroboration is different from probability-raising, and grounded in the predictive success and testability of a hypothesis. As such, corroboration becomes an indicator of the scientific value of a hypothesis and guides our practical preferences over hypotheses that have been subjected to severe tests. This article proves two impossibility results for corroboration measures based on statistical relevance. The generality of these results shows that Popper’s qualitative characterization of corroboration must be misguided. I explore what a more promising and scientifically useful concept of corroboration could look like. (shrink)

Subjective Bayesianism is a major school of uncertain reasoning and statistical inference. It is often criticized for a lack of objectivity: it opens the door to the influence of values and biases, evidence judgments can vary substantially between scientists, it is not suited for informing policy decisions. My paper rebuts these concerns by connecting the debates on scientific objectivity and statistical method. First, I show that the above concerns arise equally for standard frequentist inference with null hypothesis significance tests. Second, (...) the criticisms are based on specific senses of objectivity with unclear epistemic value. Third, I show that Subjective Bayesianism promotes other, epistemically relevant senses of scientific objectivity—most notably by increasing the transparency of scientific reasoning. (shrink)

The interpretation of tests of a point null hypothesis against an unspecified alternative is a classical and yet unresolved issue in statistical methodology. This paper approaches the problem from the perspective of Lindley's Paradox: the divergence of Bayesian and frequentist inference in hypothesis tests with large sample size. I contend that the standard approaches in both frameworks fail to resolve the paradox. As an alternative, I suggest the Bayesian Reference Criterion: it targets the predictive performance of the null hypothesis in (...) future experiments; it provides a proper decision-theoretic model for testing a point null hypothesis and it convincingly accounts for Lindley's Paradox. (shrink)

Hypothetico-deductive (H-D) confirmation builds on the idea that confirming evidence consists of successful predictions that deductively follow from the hypothesis under test. This article reviews scope, history and recent development of the venerable H-D account: First, we motivate the approach and clarify its relationship to Bayesian confirmation theory. Second, we explain and discuss the tacking paradoxes which exploit the fact that H-D confirmation gives no account of evidential relevance. Third, we review several recent proposals that aim at a sounder and (...) more comprehensive formulation of H-D confirmation. Finally, we conclude that the reputation of hypothetico-deductive confirmation as outdated and hopeless is undeserved: not only can the technical problems be addressed satisfactorily, the hypothetico-deductive method is also highly relevant for scientific practice. (shrink)

Why are conditional degrees of belief in an observation E, given a statistical hypothesis H, aligned with the objective probabilities expressed by H? After showing that standard replies are not satisfactory, I develop a suppositional analysis of conditional degree of belief, transferring Ramsey’s classical proposal to statistical inference. The analysis saves the alignment, explains the role of chance-credence coordination, and rebuts the charge of arbitrary assessment of evidence in Bayesian inference. Finally, I explore the implications of this analysis for Bayesian (...) reasoning with idealized models in science. (shrink)

A unifying framework of probabilistic reasoning Content Type Journal Article Category Book Review Pages 1-4 DOI 10.1007/s11016-011-9573-x Authors Jan Sprenger, Tilburg Center for Logic and Philosophy of Science, Tilburg University, P.O. Box 90153, 5000 LE Tilburg, The Netherlands Journal Metascience Online ISSN 1467-9981 Print ISSN 0815-0796.

This approach does not define a probability measure by syntactical structures. It reveals a link between modal logic and mathematical probability theory. This is shown (1) by adding an operator (and two further connectives and constants) to a system of lower predicate calculus and (2) regarding the models of that extended system. These models are models of the modal system S₅ (without the Barcan formula), where a usual probability measure is defined on their set of possible worlds. Mathematical probability models (...) can be seen as models of S₅. (shrink)

The aim of this contribution is to provide a rather general answer to Hume’s problem. To this end, induction is treated within a straightforward formal paradigm, i.e., several connected levels of abstraction. Within this setting, many concrete models are discussed. On the one hand, models from mathematics, statistics and information science demonstrate how induction might succeed. On the other hand, standard examples from philosophy highlight fundamental difficulties. Thus it transpires that the difference between unbounded and bounded inductive steps is crucial: (...) while unbounded leaps of faith are never justified, there may well be reasonable bounded inductive steps. In this endeavour, the twin concepts of information and probability prove to be indispensable, pinning down the crucial arguments, and, at times, reducing them to calculations. Essentially, a precise study of boundedness settles Goodman’s challenge. Hume’s more profound claim of seemingly inevitable circularity is answered by obviously non-circular hierarchical structures. (shrink)

The replication crisis has caused researchers to distinguish between exact replications, which duplicate all aspects of a study that could potentially affect the results, and direct replications, which duplicate only those aspects of the study that are thought to be theoretically essential to reproduce the original effect. The replication crisis has also prompted researchers to think more carefully about the possibility of making Type I errors when rejecting null hypotheses. In this context, the present article considers the utility of two (...) types of Type I error probability: the Neyman–Pearson long run Type I error rate and the Fisherian sample-specific Type I error probability. It is argued that the Neyman–Pearson Type I error rate is inapplicable in social science because it refers to a long run of exact replications, and social science deals with irreversible units that make exact replications impossible. Instead, the Fisherian sample-specific Type I error probability is recommended as a more meaningful way to conceptualize false positive results in social science because it can be applied to each sample-specific decision about rejecting the same substantive null hypothesis in a series of direct replications. It is concluded that the replication crisis may be partly due to researchers’ unrealistic expectations about replicability based on their consideration of the Neyman–Pearson Type I error rate across a long run of exact replications. (shrink)

Fisher criticised the Neyman-Pearson approach to hypothesis testing by arguing that it relies on the assumption of “repeated sampling from the same population.” The present article considers the responses to this criticism provided by Pearson and Neyman. Pearson interpreted alpha levels in relation to imaginary replications of the original test. This interpretation is appropriate when test users are sure that their replications will be equivalent to one another. However, by definition, scientific researchers do not possess sufficient knowledge about the relevant (...) and irrelevant aspects of their tests and populations to be sure that their replications will be equivalent to one another. Pearson also interpreted the alpha level as a personal rule that guides researchers’ behavior during hypothesis testing. However, this interpretation fails to acknowledge that the same researcher may use different alpha levels in different testing situations. Addressing this problem, Neyman proposed that the average alpha level adopted by a particular researcher can be viewed as an indicator of that researcher’s typical Type I error rate. Researchers’ average alpha levels may be informative from a metascientific perspective. However, they are not useful from a scientific perspective. Scientists are more concerned with the error rates of specific tests of specific hypotheses, rather than the error rates of their colleagues. It is concluded that neither Neyman nor Pearson adequately rebutted Fisher’s “repeated sampling” criticism. Fisher’s significance testing approach is briefly considered as an alternative to the Neyman-Pearson approach. (shrink)

Preregistration has been proposed as a useful method for making a publicly verifiable distinction between confirmatory hypothesis tests, which involve planned tests of ante hoc hypotheses, and exploratory hypothesis tests, which involve unplanned tests of post hoc hypotheses. This distinction is thought to be important because it has been proposed that confirmatory hypothesis tests provide more compelling results (less uncertain, less tentative, less open to bias) than exploratory hypothesis tests. In this article, we challenge this proposition and argue that there (...) are several advantages of exploratory hypothesis tests that can make their results more compelling than those of confirmatory hypothesis tests. We also consider some potential disadvantages of exploratory hypothesis tests and conclude that their advantages can outweigh the disadvantages. We conclude that exploratory hypothesis tests avoid researcher commitment and researcher prophecy biases, reduce the probability of data fraud, are more appropriate in the context of unplanned deviations, facilitate inference to the best explanation, and allow peer reviewers to make additional contributions at the data analysis stage. In contrast, confirmatory hypothesis tests may lead to an inappropriate level of confidence in research conclusions, less appropriate analyses in the context of unplanned deviations, and greater bias and errors in theoretical inferences. (shrink)

The pre-designationist, anti-inductivist and operationalist tenor of Neyman-Pearson theory give that theory an obvious affinity to several currently influential philosophies of science, most particularly, the Popperian. In fact, one might fairly regard Neyman-Pearson theory as the statistical embodiment of Popperian methodology. The difficulties raised in this paper have, then, wider purport, and should serve as something of a touchstone for those who would construct a theory of evidence adequate to statistics without recourse to the notion of inductive probability.

I I set out my view that all inference is essentially deductive and pinpoint what I take to be the major shortcomings of the induction rule.II The import of data depends on the probability model of the experiment, a dependence ignored by the induction rule. Inductivists admit background knowledge must be taken into account but never spell out how this is to be done. As I see it, that is the problem of induction.III The induction rule, far from providing a (...) method of discovery, does not even serve to detect pattern. Knowing that there is uniformity in the universe is no help to discovering laws. A critique of Reichenbach's justification of the straight rule is constructed along these lines.IV The induction rule, by itself, cannot account for the varying rates at which confidence in an hypothesis mounts with data. The mathematical analysis of this salient feature of inductive reasoning requires prior probabilities. We also argue, against orthodox statisticians, that prior probabilities make a substantive contribution to the objectivity of inductive methods, viz. to the design of experiments and the selection of decision rules.V Carnap's general criticisms of various estimation rules, like the straight rule and the ‘impervious rule’, are seen to be misguided when the prior densities to which they correspond are taken into account.VI Analysis of Hempel's definition of confirmation qua formalization of the enumerative (naive) conception of instancehood. We show that from the standpoint of the quantitative measure P(H/E):P(H) for the degree to which E confirms H, Hempel's classificatory concept yields correct results only for sampling at large from a finite population with a two-way classification all of whose compositions are equally probable. We extend the analysis to Goodman's paradox, finding cases in which grue-like hypotheses do receive as much confirmation as their opposite numbers. We argue, moreover, the irrelevancy of entrenchment, and maintain that Goodman's paradox is no more than a straightforward counter-example to the enumerative conception of instancehood embodied in Hempel's definition.VII We rebutt the objection that prior probabilities, qua inputs of Bayesian analysis, can only be obtained by enumerative induction (insofar as they are objective). The divergence in the prior densities of two rational agents is less a function of subjectivity, we maintain, than of vagueness.VIII Our concluding remarks stress that, for Bayesians, there is no problem of induction in the usual sense. (shrink)

This paper deals with meta-statistical questions concerning frequentist statistics. In Sections 2 to 4 I analyse the dispute between Fisher and Neyman on the so called logic of statistical inference, a polemic that has been concomitant of the development of mathematical statistics. My conclusion is that, whenever mathematical statistics makes it possible to draw inferences, it only uses deductive reasoning. Therefore I reject Fisher's inductive approach to the statistical estimation theory and adhere to Neyman's deductive one. On the other hand, (...) I assert that Neyman-Pearson's testing theory, as well as Fisher's tests of significance, properly belong to decision theory, not to logic, neither deductive nor inductive. I then also disagree with Costantini's view of Fisher's testing model as a theory of hypothetico-deductive inferences.In Section 5 I disapprove Hacking1's evidentialists criticisms of the Neyman-Pearson's theory of statistics (NPT), as well as Hacking2's interpretation of NPT as a theory of probable inference. In both cases Hacking misses the point. I conclude, by claiming that Mayo's conception of the Neyman-Pearson's testing theory, as a model of learning from experience, does not purport any advantages over Neyman's behavioristic model. (shrink)

This paper is about the differences between probabilities and beliefs and why reasoning should not always conform to probability laws. Probability is defined in terms of urn models from which probability laws can be derived. This means that probabilities are expressed in rational numbers, they suppose the existence of veridical representations and, when viewed as parts of a probability model, they are determined by a restricted set of variables. Moreover, probabilities are subjective, in that they apply to classes of events (...) that have been deemed (by someone) to be equivalent, rather than to unique events. Beliefs on the other hand are multifaceted, interconnected with all other beliefs, and inexpressible in their entirety. It will be argued that there are not sufficient rational numbers to characterise beliefs by probabilities and that the idea of a veridical set of beliefs is questionable. The concept of a complete probability model based on Fisher's notion of identifiable subsets is outlined. It is argued that to be complete a model must be known to be true. This can never be the case because whatever a person supposes to be true must be potentially modifiable in the light of new information. Thus to infer that an individual's probability estimate is biased it is necessary not only to show that the estimate differs from that given by a probability model, but also to assume that this model is complete, and completeness is not empirically verifiable. It follows that probability models and Bayes theorem are not necessarily appropriate standards for people's probability judgements. The quality of a probability model depends on how reasonable it is to treat some existing uncertainty as if it were equivalent to that in a particular urn model and this cannot be determined empirically. Bias can be demonstrated in estimates of proportions of finite populations such as in the false consensus effect. However the modification of beliefs by ad hoc methods like Tversky and Kahneman's heuristics can be justified, even though this results in biased judgements. This is because of pragmatic factors such as the cost of obtaining and taking account of additional information which are not included even in a complete probability model. Finally, an analogy is drawn between probability models and geometric figures. Both idealisations are useful but qualitatively inadequate characterisations of nature. A difference between the two is that the size of any error can be limited in the case of the geometric figure in a way that is not possible in a probability model. (shrink)

This paper evaluates attempts to defend established concepts of expertise and clinical judgement against the incursions of evidence‐based practice. Two related arguments are considered. The first suggests that standard accounts of evidence‐based practice imply an overly narrow view of ‘evidence’, and that a more inclusive concept, incorporating ‘patterns of knowing’ not recognised by the familiar evidence hierarchies, should be adopted. The second suggests that statistical generalisations cannot be applied non‐problematically to individual patients in specific contexts, and points out that this (...) is why we need clinical judgement. In evaluating the first argument, I propose a criterion for what counts as evidence. It is a minimalist criterion but the ‘patterns of knowing’, referred to in the literature, still fail to meet it. In evaluating the second argument, I will outline the powerful empirical reasons we have for thinking that decisions based on research evidence are usually better than decisions based on clinical judgement; and show that current efforts to rehabilitate clinical judgement seriously underestimate the strength of these reasons. By way of conclusion, I will sketch the ways in which the concept of expertise will have to be modified if we accept evidence‐based practice as a template for health‐care. (shrink)

Monitoring ongoing clinical trials for early signs of effectiveness is an option for improving cost-effectiveness of trials that is becoming increasingly common. Alongside the obvious advantages made possible by monitoring, however, there are some downsides. In particular, there is growing concern in the medical community that trials stopped early for benefit tend to overestimate treatment effect. In this paper, I examine this problem from the point of view of statistical methodology, starting from the observation that the overestimation is caused by (...) the statistical method adopted. Consequently, I argue that some modifications can and should be made to the present statistical framework in order not to miss the advantages the possibility of monitoring can grant. (shrink)

While the common procedure of statistical significance testing and its accompanying concept of p-values have long been surrounded by controversy, renewed concern has been triggered by the replication crisis in science. Many blame statistical significance tests themselves, and some regard them as sufficiently damaging to scientific practice as to warrant being abandoned. We take a contrary position, arguing that the central criticisms arise from misunderstanding and misusing the statistical tools, and that in fact the purported remedies themselves risk damaging science. (...) We argue that banning the use of p-value thresholds in interpreting data does not diminish but rather exacerbates data-dredging and biasing selection effects. If an account cannot specify outcomes that will not be allowed to count as evidence for a claim—if all thresholds are abandoned—then there is no test of that claim. The contributions of this paper are: To explain the rival statistical philosophies underlying the ongoing controversy; To elucidate and reinterpret statistical significance tests, and explain how this reinterpretation ameliorates common misuses and misinterpretations; To argue why recent recommendations to replace, abandon, or retire statistical significance undermine a central function of statistics in science: to test whether observed patterns in the data are genuine or due to background variability. (shrink)

Despite the widespread use of key concepts of the Neyman–Pearson (N–P) statistical paradigm—type I and II errors, significance levels, power, confidence levels—they have been the subject of philosophical controversy and debate for over 60 years. Both current and long-standing problems of N–P tests stem from unclarity and confusion, even among N–P adherents, as to how a test's (pre-data) error probabilities are to be used for (post-data) inductive inference as opposed to inductive behavior. We argue that the relevance of error probabilities (...) is to ensure that only statistical hypotheses that have passed severe or probative tests are inferred from the data. The severity criterion supplies a meta-statistical principle for evaluating proposed statistical inferences, avoiding classic fallacies from tests that are overly sensitive, as well as those not sensitive enough to particular errors and discrepancies. Introduction and overview 1.1 Behavioristic and inferential rationales for Neyman–Pearson (N–P) tests 1.2 Severity rationale: induction as severe testing 1.3 Severity as a meta-statistical concept: three required restrictions on the N–P paradigm Error statistical tests from the severity perspective 2.1 N–P test T(): type I, II error probabilities and power 2.2 Specifying test T() using p-values Neyman's post-data use of power 3.1 Neyman: does failure to reject H warrant confirming H? Severe testing as a basic concept for an adequate post-data inference 4.1 The severity interpretation of acceptance (SIA) for test T() 4.2 The fallacy of acceptance (i.e., an insignificant difference): Ms Rosy 4.3 Severity and power Fallacy of rejection: statistical vs. substantive significance 5.1 Taking a rejection of H0 as evidence for a substantive claim or theory 5.2 A statistically significant difference from H0 may fail to indicate a substantively important magnitude 5.3 Principle for the severity interpretation of a rejection (SIR) 5.4 Comparing significant results with different sample sizes in T(): large n problem 5.5 General testing rules for T(), using the severe testing concept The severe testing concept and confidence intervals 6.1 Dualities between one and two-sided intervals and tests 6.2 Avoiding shortcomings of confidence intervals Beyond the N–P paradigm: pure significance, and misspecification tests Concluding comments: have we shown severity to be a basic concept in a N–P philosophy of induction? (shrink)

It is widely held that falsification of statistical hypotheses is impossible. This view is supported by an analysis of the most important theories of statistical testing: these theories are not compatible with falsificationism. On the other hand, falsificationism yields a basically viable solution to the problems of explanation, prediction and theory testing in a deterministic context. The present paper shows how to introduce the falsificationist solution into the realm of statistics. This is done mainly by applying the concept of empirical (...) content to statistical hypotheses. It is shown that empirical content is a substitute for ‘power’ as defined by Neyman and Pearson. Since the empirical content of a hypothesis is independent of alternative hypotheses, the proposed theory of statistical testing allows for tests of isolated hypotheses. (shrink)

This paper seeks to defend the following conclusions: The program advanced by Carnap and other necessarians for probability logic has little to recommend it except for one important point. Credal probability judgments ought to be adapted to changes in evidence or states of full belief in a principled manner in conformity with the inquirer’s confirmational commitments—except when the inquirer has good reason to modify his or her confirmational commitment. Probability logic ought to spell out the constraints on rationally coherent confirmational (...) commitments. In the case where credal judgments are numerically determinate confirmational commitments correspond to Carnap’s credibility functions mathematically represented by so—called confirmation functions. Serious investigation of the conditions under which confirmational commitments should be changed ought to be a prime target for critical reflection. The necessarians were mistaken in thinking that confirmational commitments are immune to legitimate modification altogether. But their personalist or subjectivist critics went too far in suggesting that we might dispense with confirmational commitments. There is room for serious reflection on conditions under which changes in confirmational commitments may be brought under critical control. Undertaking such reflection need not become embroiled in the anti inductivism that has characterized the work of Popper, Carnap and Jeffrey and narrowed the focus of students of logical and methodological issues pertaining to inquiry. (shrink)

We show that if among the tested hypotheses the number of true hypotheses is not equal to the number of false hypotheses, then Neyman-Pearson theory of testing hypotheses does not warrant minimal epistemic reliability. We also argue that N-P does not protect from the possible negative effects of the pragmatic value-laden unequal setting of error probabilities on N-P’s epistemic reliability. Most importantly, we argue that in the case of a negative impact no methodological adjustment is available to neutralize it, so (...) in such cases the discussed pragmatic-value-ladenness of N-P inevitably compromises the goal of attaining truth. (shrink)