In guarding against inferential mistakes, experimental comparative cognition errs on the side of underattributing sophisticated cognition to animals, or what I refer to as the underattribution bias. I propose eliminating this bias by altering the method of choosing the default, or null, hypothesis. Rather than choosing the most parsimonious null hypothesis, as is current practice, I argue for choosing the best-evidenced hypothesis. Doing so at once preserves the risk-controlling structure of the current statistical paradigm and introduces a sensitivity to probability-conferring (...) empirical and theoretical information. This analysis illustrates how values like parsimony can covertly shape statistical-experimental design and inference. (shrink)

Despite the widespread use of key concepts of the Neyman–Pearson (N–P) statistical paradigm—type I and II errors, significance levels, power, confidence levels—they have been the subject of philosophical controversy and debate for over 60 years. Both current and long-standing problems of N–P tests stem from unclarity and confusion, even among N–P adherents, as to how a test's (pre-data) error probabilities are to be used for (post-data) inductive inference as opposed to inductive behavior. We argue that the relevance of error probabilities (...) is to ensure that only statistical hypotheses that have passed severe or probative tests are inferred from the data. The severity criterion supplies a meta-statistical principle for evaluating proposed statistical inferences, avoiding classic fallacies from tests that are overly sensitive, as well as those not sensitive enough to particular errors and discrepancies. Introduction and overview 1.1 Behavioristic and inferential rationales for Neyman–Pearson (N–P) tests 1.2 Severity rationale: induction as severe testing 1.3 Severity as a meta-statistical concept: three required restrictions on the N–P paradigm Error statistical tests from the severity perspective 2.1 N–P test T(): type I, II error probabilities and power 2.2 Specifying test T() using p-values Neyman's post-data use of power 3.1 Neyman: does failure to reject H warrant confirming H? Severe testing as a basic concept for an adequate post-data inference 4.1 The severity interpretation of acceptance (SIA) for test T() 4.2 The fallacy of acceptance (i.e., an insignificant difference): Ms Rosy 4.3 Severity and power Fallacy of rejection: statistical vs. substantive significance 5.1 Taking a rejection of H0 as evidence for a substantive claim or theory 5.2 A statistically significant difference from H0 may fail to indicate a substantively important magnitude 5.3 Principle for the severity interpretation of a rejection (SIR) 5.4 Comparing significant results with different sample sizes in T(): large n problem 5.5 General testing rules for T(), using the severe testing concept The severe testing concept and confidence intervals 6.1 Dualities between one and two-sided intervals and tests 6.2 Avoiding shortcomings of confidence intervals Beyond the N–P paradigm: pure significance, and misspecification tests Concluding comments: have we shown severity to be a basic concept in a N–P philosophy of induction? (shrink)

The ArgumentWhile there has been muchattention given to experiment in modern science studies, there has been astoundingly little concern spared over the practice ofquanitataivemeasurment.Thus myths about the unreasonable effectiveness of mathematice in science still abound. This paper presents: An explicit mathematical model of the stabilization of quantitative constants in a mathematical science to rival older Bayesian and classical accounts;a framework for writing a history of pracitces with regard to treatment of quantitative measurement erroe; resourece for the comparative sociology of differing (...) discipliness in this regard;and a prolegonmena to a critique of orthodox economics and accounting theories. The key to all these diverse themes is the realization that no one individual alone is capable of fixing the magnitude of a quantitative error estimate, and therefore the social construction of error must be given a more precise meaning, an therefore the social construction of error must be given a more precise menaing, and that this occurs through the istrumentality of meta-analysis. (shrink)

Some accounts of evidence regard it as an objective relationship holding between data and hypotheses, perhaps mediated by a testing procedure. Mayo’s error-statistical theory of evidence is an example of such an approach. Such a view leaves open the question of when an epistemic agent is justified in drawing an inference from such data to a hypothesis. Using Mayo’s account as an illustration, I propose a framework for addressing the justification question via a relativized notion, which I designate security , (...) meant to conceptualize practices aimed at the justification of inferences from evidence. I then show how the notion of security can be put to use by showing how two quite different theoretical approaches to model criticism in statistics can both be viewed as strategies for securing claims about statistical evidence. (shrink)

Recent breakthroughs in the history and sociology of science have begun to help us to appreciate the vast complexity and intricate character of empirical endeavours in the sciences. The days when philosophers could blandly gesture towards “observation statements” or “falsification,” as if they were some readily understood phenomena or set of procedures, are gone, happily. This does not mean we can merely use the Duhem-Quine thesis as a shibboleth, however: we are now much more sensitive to immense difficulties in establishing (...) viable claims to replication or establishing the absence of selectivity bias, or convincing others of adequacy of experimental controls. Yet, even so, I would still assert that modern science studies, with only a very few exceptions, have ignored the problem of the constitution of error in the context of precision quantitative measurement. While rather broad claims have been made concerning measurement by Hacking, Cartwright, Franklin and others, none goes so far as to examine the actual manipulation of the numerical data. More distressingly, in the areas of economics and economic methodology, we have not even got so far as have the researchers in science studies. This fact alone should provide grounds for scepticism about the claim in Cartwright that philosophy of scientific inference has something to learn from the history of econometrics. (shrink)