In light of the recent credibility crisis in psychology, this paper argues for a greater emphasis on theorizing in scientific research. Although reliable experimental evidence, preregistration, methodological rigor, and new computational frameworks for modeling are important, scientific progress also relies on properly functioning theories. However, the current understanding of the role of theorizing in psychology is lacking, which may lead to future crises. Theories should not be viewed as mere speculations or simple inductive generalizations. To address this issue, the author (...) introduces a framework called “cognitive metascience,” which studies the processes and results of evaluating scientific practice. This study should proceed both qualitatively, as in traditional science and technology studies and cognitive science, and quantitatively, by analyzing scientific discourse using language technology. By analyzing theories as cognitive artifacts that support cognitive tasks, this paper aims to shed more light on their nature. This perspective reveals that multiple distinct theories serve entirely different roles, and studying these roles, along with their epistemic vices and virtues, can provide insight into how theorizing should proceed. The author urges a change in research culture to appreciate the variety of distinct theories and to systematically advance scientific progress. . (shrink)

The dominant paradigm of experiments in the social and behavioral sciences views an experiment as a test of a theory, where the theory is assumed to generalize beyond the experiment's specific conditions. According to this view, which Alan Newell once characterized as “playing twenty questions with nature,” theory is advanced one experiment at a time, and the integration of disparate findings is assumed to happen via the scientific publishing process. In this article, we argue that the process of integration is (...) at best inefficient, and at worst it does not, in fact, occur. We further show that the challenge of integration cannot be adequately addressed by recently proposed reforms that focus on the reliability and replicability of individual findings, nor simply by conducting more or larger experiments. Rather, the problem arises from the imprecise nature of social and behavioral theories and, consequently, a lack of commensurability across experiments conducted under different conditions. Therefore, researchers must fundamentally rethink how they design experiments and how the experiments relate to theory. We specifically describe an alternative framework, integrative experiment design, which intrinsically promotes commensurability and continuous integration of knowledge. In this paradigm, researchers explicitly map the design space of possible experiments associated with a given research question, embracing many potentially relevant theories rather than focusing on just one. Researchers then iteratively generate theories and test them with experiments explicitly sampled from the design space, allowing results to be integrated across experiments. Given recent methodological and technological developments, we conclude that this approach is feasible and would generate more-reliable, more-cumulative empirical and theoretical knowledge than the current paradigm – and with far greater efficiency. (shrink)

The origins of testing scientific models with statistical techniques go back to 18th century mathematics. However, the modern theory of statistical testing was primarily developed through the work of Sir R.A. Fisher, Jerzy Neyman, and Egon Pearson in the inter-war period. Some of Fisher's papers on testing were published in economics journals (Fisher, 1923, 1935) and exerted a notable influence on the discipline. The development of econometrics and the rise of quantitative economic models in the mid-20th century made statistical significance (...) testing a commonplace, albeit controversial tool within economics. -/- In the debate about significance testing, methodological controversies intertwine with epistemological issues and sociological developments. Our aim in this chapter is to expound these connections and to show how the use of, and the debate about, significance testing in economics differs from other social sciences, such as psychology. (shrink)

The replication or reproducibility crisis in psychological science has renewed attention to philosophical aspects of its methodology. I provide herein a new, functional account of the role of replication in a scientific discipline: to undercut the underdetermination of scientific hypotheses from data, typically by hypotheses that connect data with phenomena. These include hypotheses that concern sampling error, experimental control, and operationalization. How a scientific hypothesis could be underdetermined in one of these ways depends on a scientific discipline’s epistemic goals, theoretical (...) development, material constraints, institutional context, and their interconnections. I illustrate how these apply to the case of psychological science. I then contrast this “bottom-up” account with “top-down” accounts, which assume that the role of replication in a particular science, such as psychology, must follow from a uniform role that it plays in science generally. Aside from avoiding unaddressed problems with top-down accounts, my bottom-up account also better explains the variability of importance of replication of various types across different scientific disciplines. (shrink)

The idea of replication is based on the premise that there are permanent laws to be replicated and verified, and the scientific method is adequate for doing so. Scientific truth, however, is not absolute but relative to time and context, and the method used. Time and context are inextricably interwoven, in that time creates different contexts and contexts (e.g., Christmas Day vs. New Year’s Day) create different experiences of time, rendering psychological phenomena inherently variable. This means that internal and external (...) conditions fluctuate and are different in a replication study vs. the original. Thus, a replication experiment is just another empirical investigation that has no special status in the establishment of scientific truth. It is not the final arbiter of whether or not something exists. In their pursuit of homogeneous external conditions, replications have ignored the homogeneity of internal conditions. There is not a single replication reported in the literature that would have shown participants’ feelings and thoughts—both conscious and nonconscious—to be identical to those of the original participants. Experimental instructions can create varying ratios of conscious over nonconscious processing from one study to another. Ironically, every replication is a failure at the fundamentals of human psychology. While patterns can be discovered, they are not permanent or unchangeable laws of human behavior to be proven by the pinpoint statistical verification through replication. As scientific knowledge in physics is temporary and incomplete, should it be any surprise that science can only provide “temporary winners” for psychological knowledge of human behavior? (shrink)

The replication crisis has caused researchers to distinguish between exact replications, which duplicate all aspects of a study that could potentially affect the results, and direct replications, which duplicate only those aspects of the study that are thought to be theoretically essential to reproduce the original effect. The replication crisis has also prompted researchers to think more carefully about the possibility of making Type I errors when rejecting null hypotheses. In this context, the present article considers the utility of two (...) types of Type I error probability: the Neyman–Pearson long run Type I error rate and the Fisherian sample-specific Type I error probability. It is argued that the Neyman–Pearson Type I error rate is inapplicable in social science because it refers to a long run of exact replications, and social science deals with irreversible units that make exact replications impossible. Instead, the Fisherian sample-specific Type I error probability is recommended as a more meaningful way to conceptualize false positive results in social science because it can be applied to each sample-specific decision about rejecting the same substantive null hypothesis in a series of direct replications. It is concluded that the replication crisis may be partly due to researchers’ unrealistic expectations about replicability based on their consideration of the Neyman–Pearson Type I error rate across a long run of exact replications. (shrink)

Several developments for diverse scientific goals, mostly in physics and physiology, had to take place, which eventually gave us fMRI as one of the central research paradigms of contemporary cognitive neuroscience. This technique stands on solid foundations established by the physics of magnetic resonance and the physiology of hemodynamics and is complimented by computational and statistical techniques. I argue, and support using concrete examples, that these foundations give rise to a productive theory-ladenness in fMRI, which enables researchers to identify and (...) control for the types of methodological and inferential errors. Consequently, this makes it possible for researchers to represent and investigate cognitive phenomena in terms of hemodynamic data and for experimental knowledge to grow independently of large scale theories of cognition. (shrink)

The use of correlational probability values (p‐values) as a means of evaluating evidence in nursing and health care has largely been accepted uncritically. There are reasons to be concerned about an uncritical adherence to the use of significance testing, which has been located in the natural science paradigm. p‐values have served in hypothesis and statistical testing, such as in randomized controlled trials and meta‐analyses to support what has been portrayed as the highest levels of evidence in the framework of evidence‐based (...) practice. Nursing has been minimally involved in the rich debate about the controversies of treating significance testing as evidentiary in the health and social sciences. In this paper, we join the dialogue by examining how and why this statistical mechanism has become entrenched as the gold standard for determining what constitutes legitimate scientific knowledge in the postpositivistic paradigm. We argue that nursing needs to critically reflect on the limitations associated with this tool of the evidence‐based movement, given the complexities and contextual factors that are inherent to nursing epistemology. Such reflection will inform our thinking about what constitutes substantive knowledge for the nursing discipline. (shrink)

fMRI promises to uncover the functional structure of the brain. I argue, however, that pictures of ‘brain activity' associated with fMRI experiments are poor evidence for functional claims. These neuroimages present the results of null hypothesis significance tests performed on fMRI data. Significance tests alone cannot provide evidence about the functional structure of causally dense systems, including the brain. Instead, neuroimages should be seen as indicating regions where further data analysis is warranted. This additional analysis rarely involves simple significance testing, (...) and so justified skepticism about neuroimages does not provide reason for skepticism about fMRI more generally. (shrink)

The writing of history typically involves opinions that cannot be established by historical evidence. This 'involvement' takes two main forms: first, the intimation of evaluative opinions is often the point of historical narratives; and second, as Weber maintained, opinion plays a constitutive role-the identification of historical objects, of explanatory problems, and perhaps even the selection of solutions to these problems is governed by opinions or commitments that cannot be proven historically. The comments of both Bulmer and Camic, for example, presume (...) the validity of opinions about what was and was not important in the history of American sociology. I reject these opinions. In the book, the rejection was intimated. Perhaps the most useful thing I can do in this reply is to identify and sharpen some of the differences in unprovable opinions between myself and the critics though I must say that they were charitable and mild critics indeed-and make explicit some of the things that I wished to intimate in the historical discussion but could not prove. There are two reasons for doing this: Most of the questions raised by each of the commentators can be more easily addressed if these hidden themes and differences of opinion are made more explicit. A more important reason is this: some of these conflicts might be turned into answerable historical problems. (shrink)

Professionals often believe that they must “exercise judgment” in making decisions critical to other people’s lives. The relative superiority of statistical prediction rules to intuitive judgment for combining incomparable sources of information to predict important human outcomes leads us to question this personal input belief. Some professionals hence use SPR’s to “educate” intuitive judgment, rather than replace it. In psychology in particular, such amalgamation is not justified. If a well‐validated SPR that is superior to professional judgment exists in a relevant (...) decision making context, professionals should use it, totally absenting themselves from the prediction. (shrink)

Despite the widespread use of key concepts of the Neyman–Pearson (N–P) statistical paradigm—type I and II errors, significance levels, power, confidence levels—they have been the subject of philosophical controversy and debate for over 60 years. Both current and long-standing problems of N–P tests stem from unclarity and confusion, even among N–P adherents, as to how a test's (pre-data) error probabilities are to be used for (post-data) inductive inference as opposed to inductive behavior. We argue that the relevance of error probabilities (...) is to ensure that only statistical hypotheses that have passed severe or probative tests are inferred from the data. The severity criterion supplies a meta-statistical principle for evaluating proposed statistical inferences, avoiding classic fallacies from tests that are overly sensitive, as well as those not sensitive enough to particular errors and discrepancies. Introduction and overview 1.1 Behavioristic and inferential rationales for Neyman–Pearson (N–P) tests 1.2 Severity rationale: induction as severe testing 1.3 Severity as a meta-statistical concept: three required restrictions on the N–P paradigm Error statistical tests from the severity perspective 2.1 N–P test T(): type I, II error probabilities and power 2.2 Specifying test T() using p-values Neyman's post-data use of power 3.1 Neyman: does failure to reject H warrant confirming H? Severe testing as a basic concept for an adequate post-data inference 4.1 The severity interpretation of acceptance (SIA) for test T() 4.2 The fallacy of acceptance (i.e., an insignificant difference): Ms Rosy 4.3 Severity and power Fallacy of rejection: statistical vs. substantive significance 5.1 Taking a rejection of H0 as evidence for a substantive claim or theory 5.2 A statistically significant difference from H0 may fail to indicate a substantively important magnitude 5.3 Principle for the severity interpretation of a rejection (SIR) 5.4 Comparing significant results with different sample sizes in T(): large n problem 5.5 General testing rules for T(), using the severe testing concept The severe testing concept and confidence intervals 6.1 Dualities between one and two-sided intervals and tests 6.2 Avoiding shortcomings of confidence intervals Beyond the N–P paradigm: pure significance, and misspecification tests Concluding comments: have we shown severity to be a basic concept in a N–P philosophy of induction? (shrink)

Professor Meehl [2] has pointed out a very significant problem in the methodology of psychological research, indicating that statistical tests of psychological hypotheses against a null hypothesis are loaded in favor of eventual success at rejecting the null hypothesis. In my opinion this is not, however, a contrast between physics and psychology, but rather between the method of parameter estimation and that of the null hypothesis in the tradition of Fisher. A physicist could use the null hypothesis method as well (...) as the psychologist. The fact that he doesn't is probably related to the more advanced state of his measurement techniques and theoretical constructs. (shrink)

This chapter examines the history of philosophy of psychology and philosophy of psychiatry as subfields of philosophy of science that emerged in the late twentieth and early twenty-first century. The chapter also surveys related literatures that developed in psychology and psychiatry. Philosophy of psychology (or philosophy of cognitive science) has been a well-established subfield of philosophy of mind since the 1990s and 2000s. This field of philosophy of psychology is narrowly focused on issues in cognitive psychology and cognitive science. Compared (...) to the thriving subfield of philosophy of cognitive science, there has been a lack of corresponding interest among philosophers of science in broader methodological questions about different paradigms and fields of study in psychology. These broader methodological questions about psychology have been addressed in the field of theoretical psychology, which is a subfield of psychology that materialized in the 1980s and 1990s. Philosophy of psychiatry emerged as a subfield of philosophy of science in the mid-2000s. Compared to philosophy of psychology, the philosophy of psychiatry literature in philosophy of science engaged with issues examined in an older and more interdisciplinary tradition of philosophy of psychiatry that developed after the 1960s. The participation of philosophers of science in the literature on theoretical psychology, by contrast, has been limited. (shrink)

Almaatouq et al. claim that the integrative experiment design can help “develop a reliable, cohesive, and cumulative theoretical understanding.” I will contest this claim by challenging three underlying assumptions about the nature of scientific theories. I propose that the integrative experiment design should be viewed as an exploratory framework rather than a means to build or evaluate theories.

Prior experimental studies of anomalous information reception (AIR) have been touted as strong evidence for postmortem survival of consciousness yet are plagued by several methodological weaknesses that preclude clear evidence of positive results. The present team provides an adversarial collaboration to identify and compensate for the major limitations of these previous approaches. We outline a more rigorous preregistered study design that eliminates or minimizes researcher bias in (a) data cleaning and (b) statistical analysis. Obtaining positive results with our recommended design (...) would arguably yield data that skeptics and sympathetic researchers would agree is more clearly interpretable and offers stronger support for a survivalist interpretation. However, this proposed study is not intended to be definitive but rather only a next step in a research program that aims to improve on earlier published efforts. It would also admittedly be time-consuming and expensive to implement, as well as raise ethical considerations in utilizing vulnerable research populations. However, these costs are required to achieve the rigor necessary to advance scientific knowledge in survival research. 1. Introduction -- 2. Laying a Working Foundation -- 3. Choice of "Survival Effect" -- 4. The Recommended Preregistered Protocol -- 5. Considerations and Caveats -- 5.1. Strengths of Such Protocol -- 6. Concluding Thoughts. (shrink)

In this work, we evaluate the status of both theory and empirical evidence in the field of experimental rest-break research based on a framework that combines mental-chronometry and psychometric-measurement theory. To this end, we provide a taxonomy of rest breaks according to which empirical studies can be classified. Then, we evaluate the theorizing in both the basic and applied fields of research and explain how popular concepts relate to each other in contemporary theoretical debates. Here, we highlight differences between all (...) these models in the light of two symbolic categories, termed the resource-based and satiation-based model, including aspects related to the dynamics and the control mechanisms at work. Based on a critical assessment of existing methodological and theoretical approaches, we finally provide a set of guidelines for both theory building and future empirical approaches to the experimental study of rest breaks. We conclude that a psychometrically advanced and theoretically focused research of rest and recovery has the potential to finally provide a sound scientific basis to eventually mitigate the adverse effects of ever increasing task demands on performance and well-being in a multitasking world at work and leisure. (shrink)

Most theories and hypotheses in psychology are verbal in nature, yet their evaluation overwhelmingly relies on inferential statistical procedures. The validity of the move from qualitative to quantitative analysis depends on the verbal and statistical expressions of a hypothesis being closely aligned – that is, that the two must refer to roughly the same set of hypothetical observations. Here, I argue that many applications of statistical inference in psychology fail to meet this basic condition. Focusing on the most widely used (...) class of model in psychology – the linear mixed model – I explore the consequences of failing to statistically operationalize verbal hypotheses in a way that respects researchers' actual generalization intentions. I demonstrate that although the “random effect” formalism is used pervasively in psychology to model intersubject variability, few researchers accord the same treatment to other variables they clearly intend to generalize over. The under-specification of random effects imposes far stronger constraints on the generalizability of results than most researchers appreciate. Ignoring these constraints can dramatically inflate false-positive rates, and often leads researchers to draw sweeping verbal generalizations that lack a meaningful connection to the statistical quantities they are putatively based on. I argue that failure to take the alignment between verbal and statistical expressions seriously lies at the heart of many of psychology's ongoing problems, and conclude with a discussion of several potential avenues for improvement. (shrink)

In the face of continuing assumptions by many scientists and journal editors that p-values provide a gold standard for inference, counter warnings are published periodically. But the core problem is not with p-values, per se. A finding that “p-value is less than α” could merely signal that a critical value has been exceeded. The question is why, when estimating a parameter, we provide a range (a confidence interval), but when testing a hypothesis about a parameter (e.g. µ = x) we (...) proceed as if “=” entails exact equality of the parameter with x. That standard is hard to meet, and is not a standard expected for power calculations, where we are satisfied to reject a null hypothesis H0 if the result is merely “detectably” different from (exact) H0. This paper explores, with resampling (simulation) methods, the impacts on p-values, and alternatives, if the null hypothesis is defined as a thick or thin range of values. It also examines, empirically, the extent to which the p-value may or may not be a good predictor of the probability that H0 is true, given the distribution of the data. (shrink)

Although the influence of affect on creativity has received some theoretical and empirical attention, the role of affect as a consequence of creative problem solving has been neglected. This study is the one of the first to examine empirically the affect that results from creative problem solving. In a 2 (group) × 3 (time period) × 2 (task) factorial design, 122 art and science students were randomly assigned to complete an art or science task and to report on the kind (...) and intensity of affect before, during, and after creative problem solving. It was predicted that art and science students would report different levels of affect only after the insight, not before or during, and that the effects of task, not just group, would contribute to affective variability between art and science students. Science students reported similar levels of (positive) affective intensity before and during creative insight as art students. It was only after the insight that art students reported more intense affective experiences than science students. Task differences accounted for a significant amount of variance in affective intensities, but primarily for art-oriented subjects. These findings suggest that viewing affect as a dependent variable of cognition, rather than primarily as an independent variable, is a direction that would benefit the field of empirical work on cognition and emotion. Current cognitive theories of emotion offer much potential in understanding the affective consequences of creative problem solving. (shrink)

Although the influence of affect on creativity has received some theoretical and empirical attention, the role of affect as a consequence of creative problem solving has been neglected. This study is the one of the first to examine empirically the affect that results from creative problem solving. In a 2 (group) × 3 (time period) × 2 (task) factorial design, 122 art and science students were randomly assigned to complete an art or science task and to report on the kind (...) and intensity of affect before, during, and after creative problem solving. It was predicted that art and science students would report different levels of affect only after the insight, not before or during, and that the effects of task, not just group, would contribute to affective variability between art and science students. Science students reported similar levels of (positive) affective intensity before and during creative insight as art students. It was only after the insight that art students reported more intense affective experiences than science students. Task differences accounted for a significant amount of variance in affective intensities, but primarily for art-oriented subjects. These findings suggest that viewing affect as a dependent variable of cognition, rather than primarily as an independent variable, is a direction that would benefit the field of empirical work on cognition and emotion. Current cognitive theories of emotion offer much potential in understanding the affective consequences of creative problem solving. (shrink)

L’enseignement de la méthodologie scientifique en Psychologie confère un rôle paradigmatique à l’opérationnalisation des « hypothèses générales » : une idée sans rapport précis à l’observation concrète se traduit par la tentative de rejeter une hypothèse statistique nulle au profit d’une hypothèse alternative, dite de recherche, qui opérationnalise l’idée générale. Cette démarche s’avère particulièrement inadaptée à la découverte de lois empiriques. Une loi empirique est définie comme un trou nomothétique émergeant d’un référentiel de la forme Ω x M(X) x M(Y), (...) où Ω est un ensemble d’événements ou d’objets datés dont certains états dans l’ensemble M(Y) sont par hypothèse impossibles étant données certaines conditions initiales décrites dans l’ensemble M(X). Cette approche permet de préciser le regard que l’historien des connaissances peut porter sur les avancées descriptives et nomothétiques de la Psychologie empirique contemporaine. (shrink)

L’enseignement de la méthodologie scientifique en Psychologie confère un rôle paradigmatique à l’opérationnalisation des « hypothèses générales » : une idée sans rapport précis à l’observation concrète se traduit par la tentative de rejeter une hypothèse statistique nulle au profit d’une hypothèse alternative, dite de recherche, qui opérationnalise l’idée générale. Cette démarche s’avère particulièrement inadaptée à la découverte de lois empiriques. Une loi empirique est définie comme un trou nomothétique émergeant d’un référentiel de la forme Ω x M(X) x M(Y), (...) où Ω est un ensemble d’événements ou d’objets datés dont certains états dans l’ensemble M(Y) sont par hypothèse impossibles étant données certaines conditions initiales décrites dans l’ensemble M(X). Cette approche permet de préciser le regard que l’historien des connaissances peut porter sur les avancées descriptives et nomothétiques de la Psychologie empirique contemporaine. (shrink)

Researchers display confirmation bias when they persevere by revising procedures until obtaining a theory-predicted result. This strategy produces findings that are overgeneralized in avoidable ways, and this in turn binders successful applications. (The 40-year history of an attitude-change phenomenon.

Despite the by now historical tendency to demarcate scientific epistemology sharply from virtually all others, especially theological “epistemology,” it has recently been recognized that both enterprises share a great deal in common, at least as far as the epistemology of discovery is implicated. Such a claim is founded upon a psychological analysis of figuration, where, it is argued, metaphor plays a crucial role in the mediation of discovery, in the domains of science and religion alike. Thus, although the conventionally conceived (...) scientific method is crucial to the enterprise, primacy must nonetheless be accorded to discovery, which drives virtually all disciplines. (shrink)

The APA Task Force’s treatment of research methods is critically examined. The present defense of the experiment rests on showing that (a) the control group cannot be replaced by the contrast group, (b) experimental psychologists have valid reasons to use non-randomly selected subjects, (c) there is no evidential support for the experimenter expectancy effect, (d) the Task Force had misrepresented the role of inductive and deductive logic, and (e) the validity of experimental data does not require appealing to the effect (...) size or statistical power. (shrink)

Psychologists collect empirical data with various methods for different reasons. These diverse methods have their strengths as well as weaknesses. Nonetheless, it is possible to rank them in terms of different critieria. For example, the experimental method is used to obtain the least ambiguous conclusion. Hence, it is the best suited to corroborate conceptual, explanatory hypotheses. The interview method, on the other hand, gives the research participants a kind of emphatic experience that may be important to them. It is for (...) the reason the best method to use in a clinical setting. All non-experimental methods owe their origin to the interview method. Quasi-experiments are suited for answering practical questions when ecological validity is importan. (shrink)

Professionals often believe that they must “exercise judgment” in making decisions critical to other people's lives. The relative superiority of statistical prediction rules to intuitive judgment for combining incomparable sources of information to predict important human outcomes leads us to question this personal input belief. Some professionals hence use SPR's to “educate” intuitive judgment, rather than replace it. In psychology in particular, such amalgamation is not justified. If a well-validated SPR that is superior to professional judgment exists in a relevant (...) decision making context, professionals should use it, totally absenting themselves from the prediction. (shrink)

In this essay we introduce the idea of an epistemic triangle, with factual, theoretical, and conceptual investigations at its vertices, and argue that whereas scientific progress requires a balance among the three types of investigations, psychology's epistemic triangle is stretched disproportionately in the direction of factual investigations. Expressed by a variety of different problems, this unbalance may be created by a main operative theme—the obsession of psychology with a narrow and mechanical view of the scientific method and a misguided aversion (...) to conceptual inquiries. Hence, to redress psychology's epistemic triangle, a broader and more realistic conception of method is needed and, in particular, conceptual investigations must be promoted. Using examples from different research domains, we describe the nature of conceptual investigations, relate them to theoretical investigations, and illustrate their purposes, forms, and limitations. (shrink)

How one may use descriptive statistics to give a succinct description of research data is first discussed. The probability basis of inferential statistics, namely, the random sampling distribution of the test statistic, is then introduced. The said sampling distribution is used to introduced the null-hypothesis significance-testing procedure (NHSTP). The emphasis on 'procedure' serves to highlight the fact that significance tests are about data, not about the substantive hypothesis. The distinction is made between (a) the statistical alternative hypothesis (H1) and the (...) substantive hypothesis, (b) using NHSTP to test whether or not chance is responsible for the data and using the embeddding conditional propositions to corroborate the theory. NHSTP serves to supply the minor premise for the theory-corroboration procedure. Some conceptual difficulties with effect-size, statistical power and meta-analysis are also discusse. (shrink)

Meehl's statement "in most psychological research, Improved power of a statistical design leads to a prior probability approaching 1/2 of finding a significant difference in the theoretically predicted direction" (philosophy of science, Volume 34, Pages 103-115), Is without foundation. The computation of prior probabilities of accepting or rejecting a hypothesis presupposes knowledge of the prior probabilities that this hypothesis or any of its conceivable alternatives are true. As we do not have such knowledge, We cannot give any numerical values of (...) prior probabilities of accepting or rejecting hypotheses in any statistical test procedure. Only topological statements are possible, As for example: when the region of acceptance of a hypothesis narrows, The prior probability that it will be accepted remains constant or increases. (shrink)

This paper examines three programmatic responses to the problem of cognitive bias: virtue epistemology, epistemic paternalism, and epistemic collectivism. Each of these programmes focuses on asinglelevel of epistemic analysis: virtue theorists on individuals, paternalists on environments, and collectivists on groups. I argue that this is a mistake in light of the fact that cognitive biases arise frominteractionsbetween these three domains. Consequently, epistemologists should spend less time defending these programmes, and more timecoordinatingthem. This paper offers empirically based arguments for the interactionist (...) approach, and contends that its adoption is an essential step for minimizing bias in empirical science. (shrink)

ABSTRACTPossibly, the replication crisis constitutes the most important problem in psychology. It calls into question whether psychology is a science. Existing conceptualizations of replicability depend on effect sizes; the larger the population effect size, the greater the probability of replication. This is problematic and contributes to the replication crisis. A different conceptualization, not dependent on population effect sizes, is desirable. The proposed solution features the closeness of sample means to their corresponding population means, in both the original and replication experiments. (...) If the researcher has specified the sampling precision desired, it is possible to calculate the probability of replication, prior to data collection, and without dependence on the population effect size or expected population effect size. In addition, it is not necessary to know population means or standard deviations, nor sample means or standard deviations, to employ the proposed a priori way of thinking about... (shrink)

Despite publication of many well-argued critiques of null hypothesis testing (NHT), behavioral science researchers continue to rely heavily on this set of practices. Although we agree with most critics' catalogs of NHT's flaws, this article also takes the unusual stance of identifying virtues that may explain why NHT continues to be so extensively used. These virtues include providing results in the form of a dichotomous (yes/no) hypothesis evaluation and providing an index (p value) that has a justifiable mapping onto confidence (...) in repeatability of a null hypothesis rejection. The most-criticized flaws of NHT can be avoided when the importance of a hypothesis, rather than the p value of its test, is used to determine that a finding is worthy of report, and when p = /05 is treated as insufficient basis for confidence in the replicability of an isolated non-null finding. Together with many recent critics of NHT, we also urge reporting of important hypothesis tests in enough descriptive detail to permit secondary uses such as meta-analysis. (shrink)

In rebellion against low‐dimensional (e.g., two‐factor) theories in psychology, the authors make the case for high‐dimensional theories. This change in perspective requires a shift towards a focus on computation and quantitative reasoning.

Commentaries on the target article offer diverse perspectives on integrative experiment design. Our responses engage three themes: (1) Disputes of our characterization of the problem, (2) skepticism toward our proposed solution, and (3) endorsement of the solution, with accompanying discussions of its implementation in existing work and its potential for other domains. Collectively, the commentaries enhance our confidence in the promise and viability of integrative experiment design, while highlighting important considerations about how it is used.

Significance testing is the primary method for establishing causal relationships in psychology. Meehl [1978, 1990a, 1990b] and Faust [1984] argue that significance tests and their interpretation are subject to actuarial and psychological biases, making continued adherence to these practices irrational, and even partially responsible for the slow progress of the ‘soft’ areas of psychology. I contend that familiar standards of testing and literature review, along with recently developed meta-analytic techniques, are able to correct the proposed actuarial and psychological biases. In (...) particular, psychologists embrace a principle of robustness which states that real psychological effects are (1) reproducible by similar methods, (2) detectable by diverse means, and (3) able to survive theoretical integration. By contrast, spurious significant findings perish under the strain of persistent tests of their robustness. The resulting vindication of significance testing confers on the world a role in determining the rationality of a method, and also affords us an explanation for the fast progress of ‘hard’ areas of psychology. *I would like to thank Dick Boyd and Phil Gasper for helpful comments on the ideas presented here. (shrink)

Several developments for diverse scientific goals, mostly in physics and physiology, had to take place, which eventually gave us fMRI as one of the central research paradigms of contemporary cognitive neuroscience. This technique stands on solid foundations established by the physics of magnetic resonance and the physiology of hemodynamics and is complimented by computational and statistical techniques. I argue, and support using concrete examples, that these foundations give rise to a productive theory-ladenness in fMRI, which enables researchers to identify and (...) control for the types of methodological and inferential errors. Consequently, this makes it possible for researchers to represent and investigate cognitive phenomena in terms of hemodynamic data and for experimental knowledge to grow independently of large scale theories of cognition. (shrink)