Conflicting results undermine making inferences from the empirical literature. So far, the replication crisis is mainly seen as resulting from honest errors and questionable research practices such as p-hacking or the base-rate fallacy. I discuss the malleability (researcher degrees of freedom) of quantitative research and argue that conflicting results can emerge from two studies using different but plausible designs (e.g., eligibility criteria, operationalization of concepts, outcome measures) and statistical methods. I also explore how the choices regarding study design and statistical (...) techniques bias results in a way that makes them more or less relevant for a given policy or clinical question. (shrink)

Collective procedural memory is a group’s memory of how to do things, as opposed to a group’s memory of facts. It enables groups to mount effective responses to periodic events (e.g., natural hazards) and to sustain collective projects (e.g., combatting climate change). This article presents an account of collective procedural memory called the Ability Conception. The Ability Conception has various advantages over other accounts of collective procedural memory, such as those appealing to collective know-how and collective identity. It also demonstrates (...) new applications for collective procedural memory. I develop three in this article: to social epistemology, to the ethics of memorialization, and to a pattern of group vulnerability to recurring hazardous events that I call the saeculum effect. (shrink)

Misconduct in science is often viewed and analyzed through the lenses of normative ethics and moral philosophy. However, notions and methods in the philosophy of science could also provide rather penetrative explanatory insights into the nature and causes of scientific misconduct. A brief illustration in this regard, using as examples the widely popular Popperian falsification and the Kuhnian scientific paradigm, is provided. In multiple areas of scientific research, failure to seek falsification in a Popperian manner constitutes a questionable research practice (...) and could lead to “falsification” in the context of research misconduct. On the other hand, scientific misconduct is often facilitated by its perpetrators using the familiarity, expectations and confines of a Kuhnian paradigm to blend in fabricated data/results. A rudimentary application of these philosophical notions could be useful in our understanding of the nature and cause of research misconduct, and facilitate mitigation of the latter through educational means. (shrink)

This dissertation proposes and defends a hybrid view I call Hybrid Impermissivism, which combines the following two theses: Moderate Uniqueness and Credal Permissivism. Moderate Uniqueness says that no evidence could justify both believing a proposition and its negation. However, on Moderate Uniqueness, evidence could justify both believing and suspending judgement on a proposition (hence the adjective “Moderate”). And Credal Permissivism says that more than one credal attitude could be justified on the evidence. Hybrid Impermissisim is developed into a precise theory (...) by using normative bridge principles on how rational belief and credence ought to interact. My overall conclusion is that Hybrid Impermissivism is a coherent, plausible, and conciliatory position and provides a correct characterisation of the requirements of evidence on doxastic attitudes . (shrink)

We argue that the epistemic functions of replication in science are best understood by their role in assessing kinds of experimental error. Direct replications serve to assess the reliability of an experiment through its precision: the presence and degree of random error. Conceptual replications serve to assess the validity of an experiment through its accuracy: the presence and degree of systematic errors. To illustrate the aptness of this view, we examine the Hubble constant controversy in astronomy, showing how astronomers have (...) responded to the concordances and discordances in their results by carrying out the different kinds of replication that we identify, with the aim of establishing a precise, accurate value for the Hubble constant. We contrast our view with Machery’s “re-sampling” account of replicability, which maintains that replications only assess reliability. (shrink)

Scientists often respond to failures to replicate by citing differences between the experimental components of an original study and those of its attempted replication. In this paper, we investigate these purported mismatch explanations. We assess a body of failures to replicate in neuroscience studies on spinal cord injury. We argue that a defensible mismatch explanation is one where a mismatch of components is a difference maker for a mismatch of outcomes, and the components are relevantly different in the follow-up study, (...) given the scope of the original study. With this account, we argue that not all differences between studies are meaningful, even if they are difference makers. As our examples show, focusing only on these differences results in disregarding the representativeness of the original experiment’s components and the scope of its outcomes, undercutting other epistemic aims, such as translation, in the process. (shrink)

While the term “reactivity” has come to be associated with specific phenomena in the social sciences, having to do with subjects’ awareness of being studied, this paper takes a broader stance on this concept. I argue that reactivity is a ubiquitous feature of the psychological subject matter and that this fact is a precondition of experimental research, while also posing potential problems for the experimenter. The latter are connected to the worry about distorted data and experimental artifacts. But what are (...) experimental artifacts and what is the most productive way of dealing with them? In this paper, I approach these questions by exploring the ways in which experimenters in psychology simultaneously exploit and suppress the reactivity of their subject matter in order to produce experimental data that speak to the question or subject matter at hand. Highlighting the artificiality of experimental data, I raise the question of what distinguishes a genuine experimental result from an experimental artifact. My analysis construes experimental results as the outcomes of inferences from the data that take material background assumptions as auxiliary premises. Artifacts occur when one or more of these background assumptions are false, such that the data do not reliably serve the purposes they were generated for. I conclude by laying out the ways in which my analysis of data quality is relevant to, and informed by, recent debates about the replicability of experimental results. (shrink)

There is now a general consensus amongst philosophers in the values in science literature that values necessarily play a role in core areas of scientific inquiry. We argue that attention should now be turned from debating the value-free ideal to delineating legitimate from illegitimate influences of values in science, a project we dub “The New Demarcation Problem.” First, we review past attempts to demarcate the uses of values and propose a categorization of the strategies by where they seek to draw (...) legitimacy from. Next, we propose a set of desiderata for what we take to be a satisfactory solution and present a case study where conflicting sets of values clearly impinge on science, but where the legitimacy of their influence is ambiguous. We use these desiderata and the case study to illustrate what we take to be the strengths and weaknesses of current strategies. To be clear, our goal is not to answer the question we pose, but to articulate a framework within which a solution can be judged. (shrink)

This paper advocates for making epistemic interests a central object of philosophical analysis in epistemology and philosophy of science. It is argued that the importance of epistemic interests derives from their fundamental importance for the notion of objectivity. Epistemic interests are defined as individuated by a set of objectives, each of which represents a dimension of the search for truth. Among these dimensions, specificity, sensitivity, and productivity are discussed in detail. It is argued that the relevance of productivity is often (...) overlooked in debates about the ends and means of science. A definition of the objectivity of inquiry is proposed that takes the notion of epistemic interest as its starting point. (shrink)

In this paper, I distinguish three general approaches to public trust in science, which I call the individual approach, the semi-social approach, and the social approach, and critically examine their proposed solutions to what I call the problem of harmful distrust. I argue that, despite their differences, the individual and the semi-social approaches see the solution to the problem of harmful distrust as consisting primarily in trying to persuade individual citizens to trust science and that both approaches face two general (...) problems, which I call the problem of overidealizing science and the problem of overburdening citizens. I then argue that in order to avoid these problems we need to embrace a (thoroughly) social approach to public trust in science, which emphasizes the social dimensions of the reception, transmission, and uptake of scientific knowledge in society and the ways in which social forces influence both positively and negatively the trustworthiness of science. (shrink)

The origins of testing scientific models with statistical techniques go back to 18th century mathematics. However, the modern theory of statistical testing was primarily developed through the work of Sir R.A. Fisher, Jerzy Neyman, and Egon Pearson in the inter-war period. Some of Fisher's papers on testing were published in economics journals (Fisher, 1923, 1935) and exerted a notable influence on the discipline. The development of econometrics and the rise of quantitative economic models in the mid-20th century made statistical significance (...) testing a commonplace, albeit controversial tool within economics. -/- In the debate about significance testing, methodological controversies intertwine with epistemological issues and sociological developments. Our aim in this chapter is to expound these connections and to show how the use of, and the debate about, significance testing in economics differs from other social sciences, such as psychology. (shrink)

While it is widely acknowledged that psychology is in the throes of a replication ‘crisis’, relatively little attention has been paid to the role theory plays in our evaluation of replications as ‘failed’ or ‘successful’. This paper applies well-known arguments in philosophy of science about the interplay between theory and experiment to a contemporary case study of infants’ understanding of false belief (Onishi and Baillargeon [2005]), and attempts to replicate it. It argues that the lack of consensus about over-arching theories (...) informing both the concepts under study and the methodologies used to track them means that researchers disagree over which experiments constitute replications of the original. The second part of the paper places this specific debate within a broader discussion of the replication crisis as a crisis of ‘theory’, developing work by Muthukrishna and Henrich ([2018]) and Bird ([2018]). Bird argues that the lack of agreed over-arching theories in psychology means that a high rate of replication failure is to be expected; this paper agrees with his diagnosis but challenges his proposal that more replication will resolve the problem. (shrink)

Most discussions of the reproducibility crisis focus on its epistemic aspect: the fact that the scientific community fails to follow some norms of scientific investigation, which leads to high rates of irreproducibility via a high rate of false positive findings. The purpose of this paper is to argue that there is a heretofore underappreciated and understudied dimension to the reproducibility crisis in experimental psychology and neuroscience that may prove to be at least as important as the epistemic dimension. This is (...) the communication dimension. The link between communication and reproducibility is immediate: independent investigators would not be able to recreate an experiment whose design or implementation were inadequately described. I exploit evidence of a replicability and reproducibility crisis in computational science, as well as research into quality of reporting to support the claim that a widespread failure to adhere to reporting standards, especially the norm of descriptive completeness, is an important contributing factor in the current reproducibility crisis in experimental psychology and neuroscience. (shrink)

There is widespread excitement in the literature about the method of arbitrary functions: many take it to show that it is from the dynamics of systems that the objectivity of probabilities emerge. In this paper, I differentiate three ways in which a probability function might be objective, and I argue that the method of arbitrary functions cannot help us show that dynamics objectivise probabilities in any of these senses.

Experimental moral philosophy emerged as a methodology in the last decade of the twentieth century, as a branch of the larger experimental philosophy (X-Phi) approach. Experimental moral philosophy is the empirical study of moral intuitions, judgments, and behaviors. Like other forms of experimental philosophy, it involves gathering data using experimental methods and using these data to substantiate, undermine, or revise philosophical theories. In this case, the theories in question concern the nature of moral reasoning and judgment; the extent and sources (...) of moral obligations; the nature of a good person and a good life; even the scope and nature of moral theory itself. This entry begins with a brief look at the historical uses of empirical data in moral theory and goes on to ask what, if anything, is distinctive about experimental moral philosophy—how should we distinguish it from related work in empirical moral psychology? After discussing some strategies for answering this question, the entry examines two of the main projects within experimental moral philosophy, and then discusses some of the most prominent areas of research within the field. As we will see, in some cases experimental moral philosophy has opened up new avenues of investigation, while in other cases it has influenced longstanding debates within moral theory. (shrink)

It is easier to talk frankly to a person whom one trusts. It is also easier to agree with a scientist whom one trusts. Even though in both cases the psychological state that underlies the behavior is called ‘trust’, it is controversial whether it is a token of the same psychological type. Trust can serve an affective, epistemic, or other social function, and comes to interact with other psychological states in a variety of ways. The way that the functional role (...) of trust changes across contexts and objects is further complicated when communities and individuals mediate it through technologies, and even more so when that mediation involves artificial intelligence (AI) and machine learning (ML). In this chapter I look at the ways in which trust in institutions, and specifically the medical profession, is affected by the use of AI and ML. There are two key elements of this analysis. The first is a disanalogy between institutional trust in medicine and institutional trust in science (Irzik and Kurtulmus 2021, 2019; Kitcher 2001). I note that as AI and ML become a more prominent part of medicine, trust in a medical institution becomes more like trust in a scientific institution. This is problematic for institutional trust in medicine and the practice of medicine, since institutional trust in science has been undermined by, among other things, the spread of misinformation online and the replication crisis (Romero 2019). There is also a strong analogy between the psychological state of the person who trusts a scientific report or testimony and the psychological state of a patient who trusts individual recommendations made by a medical professional in a clinical setting. In both cases, institutional trust makes it less likely that a mistake or malfeasance will result in reactive attitudes, such as blame or anger, directed at other individual members of that institution. However, it also renders people vulnerable enough to blame the institution itself. This, with time, can erode trust in the institution and naturally leads to policy recommendations that aim to preserve institutional trust. I survey two ways in which that can be done with institutional trust in medicine in the age of AI and ML. (shrink)

The replicability of a research claim is often positioned as an important step in establishing the credibility of scientific research. This expectation persists despite ongoing disagreements over how to characterise replication practices in various contexts. Rather than attempt to explain or resolve these disagreements, we propose that there is value in exploring the variable uses of the replicability concept. To this end, we treat the replicability concept as a goal-directed tool for studying scientific practices. This approach extends scholarship on the (...) goal-directed uses of concepts within investigative practices to explore the value of reflexively interrogating those concepts used to study the sciences as practiced. In doing so, we highlight the importance of considering how and when a given concept is an appropriate tool for a given goal when studying scientific practices. For instance, in the case of the replicability concept, our examination suggests that there is value in clarifying how a given characterisation of replicability is appropriate to a given context-specific analytic and/or evaluative goal. In addition, we hope to draw attention to how Metaresearch and HPS contexts provide complementary insights for those seeking to understand when and how replication practices can help in assessing a given research claim. (shrink)

Comparative psychology experiments typically test a null statistical hypothesis against an alternative. Coupled with Morgan’s canon, this is often taken to imply that the model positing the simpler psychological capacity should be treated as a ‘default’ that must be ruled out before any other model can be accepted. It has been posited that this practice neglects evidence. I argue that the problem is deeper, including the way it structures the evaluation of evidence that is considered; it frames model choice around (...) the acceptance or rejection of a default. I oppose this default framing in all its forms and develop an evidentialist alternative. Default framing fails to respect the difference between experimental statistical hypotheses and substantive hypotheses such as models. It is not actually supported by the use of null hypothesis significance testing (which I retain), it distorts the weighting of evidence, and it systematically biases practice. (shrink)

Should a scientist rely on methodological triangulation? Heesen et al. (Synthese 196(8):3067–3081, 2019) recently provided a convincing affirmative answer. However, their approach requires belief gambles if the evidence is discordant. We instead propose epistemically modest triangulation (EMT), according to which one should withhold judgement in such cases. We show that for a scientist in a methodologically diffident situation the expected utility of EMT is greater than that of Heesen et al.’s (2019) triangulation or that of using a single method. We (...) also show that EMT is more appropriate for increasing epistemic trust in science. In short: triangulate, but do not gamble with evidence. (shrink)

Experimental moral philosophy began to emerge as a methodology inthe last decade of the twentieth century, a branch of the largerexperimental philosophy approach. From the beginning,it has been embroiled in controversy on a number of fronts. Somedoubt that it is philosophy at all. Others acknowledge that it isphilosophy but think that it has produced modest results at best andconfusion at worst. Still others think it represents an important advance., Before the research program can be evaluated, we should have someconception of (...) its scope. But controversy surrounds questionsabout its boundaries as well. Uncontroversially, the distinctionbetween experimental and non-experimental philosophy is not identicalto the distinction between a posteriori and a prioriphilosophy. Experimental evidence is a proper subset of empiricalevidence, which is itself a subset of a posteriorievidence. Can any more be said?, L1: A child raised in a particular linguistic community almostinevitably ends up speaking an idiolect of the local language despitelack of sufficient explicit instruction, lack of extensive negativefeedback for mistakes, and grammatical mistakes by caretakers., character, moral | cognitivism vs. non-cognitivism, moral | emotion | ethics: virtue | intuition | moral anti-realism | moral psychology: empirical approaches | moral realism | moral relativism | well-being, Copyright © 2014 byMark Alfano Don Loeb, View this site from another server:. (shrink)

Likelihoodism is the view that the degree of evidential support should be analysed and measured in terms of likelihoods alone. The paper considers and responds to a popular criticism that a likelihoodist framework is too restrictive to guide belief. First, I show that the most detailed and rigorous version of this criticism, as put forward by Gandenberger (2016), is unsuccessful. Second, I provide a positive argument that a broadly likelihoodist framework can accommodate guidance for comparative belief, even when objectively well-grounded (...) prior probabilities are not available. As I show, the shift from non-relational to comparative probabilities opens up a new space for addressing the belief guidance problem for likelihoodism. (shrink)

The practice of HARKing---hypothesizing after results are known---is commonly maligned as undermining the reliability of scientific findings. There are several accounts in the literature as to why HARKing undermines the reliability of findings. We argue that none of these is right and that the correct account is a Bayesian one. HARKing can indeed decrease the reliability of scientific findings, but it can also increase it; which effect HARKing produces depends on the difference of the prior odds of hypotheses characteristically selected (...) ex ante and ex post to observing data. Further, we show how misdiagnosis of HARKing can lead to misprescription in the context of the replication crisis. (shrink)

In this commentary, I propose a framework for thinking about data quality in the context of scientific research. I start by analyzing conceptualizations of quality as a property of information, evidence and data and reviewing research in the philosophy of information, the philosophy of science and the philosophy of biomedicine. I identify a push for purpose dependency as one of the main results of this review. On this basis, I present a contextual approach to data quality in scientific research, whereby (...) the quality of a dataset is dependent on the context of use of the dataset as much as the dataset itself. I exemplify the approach by discussing current critiques and debates of scientific quality, thus showcasing how data quality can be approached contextually. (shrink)

Scientists are becoming increasingly aware of a “replicability crisis” in the behavioral, social, and biomedical sciences. Researchers have made progress identifying statistical and methodological...

We propose that the epistemic functions of replication in science are best understood by relating them to kinds of experimental error/uncertainty. One kind of replication, which we call “direct replications,” principally serves to assess the reliability of an experiment through its precision: the presence and degree of random error/statistical uncertainty. The other kind of replication, which we call “conceptual replications,” principally serves to assess the validity of an experiment through its accuracy: the presence and degree of systematic errors/uncertainties. To illustrate (...) the aptness of this general view, we examine the Hubble constant controversy in astronomy, showing how astronomers have responded to the concordances and discordances in their results by carrying out the different kinds of replication that we identify, with the aim of establishing a precise, accurate value for the Hubble constant. We contrast our view with Machery's “re-sampling” account of replication, which maintains that replications only assess reliability. (shrink)

The enduring replication crisis in many scientific disciplines casts doubt on the ability of science to estimate effect sizes accurately, and in a wider sense, to self-correct its findings and to produce reliable knowledge. We investigate the merits of a particular countermeasure—replacing null hypothesis significance testing with Bayesian inference—in the context of the meta-analytic aggregation of effect sizes. In particular, we elaborate on the advantages of this Bayesian reform proposal under conditions of publication bias and other methodological imperfections that are (...) typical of experimental research in the behavioral sciences. Moving to Bayesian statistics would not solve the replication crisis single-handedly. However, the move would eliminate important sources of effect size overestimation for the conditions we study. (shrink)

The replication crisis has spawned discussions on the meaning of replication. In fact, in order to determine whether an experiment fails to replicate, it is necessary to establish what replication is. This is, however, a difficult task, as it is possible to attribute different meanings to it. This paper offers a solution to this problem of ambiguity by engineering a concept of replication that, if compared to other proposals, stands out for being not only broadly applicable but also sufficiently specific. (...) It features a minimal level of operationalism, which would otherwise limit its applicability, while it heavily relies on replication’s specific epistemic functions, which are inter-disciplinary. Another merit is its context sensitivity, which enables it to differentiate instances of replication from non-instances of replication in every scientific discipline according to the discipline’s own standards. (shrink)

Replicability is usually considered to be one of the cornerstones of science; however, the growing recognition of nonreplicable experiments and studies in scientific journals—a phenomenon that has been called ‘replicability crisis’—has spurred a debate on the meaning, function, and significance of replicability in science. Amid this discussion, it has become clear that replicability is not a monolithic concept; what is still controversial is exactly how the distinction between different kinds of replicability should be laid out terminologically and conceptually, and to (...) what extent it bears on the more general debate on the centrality of replicability in science. This paper’s goals are to clarify the different uses of the terms related to replicability and, more importantly, to conceptually specify the kinds of replicability and their respective epistemic functions. (shrink)

The replicability crisis refers to the apparent failures to replicate both important and typical positive experimental claims in psychological science and biomedicine, failures which have gained increasing attention in the past decade. In order to provide evidence that there is a replicability crisis in the first place, scientists have developed various measures of replication that help quantify or “count” whether one study replicates another. In this nontechnical essay, I critically examine five types of replication measures used in the landmark article (...) “Estimating the reproducibility of psychological science” (Open Science Collaboration, Science, 349, ac4716, 2015) based on the following techniques: subjective assessment, null hypothesis significance testing, comparing effect sizes, comparing the original effect size with the replication confidence interval, and meta-analysis. The first four, I argue, remain unsatisfactory for a variety of conceptual or formal reasons, even taking into account various improvements. By contrast, at least one version of the meta-analytic measure does not suffer from these problems. It differs from the others in rejecting dichotomous conclusions, the assumption that one study replicates another or not simpliciter. I defend it from other recent criticisms, concluding however that it is not a panacea for all the multifarious problems that the crisis has highlighted. (shrink)

Data production in experimental sciences depends on localised experimental systems, but the epistemic properties of data transcend the contingencies of the processes that produce them. Philosophers often believe that experimental systems instantiate but do not produce the epistemic properties of data. In this paper, we argue that experimental systems' local functioning entails intrinsic capacities to produce the epistemic properties of data. We develop this idea by applying Derrida's model of arche-writing to study a case of theory-oriented experimental practice. Derrida's model (...) relativises or dissolves the conceptual distinction between the moment of data production and a subsequent moment of data dissemination. It thus has consequences for understanding both data production (despite being intrinsically local, data production a priori generates transferrable and modellable information) and data dissemination (when modelling information, researchers needs to refer this information to the context of its production). We study a case of data production in a non-exploratory experimental system designed to test a pre-existing hypothesis in visual neuroscience. A case of theory-oriented experimental practice should allow us to identify the autonomous functioning of experimental systems in data production more clearly, insofar as it allows us to study the limits of pre-existing theory in the activities of these systems. We suggest that pre-existing concepts, hypotheses and theories condition the relevance but not the production of experimental data. (shrink)