In a recent paper, Joshua Knobe (2019) offers a startling account of the metaphilosophical implications of findings in experimental philosophy. We argue that Knobe’s account is seriously mistaken, and that it is based on a radically misleading portrait of recent work in experimental philosophy and cultural psychology.

The cross-cultural differences in epistemic intuitions reported by Weinberg, Nichols and Stich (2001; hereafter: WNS) laid the ground for the negative program of experimental philosophy. However, most of WNS’s findings were not corroborated in further studies. The exception here is the study concerning purported differences between Westerners and Indians in knowledge ascriptions concerning the Zebra Case, which was never properly replicated. Our study replicates the above-mentioned experiment on a considerably larger sample of Westerners (n = 211) and Indians (n = (...) 204). The analysis found a significant difference between the ethnic groups in question in the predicted direction: Indians were more likely to attribute knowledge in the Zebra Case than Westerners. In this paper, we offer an explanation of our result that takes into account the fact that replications of WNS’s other experiments did not find any cross-cultural differences. We argue that the Zebra Case is unique among the vignettes tested by WNS since it should not be regarded as a Gettier case but rather as a scenario exhibiting skeptical pressure concerning the reliability of sense-perception. We argue that skepticism towards perception as a means of gaining knowledge is a trope that is deeply rooted in Western epistemology but is very much absent from Classical Indian philosophical inquiry. This line of reasoning is based on a thorough examination of the skeptical scenarios discussed by philosophers of the Indian Nyaya tradition and their adversaries. (shrink)

Fisher criticised the Neyman-Pearson approach to hypothesis testing by arguing that it relies on the assumption of “repeated sampling from the same population.” The present article considers the responses to this criticism provided by Pearson and Neyman. Pearson interpreted alpha levels in relation to imaginary replications of the original test. This interpretation is appropriate when test users are sure that their replications will be equivalent to one another. However, by definition, scientific researchers do not possess sufficient knowledge about the relevant (...) and irrelevant aspects of their tests and populations to be sure that their replications will be equivalent to one another. Pearson also interpreted the alpha level as a personal rule that guides researchers’ behavior during hypothesis testing. However, this interpretation fails to acknowledge that the same researcher may use different alpha levels in different testing situations. Addressing this problem, Neyman proposed that the average alpha level adopted by a particular researcher can be viewed as an indicator of that researcher’s typical Type I error rate. Researchers’ average alpha levels may be informative from a metascientific perspective. However, they are not useful from a scientific perspective. Scientists are more concerned with the error rates of specific tests of specific hypotheses, rather than the error rates of their colleagues. It is concluded that neither Neyman nor Pearson adequately rebutted Fisher’s “repeated sampling” criticism. Fisher’s significance testing approach is briefly considered as an alternative to the Neyman-Pearson approach. (shrink)

Appendix 1 was incomplete in the initial online publication. The original article has been corrected.

Responding to recent concerns about the reliability of the published literature in psychology and other disciplines, we formed the X-Phi Replicability Project to estimate the reproducibility of experimental philosophy. Drawing on a representative sample of 40 x-phi studies published between 2003 and 2015, we enlisted 20 research teams across 8 countries to conduct a high-quality replication of each study in order to compare the results to the original published findings. We found that x-phi studies – as represented in our sample (...) – successfully replicated about 70% of the time. We discuss possible reasons for this relatively high replication rate in the field of experimental philosophy and offer suggestions for best research practices going forward. (shrink)

Two goals - improving replicability and accountability of Machine Learning research respectively, have accrued much attention from the AI ethics and the Machine Learning community. Despite sharing the measures of improving transparency, the two goals are discussed in different registers - replicability registers with scientific reasoning whereas accountability registers with ethical reasoning. Given the existing challenge of the Responsibility Gap - holding Machine Learning scientists accountable for Machine Learning harms due to them being far from sites of application, this paper (...) posits that reconceptualizing replicability can help bridge the gap. Through a shift from model performance replicability to claim replicability, Machine Learning scientists can be held accountable for producing non-replicable claims that are prone to eliciting harm due to misuse and misinterpretation. In this paper, I make the following contributions. First, I define and distinguish two forms of replicability for ML research that can aid constructive conversations around replicability. Second, I formulate an argument for claim-replicability's advantage over model performance replicability in justifying assigning accountability to Machine Learning scientists for producing non-replicable claims and show how it enacts a sense of responsibility that is actionable. In addition, I characterize the implementation of claim replicability as more of a social project than a technical one by discussing its competing epistemological principles, practical implications on Circulating Reference, Interpretative Labor, and research communication. (shrink)

The replicability of a research claim is often positioned as an important step in establishing the credibility of scientific research. This expectation persists despite ongoing disagreements over how to characterise replication practices in various contexts. Rather than attempt to explain or resolve these disagreements, we propose that there is value in exploring the variable uses of the replicability concept. To this end, we treat the replicability concept as a goal-directed tool for studying scientific practices. This approach extends scholarship on the (...) goal-directed uses of concepts within investigative practices to explore the value of reflexively interrogating those concepts used to study the sciences as practiced. In doing so, we highlight the importance of considering how and when a given concept is an appropriate tool for a given goal when studying scientific practices. For instance, in the case of the replicability concept, our examination suggests that there is value in clarifying how a given characterisation of replicability is appropriate to a given context-specific analytic and/or evaluative goal. In addition, we hope to draw attention to how Metaresearch and HPS contexts provide complementary insights for those seeking to understand when and how replication practices can help in assessing a given research claim. (shrink)

While the term “reactivity” has come to be associated with specific phenomena in the social sciences, having to do with subjects’ awareness of being studied, this paper takes a broader stance on this concept. I argue that reactivity is a ubiquitous feature of the psychological subject matter and that this fact is a precondition of experimental research, while also posing potential problems for the experimenter. The latter are connected to the worry about distorted data and experimental artifacts. But what are (...) experimental artifacts and what is the most productive way of dealing with them? In this paper, I approach these questions by exploring the ways in which experimenters in psychology simultaneously exploit and suppress the reactivity of their subject matter in order to produce experimental data that speak to the question or subject matter at hand. Highlighting the artificiality of experimental data, I raise the question of what distinguishes a genuine experimental result from an experimental artifact. My analysis construes experimental results as the outcomes of inferences from the data that take material background assumptions as auxiliary premises. Artifacts occur when one or more of these background assumptions are false, such that the data do not reliably serve the purposes they were generated for. I conclude by laying out the ways in which my analysis of data quality is relevant to, and informed by, recent debates about the replicability of experimental results. (shrink)

We present three experiments that explore the robustness of the _authentic-apparent effect_—the finding that participants are less likely to attribute knowledge to the protagonist in apparent- than in authentic-evidence Gettier cases. The results go some way towards suggesting that the effect is robust to assessments of the justificatory status of the protagonist’s belief. However, not all of the results are consistent with an effect invariant across two demographic contexts: American and Indian nationalities.

The replication or reproducibility crisis in psychological science has renewed attention to philosophical aspects of its methodology. I provide herein a new, functional account of the role of replication in a scientific discipline: to undercut the underdetermination of scientific hypotheses from data, typically by hypotheses that connect data with phenomena. These include hypotheses that concern sampling error, experimental control, and operationalization. How a scientific hypothesis could be underdetermined in one of these ways depends on a scientific discipline’s epistemic goals, theoretical (...) development, material constraints, institutional context, and their interconnections. I illustrate how these apply to the case of psychological science. I then contrast this “bottom-up” account with “top-down” accounts, which assume that the role of replication in a particular science, such as psychology, must follow from a uniform role that it plays in science generally. Aside from avoiding unaddressed problems with top-down accounts, my bottom-up account also better explains the variability of importance of replication of various types across different scientific disciplines. (shrink)

Replications are often taken to play both epistemic and demarcating roles in science: they provide evidence about the reliability of fields’ methods and, by extension, about which fields “count” as scientific. I argue that, in a field characterized by a high degree of theoretical openness and uncertainty, like comparative cognition, replications do not sit well in these roles. Like other experiments conducted under conditions of uncertainty, replications are often equivocal and open to interpretation. As a result, they are poorly placed (...) to deliver clear judgments about the reliability of comparative cognition’s methods or its scientific bona fides. I suggest that this should encourage us to take a broader view of both the nature of scientific progress and the role of replication in comparative cognition. (shrink)

The replicability crisis refers to the apparent failures to replicate both important and typical positive experimental claims in psychological science and biomedicine, failures which have gained increasing attention in the past decade. In order to provide evidence that there is a replicability crisis in the first place, scientists have developed various measures of replication that help quantify or “count” whether one study replicates another. In this nontechnical essay, I critically examine five types of replication measures used in the landmark article (...) “Estimating the reproducibility of psychological science” (Open Science Collaboration, Science, 349, ac4716, 2015) based on the following techniques: subjective assessment, null hypothesis significance testing, comparing effect sizes, comparing the original effect size with the replication confidence interval, and meta-analysis. The first four, I argue, remain unsatisfactory for a variety of conceptual or formal reasons, even taking into account various improvements. By contrast, at least one version of the meta-analytic measure does not suffer from these problems. It differs from the others in rejecting dichotomous conclusions, the assumption that one study replicates another or not simpliciter. I defend it from other recent criticisms, concluding however that it is not a panacea for all the multifarious problems that the crisis has highlighted. (shrink)

In the first controlled, non-self-report studies to show an influence of university-level ethical instruction on everyday behavior, Schwitzgebel et al. (2020) and Jalil et al. (2020) found that students purchase less meat after exposure to material on the ethics of eating meat. We sought to extend and conceptually replicate this research. Seven hundred thirty students in three large philosophy classes read James Rachels’ (2004) “Basic Argument for Vegetarianism”, followed by 50-min small-group discussions. Half also viewed a vegetarianism advocacy video containing (...) factory farm footage. A few days after instruction, 54% of students agreed that “eating the meat of factory farmed animals is unethical”, compared to 37% before instruction, with no difference between the film and non-film conditions. Also, 39% of students anonymously pledged to avoid eating factory farmed meat for 24 h, again with no statistically detectable difference between conditions. Finally, we obtained 2828 campus food purchase receipts for 113 of the enrolled students who used their Student ID cards for purchases on campus, which we compared with 5033 purchases from a group of 226 students who did not receive the instruction. Meat purchases remained constant in the comparison group and declined among the students exposed to the material, falling from 30% to 23% of purchases overall and from 51% to 42% of purchases of $4.99 or more, with the effect possibly larger in the film condition. (shrink)

In this work, we explore the epistemic import of the value-ladenness of Neyman-Pearson’s Theory of Testing Hypotheses by reconstructing and extending Daniel Steel’s argument for the legitimate influence of pragmatic values on scientific inference. We focus on how to properly understand N-P’s pragmatic value-ladenness and the epistemic reliability of N-P. We develop an account of the twofold influence of pragmatic values on N-P’s epistemic reliability and replicability. We refer to these two distinguished aspects as “direct” and “indirect”. We discuss the (...) replicability of experiments in terms of the indirect aspect and the replicability of outcomes in terms of the direct aspect. We argue that the influence of pragmatic values is beneficial to N-P’s epistemic reliability and replicability indirectly. We show that while the direct influence of pragmatic values can be beneficial, its negative effects on reliability and replicability are also unavoidable in some cases, with the direct and indirect aspects possibly being incongruent. (shrink)

Replicability is widely taken to ground the epistemic authority of science. However, in recent years, important published findings in the social, behavioral, and biomedical sciences have failed to replicate, suggesting that these fields are facing a “replicability crisis.” For philosophers, the crisis should not be taken as bad news but as an opportunity to do work on several fronts, including conceptual analysis, history and philosophy of science, research ethics, and social epistemology. This article introduces philosophers to these discussions. First, I (...) discuss precedents and evidence for the crisis. Second, I discuss methodological, statistical, and social-structural factors that have contributed to the crisis. Third, I focus on the philosophical issues raised by the crisis. Finally, I discuss proposed solutions and highlight the gaps that philosophers could focus on. (shrink)

Matarese (2022) argued in a commentary that Machery’s (2020) resampling account of replication (RAR) is unsatisfactory because it was too quick to abolish the distinction between direct and conceptual replications. Although Matarese agrees with Machery that some interpretations of conceptual replications are misconceived, she offers an interpretation which purports to show that direct and conceptual replications have different functions, and this justifies preserving the distinction. In this commentary, I will defend the RAR from Matarese’s argument. Most importantly, I will argue (...) that Matarese’s interpretation of conceptual replications is conceptually impossible. Thus, I conclude that the direct–conceptual distinction was rightly abolished by Machery. (shrink)

Scientists often respond to failures to replicate by citing differences between the experimental components of an original study and those of its attempted replication. In this paper, we investigate these purported mismatch explanations. We assess a body of failures to replicate in neuroscience studies on spinal cord injury. We argue that a defensible mismatch explanation is one where a mismatch of components is a difference maker for a mismatch of outcomes, and the components are relevantly different in the follow-up study, (...) given the scope of the original study. With this account, we argue that not all differences between studies are meaningful, even if they are difference makers. As our examples show, focusing only on these differences results in disregarding the representativeness of the original experiment’s components and the scope of its outcomes, undercutting other epistemic aims, such as translation, in the process. (shrink)

In this paper I explore an underdiscussed factor contributing to the replication crisis: Scientists, and following them policy makers, often neglect sources of errors in the production and interpretation of data and thus overestimate what can be learnt from them. This neglect leads scientists to conduct experiments that are insufficiently informative and science consumers, including other scientists, to put too much weight on experimental results. The former leads to fragile empirical literatures, the latter to surprise and disappointment when the fragility (...) of the empirical basis of some disciplines is revealed. (shrink)

The practice of HARKing---hypothesizing after results are known---is commonly maligned as undermining the reliability of scientific findings. There are several accounts in the literature as to why HARKing undermines the reliability of findings. We argue that none of these is right and that the correct account is a Bayesian one. HARKing can indeed decrease the reliability of scientific findings, but it can also increase it; which effect HARKing produces depends on the difference of the prior odds of hypotheses characteristically selected (...) ex ante and ex post to observing data. Further, we show how misdiagnosis of HARKing can lead to misprescription in the context of the replication crisis. (shrink)

We propose that the epistemic functions of replication in science are best understood by relating them to kinds of experimental error/uncertainty. One kind of replication, which we call “direct replications,” principally serves to assess the reliability of an experiment through its precision: the presence and degree of random error/statistical uncertainty. The other kind of replication, which we call “conceptual replications,” principally serves to assess the validity of an experiment through its accuracy: the presence and degree of systematic errors/uncertainties. To illustrate (...) the aptness of this general view, we examine the Hubble constant controversy in astronomy, showing how astronomers have responded to the concordances and discordances in their results by carrying out the different kinds of replication that we identify, with the aim of establishing a precise, accurate value for the Hubble constant. We contrast our view with Machery's “re-sampling” account of replication, which maintains that replications only assess reliability. (shrink)

Cultural phylogenetics has made remarkable progress by relying on methods originally developed in biology. But biological and cultural evolution do not always proceed according to the same principles. So what, if anything, could justify the use of phylogenetic methods to reconstruct the evolutionary history of culture? In this paper, we describe models used to assess the reliability of inference methods and show how these models play an underappreciated role in addressing that question. The notion of reliability is of course central (...) to these models. As we explain, a common way of understanding reliability is in terms of low error rates. A careful look at case studies in cultural phylogenetics suggests that reliability models partly corroborate this understanding of reliability but also raises points of tension. We conclude by hinting at a few ways forward. (shrink)

What does it mean to replicate an experiment? A distinction is often drawn between ‘exact’ (or ‘direct’) and ‘conceptual’ replication. However, in recent work, Uljana Feest argues that the notion of replication in itself, whether exact or conceptual, is flawed due to the problem of systematic error, and Edouard Machery argues that, although the notion of replication is not flawed, we should nevertheless dispense with the distinction between exact and conceptual replication. My plan in this paper is to defend the (...) value of replication, along with the distinction between exact and conceptual replication, from the critiques of Feest and Machery. To that end, I provide an explication of conceptual replication, and distinguish it from what I call ‘experimental’ replication. On the basis, then, of a tripartite distinction between exact, experimental and conceptual replication, I argue in response to Feest that replication is still informative despite the prospect of systematic error. I also rebut Machery’s claim that conceptual replication is fundamentally confused and wrongly conflates replication and extension, and in turn raise some objections to his own Resampling Account of replication. (shrink)

The replication crisis has spawned discussions on the meaning of replication. In fact, in order to determine whether an experiment fails to replicate, it is necessary to establish what replication is. This is, however, a difficult task, as it is possible to attribute different meanings to it. This paper offers a solution to this problem of ambiguity by engineering a concept of replication that, if compared to other proposals, stands out for being not only broadly applicable but also sufficiently specific. (...) It features a minimal level of operationalism, which would otherwise limit its applicability, while it heavily relies on replication’s specific epistemic functions, which are inter-disciplinary. Another merit is its context sensitivity, which enables it to differentiate instances of replication from non-instances of replication in every scientific discipline according to the discipline’s own standards. (shrink)

Replicability is usually considered to be one of the cornerstones of science; however, the growing recognition of nonreplicable experiments and studies in scientific journals—a phenomenon that has been called ‘replicability crisis’—has spurred a debate on the meaning, function, and significance of replicability in science. Amid this discussion, it has become clear that replicability is not a monolithic concept; what is still controversial is exactly how the distinction between different kinds of replicability should be laid out terminologically and conceptually, and to (...) what extent it bears on the more general debate on the centrality of replicability in science. This paper’s goals are to clarify the different uses of the terms related to replicability and, more importantly, to conceptually specify the kinds of replicability and their respective epistemic functions. (shrink)

In the first controlled, non-self-report studies to show an influence of university-level ethical instruction on everyday behavior, Schwitzgebel et al. (2020) and Jalil et al. (2020) found that students purchase less meat after exposure to material on the ethics of eating meat. We sought to extend and conceptually replicate this research. Seven hundred thirty students in three large philosophy classes read James Rachels’ (2004) “Basic Argument for Vegetarianism”, followed by 50-min small-group discussions. Half also viewed a vegetarianism advocacy video containing (...) factory farm footage. A few days after instruction, 54% of students agreed that “eating the meat of factory farmed animals is unethical”, compared to 37% before instruction, with no difference between the film and non-film conditions. Also, 39% of students anonymously pledged to avoid eating factory farmed meat for 24 h, again with no statistically detectable difference between conditions. Finally, we obtained 2828 campus food purchase receipts for 113 of the enrolled students who used their Student ID cards for purchases on campus, which we compared with 5033 purchases from a group of 226 students who did not receive the instruction. Meat purchases remained constant in the comparison group and declined among the students exposed to the material, falling from 30% to 23% of purchases overall and from 51% to 42% of purchases of $4.99 or more, with the effect possibly larger in the film condition. (shrink)

De nombreux commentaires des expériences de Milgram soutiennent l’Hypothèse d’incrédulité, laquelle soutient que les participants de Milgram n’auraient en général pas cru qu’ils administraient des chocs électriques réels. Si l’Hypothèse d’incrédulité était juste, on devrait en conclure que les sujets obéissants ne croyaient pas mal agir, ce qui impliquerait que Milgram a échoué à mettre en évidence des niveaux alarmants d’obéissance destructrice. Dans cet article, nous démontrons que l’Hypothèse d’incrédulité n’est, en général, pas exacte : elle n’explique que très difficilement (...) le comportement des participants dans les expériences de Milgram et ses nombreuses réplications, et elle concorde mal avec les témoignages des participants concernant leur propre vécu. (shrink)