Two goals - improving replicability and accountability of Machine Learning research respectively, have accrued much attention from the AI ethics and the Machine Learning community. Despite sharing the measures of improving transparency, the two goals are discussed in different registers - replicability registers with scientific reasoning whereas accountability registers with ethical reasoning. Given the existing challenge of the Responsibility Gap - holding Machine Learning scientists accountable for Machine Learning harms due to them being far from sites of application, this paper (...) posits that reconceptualizing replicability can help bridge the gap. Through a shift from model performance replicability to claim replicability, Machine Learning scientists can be held accountable for producing non-replicable claims that are prone to eliciting harm due to misuse and misinterpretation. In this paper, I make the following contributions. First, I define and distinguish two forms of replicability for ML research that can aid constructive conversations around replicability. Second, I formulate an argument for claim-replicability's advantage over model performance replicability in justifying assigning accountability to Machine Learning scientists for producing non-replicable claims and show how it enacts a sense of responsibility that is actionable. In addition, I characterize the implementation of claim replicability as more of a social project than a technical one by discussing its competing epistemological principles, practical implications on Circulating Reference, Interpretative Labor, and research communication. (shrink)

Depression is a widespread and debilitating disorder, but developing effective treatments has proven challenging. Despite success in animal models, many treatments fail in human trials. While various factors contribute to this translational failure, standardization practices in animal research are often overlooked. This paper argues that certain standardization choices in behavioral neuroscience research on depression can limit the generalizability of results from rodents to humans. This raises ethical and scientific concerns, including animal waste and a lack of progress in treating human (...) patients. To address these issues, animal ethics committees can establish reasonable expectations for preclinical research and highlight the impact of standardization on generalizability. Such efforts can help improve translation from animal models to human patients and ultimately benefit those suffering from depression. (shrink)

Metascientists have studied questionable research practices in science. The present article considers the parallel concept of questionable metascience practices (QMPs). A QMP is a research practice, assumption, or perspective that has been questioned by several commentators as being potentially problematic for metascience and/or the science reform movement. The present article reviews ten QMPs that relate to criticism, replication, bias, generalization, and the characterization of science. Specifically, the following QMPs are considered: (1) rejecting or ignoring self-criticism; (2) a fast ‘n’ bropen (...) scientific criticism style; (3) overplaying the role of replication in science; (4) assuming a replication rate is “too low” without specifying an “acceptable” rate; (5) an unacknowledged metabias towards explaining the replication crisis in terms of researcher bias; (6) assuming that researcher bias can be reduced; (7) devaluing exploratory results as being more “tentative” than confirmatory results; (8) presuming that questionable research practices are problematic research practices; (9) focusing on knowledge accumulation; and (10) focusing on specific scientific methods. It is stressed that only some metascientists engage in some QMPs some of the time, and that these QMPs may not always be problematic. Research is required to estimate the prevalence and impact of QMPs. In the meantime, QMPs should be viewed as invitations to ask questions about how we go about doing better metascience. (shrink)

Replicability is widely regarded as one of the defining features of science and its pursuit is one of the main postulates of meta-research, a discipline emerging in response to the replicability crisis. At the same time, replicability is typically treated with caution by philosophers of science. In this paper, we reassess the value of replicability from an epistemic perspective. We defend the orthodox view, according to which replications are always epistemically useful, against the more prudent view that claims that it (...) is useful in very limited circumstances. Additionally, we argue that we can learn more about the original experiment and the limits of the discovered effect from replications at different levels. We hold that replicability is a crucial feature of experimental results and scientists should continue to strive to secure it. (shrink)

The replicability of a research claim is often positioned as an important step in establishing the credibility of scientific research. This expectation persists despite ongoing disagreements over how to characterise replication practices in various contexts. Rather than attempt to explain or resolve these disagreements, we propose that there is value in exploring the variable uses of the replicability concept. To this end, we treat the replicability concept as a goal-directed tool for studying scientific practices. This approach extends scholarship on the (...) goal-directed uses of concepts within investigative practices to explore the value of reflexively interrogating those concepts used to study the sciences as practiced. In doing so, we highlight the importance of considering how and when a given concept is an appropriate tool for a given goal when studying scientific practices. For instance, in the case of the replicability concept, our examination suggests that there is value in clarifying how a given characterisation of replicability is appropriate to a given context-specific analytic and/or evaluative goal. In addition, we hope to draw attention to how Metaresearch and HPS contexts provide complementary insights for those seeking to understand when and how replication practices can help in assessing a given research claim. (shrink)

Nowadays, almost everyone seems to agree that science is facing an epistemological crisis – namely the replicability crisis – and that we need to take action. But as to precisely what to do or how...

Computational reproducibility possesses its own dynamics and narratives of crisis. Alongside the difficulties of computing as an ubiquitous yet complex scientific activity, computational reproducibility suffers from a naive expectancy of total reproducibility and a moral imperative to embrace the principles of free software as a non-negotiable epistemic virtue. We argue that the epistemic issues at stake in actual practices of computational reproducibility are best unveiled by focusing on software as a pivotal concept, one that is surprisingly often overlooked in accounts (...) of reproducibility issues. Software is not only about designing and coding but also about maintaining, supporting, distributing, licensing, and governance; it is not only about developers but also about users. We focus on openness debates among computational chemists involved in molecular modeling software packages as empirical grounding for our argument. We then identify and analyse four epistemic characteristics as key to the role of software in computational reproducibility. (shrink)

In this commentary, I propose a framework for thinking about data quality in the context of scientific research. I start by analyzing conceptualizations of quality as a property of information, evidence and data and reviewing research in the philosophy of information, the philosophy of science and the philosophy of biomedicine. I identify a push for purpose dependency as one of the main results of this review. On this basis, I present a contextual approach to data quality in scientific research, whereby (...) the quality of a dataset is dependent on the context of use of the dataset as much as the dataset itself. I exemplify the approach by discussing current critiques and debates of scientific quality, thus showcasing how data quality can be approached contextually. (shrink)

Replicability is widely taken to ground the epistemic authority of science. However, in recent years, important published findings in the social, behavioral, and biomedical sciences have failed to replicate, suggesting that these fields are facing a “replicability crisis.” For philosophers, the crisis should not be taken as bad news but as an opportunity to do work on several fronts, including conceptual analysis, history and philosophy of science, research ethics, and social epistemology. This article introduces philosophers to these discussions. First, I (...) discuss precedents and evidence for the crisis. Second, I discuss methodological, statistical, and social-structural factors that have contributed to the crisis. Third, I focus on the philosophical issues raised by the crisis. Finally, I discuss proposed solutions and highlight the gaps that philosophers could focus on. (shrink)

Current debates about the replication crisis in psychology take it for granted that direct replication is valuable and focus their attention on questionable research practices in regard to statistical analyses. This paper takes a broader look at the notion of replication as such. It is argued that all experimentation/replication involves individuation judgments and that research in experimental psychology frequently turns on probing the adequacy of such judgments. In this vein, I highlight the ubiquity of conceptual and material questions in research, (...) and I argue that replication is not as central to psychological research as it is sometimes taken to be. (shrink)

Replicability is usually considered to be one of the cornerstones of science; however, the growing recognition of nonreplicable experiments and studies in scientific journals—a phenomenon that has been called ‘replicability crisis’—has spurred a debate on the meaning, function, and significance of replicability in science. Amid this discussion, it has become clear that replicability is not a monolithic concept; what is still controversial is exactly how the distinction between different kinds of replicability should be laid out terminologically and conceptually, and to (...) what extent it bears on the more general debate on the centrality of replicability in science. This paper’s goals are to clarify the different uses of the terms related to replicability and, more importantly, to conceptually specify the kinds of replicability and their respective epistemic functions. (shrink)

Discussions of reproducibility are casting doubts on the credibility of experimental outcomes in the life sciences. Although experimental evolution is not typically included in these discussions, this field is also subject to low reproducibility, partly because of the inherent contingencies affecting the evolutionary process. A received view in experimental studies more generally is that standardization (i.e., rigorous homogenization of experimental conditions) is a solution to some issues of significance and internal validity. However, this solution hides several difficulties, including a reduction (...) of external validity and reproducibility. After explaining the meaning of these two notions in the context of experimental evolution, we import from the fields of animal research and ecology and suggests that systematic heterogenization of experimental factors could prove a promising alternative. We also incorporate into our analysis some philosophical reflections on the nature and diversity of research objectives in experimental evolution. (shrink)

Data production in experimental sciences depends on localised experimental systems, but the epistemic properties of data transcend the contingencies of the processes that produce them. Philosophers often believe that experimental systems instantiate but do not produce the epistemic properties of data. In this paper, we argue that experimental systems' local functioning entails intrinsic capacities to produce the epistemic properties of data. We develop this idea by applying Derrida's model of arche-writing to study a case of theory-oriented experimental practice. Derrida's model (...) relativises or dissolves the conceptual distinction between the moment of data production and a subsequent moment of data dissemination. It thus has consequences for understanding both data production (despite being intrinsically local, data production a priori generates transferrable and modellable information) and data dissemination (when modelling information, researchers needs to refer this information to the context of its production). We study a case of data production in a non-exploratory experimental system designed to test a pre-existing hypothesis in visual neuroscience. A case of theory-oriented experimental practice should allow us to identify the autonomous functioning of experimental systems in data production more clearly, insofar as it allows us to study the limits of pre-existing theory in the activities of these systems. We suggest that pre-existing concepts, hypotheses and theories condition the relevance but not the production of experimental data. (shrink)

Irreplicability is framed as crisis, blamed on sloppy science motivated by perverse stimuli in research. Structural changes to the organization of science, targeting sloppy science (e.g., open data, pre‐registration), are proposed to prevent irreplicability. While there is an unquestionable link between sloppy science and failures to replicate/reproduce scientific studies, they are currently conflated. This position can be understood as a result of the erosion of the role of theory in science. The history, sociology, and philosophy of science reveal alternative explanations (...) for irreplicability to show it is part of proper, informative and valuable science. Irreplicability need not equate research waste. Sloppy science is the problem, also when results do replicate. Hence, the solution should focus on opposing sloppy research. (shrink)

The replication or reproducibility crisis in psychological science has renewed attention to philosophical aspects of its methodology. I provide herein a new, functional account of the role of replication in a scientific discipline: to undercut the underdetermination of scientific hypotheses from data, typically by hypotheses that connect data with phenomena. These include hypotheses that concern sampling error, experimental control, and operationalization. How a scientific hypothesis could be underdetermined in one of these ways depends on a scientific discipline’s epistemic goals, theoretical (...) development, material constraints, institutional context, and their interconnections. I illustrate how these apply to the case of psychological science. I then contrast this “bottom-up” account with “top-down” accounts, which assume that the role of replication in a particular science, such as psychology, must follow from a uniform role that it plays in science generally. Aside from avoiding unaddressed problems with top-down accounts, my bottom-up account also better explains the variability of importance of replication of various types across different scientific disciplines. (shrink)

Recently, many scientists have become concerned about an excessive number of failures to reproduce statistically significant effects. The situation has become dire enough that the situation has been named the ‘reproducibility crisis’. After reviewing the relevant literature to confirm the observation that scientists do indeed view replication as currently problematic, I explain in philosophical terms why the replication of empirical phenomena, such as statistically significant effects, is important for scientific progress. Following that explanation, I examine various diagnoses of the reproducibility (...) crisis, and argue that for the majority of scientists the crisis is due, at least in part, to a form of publication bias. This conclusion sets the stage for an assessment of the view that evidential relations in science are inherently value-laden, a view championed by Heather Douglas and Kevin Elliott. I argue, in response to Douglas and Elliott, and as motivated by the meta-scientific resistance scientists harbour to a publication bias, that if we advocate the value-ladenness of science the result would be a deepening of the reproducibility crisis. (shrink)

We propose that the epistemic functions of replication in science are best understood by relating them to kinds of experimental error/uncertainty. One kind of replication, which we call “direct replications,” principally serves to assess the reliability of an experiment through its precision: the presence and degree of random error/statistical uncertainty. The other kind of replication, which we call “conceptual replications,” principally serves to assess the validity of an experiment through its accuracy: the presence and degree of systematic errors/uncertainties. To illustrate (...) the aptness of this general view, we examine the Hubble constant controversy in astronomy, showing how astronomers have responded to the concordances and discordances in their results by carrying out the different kinds of replication that we identify, with the aim of establishing a precise, accurate value for the Hubble constant. We contrast our view with Machery's “re-sampling” account of replication, which maintains that replications only assess reliability. (shrink)

The “practice turn” in philosophy of science has strengthened the connections between philosophy and scientific practice. Apart from reinvigorating philosophy of science, this also increases the relevance of philosophical research for science, society, and science education. In this paper, we reflect on our extensive experience with teaching mandatory philosophy of science courses to science students from a range of programs at University of Copenhagen. We highlight some of the lessons we have learned in making philosophy of science “fit for teaching” (...) outside of philosophy circles by taking selected cases from the students’ own field as the starting point. We argue for adapting philosophy of science teaching to particular audiences of science students, and discuss the benefits of drawing on research within science education to inform curriculum and course design. This involves reconsidering teaching resources, assumptions about students, intended learning outcomes, and teaching formats. We also argue that to make philosophy of science relevant and engaging to science students, it is important to consider their potential career trajectories. By anticipating future contexts and situations in which methodological, conceptual, and ethical questions could be relevant, philosophy of science can demonstrate its value in the education of science students. (shrink)

This paper aims to address some methodological issues related to case-based research in the philosophy of contemporary sciences. We focus on the selection processes by which philosophers pick or generate a particular set of papers to conduct their case-based research. We illustrate how to use various quantitative and qualitative methods to improve the epistemic features of the selection processes, and help generate some potential case-based hypotheses for further philosophical investigation.

Failure to consistently reproduce experimental results, i.e. failure to reliably identify or quantify an effect — often dubbed a ‘reproducibility crisis’ when referring to a large number of studies in a given field — has become a serious concern in many communities and is widely believed to be caused by lack of systematic methodological description, poor experimental practice, or outright fraud. On the other hand, it is common knowledge of the scientific practice that replicate experiments — even when performed in (...) the same lab, by the same experimenter — will rarely show complete quantitative agreement between them. The presence of the widely believed and commonplace explanations are not mutually exclusive, but they are incompatible as justifications for irreproducibility. Invoking the former implies an anomaly, a crisis, while the latter is statistically expected and therefore amenable to quantification. Interpreting two or more studies as conflicting is often a reduction to a mechanicist view where a ground truth exists that must be observed with every properly performed experiment; a slightly less naive view is a frequentist view where statistical tests must confidently identify a true effect as significant almost always. A broader view, however, may consider that the effect can only be observed as a probability distribution; individual experiments are, therefore, not expected to differ only by sampling and power to identify a significant effect, but by variation at the level of the parameter value itself — i.e. it is accepted that there are sources of variation that cannot be controlled with infinite precision, for instance in the environment and from the experimenter, or it is acknowledged that there may be unknown, uncontrolled factors that will introduce biases. Quantitatively, that perspective is consistent with a Bayesian hierarchical formulation, where the effect parameters are under a hyperprior and above individual experiment parameters. Put another way, the Bayesian hierarchical view allows reconciliation between seemingly discordant results by interpreting each experiment as a sample itself of a distribution, which in turn sets the range and probability of expected outcomes for new individual experiments. As a corollary, a large number of replicates will increase the confidence not only in the expected value but also in the deviation for it. Thus, “validating” an experiment does not mean getting the same number every time, but establishing the range and likelihood of well-performed experiments. Conversely, once an experiment has been extensively replicated, the effect distribution is informative of how much each repetition deviates from expectation, whether they are actually extreme — and potentially contain anomalies or misconduct — or if they are probabilistically not surprising. This formulation has profound consequences for assessments and claims on reproducibility. (shrink)

Discussions about a replicability crisis in science have been driven by the normative claim that all of science should be replicable and the empirical claim that most of it isn’t. Recently, such crisis talk has been challenged by a new localism, which argues a) that serious problems with replicability are not a general occurrence in science and b) that replicability itself should not be treated as a universal standard. The goal of this article is to introduce this emerging strand of (...) the debate and to discuss some of its implications and limitations. I will in particular highlight the issue of demarcation that localist accounts have to address, i.e. the question of how we can distinguish replicable science from disciplines where replicability does not apply. (shrink)

Fact-checking agencies assess and score the truthfulness of politicians’ claims to foster their electoral accountability. Fact-checking is sometimes presented as a quasi-scientific activity, based on reproducible verification protocols that would guarantee an unbiased assessment. We will study these verification protocols and discuss under which conditions fact-checking could achieve effective reproducibility. Through an analysis of the methodological norms in verification protocols, we will argue that achieving reproducible fact-checking may not help much in rendering politicians accountable. Political fact-checkers do not deliver either (...) reproducibility or accountability today, and there are reasons to think that traditional quality journalism may serve liberal democracies better. (shrink)

Scientists are becoming increasingly aware of a “replicability crisis” in the behavioral, social, and biomedical sciences. Researchers have made progress identifying statistical and methodological...