Two goals - improving replicability and accountability of Machine Learning research respectively, have accrued much attention from the AI ethics and the Machine Learning community. Despite sharing the measures of improving transparency, the two goals are discussed in different registers - replicability registers with scientific reasoning whereas accountability registers with ethical reasoning. Given the existing challenge of the Responsibility Gap - holding Machine Learning scientists accountable for Machine Learning harms due to them being far from sites of application, this paper (...) posits that reconceptualizing replicability can help bridge the gap. Through a shift from model performance replicability to claim replicability, Machine Learning scientists can be held accountable for producing non-replicable claims that are prone to eliciting harm due to misuse and misinterpretation. In this paper, I make the following contributions. First, I define and distinguish two forms of replicability for ML research that can aid constructive conversations around replicability. Second, I formulate an argument for claim-replicability's advantage over model performance replicability in justifying assigning accountability to Machine Learning scientists for producing non-replicable claims and show how it enacts a sense of responsibility that is actionable. In addition, I characterize the implementation of claim replicability as more of a social project than a technical one by discussing its competing epistemological principles, practical implications on Circulating Reference, Interpretative Labor, and research communication. (shrink)

Science is based onfacts—facts that are systematically gathered by a community of enquirers through detailed observation and experiment. In the twentieth century, however, philosophers of science claimed that the facts that scientists “gather” in this way are shaped by the theories scientists accept, and this seemed to threaten the authority of science. Call this theold worries about science.By contrast, what seemed not to threaten that authority were other factors that shaped the facts that scientists gather—for example, the mere questions scientists (...) pursue. Call this theold nonworries about science.What I suggest is that the old nonworries are turning out to be far more worrisome than the old worries, and I use recent goings-on such as the “Death of Evidence” protests in Canada, the “replication crisis,” and the ongoing feminist critiques of science to illustrate my case. All this raises interesting new questions for philosophers of science to tackle. (shrink)

The philosophical interest in experimental practice in neuroscience has brought renewed attention to the study of the development and use of techniques and tools for data production. John Bickle has argued that the construction and progression of theories in neuroscience are entirely dependent on the development and ingenious use of research tools. In Bickle's account, theory plays a tertiary role, as it depends on what the tools allow researchers to manipulate, and the tools, in turn, are developed not in order (...) to test theories but as solutions to engineering problems. However, Bickle's account is not entirely precise in explaining what informs researchers' decision-making in their atheoretical laboratory tinkering. Identifying the sources that guide researchers in tool development and use is crucial if one wishes to contribute to the philosophical or meta-scientific understanding of experimental practice in neuroscience. In the following paper, I claim that decision-making in tools' development and use in neuroscience is doubly guided. Pre-existing theory and concepts determine information's relevance, whereas tools' functioning in controlled situations determines information's reliability. Accordingly, experimenters' decision-making is situated both in the context of analysing, modelling or interpreting information and in the context of producing information. I study the case of the tungsten microelectrode developed by David Hubel during the 1950s. First, I show that pre-existing theory and concepts determine in advance what information would be relevant to obtain from the microelectrode. Second, I show that Hubel's tinkering follows the guidelines derived from the very structure of what we recognise as reliable experimentally produced information. Finally, I suggest that data-production processes allow experimenters to assess what to expect from an experimental system in terms of concept- and theory-generation and confirmation, thereby endorsing Bickle's tenet on the tertiary role of theory in neuroscience. (shrink)

While the term “reactivity” has come to be associated with specific phenomena in the social sciences, having to do with subjects’ awareness of being studied, this paper takes a broader stance on this concept. I argue that reactivity is a ubiquitous feature of the psychological subject matter and that this fact is a precondition of experimental research, while also posing potential problems for the experimenter. The latter are connected to the worry about distorted data and experimental artifacts. But what are (...) experimental artifacts and what is the most productive way of dealing with them? In this paper, I approach these questions by exploring the ways in which experimenters in psychology simultaneously exploit and suppress the reactivity of their subject matter in order to produce experimental data that speak to the question or subject matter at hand. Highlighting the artificiality of experimental data, I raise the question of what distinguishes a genuine experimental result from an experimental artifact. My analysis construes experimental results as the outcomes of inferences from the data that take material background assumptions as auxiliary premises. Artifacts occur when one or more of these background assumptions are false, such that the data do not reliably serve the purposes they were generated for. I conclude by laying out the ways in which my analysis of data quality is relevant to, and informed by, recent debates about the replicability of experimental results. (shrink)

Data production in experimental sciences depends on localised experimental systems, but the epistemic properties of data transcend the contingencies of the processes that produce them. Philosophers often believe that experimental systems instantiate but do not produce the epistemic properties of data. In this paper, we argue that experimental systems' local functioning entails intrinsic capacities to produce the epistemic properties of data. We develop this idea by applying Derrida's model of arche-writing to study a case of theory-oriented experimental practice. Derrida's model (...) relativises or dissolves the conceptual distinction between the moment of data production and a subsequent moment of data dissemination. It thus has consequences for understanding both data production (despite being intrinsically local, data production a priori generates transferrable and modellable information) and data dissemination (when modelling information, researchers needs to refer this information to the context of its production). We study a case of data production in a non-exploratory experimental system designed to test a pre-existing hypothesis in visual neuroscience. A case of theory-oriented experimental practice should allow us to identify the autonomous functioning of experimental systems in data production more clearly, insofar as it allows us to study the limits of pre-existing theory in the activities of these systems. We suggest that pre-existing concepts, hypotheses and theories condition the relevance but not the production of experimental data. (shrink)

Whether we live in a world of autonomous things, or a world of interconnected processes in constant flux, is an ancient philosophical debate. Modern biology provides decisive reasons for embracing the latter view. How does one understand the practices and outputs of science in such a dynamic, ever-changing world - and particularly in an emergency situation such as the COVID-19 pandemic, where scientific knowledge has been regarded as bedrock for decisive social interventions? We argue that key to answering this question (...) is to consider the role of the activity of reification within the research process. Reification consists in the identification of more or less stable features of the flux, and treating these as constituting stable things. As we illustrate with reference to biological and biomedical research on COVID-19, reification is a necessary component of any process of inquiry and comes in at least two forms: means reification, when researchers create objects meant to capture features of the world, or phenomena, in order to be able to study them; and target reification, when researchers infer an understanding of phenomena from an investigation of the epistemic objects created to study them. We note that both objects and phenomena are dynamic processes and argue that have no reason to assume that changes in objects and phenomena track one another. We conclude that failure to acknowledge these forms of reification and their epistemic role in scientific inquiry can have dire consequences for how the resulting knowledge is interpreted and used. (shrink)

The replicability of a research claim is often positioned as an important step in establishing the credibility of scientific research. This expectation persists despite ongoing disagreements over how to characterise replication practices in various contexts. Rather than attempt to explain or resolve these disagreements, we propose that there is value in exploring the variable uses of the replicability concept. To this end, we treat the replicability concept as a goal-directed tool for studying scientific practices. This approach extends scholarship on the (...) goal-directed uses of concepts within investigative practices to explore the value of reflexively interrogating those concepts used to study the sciences as practiced. In doing so, we highlight the importance of considering how and when a given concept is an appropriate tool for a given goal when studying scientific practices. For instance, in the case of the replicability concept, our examination suggests that there is value in clarifying how a given characterisation of replicability is appropriate to a given context-specific analytic and/or evaluative goal. In addition, we hope to draw attention to how Metaresearch and HPS contexts provide complementary insights for those seeking to understand when and how replication practices can help in assessing a given research claim. (shrink)