Given the importance of scientific research in shaping our perception of the world, and our senses of what policies will and won’t succeed in altering that world, it is of great practical, political, and moral importance that we carry out scientific research with integrity. The phenomenon of scientific fraud stands in the way of that, as scientists may knowingly enter claims they take to be false into the scientific literature, often knowingly doing so in defiance of norms they profess allegiance (...) to. In this chapter we take a look at some of the causes of scientific fraud, and how it might be manifested in large-scale research teams and situations of anonymous authorship. We find that such cases make trouble for what might seem like intuitive answers to the question “who should be held responsible for this fraud?”, and we argue that in such cases it would be better to hold the entire community responsible for seeing to it that there is less fraud. (shrink)

In their recent book, Oreskes and Conway describe the ‘tobacco strategy’, which was used by the tobacco industry to influence policymakers regarding the health risks of tobacco products. The strategy involved two parts, consisting of promoting and sharing independent research supporting the industry’s preferred position and funding additional research, but selectively publishing the results. We introduce a model of the tobacco strategy, and use it to argue that both prongs of the strategy can be extremely effective—even when policymakers rationally update (...) on all evidence available to them. As we elaborate, this model helps illustrate the conditions under which the tobacco strategy is particularly successful. In addition, we show how journalists engaged in ‘fair’ reporting can inadvertently mimic the effects of industry on public belief. 1Introduction2Epistemic Network Models3Selective Sharing4Biased Production5Journalists as Unwitting Propagandists6ConclusionAppendix. (shrink)

In the last decade, many problematic cases of scientific conduct have been diagnosed; some of which involve outright fraud others are more subtle. These and similar problems can be interpreted as caused by lack of scientific objectivity. The current philosophical theories of objectivity do not provide scientists with conceptualizations that can be effectively put into practice in remedying these issues. We propose a novel way of thinking about objectivity for individual scientists; a negative and dynamic approach.We provide a philosophical conceptualization (...) of objectivity that is informed by empirical research. In particular, it is our intention to take the first steps in providing an empirically and methodologically informed inventory of factors that impair the scientific practice. The inventory will be compiled into a negative conceptualization, which could in principle be used by individual scientists to assess objectivity of scientific practice. We propose a preliminary outline of a usable and testable instrument for indicating the objectivity of scientific practice. (shrink)

This paper presents an account of how to evaluate formal models of science: models and simulations in social epistemology designed to draw normative conclusions about the social structure of scientific research. I argue that such models should be evaluated according to their representational and predictive accuracy. Using these criteria and comparisons with familiar models from science, I argue that most formal models of science are incapable of supporting normative conclusions.

The enduring replication crisis in many scientific disciplines casts doubt on the ability of science to estimate effect sizes accurately, and in a wider sense, to self-correct its findings and to produce reliable knowledge. We investigate the merits of a particular countermeasure—replacing null hypothesis significance testing with Bayesian inference—in the context of the meta-analytic aggregation of effect sizes. In particular, we elaborate on the advantages of this Bayesian reform proposal under conditions of publication bias and other methodological imperfections that are (...) typical of experimental research in the behavioral sciences. Moving to Bayesian statistics would not solve the replication crisis single-handedly. However, the move would eliminate important sources of effect size overestimation for the conditions we study. (shrink)

The enduring replication crisis in many scientific disciplines casts doubt on the ability of science to estimate effect sizes accurately, and in a wider sense, to self-correct its findings and to produce reliable knowledge. We investigate the merits of a particular countermeasure—replacing null hypothesis significance testing with Bayesian inference—in the context of the meta-analytic aggregation of effect sizes. In particular, we elaborate on the advantages of this Bayesian reform proposal under conditions of publication bias and other methodological imperfections that are (...) typical of experimental research in the behavioral sciences. Moving to Bayesian statistics would not solve the replication crisis single-handedly. However, the move would eliminate important sources of effect size overestimation for the conditions we study. (shrink)

Replicability is widely taken to ground the epistemic authority of science. However, in recent years, important published findings in the social, behavioral, and biomedical sciences have failed to replicate, suggesting that these fields are facing a “replicability crisis.” For philosophers, the crisis should not be taken as bad news but as an opportunity to do work on several fronts, including conceptual analysis, history and philosophy of science, research ethics, and social epistemology. This article introduces philosophers to these discussions. First, I (...) discuss precedents and evidence for the crisis. Second, I discuss methodological, statistical, and social-structural factors that have contributed to the crisis. Third, I focus on the philosophical issues raised by the crisis. Finally, I discuss proposed solutions and highlight the gaps that philosophers could focus on. (shrink)

Background: Evidence suggesting adverse drug reactions often emerges unsystematically and unpredictably in form of anecdotal reports, case series and survey data. Safety trials and observational studies also provide crucial information regarding the (un-)safety of drugs. Hence, integrating multiple types of pharmacovigilance evidence is key to minimising the risks of harm. Methods: In previous work, we began the development of a Bayesian framework for aggregating multiple types of evidence to assess the probability of a putative causal link between drugs and side (...) effects. This framework arose out of a philosophical analysis of the Bradford Hill Guidelines. In this article, we expand the Bayesian framework and add “evidential modulators,” which bear on the assessment of the reliability of incoming study results. The overall framework for evidence synthesis, “E-Synthesis”, is then applied to a case study. Results: Theoretically and computationally, E-Synthesis exploits coherence of partly or fully independent evidence converging towards the hypothesis of interest (or of conflicting evidence with respect to it), in order to update its posterior probability. With respect to other frameworks for evidence synthesis, our Bayesian model has the unique feature of grounding its inferential machinery on a consolidated theory of hypothesis confirmation (Bayesian epistemology), and in allowing any data from heterogeneous sources (cell-data, clinical trials, epidemiological studies), and methods (e.g., frequentist hypothesis testing, Bayesian adaptive trials, etc.) to be quantitatively integrated into the same inferential framework. Conclusions: E-Synthesis is highly flexible concerning the allowed input, while at the same time relying on a consistent computational system, that is philosophically and statistically grounded. Furthermore, by introducing evidential modulators, and thereby breaking up the different dimensions of evidence (strength, relevance, reliability), E-Synthesis allows them to be explicitly tracked in updating causal hypotheses. (shrink)

Amalgamating evidence from heterogeneous sources and across levels of inquiry is becoming increasingly important in many pure and applied sciences. This special issue provides a forum for researchers from diverse scientific and philosophical perspectives to discuss evidence amalgamation, its methodologies, its history, its pitfalls, and its potential. We situate the contributions therein within six themes from the broad literature on this subject: the variety-of-evidence thesis, the philosophy of meta-analysis, the role of robustness/sensitivity analysis for evidence amalgamation, its bearing on questions (...) of extrapolation and external validity of experiments, its connection with theory development, and its interface with causal inference, especially regarding causal theories of cancer. (shrink)

According to the variety of evidence thesis items of evidence from independent lines of investigation are more confirmatory, ceteris paribus, than, for example, replications of analogous studies. This thesis is known to fail (Bovens and Hartmann; Claveau). However, the results obtained by Bovens and Hartmann only concern instruments whose evidence is either fully random or perfectly reliable; instead, for Claveau, unreliability is modelled as deterministic bias. In both cases, the unreliable instrument delivers totally irrelevant information. We present a model that (...) formalizes both reliability and unreliability differently. Our instruments either are reliable, but affected by random error, or are biased but not deterministically so.Bovens and Hartmann’s results are counter-intuitive in that in their model a long series of consistent reports from the same instrument does not raise suspicion of ‘too-good-to-be-true’ evidence. This happens precisely because they contemplate neither the role of systematic bias, nor unavoidable random error of reliable instruments. In our model, the variety of evidence thesis fails as well, but the area of failure is considerably smaller than for Bovens and Hartmann and Claveau, and holds for (the majority of) realistic cases (that is, where biased instruments are very biased). The essential mechanism that triggers variety of evidence thesis failure is the rate of false to true positives for the two kinds of instruments. Our emphasis is on modelling beliefs about sources of knowledge and their role in hypothesis confirmation in interaction with dimensions of evidence, such as variety and consistency. (shrink)

This article develops a new, general account of replication. I argue that a replication is an experiment that resamples the experimental components of an ori...

Arguments that medical decision making should rely on a variety of evidence often begin from the claim that meta-analysis has been shown to be problematic. In this paper, I first examine Stegenga’s argument that meta-analysis requires multiple decisions and thus fails to provide an objective ground for medical decision making. Next, I examine three arguments from social epistemologists that contend that meta-analyses are systematically biased in ways not appreciated by standard epistemology. In most cases I show that critiques of meta-analysis (...) fail to account for the full range of meta-analytic procedures. In the remainder of cases, I argue that the critiques identify problems that do not uniquely cut against meta-analysis. I close by suggesting one reason why it may be pragmatically rational to violate the principle of total evidence and by outlining the criteria for a successful argument against meta-analysis. A set of criteria I contend remain unmet. (shrink)

The replicability crisis refers to the apparent failures to replicate both important and typical positive experimental claims in psychological science and biomedicine, failures which have gained increasing attention in the past decade. In order to provide evidence that there is a replicability crisis in the first place, scientists have developed various measures of replication that help quantify or “count” whether one study replicates another. In this nontechnical essay, I critically examine five types of replication measures used in the landmark article (...) “Estimating the reproducibility of psychological science” (Open Science Collaboration, Science, 349, ac4716, 2015) based on the following techniques: subjective assessment, null hypothesis significance testing, comparing effect sizes, comparing the original effect size with the replication confidence interval, and meta-analysis. The first four, I argue, remain unsatisfactory for a variety of conceptual or formal reasons, even taking into account various improvements. By contrast, at least one version of the meta-analytic measure does not suffer from these problems. It differs from the others in rejecting dichotomous conclusions, the assumption that one study replicates another or not simpliciter. I defend it from other recent criticisms, concluding however that it is not a panacea for all the multifarious problems that the crisis has highlighted. (shrink)

Formal models of scientific inquiry, aimed at capturing socio-epistemic aspects underlying the process of scientific research, have become an important method in formal social epistemology and philosophy of science. In this introduction to the special issue we provide a historical overview of the development of formal models of this kind and analyze their methodological contributions to discussions in philosophy of science. In particular, we show that their significance consists in different forms of ‘methodological iteration’ whereby the models initiate new lines (...) of inquiry, isolate and clarify problems with existing knowledge claims, and stimulate further research. (shrink)

We argue that the main results of scientific papers may appropriately be published even if they are false, unjustified, and not believed to be true or justified by their author. To defend this claim we draw upon the literature studying the norms of assertion, and consider how they would apply if one attempted to hold claims made in scientific papers to their strictures, as assertions and discovery claims in scientific papers seem naturally analogous. We first use a case study of (...) William H. Bragg’s early 20th century work in physics to demonstrate that successful science has in fact violated these norms. We then argue that features of the social epistemic arrangement of science which are necessary for its long run success require that we do not hold claims of scientific results to their standards. We end by making a suggestion about the norms that it would be appropriate to hold scientific claims to, along with an explanation of why the social epistemology of science—considered as an instance of collective inquiry—would require such apparently lax norms for claims to be put forward. (shrink)

Financial conflicts of interest in medical research foster deviations from research standards and evidentially lead to the suppression of research findings that are at odds with commercial interests of pharmaceutical companies. Questionable research practices prevent data from being created, made available, or given suitable recognition. They run counter to codified principles of responsible conduct of research, such as honesty, openness or respect for the law. Resulting in ignorance, misrepresentation and suspension of scientific self-correction, suppression of medical evidence in its various (...) forms is both a threat to the epistemic and the moral integrity of medical science. This paper discusses an example of the suppression of medical evidence in recent influenza research and develops a conceptual framework for the description and assessment of questionable research practices applied in research and publication processes to suppress evidence. (shrink)

It's natural to think of scientists as truth seekers, people driven by an intense curiosity to understand the natural world. Yet this picture of scientists and scientific inquiry sits uncomfortably with the reality and prevalence of scientific fraud. If one wants to get at the truth about nature, why lie? Won't that just set inquiry back, as people pursue false leads? To understand why this occurs – and what can be done about it – we need to understand the social (...) structures scientists work within, and how some of the institutions which enable science to be such a successful endeavour all things considered, also abet and encourage fraud. (shrink)

How we can reliably draw inferences from data, evidence and/or experience has been and continues to be a pressing question in everyday life, the sciences, politics and a number of branches in philosophy (traditional epistemology, social epistemology, formal epistemology, logic and philosophy of the sciences). In a world in which we can now longer fully rely on our experiences, interlocutors, measurement instruments, data collection and storage systems and even news outlets to draw reliable inferences, the issue becomes even more pressing. (...) While we were working on this question using a formal epistemology approach Landes and Osimani (2020); De Pretis et al. (2019); Osimani and Landes (2020); Osimani (2020), we realised that the width of current active interests in the notion of reliability was much broader than we initially thought. Given the breadth of approaches and angles present in philosophy (even in this journal Schubert 2012; Avigad 2021; Claveau and Grenier 2019; Kummerfeld and Danks 2014; Landes 2021; Trpin et al. 2021; Schippers 2014; Schindler 2011; Kelly et al. 2016; Mayo-Wilson 2014; Olsson and Schubert 2007; Pittard 2017), we thought that it would be beneficial to provide a forum for an open exchange of ideas, in which philosophers working in different paradigms could come together. Our call for expression of interest received a great variety of promised manuscripts, the variety is reflected in the published papers. They range from fields far away from our own interests such as quantum probabilities de Ronde et al. (2021), evolvable software systems Primiero et al. (2021), to topics closer to our own research in the philosophy of medicine Lalumera et al. (2020), psychology Dutilh et al. (2021), traditional epistemology Dunn (2021); Tolly (2021) to finally close shared interests in formal epistemology Romero and Sprenger (2021) even within our own department Merdes et al. (2021). Our job is now to reliably inform you about all the contributions in the papers in this special issue. Unfortunately, that task is beyond our capabilities. What we can and instead do is to summarise the contributed papers to inform your reliable inference to read them all in great detail. (shrink)

The replication (replicability, reproducibility) crisis in social psychology and clinical medicine arises from the fact that many apparently well-confirmed experimental results are subsequently overturned by studies that aim to replicate the original study. The culprit is widely held to be poor science: questionable research practices, failure to publish negative results, bad incentives, and even fraud. In this article I argue that the high rate of failed replications is consistent with high-quality science. We would expect this outcome if the field of (...) science in question produces a high proportion of false hypotheses prior to testing. If most of the hypotheses under test are false, then there will be many false hypotheses that are apparently supported by the outcomes of well conducted experiments and null hypothesis significance tests with a type-I error rate (α) of 5%. Failure to recognize this is to commit the fallacy of ignoring the base rate. I argue that this is a plausible diagnosis of the replication crisis and examine what lessons we thereby learn for the future conduct of science. (shrink)

Peer review is often taken to be the main form of quality control on academic research. Usually journals carry this out. However, parts of maths and physics appear to have a parallel, crowd-sourced model of peer review, where papers are posted on the arXiv to be publicly discussed. In this paper we argue that crowd-sourced peer review is likely to do better than journal-solicited peer review at sorting papers by quality. Our argument rests on two key claims. First, crowd-sourced peer (...) review will lead on average to more reviewers per paper than journal-solicited peer review. Second, due to the wisdom of the crowds, more reviewers will tend to make better judgments than fewer. We make the second claim precise by looking at the Condorcet Jury Theorem as well as two related jury theorems developed specifically to apply to peer review. (shrink)

Nowadays, almost everyone seems to agree that science is facing an epistemological crisis – namely the replicability crisis – and that we need to take action. But as to precisely what to do or how...

In my thesis, I develop two distinct themes. The first part of my thesis is devoted to indicative conditionals and approaching them from an empirically informed perspective. In the second part, I am developing classical topics of philosophy of science, specifically, scientific objectivity and the role of values in science, in connection to recent methodological developments, revolving around the Replication Crisis.

The replicability of a research claim is often positioned as an important step in establishing the credibility of scientific research. This expectation persists despite ongoing disagreements over how to characterise replication practices in various contexts. Rather than attempt to explain or resolve these disagreements, we propose that there is value in exploring the variable uses of the replicability concept. To this end, we treat the replicability concept as a goal-directed tool for studying scientific practices. This approach extends scholarship on the (...) goal-directed uses of concepts within investigative practices to explore the value of reflexively interrogating those concepts used to study the sciences as practiced. In doing so, we highlight the importance of considering how and when a given concept is an appropriate tool for a given goal when studying scientific practices. For instance, in the case of the replicability concept, our examination suggests that there is value in clarifying how a given characterisation of replicability is appropriate to a given context-specific analytic and/or evaluative goal. In addition, we hope to draw attention to how Metaresearch and HPS contexts provide complementary insights for those seeking to understand when and how replication practices can help in assessing a given research claim. (shrink)

. Scientists, for the most part, want to get it right. However, the social structures that govern their work undermine that aim, and this leads to nonreplicable findings in many fields. Because the social structure of science is a decentralized system, it is difficult to intervene. In this article, I discuss how we might do so, focusing on self-corrective-labor schemes. First, I argue that we need to implement a scheme that makes replication work outcome independent, systematic, and sustainable. Second, I (...) use these three criteria to evaluate extant proposals, which place the responsibility for replication on original researchers, consumers of their research, students, or many labs. Third, on the basis of a philosophical analysis of the reward system of science and the benefits of the division of cognitive labor, I propose a scheme that satisfies the criteria better: the professional scheme. This scheme has two main components. First, the scientific community is organized into two groups: discovery researchers, who produce new findings, and confirmation researchers, whose primary function is to do confirmation work. Second, a distinct reward system is established for confirmation researchers so that their career advancement is separated from whether they obtain positive experimental results. (shrink)