Case Study research is characterized by the employment of multiple data gathering methods. In this paper, I examine the concurrent use of participant observation and qualitative interviews. The question I examine is: what is the rationale behind their combination in case study research? In the literature on case study research, the two most common reasons for using multiple methods appeal to comprehensiveness and convergent confirmation respectively. I argue that there is a third significant, yet overlooked, way to motivate the joint (...) use of participant observation and qualitative interviews: the methods may generate complementary evidence and this puts the researcher in a better position to confirm that her data manifest central epistemic values and so are suitable as basis for providing an adequate answer to her research question. I refer to this as the rationale of blended epistemic value validation. (shrink)

Should a scientist rely on methodological triangulation? Heesen et al. (Synthese 196(8):3067–3081, 2019) recently provided a convincing affirmative answer. However, their approach requires belief gambles if the evidence is discordant. We instead propose epistemically modest triangulation (EMT), according to which one should withhold judgement in such cases. We show that for a scientist in a methodologically diffident situation the expected utility of EMT is greater than that of Heesen et al.’s (2019) triangulation or that of using a single method. We (...) also show that EMT is more appropriate for increasing epistemic trust in science. In short: triangulate, but do not gamble with evidence. (shrink)

One important strategy for dealing with error in our methods is triangulation, or the use multiple methods to investigate the same object. Current accounts of triangulation assume that its primary function is to provide a confirmatory boost to hypotheses beyond what confirmation of each method alone could produce. Yet, researchers often use multiple methods to examine new constructs about which they are uncertain. For example, social psychologists use multiple indirect measures to provide convergent evidence about implicit attitudes, but how to (...) characterize implicit attitudes is an open question. To make sense of triangulation under uncertainty about constructs, I suggest two changes: first, triangulation can serve multiple epistemic functions, including some that are non-confirmatory, and second, researchers should assess the epistemic risk in claims about evidence and the acceptance/rejection of hypotheses. (shrink)

We show that previous results from epistemic network models showing the benefits of decreased connectivity in epistemic networks are not robust across changes in parameter values. Our findings motivate discussion about whether and how such models can inform real-world epistemic communities. As we argue, only robust results from epistemic network models should be used to generate advice for the real-world, and, in particular, decreasing connectivity is a robustly poor recommendation.

Amalgamating evidence from heterogeneous sources and across levels of inquiry is becoming increasingly important in many pure and applied sciences. This special issue provides a forum for researchers from diverse scientific and philosophical perspectives to discuss evidence amalgamation, its methodologies, its history, its pitfalls, and its potential. We situate the contributions therein within six themes from the broad literature on this subject: the variety-of-evidence thesis, the philosophy of meta-analysis, the role of robustness/sensitivity analysis for evidence amalgamation, its bearing on questions (...) of extrapolation and external validity of experiments, its connection with theory development, and its interface with causal inference, especially regarding causal theories of cancer. (shrink)

According to the variety of evidence thesis items of evidence from independent lines of investigation are more confirmatory, ceteris paribus, than, for example, replications of analogous studies. This thesis is known to fail (Bovens and Hartmann; Claveau). However, the results obtained by Bovens and Hartmann only concern instruments whose evidence is either fully random or perfectly reliable; instead, for Claveau, unreliability is modelled as deterministic bias. In both cases, the unreliable instrument delivers totally irrelevant information. We present a model that (...) formalizes both reliability and unreliability differently. Our instruments either are reliable, but affected by random error, or are biased but not deterministically so.Bovens and Hartmann’s results are counter-intuitive in that in their model a long series of consistent reports from the same instrument does not raise suspicion of ‘too-good-to-be-true’ evidence. This happens precisely because they contemplate neither the role of systematic bias, nor unavoidable random error of reliable instruments. In our model, the variety of evidence thesis fails as well, but the area of failure is considerably smaller than for Bovens and Hartmann and Claveau, and holds for (the majority of) realistic cases (that is, where biased instruments are very biased). The essential mechanism that triggers variety of evidence thesis failure is the rate of false to true positives for the two kinds of instruments. Our emphasis is on modelling beliefs about sources of knowledge and their role in hypothesis confirmation in interaction with dimensions of evidence, such as variety and consistency. (shrink)

In this paper we argue for a naturalistic solution to some of the methodological controversies in regulatory science, on the basis of two case studies: toxicology and health claim regulation. We analyze the debates related to the scientific evidence that is considered necessary for regulatory decision making in each of those two fields, with a particular attention to the interactions between scientific and regulatory aspects. This analysis allows us to identify two general stances in the debate: a) one that argues (...) for more permissive standards of evidence and for methodological pluralism, and b) an opposing one that not only defends strict evidence requirements but also stipulates the use of one particular scientific methodologies for data generation. We argue that the real-world outcomes produced by alternative regulatory options are a vital piece of information that allows for the empirical assessment of these two stances. In particular, this information on outcomes makes it possible to analyze which standards of evidence and scientific methods generate the most useful knowledge as input for regulatory decision making. Our conclusion is that instead of an a priori selection of methodologies and standards, such decisions ought to be based on empirical evidence related to real-world outcomes. (shrink)

The intuitive Variety of Evidence Thesis states that, ceteris paribus, more varied evidence for a hypothesis confirms it more strongly than less varied evidence. Recent Bayesian analyses have raised serious doubts in its validity. Claveau suggests the existence of a novel type of counter-example to this thesis: a gradual increase in source independence can lead to a decrease in hypothesis confirmation. I show that Claveau’s measure of gradual source independence suffers from two unsuspected types of inconsistencies. I hence put forward (...) a more natural measure of gradual source independence which is not plagued by inconsistencies. Claveau’s counter-examples to the variety of evidence thesis disappear with the measure I suggest. I hence argue that my measure is preferable and that this thesis does at least not seem to be troubled by Claveau’s arguments. (shrink)

Prepublication peer review should be abolished. We consider the effects that such a change will have on the social structure of science, paying particular attention to the changed incentive structure and the likely effects on the behaviour of individual scientists. We evaluate these changes from the perspective of epistemic consequentialism. We find that where the effects of abolishing prepublication peer review can be evaluated with a reasonable level of confidence based on presently available evidence, they are either positive or neutral. (...) We conclude that on present evidence abolishing peer review weakly dominates the status quo. (shrink)

I argue that collaborators do not need to reach broad agreement over the justification of a consensus claim. This is because maintaining a diversity of justifiers within a scientific collaboration has important epistemic value. I develop a view of collective justification that depends on the diversity of epistemic perspectives present in a group. I argue that a group can be collectively justified in asserting that P as long as the disagreement among collaborators over the reasons for P is itself justified. (...) In conclusion, I make a case for multimethod collaborative research and work through an example in the social sciences. (shrink)

Machine learning (ML) models recently led to major breakthroughs in predictive tasks in the natural sciences. Yet their benefits for the social sciences are less evident, as even high-profile studies on the prediction of life trajectories have shown to be largely unsuccessful – at least when measured in traditional criteria of scientific success. This paper tries to shed light on this remarkable performance gap. Comparing two social science case studies to a paradigm example from the natural sciences, we argue that, (...) in addition to explanation, prediction is an important goal of social science – and we identify constraints that impede pure ML prediction from being successful in that field. As a remedy, we outline elements of an integrative modelling approach that combines explanatory models and predictive ML models. (shrink)

Peer review is often taken to be the main form of quality control on academic research. Usually journals carry this out. However, parts of maths and physics appear to have a parallel, crowd-sourced model of peer review, where papers are posted on the arXiv to be publicly discussed. In this paper we argue that crowd-sourced peer review is likely to do better than journal-solicited peer review at sorting papers by quality. Our argument rests on two key claims. First, crowd-sourced peer (...) review will lead on average to more reviewers per paper than journal-solicited peer review. Second, due to the wisdom of the crowds, more reviewers will tend to make better judgments than fewer. We make the second claim precise by looking at the Condorcet Jury Theorem as well as two related jury theorems developed specifically to apply to peer review. (shrink)

Pluralism about scientific method is more-or-less accepted, but the consequences have yet to be drawn out. Scientists adopt different methods in response to different epistemic situations: depending on the system they are interested in, the resources at their disposal, and so forth. If it is right that different methods are appropriate in different situations, then mismatches between methods and situations are possible. This is most likely to occur due to method bias: when we prefer a particular kind of method, despite (...) that method clashing with evidential context or our aims. To explore these ideas, we sketch a kind of method pluralism which turns on two properties of evidence, before using agent-based models to examine the relationship between methods, epistemic situations, and bias. Based on our results, we suggest that although method bias can undermine the efficiency of a scientific community, it can also be productive through preserving a diversity of evidence. We consider circumstances where method bias could be particularly egregious, and those where it is a potential virtue, and argue that consideration of method bias reveals that community standards deserve a central place in the epistemology of science. (shrink)