Reactivity, or the phenomenon by which subjects tend to modify their behavior in virtue of their being studied upon, is often cited as one of the most important difficulties involved in social scientific experiments, and yet, there is to date a persistent conceptual muddle when dealing with the many dimensions of reactivity. This paper offers a conceptual framework for reactivity that draws on an interventionist approach to causality. The framework allows us to offer an unambiguous definition of reactivity and distinguishes (...) it from placebo effects. Further, it allows us to distinguish between benign and malignant forms of the phenomenon, depending on whether reactivity constitutes a danger to the validity of the causal inferences drawn from experimental data. (shrink)

In this commentary, I propose a framework for thinking about data quality in the context of scientific research. I start by analyzing conceptualizations of quality as a property of information, evidence and data and reviewing research in the philosophy of information, the philosophy of science and the philosophy of biomedicine. I identify a push for purpose dependency as one of the main results of this review. On this basis, I present a contextual approach to data quality in scientific research, whereby (...) the quality of a dataset is dependent on the context of use of the dataset as much as the dataset itself. I exemplify the approach by discussing current critiques and debates of scientific quality, thus showcasing how data quality can be approached contextually. (shrink)

This paper proposes an account of scientific data that makes sense of recent debates on data-driven and ‘big data’ research, while also building on the history of data production and use particularly within biology. In this view, ‘data’ is a relational category applied to research outputs that are taken, at specific moments of inquiry, to provide evidence for knowledge claims of interest to the researchers involved. They do not have truth-value in and of themselves, nor can they be seen as (...) straightforward representations of given phenomena. Rather, they are fungible objects defined by their portability and prospective usefulness as evidence. (shrink)

This paper discusses the so-called non-interference assumption (NIA) grounding causal inference in trials in both medicine and the social sciences. It states that for each participant in the experiment, the value of the potential outcome depends only upon whether she or he gets the treatment. Drawing on methodological discussion in clinical trials and laboratory experiments in economics, I defend the necessity of partial forms of blinding as a warrant of the NIA, to control the participants’ expectations and their strategic interactions (...) with the experimenter. (shrink)

This paper provides a restatement and defense of the data/ phenomena distinction introduced by Jim Bogen and me several decades ago (e.g., Bogen and Woodward, The Philosophical Review, 303–352, 1988). Additional motivation for the distinction is introduced, ideas surrounding the distinction are clarified, and an attempt is made to respond to several criticisms.

Many philosophers have claimed that evidence for a theory is better when multiple independent tests yield the same result, i.e., when experimental results are robust. Little has been said about the grounds on which such a claim rests, however. The present essay presents an analysis of the evidential value of robustness that rests on the fallibility of assumptions about the reliability of testing procedures and a distinction between the strength of evidence and the security of an evidence claim. Robustness can (...) enhance the security of an evidence claim either by providing what I call second-order evidence, or by providing back-up evidence for a hypothesis. (shrink)

If a human subject knows they are being measured, this knowledge may affect their attitudes and behaviour to such an extent that it affects the measurement results as well. This broad range of effects is shared under the term ‘reactivity’. Although reactivity is often seen by methodologists as a problem to overcome, in this paper I argue that some quite extreme reactive changes may be legitimate, as long as we are measuring phenomena that are not simple biological regularities. Legitimate reactivity (...) is reactivity which does not undermine the accuracy of a measure; I show that if such reactivity were corrected for, this would unjustifiably ignore the authority of the research subject. Applying this argument to the measurement of depression, I show that under the most commonly accepted models of depression there is room for legitimate reactivity. In the first part of the paper, I provide an inventory of the different types of reactivity that exist in the literature, as well as the different types of phenomena that one could measure. In the second part, I apply my argument to the measurement of depression with the PHQ-9 survey. I argue that depending on what kind of phenomenon we consider depression to be, we will accept different kinds of reactivity. I show that both under the harmful dysfunction model and the practical kinds model, certain reactive changes in measuring depression are best seen as legitimate recharacterizations of the underlying phenomenon, and define what legitimate means in this context. I conclude that in both models, biological aspects constrain characterization, but the models are not so strict that only one concept is acceptable, leaving room for reactivity. (shrink)

Qualitative researchers sometimes talk about objectivity in relation to qualitative data sets. In this paper, I defend a reconstructed notion of objective qualitative data sets that may serve as a useful and reachable guiding ideal in qualitative data generation. In the first part of the paper, I develop the ideal. According to it, a qualitative data set is objective to the extent that it, in conjunction with true assumptions, possesses a combination of good-making features in virtue of which the data (...) set is suited to serve as evidence base for a satisfying answer to the research question under study. In the second part of the paper, I examine and reject two possible lines of objection to this ideal: One is that it picks out the wrong good-making features. The other is that the very focus on good-making features is misguided: the objectivity of a qualitative data set should instead be seen as a matter of how it was generated or evaluated. (shrink)

Case Study research is characterized by the employment of multiple data gathering methods. In this paper, I examine the concurrent use of participant observation and qualitative interviews. The question I examine is: what is the rationale behind their combination in case study research? In the literature on case study research, the two most common reasons for using multiple methods appeal to comprehensiveness and convergent confirmation respectively. I argue that there is a third significant, yet overlooked, way to motivate the joint (...) use of participant observation and qualitative interviews: the methods may generate complementary evidence and this puts the researcher in a better position to confirm that her data manifest central epistemic values and so are suitable as basis for providing an adequate answer to her research question. I refer to this as the rationale of blended epistemic value validation. (shrink)

This paper discusses the so-called non-interference assumption (NIA) grounding causal inference in trials in both medicine and the social sciences. It states that for each participant in the experiment, the value of the potential outcome depends only upon whether she or he gets the treatment. Drawing on methodological discussion in clinical trials and laboratory experiments in economics, I defend the necessity of partial forms of blinding as a warrant of the NIA, to control the participants’ expectations and their strategic interactions (...) with the experimenter. (shrink)

The Dictator Game, one of the best-known designs in experimental social science, has been extensively criticized, and declared by some to be defunct, on the grounds that its results are the product of a research artefact. Critics of the DG argue that the behaviour observed in the game is not the outcome of genuine pro-social preferences but must, instead, be interpreted as a response to the cues given by the experimental design, where these cues signal that the game is about (...) ‘sharing’. Despite this criticism, the DG continues to be extensively used, and some have defended its validity as an instrument capable of measuring the role of social pressure and social norms against economic motivations. This article examines the assumptions implicit in the claim that the DG results are artefactual and spells out the conditions under which the game can be used to test hypotheses about pro-social... (shrink)

A series of recent debates in experimental economics have associated demand effects with the artificiality of the experimental setting and have linked it to the problem of external validity. In this paper, we argue that these associations can be misleading, partly because of the ambiguity with which “artificiality” has been defined, but also because demand effects and external validity are related in complex ways. We argue that artificiality may be directly as well as inversely correlated with demand effects. We also (...) distinguish between the demand effects of experimentation and the reactions that they may trigger and that might endanger experimental validity. We conclude that economists should pay more attention to the way in which subjects construe the experimental task and learn to exploit subjects’ reactivity to expectations in their experiments. (shrink)