The study of human behavior is severely hampered by logistical problems, ethical and legal constraints, and funding shortfalls. However, the biggest difficulty of conducting social and behavioral research is the extraordinary complexity of the study phenomena. In this article, we review the impact of complexity on research design, hypothesis testing, measurement, data analyses, reproducibility, and the communication of findings in psychological science. The systematic investigation of the world often requires different approaches because of the variability in complexity. Confirmatory testing, multi-factorial (...) designs, survey methods, large samples, and modeling are frequently needed to study complex social and behavioral topics. Complexity impedes the measurement of general constructs, the reproducibility of results and scientific reporting, and the general rigor of research. Many of the benchmarks established by classic work in physical science are not attainable in studies of more complex phenomena. Consequently, the standards used to evaluate scientific research should be tethered to the complexity of the study topic. (shrink)

The replication crisis has caused researchers to distinguish between exact replications, which duplicate all aspects of a study that could potentially affect the results, and direct replications, which duplicate only those aspects of the study that are thought to be theoretically essential to reproduce the original effect. The replication crisis has also prompted researchers to think more carefully about the possibility of making Type I errors when rejecting null hypotheses. In this context, the present article considers the utility of two (...) types of Type I error probability: the Neyman–Pearson long run Type I error rate and the Fisherian sample-specific Type I error probability. It is argued that the Neyman–Pearson Type I error rate is inapplicable in social science because it refers to a long run of exact replications, and social science deals with irreversible units that make exact replications impossible. Instead, the Fisherian sample-specific Type I error probability is recommended as a more meaningful way to conceptualize false positive results in social science because it can be applied to each sample-specific decision about rejecting the same substantive null hypothesis in a series of direct replications. It is concluded that the replication crisis may be partly due to researchers’ unrealistic expectations about replicability based on their consideration of the Neyman–Pearson Type I error rate across a long run of exact replications. (shrink)

Kerr coined the term ‘HARKing’ to refer to the practice of ‘hypothesizing after the results are known’. This questionable research practice has received increased attention in recent years because it is thought to have contributed to low replication rates in science. The present article discusses the concept of HARKing from a philosophical standpoint and then undertakes a critical review of Kerr’s twelve potential costs of HARKing. It is argued that these potential costs are either misconceived, misattributed to HARKing, lacking evidence, (...) or that they do not take into account pre- and post-publication peer review and public availability to research materials and data. It is concluded that it is premature to conclude that HARKing has led to low replication rates. (shrink)

This article criticizes what it calls perspectival thought experiments, which require subjects to mentally simulate a perspective before making judgments from within it. Examples include Judith Thomson's violinist analogy, Philippa Foot's trolley problem, and Bernard Williams's Jim case. The article argues that advances in the philosophical and psychological study of empathy suggest that the simulative capacities required by perspectival thought experiments are all but impossible. These thought experiments require agents to consciously simulate necessarily unconscious features of subjectivity. To complete these (...) experiments subjects must deploy theory-theoretical frameworks to predict what they think they would do. These outputs, however, systematically mislead subjects and are highly prone to error. They are of negligible probative value, and this bodes poorly for their continued use. The article ends with two suggestions. First, many thought experiments are not problematically perspectival. Second, it should be possible to carry out “in-their-shoes” perspectival thought experiments by off-loading simulations onto virtual environments into which philosophers place subjects. (shrink)

Whether people possess free will has been a long-lasting philosophical debate. Recent attention in social psychology has been given to the behavioral consequences of believing in free will. Research has demonstrated that manipulating free will beliefs has implications for many social behaviors. For example, free will belief manipulations have been associated with cheating, aggressiveness, and prejudice. Despite this work, some of these findings have failed to replicate. Testing theoretical predictions, such as whether believing in free will influences behavior, depends on (...) theoretical, auxiliary, statistical, and inferential assumptions (TASI). In this paper we apply the TASI category of assumptions to the free will belief debate. In doing so we demonstrate why these assumptions should be considered when appraising the influence of free will belief manipulations on free will beliefs and behavior. To provide a nuanced view of free will beliefs, we believe researchers should pay careful attention to these critical assumptions. (shrink)

A large number of scientists and several news platforms have, over the last few years, been speaking of a replication crisis in various academic disciplines, especially the biomedical and social sciences. This paper answers the novel question of whether we should also pursue replication in the humanities. First, I create more conceptual clarity by defining, in addition to the term “humanities,” various key terms in the debate on replication, such as “reproduction” and “replicability.” In doing so, I pay attention to (...) what is supposed to be the object of replication: certain studies, particular inferences, of specific results. After that, I spell out three reasons for thinking that replication in the humanities is not possible and argue that they are unconvincing. Subsequently, I give a more detailed case for thinking that replication in the humanities is possible. Finally, I explain why such replication in the humanities is not only possible, but also desirable. (shrink)

A key source of support for the view that challenging people’s beliefs about free will may undermine moral behavior is two classic studies by Vohs and Schooler (2008). These authors reported that exposure to certain prompts suggesting that free will is an illusion increased cheating behavior. In the present paper, we report several attempts to replicate this influential and widely cited work. Over a series of five studies (sample sizes of N = 162, N = 283, N = 268, N (...) = 804, N = 982) (four preregistered) we tested the relationship between (1) anti-free-will prompts and free will beliefs and (2) free will beliefs and immoral behavior. Our primary task was to closely replicate the findings from Vohs and Schooler (2008) using the same or highly similar manipulations and measurements as the ones used in their original studies. Our efforts were largely unsuccessful. We suggest that manipulating free will beliefs in a robust way is more difficult than has been implied by prior work, and that the proposed link with immoral behavior may not be as consistent as previous work suggests. (shrink)

Moral, social, political, and other “nonepistemic” values can lead to bias in science, from prioritizing certain topics over others to the rationalization of questionable research practices. Such values might seem particularly common or powerful in the social sciences, given their subject matter. However, I argue first that the well-documented phenomenon of motivated reasoning provides a useful framework for understanding when values guide scientific inquiry (in pernicious or productive ways). Second, this analysis reveals a parity thesis: values influence the social and (...) natural sciences about equally, particularly because both are so prominently affected by desires for social credit and status, including recognition and career advancement. Ultimately, bias in natural and social science is both natural and social— that is, a part of human nature and considerably motivated by a concern for social status (and its maintenance). Whether the pervasive influence of values is inimical to the sciences is a separate question. (shrink)

The idea of replication is based on the premise that there are permanent laws to be replicated and verified, and the scientific method is adequate for doing so. Scientific truth, however, is not absolute but relative to time and context, and the method used. Time and context are inextricably interwoven, in that time creates different contexts and contexts (e.g., Christmas Day vs. New Year’s Day) create different experiences of time, rendering psychological phenomena inherently variable. This means that internal and external (...) conditions fluctuate and are different in a replication study vs. the original. Thus, a replication experiment is just another empirical investigation that has no special status in the establishment of scientific truth. It is not the final arbiter of whether or not something exists. In their pursuit of homogeneous external conditions, replications have ignored the homogeneity of internal conditions. There is not a single replication reported in the literature that would have shown participants’ feelings and thoughts—both conscious and nonconscious—to be identical to those of the original participants. Experimental instructions can create varying ratios of conscious over nonconscious processing from one study to another. Ironically, every replication is a failure at the fundamentals of human psychology. While patterns can be discovered, they are not permanent or unchangeable laws of human behavior to be proven by the pinpoint statistical verification through replication. As scientific knowledge in physics is temporary and incomplete, should it be any surprise that science can only provide “temporary winners” for psychological knowledge of human behavior? (shrink)

Based on an analysis of ten popular introductions to social psychology, we will show that Karl Popper's philosophy of ‘critical rationalism’ so far has had little to no traceable influence on the epistemology and practice of social psychology. If Popper is quoted or mentioned in the textbooks at all, the guiding principle of ‘falsificationism’ is reduced to a mere ‘falsifiability’ and some central elements of critical rationalism are left out – those that are incompatible with positivism and inductivism. Echoing earlier (...) attempts to introduce Popper to social psychology by Paul Meehl and Tom Pettigrew, we will argue that a discussing Popper's ideas in more depth could help social psychology to move forward in view of the ‘crisis of confidence’ that has emerged recently in view of the ‘Stapel affair’ and the reports of failures to replicate social psychological experiments in high-powered replication attempts. (shrink)

The replicability crisis refers to the apparent failures to replicate both important and typical positive experimental claims in psychological science and biomedicine, failures which have gained increasing attention in the past decade. In order to provide evidence that there is a replicability crisis in the first place, scientists have developed various measures of replication that help quantify or “count” whether one study replicates another. In this nontechnical essay, I critically examine five types of replication measures used in the landmark article (...) “Estimating the reproducibility of psychological science” (Open Science Collaboration, Science, 349, ac4716, 2015) based on the following techniques: subjective assessment, null hypothesis significance testing, comparing effect sizes, comparing the original effect size with the replication confidence interval, and meta-analysis. The first four, I argue, remain unsatisfactory for a variety of conceptual or formal reasons, even taking into account various improvements. By contrast, at least one version of the meta-analytic measure does not suffer from these problems. It differs from the others in rejecting dichotomous conclusions, the assumption that one study replicates another or not simpliciter. I defend it from other recent criticisms, concluding however that it is not a panacea for all the multifarious problems that the crisis has highlighted. (shrink)

The past ten years have seen multiple attempts to estimate the relation between the global personality trait extraversion and compatibilist free will judgments. Here, we contribute to that line of research by conducting a meta-analysis of 17 published and eight unpublished studies (N = 2,811) estimating that relation. Overall, the mean effect size was modest but remarkably robust across materials, locations, and labs (z =.19, 95% CI.15-.24, p <.001). No significant publication bias was detected in the studies (t (23) = (...) 1.88, p =.07). While there was no significant heterogeneity in the studies (Q (24) = 34.42, p =.08, I2 = 26.05), a moderator analysis suggested that the effect is strongest in cases that contain highly affective actions (e.g., murder) (z =.22, 95% CI.17-.28, p <.001) and weakest in cases that contain actions with low affect (e.g., asking whether free will is compatible with determinism) (z =.09, 95% CI -.05-.23, p =.22). The meta-analysis provides additional evidence that extraversion is related to compatibilist free will judgments and helps to identify opportunities to discover boundary conditions and more proximal causal mechanisms for the relation. The results of the meta-analysis also have implications for informed decision making. (shrink)

Current debates about the replication crisis in psychology take it for granted that direct replication is valuable and focus their attention on questionable research practices in regard to statistical analyses. This paper takes a broader look at the notion of replication as such. It is argued that all experimentation/replication involves individuation judgments and that research in experimental psychology frequently turns on probing the adequacy of such judgments. In this vein, I highlight the ubiquity of conceptual and material questions in research, (...) and I argue that replication is not as central to psychological research as it is sometimes taken to be. (shrink)

This paper revisits the debate between Harry Collins and Allan Franklin, concerning the experimenters’ regress. Focusing my attention on a case study from recent psychology (regarding experimental evidence for the existence of a Mozart Effect), I argue that Franklin is right to highlight the role of epistemological strategies in scientific practice, but that his account does not sufficiently appreciate Collins’s point about the importance of tacit knowledge in experimental practice. In turn, Collins rightly highlights the epistemic uncertainty (and skepticism) surrounding (...) much experimental research. However, I will argue that his analysis of tacit knowledge fails to elucidate the reasons why scientists often are (and should be) skeptical of other researchers’ experimental results. I will present an analysis of tacit knowledge in experimental research that not only answers to this desideratum, but also shows how such skepticism can in fact be a vital enabling factor for the dynamic processes of experimental knowledge generation. (shrink)

Sam Harris, in his book The Moral Landscape, argues that "science can determine human values." Against this view, I argue that while secular moral philosophy can certainly help us to determine our values, science must play a subservient role. To the extent that science can what we ought to do, it is only by providing us with empirical information, which can then be slotted into a chain of deductive reasoning. The premises of such reasoning, however, can in no way be (...) derived from the scientific method: they come, instead, from philosophy – and common sense. (shrink)

The moral enhancement (or bioenhancement) debate seems stuck in a dilemma. On the one hand, the more radical proposals, while certainly novel and interesting, seem unlikely to be feasible in practice, or if technically feasible then most likely imprudent. But on the other hand, the more sensible proposals – sensible in the sense of being both practically achievable and more plausibly ethically justifiable – can be rather hard to distinguish from both traditional forms of moral enhancement, such as non-drug-mediated social (...) or moral education, and non-moral forms of bioenhancement, such as smart-drug style cognitive enhancement. In this essay, I argue that bioethicists have paid insufficient attention to an alternative form of moral bioenhancement – or at least a likely candidate – that falls somewhere between these two extremes, namely the (appropriately qualified) use of certain psychedelic drugs. (shrink)

What to do with the missing shade of blue? Some have argued that Hume's famous thought experiment undermines his central doctrine – the ‘copy principle’ – such that he should have revised his whole theory in light of it. Others contend that the MSB is not a true or actual counterexample to the copy principle, but merely an apparent or conceivable one, so that he had no such obligation to revise. In this essay, I argue that even if the MSB (...) is a true counterexample, Hume would not have been obligated to revise his theory, given methodological resources he makes available to himself elsewhere in his work and his overarching empiricist aims. And I offer a new suggestion for why he might have included and even highlighted the MSB in his writings despite its being so very vexing and peculiar. (shrink)

This paper explores an emerging sub-field of both empirical bioethics and experimental philosophy, which has been called “experimental philosophical bioethics” (bioxphi). As an empirical discipline, bioxphi adopts the methods of experimental moral psychology and cognitive science; it does so to make sense of the eliciting factors and underlying cognitive processes that shape people’s moral judgments, particularly about real-world matters of bioethical concern. Yet, as a normative discipline situated within the broader field of bioethics, it also aims to contribute to substantive (...) ethical questions about what should be done in a given context. What are some of the ways in which this aim has been pursued? In this paper, we employ a case study approach to examine and critically evaluate four strategies from the recent literature by which scholars in bioxphi have leveraged empirical data in the service of normative arguments. (shrink)

Male circumcision—partial or total removal of the penile prepuce—has been proposed as a public health measure in Sub-Saharan Africa, based on the results of three randomized control trials showing a relative risk reduction of approximately 60 per cent for voluntary, adult male circumcision against female-to-male human immunodeficiency virus transmission in that context. More recently, long-time advocates of infant male circumcision have argued that these findings justify involuntary circumcision of babies and children in dissimilar public health environments, such as the USA, (...) Australasia and Europe. In this article, we take a close look at the necessary ethical and empirical steps that would be needed to bridge the gap between the African RCTs and responsible public health policy in developed countries. In the course of doing so, we discuss some of the main disagreements about the moral permissibility of performing a nontherapeutic surgery on a child to benefit potential future sexual partners of his. In this context, we raise concerns not only about weaknesses in the available evidence concerning such claims of benefit, but also about a child’s moral interest in future autonomy and the preservation of his bodily integrity. We conclude that circumcision of minors in developed countries on public health grounds is much harder to justify than proponents of the surgery suggest. (shrink)

Background: Recent literature on addiction and judgments about the characteristics of agents has focused on the implications of adopting a ‘brain disease’ versus ‘moral weakness’ model of addiction. Typically, such judgments have to do with what capacities an agent has (e.g., the ability to abstain from substance use). Much less work, however, has been conducted on the relationship between addiction and judgments about an agent’s identity, including whether or to what extent an individual is seen as the same person after (...) becoming addicted. Methods: We conducted a series of vignette-based experiments (total N = 3,620) to assess lay attitudes concerning addiction and identity persistence, systematically manipulating key characteristics of agents and their drug of addiction. Conclusions: In Study 1, we found that US participants judged an agent who became addicted to drugs as being closer to ‘a completely different person’ than ‘completely the same person’ as the agent who existed prior to the addiction. In Studies 2-6, we investigated the intuitive basis for this result, finding that lay judgments of altered identity as a consequence of drug use and addiction are driven primarily by perceived negative changes in the moral character of drug users, who are seen as having deviated from their good true selves. (shrink)

Responding to recent concerns about the reliability of the published literature in psychology and other disciplines, we formed the X-Phi Replicability Project to estimate the reproducibility of experimental philosophy. Drawing on a representative sample of 40 x-phi studies published between 2003 and 2015, we enlisted 20 research teams across 8 countries to conduct a high-quality replication of each study in order to compare the results to the original published findings. We found that x-phi studies – as represented in our sample (...) – successfully replicated about 70% of the time. We discuss possible reasons for this relatively high replication rate in the field of experimental philosophy and offer suggestions for best research practices going forward. (shrink)

Appendix 1 was incomplete in the initial online publication. The original article has been corrected.

Since the advent of the twenty-first century, science has experienced a crisis pertaining to the replicability of quantitative research findings, which has become known as the ‘replication crisis’. The replication crisis has particularly afflicted research in the behavioural sciences, and psychology in particular. Given the relevance of psychology to education, it is unsurprising that the replication crisis also presents an issue for quantitative educational research, thus potentially compromising its practical usefulness. This article outlines the replication crisis in psychology, and highlights (...) its significance for quantitative educational research. Greater methodological rigour, aimed at addressing methodological deficiencies in the conduct of scientific research, has been suggested as a response to the replication crisis. Following a review of various calls for greater methodological rigour, the current article argues that, whilst methodological deficiencies may be a contributory factor, there is potentially a more fundamental reason for the replication crisis. The case is made that the non-separability of a measured attribute from the measurement instrument, and the irreducible uncertainty in unmeasured attributes, may be the principal reasons for the replication crisis in psychology and education. Implications of such an explanation for the crisis are articulated in relation to the evidence-based policy agenda in education. (shrink)

Cranial electrotherapy stimulation is a neuromodulation tool used for treating several clinical disorders, including insomnia, anxiety, and depression. More recently, a limited number of studies have examined CES for altering affect, physiology, and behavior in healthy, non-clinical samples. The physiological, neurochemical, and metabolic mechanisms underlying CES effects are currently unknown. Computational modeling suggests that electrical current administered with CES at the earlobes can reach cortical and subcortical regions at very low intensities associated with subthreshold neuromodulatory effects, and studies using electroencephalography (...) and functional magnetic resonance imaging show some effects on alpha band EEG activity, and modulation of the default mode network during CES administration. One theory suggests that CES modulates brain stem, limbic, and cortical regions and increases relative parasympathetic to sympathetic drive in the autonomic nervous system. There is no direct evidence supporting this theory, but one of its assumptions is that CES may induce its effects by stimulating afferent projections of the vagus nerve, which provides parasympathetic signals to the cardiorespiratory and digestive systems. In our critical review of studies using CES in clinical and non-clinical populations, we found severe methodological concerns, including potential conflicts of interest, risk of methodological and analytic biases, issues with sham credibility, lack of blinding, and a severe heterogeneity of CES parameters selected and employed across scientists, laboratories, institutions, and studies. These limitations make it difficult to derive consistent or compelling insights from the extant literature, tempering enthusiasm for CES and its potential to alter nervous system activity or behavior in meaningful or reliable ways. The lack of compelling evidence also motivates well-designed and relatively high-powered experiments to assess how CES might modulate the physiological, affective, and cognitive responses to stress. Establishing reliable empirical links between CES administration and human performance is critical for supporting its prospective use during occupational training, operations, or recovery, ensuring reliability and robustness of effects, characterizing if, when, and in whom such effects might arise, and ensuring that any benefits of CES outweigh the risks of adverse events. (shrink)

Introduction: Publications arguing against the null hypothesis significance testing (NHST) procedure and in favor of good statistical practices have increased. The most frequently mentioned alternatives to NHST are effect size statistics (ES), confidence intervals (CIs), and meta-analyses. A recent survey conducted in Spain found that academic psychologists have poor knowledge about effect size statistics, confidence intervals, and graphic displays for meta-analyses, which might lead to a misinterpretation of the results. In addition, it also found that, although the use of ES (...) is becoming generalized, the same thing is not true for CIs. Finally, academics with greater knowledge about ES statistics presented a profile closer to good statistical practice and research design. Our main purpose was to analyze the extension of these results to a different geographical area through a replication study. Method: For this purpose, we elaborated an on-line survey that included the same items as the original research, and we asked academic psychologists to indicate their level of knowledge about ES, their CIs, and meta-analyses, and how they use them. The sample consisted of 159 Italian academic psychologists (54.09% women, mean age of 47.65 years). The mean number of years in the position of professor was 12.90 (SD = 10.21). Results: As in the original research, the results showed that, although the use of effect size estimates is becoming generalized, an under-reporting of CIs for ES persists. The most frequent ES statistics mentioned were Cohen’s d and R2/η2, which can have outliers or show non-normality or violate statistical assumptions. In addition, academics showed poor knowledge about meta-analytic displays (e.g., forest plot and funnel plot) and quality checklists for studies. Finally, academics with higher-level knowledge about ES statistics seem to have a profile closer to good statistical practices. Conclusions: Changing statistical practice is not easy. This change requires statistical training programs for academics, both graduate and undergraduate. (shrink)

Popper’s (1983, 2002) philosophy of science has enjoyed something of a renaissance in the wake of the replication crisis, offering a philosophical basis for the ensuing science reform movement. However, adherence to Popper’s approach may also be at least partly responsible for the sense of “crisis” that has developed following multiple unexpected replication failures. In this article, I contrast Popper’s approach with Lakatos’ (1978) approach and a related approach called naïve methodological falsificationism (NMF; Lakatos, 1978). The Popperian approach is powerful (...) because it is based on logical refutation, but its theories are noncausal and, therefore, lacking in scientific value. In contrast, the Lakatosian approach considers causal theories, but it concedes that these theories are not logically refutable. Finally, the NMF approach subjects Lakatosian causal theories to Popperian logical refutations. However, its approach of temporarily accepting a ceteris paribus clause during theory testing may be viewed as scientifically inappropriate, epistemically inconsistent, and “completely redundant” (Lakatos, 1978, p. 40). I conclude that the replication “crisis” makes the most sense in the context of the Popperian and NMF approaches because it is only in these two approaches that replication failures represent logical refutations of theories. In contrast, replication failures are less problematic in the Lakatosian approach because they do not logically refute theories. Indeed, in the Lakatosian approach, replication failures can be legitimately ignored or used to motivate theory development. (shrink)

A heated debate surrounds the significance of reproducibility as an indicator for research quality and reliability, with many commentators linking a "crisis of reproducibility" to the rise of fraudulent, careless and unreliable practices of knowledge production. Through the analysis of discourse and practices across research fields, I point out that reproducibility is not only interpreted in different ways, but also serves a variety of epistemic functions depending on the research at hand. Given such variation, I argue that the uncritical pursuit (...) of reproducibility as an overarching epistemic value is misleading and potentially damaging to scientific advancement. Requirements for reproducibility, however they are interpreted, are one of many available means to secure reliable research outcomes. Furthermore, there are cases where the focus on enhancing reproducibility turns out not to foster high-quality research. Scientific communities and Open Science advocates should learn from inferential reasoning from irreproducible data, and promote incentives for all researchers to explicitly and publicly discuss their methodological commitments, the ways in which they learn from mistakes and problems in everyday practice, and the strategies they use to choose which research component of any project needs to be preserved in the long term, and how. (shrink)