Researchers misunderstand their role in creating ethical problems when they allow dogmas to purportedly divorce scientists and scientific practices from the values that they embody. Cortina, Edwards, and Powell help us clarify and further develop our position by responding to our critique of, and alternatives to, this misleading separation. In this rebuttal, we explore how the desire to achieve the separation of facts and values is unscientific on the very terms endorsed by its advocates—this separation is refuted by empirical observation. (...) We show that positivists like Cortina and Edwards offer no rigorous theoretical or empirical justifications to substantiate their claims, let alone critique ours. Following Powell, we point to how classical pragmatism understands ‘purpose’ in scientific pursuits while also providing an alternative to the dogmas of positivism and related philosophical positions. In place of dogmatic, unscientific cries about an abstract and therefore always-unobservable ‘reality,’ we invite all organizational scholars to join us in shifting the discussion about quantitative research towards empirically grounded scientific inquiry. This makes the ethics of actual people and their practices central to quantitative research, including the thoughts, discourses, and behaviors of researchers who are always in particular places doing particular things. We propose that quantitative researchers can thus start to think about their research practices as a kind of work, rather than having the status of a kind of dogma. We conclude with some implications that this has for future research and education, including the relevance of research and research methods. (shrink)

Quantitative researchers often discuss research ethics as if specific ethical problems can be reduced to abstract normative logics (e.g., virtue ethics, utilitarianism, deontology). Such approaches overlook how values are embedded in every aspect of quantitative methods, including ‘observations,’ ‘facts,’ and notions of ‘objectivity.’ We describe how quantitative research practices, concepts, discourses, and their objects/subjects of study have always been value-laden, from the invention of statistics and probability in the 1600s to their subsequent adoption as a logic made to appear as (...) if it exists prior to, and separate from, ethics and values. This logic, which was embraced in the Academy of Management from the 1960s, casts management researchers as ethical agents who ought to know about a reality conceptualized as naturally existing in the image of statistics and probability (replete with ‘constructs’), while overlooking that S&P logic and practices, which researchers made for themselves, have an appreciable role in making the world appear this way. We introduce a different way to conceptualize reality and ethics, wherein the process of scientific inquiry itself requires an examination of its own practices and commitments. Instead of resorting to decontextualized notions of ‘rigor’ and its ‘best practices,’ quantitative researchers can adopt more purposeful ways to reason about the ethics and relevance of their methods and their science. We end by considering implications for addressing ‘post truth’ and ‘alternative facts’ problems as collective concerns, wherein it is actually the pluralistic nature of description that makes defending a collectively valuable version of reality so important and urgent. (shrink)

The field of psychology, including cognitive science, is vexed by a crisis of confidence. Although the causes and solutions are varied, we focus here on a common logical problem in inference. The default mode of inference is significance testing, which has a free lunch property where researchers need not make detailed assumptions about the alternative to test the null hypothesis. We present the argument that there is no free lunch; that is, valid testing requires that researchers test the null against (...) a well-specified alternative. We show how this requirement follows from the basic tenets of conventional and Bayesian probability. Moreover, we show in both the conventional and Bayesian framework that not specifying the alternative may lead to rejections of the null hypothesis with scant evidence. We review both frequentist and Bayesian approaches to specifying alternatives, and we show how such specifications improve inference. The field of cognitive science will benefit because consideration of reasonable alternatives will undoubtedly sharpen the intellectual underpinnings of research. (shrink)

The study of human behavior is severely hampered by logistical problems, ethical and legal constraints, and funding shortfalls. However, the biggest difficulty of conducting social and behavioral research is the extraordinary complexity of the study phenomena. In this article, we review the impact of complexity on research design, hypothesis testing, measurement, data analyses, reproducibility, and the communication of findings in psychological science. The systematic investigation of the world often requires different approaches because of the variability in complexity. Confirmatory testing, multi-factorial (...) designs, survey methods, large samples, and modeling are frequently needed to study complex social and behavioral topics. Complexity impedes the measurement of general constructs, the reproducibility of results and scientific reporting, and the general rigor of research. Many of the benchmarks established by classic work in physical science are not attainable in studies of more complex phenomena. Consequently, the standards used to evaluate scientific research should be tethered to the complexity of the study topic. (shrink)