A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published (...) research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated. (shrink)

Speculations about the role of consciousness in physical systems are frequently observed in the literature concerned with the interpretation of quantum mechanics. While only three experimental investigations can be found on this topic in physics journals, more than 800 relevant experiments have been reported in the literature of parapsychology. A well-defined body of empirical evidence from this domain was reviewed using meta-analytic techniques to assess methodological quality and overall effect size. Results showed effects conforming to chance expectation in control conditions (...) and unequivocal non-chance effects in experimental conditions. This quantitative literature review agrees with the findings of two earlier reviews, suggesting the existence of some form of consciousness-related anomaly in random physical systems. (shrink)

It has long been known that scientists have a tendency to conduct experiments in a way that brings about the expected outcome. Here, we provide the first direct demonstration of this type of experimenter bias in experimental philosophy. Opposed to previously discovered types of experimenter bias mediated by face-to-face interactions between experimenters and participants, here we show that experimenters also have a tendency to create stimuli in a way that brings about expected outcomes. We randomly assigned undergraduate experimenters to receive (...) two different hypotheses about folk intuitions of consciousness, and then asked them to design experiments based on their hypothesis. Specifically, experimenters generated sentences ascribing intentional and phenomenal mental states to groups, which were later rated by online participants for naturalness. We found a significant interaction between experimenter hypothesis and participant ratings indicating a general tendency for experimenters to obtain the result that they expected. These results indicate that experimenter bias is a real problem in experimental philosophy since the methods and design employed here mirror the predominant survey methods of the field as a whole. The bearing of the current results on Knobe and Prinz’s :67–83, 2008) group mind hypothesis is discussed, and new methods for avoiding experimenter bias are proposed. (shrink)

Over the past hundred years, a number of scientific investigators claim to have adduced experimental evidence for phenomena information” seems to behave like a weak signal that has to compete for the information-processing resources of the organism, a reduction of ongoing sensorimotor activity may facilitate ESP detection. Such a meaningful convergence of results suggests that psi phenomena may represent a unitary, coherent process whose nature and compatibility with current physical theory have yet to be determined. The theoretical implications and potential (...) practical applications of psi could be significant, irrespective of the small magnitude of psi effects in laboratory settings. (shrink)

De nombreux commentaires des expériences de Milgram soutiennent l’Hypothèse d’incrédulité, laquelle soutient que les participants de Milgram n’auraient en général pas cru qu’ils administraient des chocs électriques réels. Si l’Hypothèse d’incrédulité était juste, on devrait en conclure que les sujets obéissants ne croyaient pas mal agir, ce qui impliquerait que Milgram a échoué à mettre en évidence des niveaux alarmants d’obéissance destructrice. Dans cet article, nous démontrons que l’Hypothèse d’incrédulité n’est, en général, pas exacte : elle n’explique que très difficilement (...) le comportement des participants dans les expériences de Milgram et ses nombreuses réplications, et elle concorde mal avec les témoignages des participants concernant leur propre vécu. (shrink)

A self-fulfilling prophecy is, roughly, a prediction that brings about its own truth. Although true predictions are hard to fault, self-fulfilling prophecies are often regarded with suspicion. In this article, we vindicate this suspicion by explaining what self-fulfilling prophecies are and what is problematic about them, paying special attention to how their problems are exacerbated through automated prediction. Our descriptive account of self-fulfilling prophecies articulates the four elements that define them. Based on this account, we begin our critique by showing (...) that typical self-fulfilling prophecies arise due to mistakes about the relationship between a prediction and its object. Such mistakes—along with other mistakes in predicting or in the larger practical endeavor—are easily overlooked when the predictions turn out true. Thus we note that self-fulfilling prophecies prompt no error signals; truth shrouds their mistakes from humans and machines alike. Consequently, self-fulfilling prophecies create several obstacles to accountability for the outcomes they produce. We conclude our critique by showing how failures of accountability, and the associated failures to make corrections, explain the connection between self-fulfilling prophecies and feedback loops. By analyzing the complex relationships between accuracy and other evaluatively significant features of predictions, this article sheds light both on the special case of self-fulfilling prophecies and on the ethics of prediction more generally. (shrink)

Experimental philosophers have empirically challenged the connection between intuition and philosophical expertise. This paper reviews these challenges alongside other research findings in cognitive science on expert performance and argues for three claims. First, evidence taken to challenge philosophical expertise may also be explained by the well-researched failures and limitations of genuine expertise. Second, studying the failures and limitations of experts across many fields provides a promising research program upon which to base a new model of philosophical expertise. Third, a model (...) of philosophical expertise based on the limitations of genuine experts may suggest a series of constraints on the reliability of professional philosophical intuition. Even when the experts all agree, they may well be mistaken. — Bertrand Russell, On the Value of Scepticism. (shrink)