A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published (...) research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated. (shrink)

Peer review is embedded in the core of our knowledge generation systems, perceived as a method for establishing quality or scholarly legitimacy for research, while also often distributing academic prestige and standing on individuals. Despite its critical importance, it curiously remains poorly understood in a number of dimensions. In order to address this, we have analysed peer review to assess where the major gaps in our theoretical and empirical understanding of it lie. We identify core themes including editorial responsibility, the (...) subjectivity and bias of reviewers, the function and quality of peer review, and the social and epistemic implications of peer review. The high-priority gaps are focused around increased accountability and justification in decision-making processes for editors and developing a deeper, empirical understanding of the social impact of peer review. Addressing this at the bare minimum will require the design of a consensus for a minimal set of standards for what constitutes peer review, and the development of a shared data infrastructure to support this. Such a field requires sustained funding and commitment from publishers and research funders, who both have a commitment to uphold the integrity of the published scholarly record. We use this to present a guide for the future of peer review, and the development of a new research discipline based on the study of peer review. (shrink)

The Dynamics of Social Practice -- Introducing Theories of Practice -- Materials and Resources -- Sequence and Structure -- Making and Breaking Links -- Material, Competence and Meaning -- Car-Driving: Elements and Linkages Making Links -- Breaking Links -- Elements Between Practices -- Standardization and Diversity -- Individual and Collective Careers -- The Life of Elements -- Modes of Circulation -- Transportation and Access: Material -- Abstraction, Reversal and Migration: Competence -- Association and Classification: Meaning -- Packing and Unpacking -- (...) Emergence, Disappearance and Persistence -- Recruitment, Defection and Reproduction -- First Encounters: Networks and Communities -- Capture and Commitment: Careers and Carriers -- Collapse and Transformation: The Dynamics of Defection -- Daily Paths, Life Paths and Dominant Projects -- Connections Between Practices -- Bundles and Complexes -- Collaboration and Competition -- Selection and Integration -- Coordinating Daily Life -- Circuits of Reproduction -- Monitoring Practices-as-Performances -- Monitoring Practices-as-Entities -- Cross-Referencing Practices-as-Performances -- Cross-Referencing Practices-as-Entities -- Aggregation -- Elements of Coordination -- Intersecting Circuits -- Representing the Dynamics of Social Practice -- Representing Elements and Practices -- Characterizing Circulation -- Competition, Transformation and Convergence -- Reproducing Elements, Practices and Relations between Them -- Time and Practice -- Space and Practice -- Dominant Projects and Power -- Promoting Transitions in Practice -- Climate Change and Behaviour Change -- Basis of Action -- Processes of Change -- Positioning Policy -- Transferable Lessons -- Practice Theory and Climate Change Policy -- Configuring Elements of Practice -- Configuring Relations between Practices -- Configuring Careers: Carriers and Practices -- Configuring Connections -- Practice Oriented Policy Making. (shrink)