In light of the widespread existence of financial and non-financial issues that contribute to the appearance or fact of conflict of interest, it is proposed that conflict of interest should generally be assumed, no matter the source of financial support or the expressed declarations of conflicts and even with respect to one’s own work. No new model is advanced for modification of peer-review processes or for elaboration of author declarations of interest. Researchers should be assessing the quality of published work (...) as best they can and make their own decisions on the appropriate use of the work. While some apparent sources of conflict are likely more obvious and serious than others, even subtler biases can influence scientific reports. Ignoring peer-reviewed contributions because of conflict-of-interest concerns is discouraged. Listening skeptically to all sources, including yourself, is encouraged. (shrink)

A growing body of literature has identified potential problems that can compromise the quality, fairness, and integrity of journal peer review, including inadequate review, inconsistent reviewer reports, reviewer biases, and ethical transgressions by reviewers. We examine the evidence concerning these problems and discuss proposed reforms, including double-blind and open review. Regardless of the outcome of additional research or attempts at reforming the system, it is clear that editors are the linchpin of peer review, since they make decisions that have a (...) significant impact on the process and its outcome. We consider some of the steps editors should take to promote quality, fairness and integrity in different stages of the peer review process and make some recommendations for editorial conduct and decision-making. (shrink)

On the surface, developing a social psychology of science seems compelling as a way to understand how individual social cognition – in aggregate – contributes towards individual and group behavior within scientific communities (Kitcher, 2002). However, in cases where the functional input-output profile of psychological processes cannot be mapped directly onto the observed behavior of working scientists, it becomes clear that the relationship between psychological claims and normative philosophy of science should be refined. For example, a robust body of social (...) psychological research demonstrates implicit gender bias in the evaluation of others (e.g., Steinpreis, Anders, & Ritzke, 1999). Many expected implicit bias to be a major cause of women’s underrepresentation in math intensive fields of science; however, quantitative sociological research of hiring and manuscript and grant evaluation has discovered no gender disparity in outcomes (Ceci & Williams, 2011). Why might this be so? This paper will discuss methodological challenges that go beyond classic problems of external validity in extrapolating psychological effects and explanations to scientific communities. These problems include more complex external validity issues raised by the introduction of multi-process models of cognition (e.g., implicit versus explicit social cognition) as well as the reflexive role that folk and experimental theories of social psychology play in guiding the behavior of scientists at the individual and community level. (shrink)

Under the traditional system of peer-reviewed publication, the degree of prestige conferred to authors by successful publication is tied to the degree of the intellectual rigor of its peer review process: ambitious scientists do well professionally by doing well epistemically. As a result, we should expect journal editors, in their dual role as epistemic evaluators and prestige-allocators, to have the power to motivate improved author behavior through the tightening of publication requirements. Contrary to this expectation, I will argue that the (...) publication bias literature in academic medicine demonstrates that editor interventions have had limited effectiveness in improving the health of the publication and trial registration record, suggesting that much stronger interventions are needed. (shrink)

In her account of scientific objectivity, feminist philosopher of science Helen Longino shows how scientific objectivity is not so much of individual practice, but rather a social commitment practiced by a scientific community, provided by the necessary accommodations for critical discourse. However, is this conception of scientific objectivity truly capable of living up to the social realities of critical discourse and deliberation within a scientific community? Drawing from Dutilh Novaes’ social epistemological account of argumentation, this paper highlights the challenges Longino’s (...) scientific objectivity faces on a prescriptive and descriptive level, specifically in overcoming the various epistemic injustices Longino’s proposed structural accommodations for objectivity are still sensitive to. Dutilh Novaes’ social epistemological model of argumentation illustrates how the realities of critical debate too often don’t consist of true epistemic or knowledge exchange, even though such exchanges are essential to achieve Longino’s primary goal when redefining scientific objectivity: to wield out and address idiosyncratic background assumptions and individual bigotry that possibly influence a researcher’s scientific conduct. (shrink)

This paper presents the results of an experiment designed to test for nationality bias among members of the Brazilian philosophical community. Faculty members and postgraduate students from philosophy departments at seven Brazilian universities evaluated texts attributed to authors of European and Latin American nationalities. Results showed a clear preference for French nationality over Brazilian. They were inconclusive, however, when contrasting other Latin American nationalities with European nationalities, which likely relates to the academic background of the participants. These overall results support (...) the claim that Brazilian academic philosophers are highly critical of Brazilians who produce theories of their own. The paper calls on philosophers to carefully consider the ways in which Eurocentrism impacts their very ability to reason. (shrink)

The enduring replication crisis in many scientific disciplines casts doubt on the ability of science to estimate effect sizes accurately, and in a wider sense, to self-correct its findings and to produce reliable knowledge. We investigate the merits of a particular countermeasure—replacing null hypothesis significance testing with Bayesian inference—in the context of the meta-analytic aggregation of effect sizes. In particular, we elaborate on the advantages of this Bayesian reform proposal under conditions of publication bias and other methodological imperfections that are (...) typical of experimental research in the behavioral sciences. Moving to Bayesian statistics would not solve the replication crisis single-handedly. However, the move would eliminate important sources of effect size overestimation for the conditions we study. (shrink)

Publishing research in respected journals, such as Nature or Science, is a key way for academics to share their findings with the scientific community and beyond.

Nowadays, almost everyone seems to agree that science is facing an epistemological crisis – namely the replicability crisis – and that we need to take action. But as to precisely what to do or how...

Scientific facts are constituted as consensus about observable phenomena against the background of an accepted, or at least plausible, theory. Empirical data without a theoretical framework are at best curiosities and anomalies, at worst they are neglected. The problem of parapsychological research since its inception with the foundation of the Society of Psychical Research in 1882 was that no sound theoretical basis existed. On the contrary, the proponents of the SPR often indulged in a theoretical model that ran contrary to (...) the perceived materialism of mainstream science, and many tried to use the data of parapsychological research to bolster the case of “mind over matter,” yet without producing a good model of how such effects could be conceptualized. In general, parapsychological (PSI) research has been rather devoid of theorizing and, if anything, assumed a tacit signal-theoretical, local-causal model of some sort of subtle energy that would be vindicated, once enough empirical data were amassed. History, and data, proved this stance wrong. We will present a theoretical approach that challenges this local-causal, signal-theoretical approach by proposing that parapsychological phenomena are instances of a larger class of phenomena that are examples of nonlocal correlations. These are predicted by Generalized Quantum Theory (GQT) and can be expected to occur, whenever global descriptions of a system are complementary to or incompatible with local descriptions of elements of such a system. We will analyze the standard paradigms of PSI-research along those lines and describe how they can be reconceptualized as instances of such generalized nonlocal correlations. A direct consequence of this conceptual framework is that misrepresentations of these phenomena as local causes, as is done in direct experimentation, is bound to fail long-term. Strategies to escape this problem are discussed. (shrink)

Scientific realists have suggested that changes in our scientific communities over the course of their history have rendered those communities progressively less vulnerable to the problem of unconcieved alternatives over time. I argue in response not only that the most fundamental historical transformations of the scientific enterprise have generated steadily mounting obstacles to revolutionary, transformative, or unorthodox scientific theorizing, but also that we have substantial independent evidence that the institutional apparatus of contemporary scientific inquiry fosters an exceedingly and increasingly theoretically (...) conservative form of that inquiry. I conclude that contemporary scientific communities are actually more vulnerable to the problem of unconceived alternatives than their historical predecessors, and I briefly suggest how we might seek to pursue scientific inquiry in a less theoretically conservative way. (shrink)

This paper offers a new angle on the common idea that the process of science does not support epistemic diversity. Under minimal assumptions on the nature of journal editing, we prove that editorial procedures, even when impartial in themselves, disadvantage less prominent research programs. This purely statistical bias in article selection further skews existing differences in the success rate and hence attractiveness of research programs, and exacerbates the reputation difference between the programs. After a discussion of the modeling assumptions, the (...) paper ends with a number of recommendations that may help promote scientific diversity through editorial decision making. (shrink)

Peer review is often taken to be the main form of quality control on academic research. Usually journals carry this out. However, parts of maths and physics appear to have a parallel, crowd-sourced model of peer review, where papers are posted on the arXiv to be publicly discussed. In this paper we argue that crowd-sourced peer review is likely to do better than journal-solicited peer review at sorting papers by quality. Our argument rests on two key claims. First, crowd-sourced peer (...) review will lead on average to more reviewers per paper than journal-solicited peer review. Second, due to the wisdom of the crowds, more reviewers will tend to make better judgments than fewer. We make the second claim precise by looking at the Condorcet Jury Theorem as well as two related jury theorems developed specifically to apply to peer review. (shrink)

Peer reviewers at many funding agencies and scientific journals are asked to score submissions both on individual criteria and overall. The overall scores should be some kind of aggregate of the criteria scores. Carole Lee identifies this as a potential locus for bias to enter the peer review process, which she calls commensuration bias. Here I view the aggregation of scores through the lens of social choice theory. I argue that, when reviewing grant proposals, it is in many cases impossible (...) to avoid commensuration bias. (shrink)

Prepublication peer review should be abolished. We consider the effects that such a change will have on the social structure of science, paying particular attention to the changed incentive structure and the likely effects on the behaviour of individual scientists. We evaluate these changes from the perspective of epistemic consequentialism. We find that where the effects of abolishing prepublication peer review can be evaluated with a reasonable level of confidence based on presently available evidence, they are either positive or neutral. (...) We conclude that on present evidence abolishing peer review weakly dominates the status quo. (shrink)

While some form of evaluation has always been employed in science (e.g. peer review, hiring), formal systems of evaluation of research and researchers have recently come to play a more prominent role in many countries because of the adoption of new models of governance. According to such models, the quality of the output of both researchers and their institutions is measured, and issues such as eligibility for tenure or the allocation of public funding to research institutions crucially depends on the (...) outcomes of such measures. However, concerns have been raised over the risk that such evaluation may be threatening epistemic pluralism by penalizing the existent heterodox schools of thought and discouraging the pursuit of new ones. It has been proposed that this may happen because of epistemic bias favouring mainstream research programmes. In this paper, I claim that (1) epistemic pluralism is desirable and should be preserved; (2) formal evaluation exercises may threaten epistemic pluralism because they may be affected by some form of epistemic bias; therefore, (3) to preserve epistemic pluralism, we need some strategy to actively dampen epistemic bias. (shrink)

Naturalistic philosophers rely on literature search and review in a number of ways and for different purposes. Yet this article shows how processes of literature search and review are likely to be affected by widespread and systematic biases. A solution to this problem is offered here. Whilst the tradition of systematic reviews of literature from scientific disciplines has been neglected in philosophy, systematic reviews are important tools that minimize bias in literature search and review and allow for greater reproducibility and (...) transparency. If naturalistic philosophers wish to reduce bias in their research, they should then supplement their traditional tools for literature search and review by including systematic methodologies. (shrink)

Recently, mainstream philosophy journals have tended to implement more and more stringent forms of peer review, probably in an attempt to prevent editorial decisions that are based on factors other than quality. Against this trend, we propose that journals should relax their standards of acceptance, as well as be less restrictive about whom is to decide what is admitted into the debate. We start by arguing, partly on the basis of the history of peer review in the journal Mind, that (...) past and current peer review practices attest to partisanship with respect to philosophical approach. Then, we explain that such partisanship conflicts with the standard aims of peer review, and that it is both epistemically and morally problematic. This assessment suggests that, if feasible, journals should treat all available and proposed standards of acceptance in philosophy as epistemically equal, and that philosophical work should be evaluated in terms of the novelty and significance of its contribution to developing thought in ways that are of value. Finally, we show, in a programmatic way, that improving the current situation is feasible, and can be done fairly easily. (shrink)

This paper shows that, for a large range of parameters, the journal editor prefers to delegate the choice to review the manuscript to the biased referee. If the peer review process is informative and the review reports are costly for the reviewers, even biased referees with extreme scientific preferences may choose to become informed about the manuscript’s quality. On the contrary, if the review process is potentially informative but the reviewer reports are not costly for the referees, the biased reviewer (...) has no incentive to become informed about the manuscript. Furthermore, if the reports are costly for referees but the peer review processes are not potentially informative, the biased reviewers will never become informed. In this paper, we also present a web resource that helps editors to experiment with the review process as a device for information transmission. (shrink)

Should editors of scientific journals practice triple-anonymous reviewing? I consider two arguments in favor. The first says that insofar as editors’ decisions are affected by information they would not have had under triple-anonymous review, an injustice is committed against certain authors. I show that even well-meaning editors would commit this wrong and I endorse this argument. The second argument says that insofar as editors’ decisions are affected by information they would not have had under triple-anonymous review, it will negatively affect (...) the quality of published papers. I distinguish between two kinds of biases that an editor might have. I show that one of them has a positive effect on quality and the other a negative one, and that the combined effect could be either positive or negative. Thus I do not endorse the second argument in general. However, I do endorse this argument for certain fields, for which I argue that the positive effect does not apply. (shrink)

As part of a continuous process to explore the factors that might weaken or corrupt traditional peer review, in this paper, we query the ethics, fairness and validity of the request, by editors, of authors to suggest peer reviewers during the submission process. One of the reasons for the current crisis in science pertains to a loss in trust as a result of a flawed peer review which is by nature biased unless it is open peer review. As we indicate, (...) the fact that some editors and journals rely on authors’ suggestions in terms of who should peer review their paper already instills a potential way to abuse the trust of the submission and publishing system. An author-suggested peer reviewer choice might also tempt authors to seek reviewers who might be more receptive or sympathetic to the authors’ message or results, and thus favor the outcome of that paper. Authors should thus not be placed in such a potentially ethically compromising situation, especially as a mandatory condition for submission. However, the fact that they do not have an opt-out choice during the submission process—especially when using an online submission system that makes such a suggestion compulsory—may constitute a violation of authors’ rights. (shrink)

Born in the 17th century, journal peer review is an extremely diverse technology, constantly torn between two often incompatible goals: the validation of manuscripts conceived as a collective industrial-like reproducible process performed to assert scientific statements, and the dissemination of articles considered as a means to spur scientific discussion, raising controversies, and civically challenging a state of knowledge. Such a situation is particularly conducive to clarifying the processes of valuation and evaluation in journal peer review. In this article, such processes (...) are considered as specific tests in order to emphasize the uncertain properties of pre-tests manuscripts. On the one hand, evaluation tests are examined at the core of the validation of manuscripts, such as defining the coordination of judging instances or controlling the modalities of inter-knowledge between reviewers and authors. They are also studied regarding the dissemination of articles, notably through the contemporary conception of a continuing evaluation test termed “post publication peer review”. On the other hand, valuation tests are both part of the validation of manuscripts, such as the weighting of different judgments of the same manuscript and the tensions that these hierarchies cause, and of the dissemination of articles, such as attention metrics recording the uses of articles. The conclusion sketches out how the articulation of these different tests has recently empowered readers as a new key judging instance for dissemination and for validation, potentially transforming the definition of peers, and thus the whole process of journal peer review. (shrink)

Although science is our most reliable producer of knowledge, it can also be used to create ignorance, unjustified doubt, and misinformation. In doing so, agnotological practices result not only in epistemic harms but also in social ones. A way to prevent or minimise such harms is to impede these ignorance-producing practices. In this paper, I explore various challenges to such a proposal. I first argue that reliably identifying agnotological practices in a way that permits the prevention of relevant harms is (...) more difficult than it might appear. I focus on an identifying criterion that many find apt for the task: bad faith motives. I then consider an objection—that reliable criteria are unnecessary to successfully address the concerns raised by agnotological strategies—and I show that it fails. I conclude by exploring other ways of conceptualising the problems attributed to agnotological practices. In particular, I challenge the focus on misinformation as the main problem of concern. (shrink)

This paper investigates how the credit incentive to engage in questionable research practices interacts with cumulative advantage, the process whereby high-status academics more easily increase their status than low-status academics. I use a mathematical model to highlight two dynamics that have not yet received much attention. First, due to cumulative advantage, questionable research practices may pay off over the course of an academic career even if they do not appear attractive at the level of individual publications. Second, because of the (...) role of bottleneck moments in academic careers, questionable research practices may be selected for even if they do not provide a benefit in expectation. I also observe that, within the model, the most successful academics are the most likely to have benefited from fraud. -/- This paper has been published under a different title. See PhilPapers record HEECAA or DOI 10.1086/716235. (shrink)

This paper investigates the representation of authors with different linguistic backgrounds in academic publishing. We first review some common rebuttals of concerns about linguistic injustice. We then analyze 1039 authors of philosophy journals, primarily selected from the 2015 Leiter Report. While our data show that Anglophones dominate the output of philosophy papers, this unequal distribution cannot be solely attributed to language capacities. We also discover that ethics journals have more Anglophone authors than logic journals and that most authors are affiliated (...) with English-speaking universities, suggesting other factors may also play significant roles. Moreover, some interesting results are revealed when we combine the factor of sex with place of affiliation and linguistic background. It indicates that while certain linguistic injustice is inevitable in academic publishing, it may be more complex than thought. We next introduce Broadbent’s :457–476, 2012, Philosophy of epidemiology, Palgrave Macmillan, New York, 2013, Stud Hist Philos Biol Biomed Sci 48:250–257, 2014) contrastive account of causation to give a causal explanation of our findings. Broadbent’s account not only well characterizes the multifaceted causality in academic publishing but also provides a methodological guideline for further investigation. (shrink)

These preprints were automatically compiled into a PDF from the collection of papers deposited in PhilSci-Archive in conjunction with the PSA 2018.

This article examines ethical concerns surrounding research on minority issues. Specifically, it addresses whether researchers have an obligation to consider the impact that minority research can have on vulnerable populations and their own backgrounds before conducting or assessing minority research for publication. The article argues for such an obligation mainly from a consequentialist perspective and then explores possible strategies for assessing and meeting that obligation. Finally, it explores the possible negative secondary effects of those strategies and censorship concerns.

This study examines the tension between the academic evaluation system and the development of unconventional research agendas. While scholars have studied the evaluation of research that crosses disciplinary boundaries from the perspective of peer reviewers, they have paid comparatively little attention to the experiences of the performers of unconventional science. This study asks how researchers develop unconventional research agendas to address a long-standing health problem and, in the process, make sense of the actions of a site visit committee organized to (...) advise a foundation funding the project. This study develops a process narrative on the development of a specific transdisciplinary team proposing to study premature birth. The findings show that when the performers and the evaluators of unconventional science developed competing understandings of the research agenda, transdisciplinary discovery became limited as a particular research topic became taboo. Yet, the study also reveals how the performers of unconventional science challenged the power of the site visit committee by making it seem as though they followed the committee’s decisions. These findings raise questions about the role of private foundations as funders of academic research and the suitability of traditional evaluation procedures for assessing transdisciplinary discovery. (shrink)

Peer review is embedded in the core of our knowledge generation systems, perceived as a method for establishing quality or scholarly legitimacy for research, while also often distributing academic prestige and standing on individuals. Despite its critical importance, it curiously remains poorly understood in a number of dimensions. In order to address this, we have analysed peer review to assess where the major gaps in our theoretical and empirical understanding of it lie. We identify core themes including editorial responsibility, the (...) subjectivity and bias of reviewers, the function and quality of peer review, and the social and epistemic implications of peer review. The high-priority gaps are focused around increased accountability and justification in decision-making processes for editors and developing a deeper, empirical understanding of the social impact of peer review. Addressing this at the bare minimum will require the design of a consensus for a minimal set of standards for what constitutes peer review, and the development of a shared data infrastructure to support this. Such a field requires sustained funding and commitment from publishers and research funders, who both have a commitment to uphold the integrity of the published scholarly record. We use this to present a guide for the future of peer review, and the development of a new research discipline based on the study of peer review. (shrink)

In September 2012, Gilles-Eric Séralini and seven coauthors published an article in Food and Chemical Toxicology claiming that rats fed Roundup©-resistant genetically modified maize alone, genetically modified maize with Roundup©, or Roundup© for 2 years had a higher percentage of tumors and kidney and liver damage than normal controls. Shortly after this study was published, numerous scientists and several scientific organizations criticized the research as methodologically and ethically flawed. In January 2014, the journal retracted the article without the authors’ consent (...) on the grounds that the research was inconclusive. In June 2014, Environmental Sciences Europe published a slightly modified version of the retracted paper. The publication, retraction and subsequent republication of the Séralini study raise important scientific and ethical issues for journal editors. Decisions to retract an article should be made on the basis of well-established policies. Articles should be retracted only for serious errors that undermine the reliability of the data or results, or for serious ethical lapses, such as research misconduct or mistreatment of animal or human subjects. Inconclusiveness, by itself, is not a sufficient reason for retracting an article, though a flawed study design might be. Retracted articles that are submitted for republication should undergo scientific review to ensure that they meet appropriate standards. Republished articles should be linked to the original, retracted publication. Journals that are reviewing studies with significant scientific and social implications should take special care to ensure that peer review is rigorous and fair. (shrink)

The way research is, and should be, funded by the public sphere is the subject of renewed interest for sociology, economics, management sciences, and more recently, for the philosophy of science. In this contribution, I propose a qualitative, epistemological criticism of the funding by lottery model, which is advocated by a growing number of scholars as an alternative to peer-review. This lottery scheme draws on the lack of efficiency and of robustness of the peer-review based evaluation to argue that the (...) majority of public resources for basic science should be allocated randomly. I first differentiate between two distinct arguments used to defend this alternative funding scheme based on considerations about the logic of scientific research. To assess their epistemological limits, I then present and develop a conceptual frame, grounded on the notion of “system of practice”, which can be used to understand what precisely it means, for a research project, to be interesting or significant. I use this epistemological analysis to show that the lottery model is not theoretically optimal, since it underestimates the integration of all scientific projects in densely interconnected systems of conceptual, experimental, or technical practices which confer their proper interest to them. I also apply these arguments in order to criticize the classical peer-review process. I finally suggest, as a discussion, that some recently proposed models that bring to the fore a principle of decentralization of the evaluation and selection process may constitute a better alternative, if the practical conditions of their implementation are adequately settled. (shrink)

The trustworthiness of nutrition science has been questioned recently. According to the critics, the food industry has corrupted scientists in the field. I argue that the worries that commercialization threatens the epistemic trustworthiness of nutrition science are indeed well-founded. However, it is problematic that the discussion has revolved around how funding can threaten the integrity of researchers and the methodological quality of the studies. By extending Wilholt’s :233–253, 2013) account of epistemic trustworthiness, I argue that when assessing the epistemic trustworthiness (...) of research that forms the basis for different health policy measures, it is necessary to evaluate research at the macro-level and whether agenda setting advances the goals that are assigned to the field. The prevalence of commercial funding becomes problematic if it leads to a situation where the body of available evidence that is used for making health policy decisions does not reflect the shared sense of what epistemic and non-epistemic goals of the inquiry are important. (shrink)

Psychometrically oriented researchers construe low inter-rater reliability measures for expert peer reviewers as damning for the practice of peer review. I argue that this perspective overlooks different forms of normatively appropriate disagreement among reviewers. Of special interest are Kuhnian questions about the extent to which variance in reviewer ratings can be accounted for by normatively appropriate disagreements about how to interpret and apply evaluative criteria within disciplines during times of normal science. Until these empirical-cum-philosophical analyses are done, it will remain (...) unclear the extent to which low inter-rater reliability measures represent reasonable disagreement rather than arbitrary differences between reviewers. (shrink)

There is an increasing push by journals to ensure that data and products related to published papers are shared as part of a cultural move to promote transparency, reproducibility, and trust in the scientific literature. Yet few journals commit to evaluating their effectiveness in implementing reporting standards aimed at meeting those goals (1, 2). Similarly, though the vast majority of journals endorse peer review as an approach to ensure trust in the literature, few make their peer review data available to (...) evaluate effectiveness toward achieving concrete measures of quality, including consistency and completeness in meeting reporting standards. Remedying these apparent disconnects is critical for closing the gap between guidance recommendations and actual reporting behavior. We see this as a collective action problem requiring leadership and investment by publishers, who can be incentivized through mechanisms that allow them to manage reputational risk and through continued innovation in journal assessment. (shrink)

Despite general agreement within philosophy that peer review is indispensable, its fairness and reliability is often questioned. This article suggests that such worries can to a large extent be met by adopting the practice that reviews as well as earlier versions of papers are made publicly available when the final version of a paper is published. This suggestion combines the advantages of transparency with the merits of anonymity of reviewers. While there are obstacles to this suggestion, the article argues that (...) it would be worthwhile to implement it because it can help map patterns of conduct and secure confidence in the fairness and reliability of review procedures and journal editing within philosophy. (shrink)

Peer-review is neither reliable, fair, nor a valid basis for predicting ‘impact’: as quality control, peer-review is not fit for purpose. Endorsing the consensus, I offer a reframing: while a normative social process, peer-review also shapes the writing of a scientific paper. In so far as ‘cognition’ describes enabling conditions for flexible behavior, the practices of peer-review thus constrain knowledge-making. To pursue cognitive functions of peer-review, however, manuscripts must be seen as ‘symbolizations’, replicable patterns that use technologically enabled activity. On (...) this bio-cognitive view, peer-review constrains knowledge-making by writers, editors, reviewers. Authors are prompted to recursively re-aggregate symbolizations to present what are deemed acceptable knowledge claims. How, then, can recursive re-embodiment be explored? In illustration, I sketch how the paper’s own content came to be re-aggregated: agonistic review drove reformatting of argument structure, changes in rhetorical ploys and careful choice of wordings. For this reason, the paper’s knowledge-claims can be traced to human activity that occurs in distributed cognitive systems. Peer-review is on the frontline in the knowledge sector in that it delimits what can count as knowing. Its systemic nature is therefore crucial to not only discipline-centered ‘real’ science but also its ‘post-academic’ counterparts. (shrink)

Naturalistic philosophers rely on literature search and review in a number of ways and for different purposes. Yet this article shows how processes of literature search and review are likely to be affected by widespread and systematic biases. A solution to this problem is offered here. Whilst the tradition of systematic reviews of literature from scientific disciplines has been neglected in philosophy, systematic reviews are important tools that minimize bias in literature search and review and allow for greater reproducibility and (...) transparency. If naturalistic philosophers wish to reduce bias in their research, they should then supplement their traditional tools for literature search and review by including systematic methodologies. (shrink)

Advocates of the self-corrective thesis argue that scientific method will refute false theories and find closer approximations to the truth in the long run. I discuss a contemporary interpretation of this thesis in terms of frequentist statistics in the context of the behavioral sciences. First, I identify experimental replications and systematic aggregation of evidence (meta-analysis) as the self-corrective mechanism. Then, I present a computer simulation study of scientific communities that implement this mechanism to argue that frequentist statistics may converge upon (...) a correct estimate or not depending on the social structure of the community that uses it. Based on this study, I argue that methodological explanations of the “replicability crisis” in psychology are limited and propose an alternative explanation in terms of biases. Finally, I conclude suggesting that scientific self-correction should be understood as an interaction effect between inference methods and social structures. (shrink)

The social identity of a researcher can affect their position in a community, as well as the uptake of their ideas. In many fields, members of underrepresented or minority groups are less likely to be cited, leading to citation gaps. Though this empirical phenomenon has been well-studied, empirical work generally does not provide insight into the causes of citation gaps. I will argue, using mathematical models, that citation gaps are likely due in part to the structure of academic communities. The (...) existence of these ‘structural causes’ has implications for attempts to lessen citation gaps, and for proposals to make academic communities more efficient. These proposals have the potential to create feedback loops, amplifying current structural inequities. (shrink)

The purpose of grant peer review is to identify the most excellent and promising research projects. However, sociologists of science and STS scholars have shown that peer review tends to promote solid low-risk projects at the expense of more original and innovative projects that often come with higher risk. It has also been shown that the review process is affected by significant measures of chance. Against this background, the aim of this study is to theorize the notions of academic judgment (...) and agonistic chance, and to present and analyze situations in which expert reviewers are faced with the challenge of trying to decide which grant proposals to select when there is strong disagreement. The empirical analysis is based on ethnographic observations of ten panel groups at the Swedish Research Council in the areas of natural and engineering sciences. By focusing on disagreement, the study provides a more in-depth understanding of how agonistic chance creeps into the peer review process and becomes part of the consensus that is created. (shrink)

The primary goal of the peer review of research grant proposals is to evaluate their quality for the funding agency. An important secondary goal is to provide constructive feedback to applicants for their resubmissions. However, little is known about whether review feedback achieves this goal. In this paper, we present a multi-methods analysis of responses from grant applicants regarding their perceptions of the effectiveness and appropriateness of peer review feedback they received from grant submissions. Overall, 56–60% of applicants determined the (...) feedback to be appropriate, although their judgments were more favorable if their recent application was funded. Importantly, independent of funding success, women found the feedback better written than men, and more white applicants found the feedback to be fair than non-white applicants. Also, perceptions of a variety of biases were specifically reported in respondents’ feedback. Less than 40% of applicants found the feedback to be very useful in informing their research and improving grantsmanship and future submissions. Further, negative perceptions of the appropriateness of review feedback were positively correlated with more negative perceptions of feedback usefulness. Importantly, respondents suggested that highly competitive funding pay-lines and poor inter-panel reliability limited the usefulness of review feedback. Overall, these results suggest that more effort is needed to ensure that appropriate and useful feedback is provided to all applicants, bolstering the equity of the review process and likely improving the quality of resubmitted proposals. (shrink)

Peer review of journal submissions has become one of the most important pillars of quality management in academic publishing. Because of growing concerns with the quality and effectiveness of the system, a host of enthusiastic innovators has proposed and experimented with new procedures and technologies. However, little is known about whether these innovations manage to convince other journal editors. This paper will address open questions regarding the implementation of new review procedures, the occurrence rate of various peer review procedures and (...) their distribution over scientific disciplines or academic publishers, as well as the motivations for editors or publishers to engage in novel review procedures. It shows that in spite of enthusiastic innovation, the adoption of new peer review procedures is in fact very slow, with the exception of text similarity scanners. For now, peer review innovations appear to be restricted to specific niches in academic publishing. Analysing these niches, the article concludes with a reflection on the circumstances in which innovations might be more widely implemented. (shrink)

Journal peer review is an essential part of academic practices.1 But how well does it serve its purpose and which factors have an influence on how close it comes to achieving its aims? Peer review has been widely discussed in empirical literature: it has been studied both qualitatively and quantitatively (e.g., by Cole, who in his 1992 book uses data on how grant applications submitted to National Science Foundation were...

This study seeks to examine the impact of alumni connections between the evaluators and evaluatees on the results of peer review ratings for the Korean national R&D project and selection success rate. Specifically, this study analyzed the evaluation results of 8,402 research proposal entries submitted between 2007 and 2011 for the “general researchers support project,” all in the Natural Science and Engineering areas and sponsored by the National Research Foundation of Korea. Each proposal entry was evaluated by three evaluators, and (...) approximately 39 percent of the proposals had at least one evaluator from the same university. The results of this study showed that evaluators have a tendency to give relatively high scores to research proposals submitted by the alumni of the same universities as their alma mater. Also, when an evaluator from the same university as an evaluatee was included in the evaluator group, the results of this study showed that the percentage of entry submissions was higher compared to the contrary. Such results show that in the process of peer review–based research proposal evaluations for national R&D projects, alumni connections have significant influence on evaluation results in South Korea. (shrink)

This article examines the evolution of peer review and the modern editorial processes of scholarly journals by analyzing a novel data set derived from the Royal Society’s archives and covering 1865-1965, that is, the historical period in which refereeing became firmly established. Our analysis reveals how the Royal Society’s editorial processes coped with both an increasing reliance on refereeing and a growth in submissions, while maintaining collective responsibility and minimizing research waste. By engaging more of its fellows in editorial activity, (...) the society was able to establish an equilibrium of number of submissions per reviewer that was relatively stable over time. Nevertheless, our analysis shows that the distribution of editorial work was significantly uneven. Our findings reveal interesting parallels with current concerns about the scale and distribution of peer review work and suggest the strategic importance of the management of the editorial process to achieve a creative mix of community commitment and professional responsibility that is essential in contemporary journals. (shrink)

In a recent editorial published in Nature, the journal's editors comment on a new automated software that has been used to check findings in psychology publications. The editors express concern with the way in which the anonymous fact-checkers have proceeded, but at the same time, they underscore the crucial role of peer criticism for scientific progress and insist: "self-correction is at the heart of science." Brief as it is, the editorial showcases that peer criticism and the application of norms of (...) good research practice are very thorny issues indeed.It is essential for the functioning of science that empirical findings, arguments, as well as methodological approaches are scrutinized... (shrink)