As more objections have been raised against grant peer-review for being costly and time-consuming, the legitimate question arises whether machine learning algorithms could help assess the epistemic efficiency of the proposed projects. As a case study, we investigated whether project efficiency in high energy physics can be algorithmically predicted based on the data from the proposal. To analyze the potential of algorithmic prediction in HEP, we conducted a study on data about the structure and outcomes of HEP experiments with the (...) goal of predicting their efficiency. In the first step, we assessed the project efficiency using Data Envelopment Analysis of 67 experiments conducted in the HEP laboratory Fermilab. In the second step, we employed predictive algorithms to detect which team structures maximize the epistemic performance of an expert group. For this purpose, we used the efficiency scores obtained by DEA and applied predictive algorithms – lasso and ridge linear regression, neural network, and gradient boosted trees – on them. The results of the predictive analyses show moderately high accuracy, indicating that they can be beneficial as one of the steps in grant review. Still, their applicability in practice should be approached with caution. Some of the limitations of the algorithmic approach are the unreliability of citation patterns, unobservable variables that influence scientific success, and the potential predictability of the model. (shrink)

The question of when to stop an unsuccessful experiment can be difficult to answer from an individual perspective. To help to guide these decisions, we turn to the social epistemology of science and investigate knowledge inquisition within a group. We focused on the expensive and lengthy experiments in high energy physics, which were suitable for citation-based analysis because of the relatively quick and reliable consensus about the importance of results in the field. In particular, we tested whether the time spent (...) on a scientific project correlates with the project output. Our results are based on data from the high energy physics laboratory Fermilab. They point out that there is an epistemic saturation point in experimenting, after which the likelihood of obtaining major results drops. With time the number of less significant publications does increase, but highly cited ones do not get published. Since many projects continue to run after the epistemic saturation point, it becomes clearer that decisions made about continuing them are not always rational. (shrink)

Prepublication peer review should be abolished. We consider the effects that such a change will have on the social structure of science, paying particular attention to the changed incentive structure and the likely effects on the behaviour of individual scientists. We evaluate these changes from the perspective of epistemic consequentialism. We find that where the effects of abolishing prepublication peer review can be evaluated with a reasonable level of confidence based on presently available evidence, they are either positive or neutral. (...) We conclude that on present evidence abolishing peer review weakly dominates the status quo. (shrink)

We argue that inductive analysis and operational assessment of the scientific process can be justifiably and fruitfully brought together, whereby the citation metrics used in the operational analysis can effectively track the inductive dynamics and measure the research efficiency. We specify the conditions for the use of such inductive streamlining, demonstrate it in the cases of high energy physics experimentation and phylogenetic research, and propose a test of the method’s applicability.

The organization of cutting-edge HEP laboratories has evolved in the intersection of academia, state agencies, and industry. Exponentially ever-larger and more complex knowledge-intensive operations, the laboratories have often faced the challenges of, and required organizational solutions similar to, those identified by a cluster of diverse theories falling under the larger heading of organization theory. The cluster has either shaped or accounted for the organization of industry and state administration. The theories also apply to HEP laboratories, as they have gradually and (...) uniquely hybridized their principles and solutions. Yet scholarship has virtually ignored this linkage and has almost exclusively focused on the laboratories’ presumably unique egalitarian organizational aspects. Guided by the principles developed in the organization theory cluster, we identify the basic organizational features of HEP laboratories in relation to their pursuit of narrow and broad epistemic goals. We also provide a set of criteria and methods for assessing the efficiency of the identified organizational features in achieving such goals. (shrink)

This paper presents an account of how to evaluate formal models of science: models and simulations in social epistemology designed to draw normative conclusions about the social structure of scientific research. I argue that such models should be evaluated according to their representational and predictive accuracy. Using these criteria and comparisons with familiar models from science, I argue that most formal models of science are incapable of supporting normative conclusions.

Varied evidence confirms more strongly than less varied evidence, ceteris paribus. This epistemological Variety of Evidence Thesis enjoys widespread intuitive support. We put forward a novel explication of one notion of varied evidence and the Variety of Evidence Thesis within Bayesian models of scientific inference by appealing to measures of entropy. Our explication of the Variety of Evidence Thesis holds in many of our models which also pronounce on disconfirmatory and discordant evidence. We argue that our models pronounce rightly. Against (...) a backdrop of failures of the Variety of Evidence Thesis, the intuitive case for the Variety of Evidence Thesis emerges strengthened. Our models do however not support the general case for the thesis since our explication of it fails to hold in certain cases. The parameter space of this failure is explored and an explanation for the failure is offered. (shrink)