Supergrading: how diverse standards can improve collective performance in ranking tasks

Theory and Decision 88 (4):541-565 (2020)
  Copy   BIBTEX

Abstract

The method of supergrading is introduced for deriving a ranking of items from scores or grades awarded by several people. Individual inputs may come in different languages of grades. Diversity in grading standards is an advantage, enabling rankings derived by this method to separate more items from one another. A framework is introduced for studying grading on the basis of observations. Measures of accuracy, reliability and discrimination are developed within this framework. Ability in grading is characterized for individuals and groups as the capacity to grade reliably, accurately and at a high level of discrimination. It is shown that the collective ability of a supergrading group with diverse standards can be greater than that of a less diverse group whose members have greater ability.

Author's Profile

Michael Morreau
University of Tromsø

Analytics

Added to PP
2019-12-18

Downloads
284 (#50,278)

6 months
92 (#38,288)

Historical graph of downloads since first upload
This graph includes both downloads from PhilArchive and clicks on external links on PhilPapers.
How can I increase my downloads?