Gigerenzer’s ‘external validity argument’ plays a pivotal role in his critique of the heuristics and biases research program (HB). The basic idea is that (a) the experimental contexts deployed by HB are not representative of the real environment and that (b) the differences between the setting and the real environment are causally relevant, because they result in different performances by the subjects. However, by considering Gigerenzer’s work on frequencies in probability judgments, this essay attempts to show that there are fatal (...) flaws in the argument. Specifically, each of the claims is controversial: whereas (b) is not adequately empirically justified, (a) is inconsistent with the ‘debiasing’ program of Gigerenzer’s ABC group. Therefore, whatever reason we might have for believing that the experimental findings of HB lack experimental validity, this should not be based on Gigerenzer’s version of the argument. (shrink)

Psychometrically oriented researchers construe low inter-rater reliability measures for expert peer reviewers as damning for the practice of peer review. I argue that this perspective overlooks different forms of normatively appropriate disagreement among reviewers. Of special interest are Kuhnian questions about the extent to which variance in reviewer ratings can be accounted for by normatively appropriate disagreements about how to interpret and apply evaluative criteria within disciplines during times of normal science. Until these empirical-cum-philosophical analyses are done, it will remain (...) unclear the extent to which low inter-rater reliability measures represent reasonable disagreement rather than arbitrary differences between reviewers. (shrink)