This work focused on the investigation of the question how the concept of relevance in Information Retrieval can be validated. The work is motivated by the consistent difficulties of defining the meaning of the concept, and by advances in the field of cognitive science. Analytical and empirical investigations are carried out with the aim of devising a principled approach to the validation of the concept. The foundation for this work was set by interpreting relevance as a phenomenon occurring within the (...) context of two systems: An IR system and the cognitive processing system of the user. In light of the cognitive interpretation of relevance, an analysis of the learnt lessons in cognitive science with regard to the validation of cognitive phenomena was conducted. It identified that construct validity constitutes the dominant approach to the validation of constructs in cognitive science. Construct validity constitutes a proposal for the conduction of validation in scenarios, where no direct observation of a phenomenon is possible. With regard to the limitations on direct observation of a construct (i.e. a postulated theoretic concept), it bases validation on the evaluation of its relations to other constructs. Based on the interpretation of relevance as a product of cognitive processing it was concluded, that the limitations with regard to direct observation apply to its investigation. The evaluation of its applicability to an IR context, focused on the exploration of the nomological network methodology. A nomological network constitutes an analytically constructed set of constructs and their relations. The construction of such a network forms the basis for establishing construct validity through investigation of the relations between constructs. An analysis focused on contemporary insights to the nomological network methodology identified two important aspects with regard to its application in IR. The first aspect is given by a choice of context and the identification of a pool of candidate constructs for the inclusion in the network. The second consists of identifying criteria for the selection of a set of constructs from the candidate pool. The identification of the pertinent constructs for the network was based on a review of the principles of cognitive exploration, and an analysis of the state of the art in text based discourse processing and reasoning. On that basis, a listing of known sub-processes contributing to the pertinent cognitive processing was presented. Based on the identification of a large number of potential candidates, the next step consisted of the inference of criteria for the selection of an initial set of constructs for the network. The investigation of these criteria focused on the consideration of pragmatic and meta-theoretical aspects. Based on a survey of experimental means in cognitive science and IR, five pragmatic criteria for the selection of constructs were presented. Consideration of meta-theoretically motivated criteria required to investigate what the specific challenges with regard to the validation of highly abstract constructs are. This question was explored based on the underlying considerations of the Information Processing paradigm and Newell’s (1994) cognitive bands. This led to the identification of a set of three meta-theoretical criteria for the selection of constructs. Based on the criteria and the demarcated candidate pool, an IR focused nomological network was defined. The network consists of the constructs of relevance and type and grade of word relatedness. A necessary prerequisite for making inferences based on a nomological network consists of the availability of validated measurement instruments for the constructs. To that cause, two validation studies targeting the measurement of the type and grade of relations between words were conducted. The clarification of the question of the validity of the measurement instruments enabled the application of the nomological network. A first step of the application consisted of testing if the constructs in the network are related to each other. Based on the alignment of measurements of relevance and the word related constructs it was concluded to be true. The relation between the constructs was characterized by varying the word related constructs over a large parameter space and observing the effect of this variation on relevance. Three hypotheses relating to different aspects of the relations between the word related constructs and relevance. It was concluded, that the conclusive confirmation of the hypotheses requires an extension of the experimental means underlying the study. Based on converging observations from the empirical investigation of the three hypotheses it was concluded, that semantic and associative relations distinctly differ with regard to their impact on relevance estimation. (shrink)

In this paper we attempt to understand the intellectual origins of Albert Ellis' Rational Psychotherapy (now known as Rational Emotive Behavior Therapy). In his therapeutic practice Ellis used a 'lumper' argument to replace the focus of change in psychoanalysis: not the lengthy uncovering and reworking of the individual's personal history, but the demands in self-talk through which the client is currently dis turbed. In constructing around this the persuasive (rhetorical) package that became his therapy, Ellis drew on a number of (...) popular intellectual movements, operationalism, General Semantics, the holistic theory of emotion, cognitive psychology, and psychoanalysis itself. (shrink)

The rhetoric-analytic critique of experimental psychology owes its apparent attractiveness to (a) some erroneous ideas about cognitive psychology and the rationale of experimentation, (b) the failure to distinguish between prior data and evidential data vis-à-vis the to-be-corroborated explanatory theory, and (c) evidential data owes their identity to a theory that is independent of the theory being tested. Theories in cognitive psychology are accepted because they can withstand concerted efforts to falsify them.

A representation is a remnant of previous experience that allows that experience to affect later behavior. This paper develops a metatheoretical view of representation and applies it to issues concerning representation in animals. To describe a representational system one must specify the following: thedomainor range of situations in the represented world to which the system applies; thecontentor set of features encoded and preserved by the system; thecodeor transformational rules relating features of the representation to the corresponding features of the represented (...) world; themedium, or the representation's physical instantiation; and thedynamics, or how the system changes with time. In part because of the behaviorist assumption that the hypothetical, covert changes occurring in an organism during learning correspond to the overt physical changes that are observed, issues of representation in animal behavior have been largely ignored as irrelevant or misleading. However, it can be inferred that representations, acting as models of environmental regularities, operate at many levels of behavioral functioning, both cognitive and noncognitive. Objections to the use of this concept in explanations of animal behavior, based on the claim that it is indeterminate and on behaviorist considerations of parsimony, can be answered. Animal representations may be specialized in terms of tasks and species. Data from tasks involving spatial memory, delayed matching-to-sample, and sequence learning suggest some foundations for a general theory of animal representations. (shrink)

Brain damage may doubly dissociate cognitive modules, but the practice of revealing dissociations is predicated on modularity being true (T. Shallice, 1988). This article questions the utility of assuming modularity, as it examines a paradigmatic double dissociation of reading modules. Reading modules illustrate two general problems. First, modularity fails to converge on a fixed set of exclusionary criteria that define pure cases. As a consequence, competing modular theories force perennial quests for purer cases, which simply perpetuates growth in the list (...) of exclusionary criteria. The first problem leads, in part, to the second problem. Modularity fails to converge on a fixed set of pure cases. The second failure perpetuates unending fractionation into more modules. (shrink)

Accurately assessing youth mental health involves obtaining reports from multiple informants who typically display low levels of correspondence. This low correspondence may reflect situational specificity. That is, youth vary as to where they display mental health concerns and informants vary as to where and from what perspective they observe youth. Despite the frequent need to understand and interpret these informant discrepancies, no consensus guidelines exist for integrating informants’ reports. The path to building these guidelines starts with identifying factors that reliably (...) predict the level and form of these informant discrepancies, and do so for theoretically and empirically relevant reasons. Yet, despite the knowledge of situational specificity, few approaches to integrating multi-informant data are well-equipped to account for these factors in measurement, and those that claim to be well-positioned to do so have undergone little empirical scrutiny. One promising approach was developed roughly 20 years ago by Kraemer and colleagues. Their Satellite Model leverages principal components analysis and strategic selection of informants to instantiate situational specificity in measurement, namely components reflecting variance attributable to the context in which informants observe behavior, the perspective from which they observe behavior, and behavior that manifests across contexts and perspectives. The current study represents the first construct validation test of the Satellite Model. A mixed-clinical/community sample of 134 adolescents and their parents completed six parallel surveys of adolescent mental health. Adolescents also participated in a series of simulated social interactions with research personnel trained to act as same-age, unfamiliar peers. A third informant viewed these interactions and completed the same surveys as parents and adolescents. We applied the Satellite Model to each set of surveys and observed high internal consistency estimates for each of the six-item trait, context, and perspective components. Scores reflecting the trait, context, and perspective components displayed distinct patterns of relations to a battery of criterion variables that varied in the context, perspective, and source of measurement. The Satellite Model instantiates situational specificity in measurement and facilitates unifying conceptual and measurement models of youth mental health. (shrink)

Brain damage may doubly dissociate cognitive modules, but the practice of revealing dissociations is predicated on modularity being true (T. Shallice, 1988). This article questions the utility of assuming modularity, as it examines a paradigmatic double dissociation of reading modules. Reading modules illustrate two general problems. First, modularity fails to converge on a fixed set of exclusionary criteria that define pure cases. As a consequence, competing modular theories force perennial quests for purer cases, which simply perpetuates growth in the list (...) of exclusionary criteria. The first problem leads, in part, to the second problem. Modularity fails to converge on a fixed set of pure cases. The second failure perpetuates unending fractionation into more modules. (shrink)

On page 1 of his classic text, Millsap states, “Measurement invariance is built on the notion that a measuring device should function the same way across varied conditions, so long as those varied conditions are irrelevant [emphasis added] to the attribute being measured.” By construction, measurement invariance techniques require not only detecting varied conditions but also ruling out that these conditions inform our understanding of measured domains. In fact, measurement invariance techniques possess great utility when theory and research inform their (...) application to specific, varied conditions that, if not detected, introduce measurement biases, and, thus, depress measurement validity. Yet, we see emerging bodies of work where scholars have “put the cart before the horse” when it comes to measurement invariance, and they apply these techniques to varied conditions that, in fact, may reflect domain-relevant information. These bodies of work highlight a larger problem in measurement that likely cuts across many areas of scholarship. In one such area, youth mental health, researchers commonly encounter a set of conditions that nullify the use of measurement invariance, namely discrepancies between survey reports completed by multiple informants, such as parents, teachers, and youth themselves. In this paper, we provide an overview of conceptual, methodological, and measurement factors that should prevent researchers from applying measurement invariance techniques to detect informant discrepancies. Along the way, we cite evidence from the last 15 years indicating that informant discrepancies reflect domain-relevant information. We also apply this evidence to recent uses of measurement invariance techniques in youth mental health. Based on prior evidence, we highlight the implications of applying these techniques to multi-informant data, when the informant discrepancies observed within these data might reflect domain-relevant information. We close by calling for a moratorium on applying measurement invariance techniques to detect informant discrepancies in youth mental health assessments. In doing so, we describe how the state of the science would need to fundamentally “flip” to justify applying these techniques to detect informant discrepancies in this area of work. (shrink)

Participants were asked to draw inferences about correlation from single x,y observations. In Experiment 1 statistically sophisticated participants were given the univariate characteristics of distributions of x and y and asked to infer whether a single x, y observation came from a correlated or an uncorrelated population. In Experiment 2, students with a variety of statistical backgrounds assigned posterior probabilities to five possible populations based on single x, y observations, again given knowledge of the univariate statistics. In Experiment 3, statistically (...) naïve participants were given a problem analogous to that given in Experiment 1, framed verbally. Experiment 4 replicated Experiment 3 but added an “impossible to determine” response option. Models that rely on computing sample correlations make no predictions about these investigations. From a Bayesian perspective, participants’ inferences in all four experiments tended to make probabilistically valid inferences as long as the single datum was directional. The results are discussed in light of the Brunswikian notion of vicarious functioning. (shrink)

Introductory Abstract Philosophers of science, in the course of making a sharp distinction between the tasks of the philosopher and those of the scientist, have pointed to the possibility of an empirical science of induction. A comparative psychology of knowledge processes is offered as one aspect of this potential enterprise. From fragments of such a psychology, methodological suggestions are drawn relevant to several chronic problems in the social sciences, including the publication of negative results from novel explorations, the operational diagnosis (...) of dispositions, the status of aggregates of persons as social entities, and the validation of psychological tests. (shrink)