Research into expertise is relatively common in cognitive science concerning expertise existing across many domains. However, much less research has examined how experts within the same domain assess the performance of their peer experts. We report the results of a modified think-aloud study conducted with 18 pilots . Pairs of same-ranked pilots were asked to rate the performance of a captain flying in a critical pre-recorded simulator scenario. Findings reveal considerable variance within performance categories, differences in the process used as (...) evidence in support of a performance rating, different numbers and types of facts identified, and differences in how specific performance events affect choice of performance category and gravity of performance assessment. Such variance is consistent with low inter-rater reliability. Because raters exhibited good, albeit imprecise, reasons and facts, a fuzzy mathematical model of performance rating was developed. The model provides good agreement with observed variations. (shrink)

The current study had three goals: to investigate strategies, tactics, and errors as predictors of success and failure under uncertainty following the dynamic decision making and complex problem solving framework; to use observation and to examine its reliability and potential as a data collection method when using microworlds; and to investigate the applicability and validity of a microworld developed in the West, to an Asian sample. One hundred three participants in the Philippines took the role of fire chief in the (...) microworld WINFIRE. Their strategies, tactics, and errors were observed and coded by experimenters as they worked individually on the simulation twice. Results showed that DDM strategies, tactics, and errors predicted success and failure in WINFIRE, and strategies and tactics that led to success increased while errors decreased over time; strategies, tactics, and errors can be validly assessed through observation by experimenters, specifically that two types of decision makers were identified: the active, flexible, and big picture planners and the slow or cautious, and singlefocused decision makers; these findings together with participants’ survey ratings speak for the applicability of the microworld in an East Asian sample and for its validity. Findings are potentially relevant for experts and for training programs, highlighting the benefits of virtual environments during DDM. (shrink)

Thinking aloud is the concurrent verbalization of thoughts while performing a task. The study of thinking-aloud protocols has a long tradition in cognitive psychology, the field of education, and the industrial-organizational context. It has been used rarely in cultural and cross-cultural psychology. This paper will describe thinking aloud as a useful method in cultural and cross-cultural psychology referring to a few studies in general and one study in particular to show the wide applications of this method. Thinking-aloud protocols can be (...) applied for a) improving the validity of cross-cultural surveys, b) process analysis of thoughts and the analysis of changes over time, c) theory development across cultures, d) the study of cultural meaning systems, and e) individual as well as group level analyses allowing hypothesis testing cross-culturally. Limitations of the thinking-aloud method are also discussed. (shrink)

The Role of Motivation in Complex Problem SolvingPrevious research on Complex Problem Solving (CPS) has primarily focused on cognitive factors as outlined below. The current paper discusses the role of motivation during CPS and argues that motivation, emotion, and cognition interact and cannot be studied in an isolated manner. Motivation is the process that determines the energization and direction of behavior (Heckhausen, 1991).Three motivation theories and their relation to CPS are examined: McClelland’s achievement motivation, Maslow’s hierarchy of needs, and Dörner’s (...) needs as outlined in PSI-theory. We chose these three theories for several reasons. First, space forces us to be selective. Second, the three theories are among the most prominent motivational theories. Finally, they are need theories postulating several motivations and not just one. A thinking-aloud protocol is provided to illustrate the role of motivational and cognitive dynamics in CPS. Problems are part of all the domains of human life. The field of CPS investigates problems that are complex, dynamic, and non-transparent (Dörner, 1996). Complex problems consist of many interactively interrelated variables. Dynamic ones change and develop further over time, regardless of whether the involved people take action. And non-transparent problems have many aspects of the problem situation that are unclear or unknown to the involved people. CPS researchers focus exactly on such kinds of problems. Under a narro... (shrink)