Sections include: experiments and generalised causal inference; statistical conclusion validity and internal validity; construct validity and external validity; quasi-experimental designs that either lack a control group or lack pretest observations on the outcome; quasi-experimental designs that use both control groups and pretests; quasi-experiments: interrupted time-series designs; regresssion discontinuity designs; randomised experiments: rationale, designs, and conditions conducive to doing them; practical problems 1: ethics, participation recruitment and random assignment; practical problems 2: treatment implementation and attrition; generalised causal inference: a grounded theory; (...) generalised causal inference: methods for single studies; generalised causal inference: methods for multiple studies; a critical assessment of our assumptions. (shrink)

Theories of skill acquisition have made radically different predictions about the role of general problem‐solving methods in acquiring rules that promote effective transfer to new problems. Under one view, methods that focus on reaching specific goals, such as means‐ends analysis, are assumed to provide the basis for efficient knowledge compilation (Anderson, 1987), whereas under an alternative view such methods are believed to disrupt rule induction (Sweller, 1988). We suggest that the role of general methods in learning varies with both the (...) specificity of the problem solver's goal and the systematicity of the strategies used for testing hypotheses about rules. In the absence of a specific goal people are more likely to use a rule‐induction learning strategy, whereas provision of a specific goal fosters use of difference reduction, which tends to be a non‐rule‐induction strategy. We performed two experiments to investigate the impact of goal specificity and systematicity of rule‐induction strategies in learning and transfer within a complex dynamic system. The results of Experiment 1 indicated that during free exploration of a problem space, greater learning occurred if participants adopted more systematic strategies for rule induction, and that participants come to favor such strategies. Experiment 2 revealed that participants who were provided with a specific goal performed well on the initial problem but were impaired on a transfer test using a similar problem with a different goal. Instruction on a systematic rule‐induction strategy facilitated solution for both the initial and transfer problems, but participants' use of this strategy declined if they had a specific goal. Our results support Sweller's (1988) proposal that general problemsolving methods applied to a specific goal foster acquisition of knowledge about an isolated solution path but do not provide an effective way of learning the overall structure of a problem space. We interpret these results in terms of dualspace theories of search through problem space. (shrink)

The assumption that positive affect leads to a better performance in simple cognitive tasks has become well established. We address the question whether positive and negative emotions differentially influence performance in complex problem-solving in the same way. Emotions were induced by positive or negative feedback in 74 participants who had to manage a computer-simulated complex problem-solving scenario. Results show that overall scenario performance is not affected, but positive and negative emotions elicit distinguishable problem-solving strategies: Participants with negative emotions are more (...) focused on the seeking and use of information. We discuss methodological requirements for investigating emotion influences in complex and dynamic cognitive tasks. (shrink)

This study examined whether experience with different computer-based complex problem situations would improve complex problem solving competence in an unknown problem situation. Therefore, university students took part in a control group study. They were trained in five different complex problem situations for up to 7 hr, and their performance was tested in a sixth complex problem situation. The data analyses revealed that the training influenced the CPS process of knowledge acquisition. However, the CPS process of knowledge application was not impacted (...) by experience with other problem situations. Implications for the concept of CPS as a trainable competence as well as the training of CPS in general are discussed. (shrink)

In this paper we propose the multiple complex systems approach for assessing domain-general complex problem-solving skills and its processes knowledge acquisition and knowledge application. After defining the construct and the formal frameworks for describing complex problems, we emphasise some of the measurement issues inherent in assessing CPS skills with single tasks. With examples of the MicroDYN test and the MicroFIN test, we show how to adequately score problem-solving skills by using multiple tasks. We discuss implications for problem-solving research and the (...) assessment of CPS skills in general. (shrink)

Many situations require people to acquire knowledge about, and learn how to control, complex dynamic systems of inter-connected variables. Numerous studies have found that most problem solvers are unable to acquire complete knowledge of the underlying structure of a system through an unguided exploration of the system variables; additional instruction or guidance is required. This paper examines whether providing structural information following an unguided exploration also improves control performance, and the extent to which any improvements are moderated by problem solvers’ (...) fluid intelligence. A sample of 98 individuals attempted to discover the underlying structure of a computer-simulated complex dynamic system. After initially controlling the system with their limited knowledge, half of the sample received guidance concerning the actual underlying structure of the system. All participants then controlled the system again. In contrast to the results of previous studies, guidance did result in immediate improvements in control performance. Fluid intelligence moderated the extent to which problem solvers benefited from structural information. These results indicate that guidance in the form of structural information is critical in promoting knowledge and control over complex and dynamic systems. (shrink)

This paper presents a learning theory pertinent to dynamic decision making (DDM) called instancebased learning theory (IBLT). IBLT proposes five learning mechanisms in the context of a decision‐making process: instance‐based knowledge, recognition‐based retrieval, adaptive strategies, necessity‐based choice, and feedback updates. IBLT suggests in DDM people learn with the accumulation and refinement of instances, containing the decision‐making situation, action, and utility of decisions. As decision makers interact with a dynamic task, they recognize a situation according to its similarity to past instances, (...) adapt their judgment strategies from heuristic‐based to instance‐based, and refine the accumulated knowledge according to feedback on the result of their actions. The IBLT's learning mechanisms have been implemented in an ACT‐R cognitive model. Through a series of experiments, this paper shows how the IBLT's learning mechanisms closely approximate the relative trend magnitude and performance of human data. Although the cognitive model is bounded within the context of a dynamic task, the IBLT is a general theory of decision making applicable to other dynamic environments. (shrink)

In the last 20 years, a stream of research emerged under the label of „complex problem solving“ (CPS). This research was intended to describe the way people deal with complex, dynamic, and intransparent situations. Complex computer-simulated scenarios were as stimulus material in psychological experiments. This line of research lead to subtle insights into the way how people deal with complexity and uncertainty. Besides these knowledge-rich, realistic, intransparent, complex, dynamic scenarios with many variables, a second line of research used more simple, (...) knowledge-lean scenarios with a low number of variables („minimal complex systems“, MCS) that have been proposed recently in problem-solving research for the purpose of educational assessment. In both cases, the idea behind the use of microworlds is to increase validity of problem solving tasks by presenting interactive environments that can be explored and controlled by participants while pursuing certain action goals. The main argument presented here is: both types of systems - CPS and MCS – can only be dealt with successfully if causal dependencies between input and output variables are identified and used for system control. System knowledge is necessary for control and intervention. But CPS and MCS differ in their way of how causal dependencies are identified and how the mental model is constructed; therefore, they cannot be compared directly to each other with respect to the cognitive processes that are necessary for solving the tasks. Knowledge-poor MCS tasks address only a small fraction of the cognitive processes and structures needed for knowledge-rich CPS situations. (shrink)

The question “How can humans learn efficiently to make decisions in a complex, dynamic, and uncertain environment” is still a very open question. We investigate what effects arise when feedback is given in a computer-simulated microworld that is controlled by participants. This has a direct impact on training simulators that are already in standard use in many professions, e.g., for flight simulators for pilots, and a potential impact on a better understanding of human decision making in general. Our study is (...) based on a benchmark microworld with an economic framing, the IWR Tailorshop. N=94 participants played four rounds of the microworld, each 10 months, via a web interface. We propose a new approach to quantify performance and learning, which is based on a mathematical model of the microworld and optimization. Six participant groups receive different kinds of feedback in a training phase, then results in a performance phase without feedback are analyzed. As a main result, feedback of optimal solutions in training rounds improved model knowledge, early learning, and performance, especially when this information is encoded in a graphical representation. (shrink)

Computational models of learning and the theories they represent are often validated by calibrating them to human data on decision outcomes. However, only a few models explain the process by which these decision outcomes are reached. We argue that models of learning should be able to reflect the process through which the decision outcomes are reached, and validating a model on the process is likely to help simultaneously explain both the process as well as the decision outcome. To demonstrate the (...) proposed validation, we use a large dataset from the Technion Prediction Tournament and an existing Instance-based Learning model. We present two ways of calibrating the model’s parameters to human data: on an outcome measure and on a process measure. In agreement with our expectations, we find that calibrating the model on the process measure helps to explain both the process and outcome measures compared to calibrating the model on the outcome measure. These results hold when the model is generalized to a different dataset. We discuss implications for explaining the process and the decision outcomes in computational models of learning. (shrink)

Computer-simulated scenarios have been part of psychological research on problem solving for more than 40 years. The shift in emphasis from simple toy problems to complex, more real-life oriented problems has been accompanied by discussions about the best ways to assess the process of solving complex problems. Psychometric issues such as reliable assessments and addressing correlations with other instruments have been in the foreground of these discussions and have left the content validity of complex problem solving in the background. In (...) this paper, we return the focus to content issues and address the important features that define complex problems. (shrink)

This book presents an overview of these studies and attempts to clarify apparently disparate results by placing them in a coherent theoretical framework.

The assumption that positive affect leads to a better performance in simple cognitive tasks has become well established. We address the question whether positive and negative emotions differentially influence performance in complex problem-solving in the same way. Emotions were induced by positive or negative feedback in 74 participants who had to manage a computer-simulated complex problem-solving scenario. Results show that overall scenario performance is not affected, but positive and negative emotions elicit distinguishable problem-solving strategies: Participants with negative emotions are more (...) focused on the seeking and use of information. We discuss methodological requirements for investigating emotion influences in complex and dynamic cognitive tasks. (shrink)