We review how leaky competing accumulators (LCAs) can be used to model decision making in two‐alternative, forced‐choice tasks, and we show how they reduce to drift diffusion (DD) processes in special cases. As continuum limits of the sequential probability ratio test, DD processes are optimal in producing decisions of specified accuracy in the shortest possible time. Furthermore, the DD model can be used to derive a speed–accuracy trade‐off that optimizes reward rate for a restricted class of two alternative forced‐choice decision (...) tasks. We review findings that compare human performance with this benchmark, and we reveal both approximations to and deviations from optimality. We then discuss three potential sources of deviations from optimality at the psychological level—avoidance of errors, poor time estimation, and minimization of the cost of control—and review recent theoretical and empirical findings that address these possibilities. We also discuss the role of cognitive control in changing environments and in modulating exploitation and exploration. Finally, we consider physiological factors in which nonlinear dynamics may also contribute to deviations from optimality. (shrink)

Why does performing certain tasks cause the aversive experience of mental effort and concomitant deterioration in task performance? One explanation posits a physical resource that is depleted over time. We propose an alternative explanation that centers on mental representations of the costs and benefits associated with task performance. Specifically, certain computational mechanisms, especially those associated with executive function, can be deployed for only a limited number of simultaneous tasks at any given moment. Consequently, the deployment of these computational mechanisms carries (...) an opportunity cost – that is, the next-best use to which these systems might be put. We argue that the phenomenology of effort can be understood as the felt output of these cost/benefit computations. In turn, the subjective experience of effort motivates reduced deployment of these computational mechanisms in the service of the present task. These opportunity cost representations, then, together with other cost/benefit calculations, determine effort expended and, everything else equal, result in performance reductions. In making our case for this position, we review alternative explanations for both the phenomenology of effort associated with these tasks and for performance reductions over time. Likewise, we review the broad range of relevant empirical results from across sub-disciplines, especially psychology and neuroscience. We hope that our proposal will help to build links among the diverse fields that have been addressing similar questions from different perspectives, and we emphasize ways in which alternative models might be empirically distinguished. (shrink)

One way to learn about the world is by asking questions. We investigate how younger children (7- to 8-year-olds), older children (9- to 11-year-olds), and young adults (17- to 18-year-olds) ask questions to identify the cause of an event. We find a developmental shift in children’s reliance on hypothesis-scanning questions (which test hypotheses directly) versus constraint-seeking questions (which reduce the space of hypotheses), but also that all age groups ask more constraint-seeking questions when hypothesis-scanning questions are least likely to pay (...) off: When the solution is one among equally likely alternatives (Study 1) or when the problem is difficult (Studies 1 and 2). These findings are the first to demonstrate that even young children dynamically adapt their strategies for inquiry to increase the efficiency of information search. (shrink)

Decision making in noisy and changing environments requires a fine balance between exploiting knowledge about good courses of action and exploring the environment in order to improve upon this knowledge. We present an experiment on a restless bandit task in which participants made repeated choices between options for which the average rewards changed over time. Comparing a number of computational models of participants’ behavior in this task, we find evidence that a substantial number of them balanced exploration and exploitation by (...) considering the probability that an option offers the maximum reward out of all the available options. (shrink)

When we learn how to throw darts we adjust how we throw based on where the darts stick. Much of skill learning is computationally similar in that we learn using feedback obtained after the completion of individual actions. We can formalize such tasks as a search problem; among the set of all possible actions, find the action that leads to the highest reward. In such cases our actions have two objectives: we want to best utilize what we already know (exploitation), (...) but we also want to learn to be more successful in the future (exploration). Here we tested how participants learn movement trajectories where feedback is provided as a monetary reward that depends on the chosen trajectory. We mathematically derived the optimal search policy for our experiment using decision theory. The search behavior of participants is well predicted by an ideal searcher model that optimally combines exploration and exploitation. (shrink)

Numerous studies have documented individual differences in exploratory tendencies and other phenomena related to search, and these differences have been linked to fitness. Here, I discuss the origins of these differences, focusing on how experience shapes animal search and exploration. The origin of individual differences will also depend upon the alternatives to exploration that are available. Given that search and exploration frequently carry significant costs, we might expect individuals to utilize cues indicating the potential net payoffs of exploration versus the (...) exploitation of known acts. Informative cues could arise from both recent and early-life experiences, from both the social and physical environment. Open questions are the extent to which an individual's exploratory tendencies are fixed throughout life versus being flexibly adjusted according to prevailing conditions and the actions of other individuals, and the extent to which individual differences in exploration extend across domains and are independent of other processes. (shrink)

Mata and von Helversen's integrative review of adult age differences in search performance makes a good case that cognitive control may impact certain aspects of self-regulation of search. However, information foraging as a framework also offers an avenue to consider how adults of different ages adapt to age-related changes in cognition, such as in cognitive control.

In reinforcement learning, a decision maker searching for the most rewarding option is often faced with the question: What is the value of an option that has never been tried before? One way to frame this question is as an inductive problem: How can I generalize my previous experience with one set of options to a novel option? We show how hierarchical Bayesian inference can be used to solve this problem, and we describe an equivalence between the Bayesian model and (...) temporal difference learning algorithms that have been proposed as models of RL in humans and animals. According to our view, the search for the best option is guided by abstract knowledge about the relationships between different options in an environment, resulting in greater search efficiency compared to traditional RL algorithms previously applied to human cognition. In two behavioral experiments, we test several predictions of our model, providing evidence that humans learn and exploit structured inductive knowledge to make predictions about novel options. In light of this model, we suggest a new interpretation of dopaminergic responses to novelty. (shrink)

The effects of striatal dopamine (DA) on behavior have been widely investigated over the past decades, with “phasic” burst firings considered as the key expression of a reward prediction error responsible for reinforcement learning. Less well studied is “tonic” DA, where putative functions include the idea that it is a regulator of vigor, incentive salience, disposition to exert an effort and a modulator of approach strategies. We present a model combining tonic and phasic DA to show how different outflows triggered (...) by either intrinsically or extrinsically motivating stimuli dynamically affect the basal ganglia by impacting on a selection process this system performs on its cortical input. The model, which has been tested on the simulated humanoid robot iCub interacting with a mechatronic board, shows the putative functions ascribed to DA emerging from the combination of a standard computational mechanism coupled to a differential sensitivity to the presence of DA across the striatum. (shrink)

Search is a prerequisite for successful performance in a broad range of tasks ranging from making decisions between consumer goods to memory retrieval. How does aging impact search processes in such disparate situations? Aging is associated with structural and neuromodulatory brain changes that underlie cognitive control processes, which in turn have been proposed as a domain-general mechanism controlling search in external environments as well as memory. We review the aging literature to evaluate the cognitive control hypothesis that suggests that age-related (...) change in cognitive control underlies age differences in both external and internal search. We also consider the limits of the cognitive control hypothesis and propose additional mechanisms such as changes in strategy use and affect that may be necessary to understand how aging affects search. (shrink)