How do individuals guide their activities towards some goal? Harry Frankfurt once identified the task of explaining guidance as the central problem in action theory. An explanation has proved to be elusive, however. In this paper, I show how we can marshal empirical research to make explanatory progress. I contend that human agents have a primitive capacity to guide visual attention, and that this capacity is actually constituted by a sub-individual psychological control-system: the executive system. I thus illustrate how we (...) can explain exercises of individual-level guidance by appeal to its sub-individual constitution. This opens up a new avenue for explicating agency. (shrink)

In paradigm exercises of agency, individuals guide their activities toward some goal. A central challenge for action theory is to explain how individuals guide. This challenge is an instance of the more general problem of how to accommodate individuals and their actions in the natural world, as explained by natural science. Two dominant traditions–primitivism and the causal theory–fail to address the challenge in a satisfying way. Causal theorists appeal to causation by an intention, through a feedback mechanism, in explaining guidance. (...) Primitivists postulate primitive agential capacities in their explanations. The latter neglect to explain how primitive capacities integrate with findings from natural science. The former do not explain why some feedback mechanism’s activity amounts to the agent’s guidance. In this paper I argue that both traditions should acknowledge a capacity to guide, as actually constituted by the executive system. I argue that appeals to this empirically discovered psychological system explain how individuals guide in a way that integrates with explanations from cognitive science. Individuals’ capacity to guide is embedded in the natural world through the activity of its constituent (mechanistic) components. (shrink)

The ability to group perceptual objects into functionally relevant categories is vital to our comprehension of the world. Such categorisation aids in how we search for objects in familiar scenes and how we identify an object and its likely uses despite never having seen that specific object before. The systems that mediate this process are only now coming to be understood through considerable research efforts combining neurological, psychological and behavioural studies. What is much less well understood are the differences between (...) the categories, how they are formed and how they are used by experts and non-experts in a complex task that can take decades to master. In a quite different direction to previous studies, this work infers the different categorical structures that might be used by amateurs and professionals in the oriental game of Go. This is achieved by using a newly developed combination of artificial neural networks (Self-organising Maps) and perceptual inference to show that categories of strategic scenes can be learned while playing games using a model of ‘conditional perceptual learning’. Applying this technique to two databases of games, one of amateurs and one of professionals, shows that a structural hierarchy of scene information develops that can be readily incorporated into traditional psychological models of decisions and readily implemented in computational systems. The results are discussed in terms of the heuristics and biases literature, emphasising where the significant similarities and differences lie between this work and previous work. (shrink)

We provide the first demonstration of an artificial neural network encoding the perceptual templates that form an important component of the high level strategic understanding developed by experts. Experts have a highly refined sense of knowing where to look, what information is important and what information to ignore. The conclusions these experts reach are of a higher quality and typically made in a shorter amount of time than those of non-experts. Understanding the manifestation of such abilities in terms of both (...) the psychology of expert performance and the underlying neural mechanisms constitutes one of the most challenging problems in the cognitive sciences. Using perceptual templates we show how the amount of contextual information can change significantly even within a given task, the relationship between local and non-local contexts and finally why there is very little correlation between measures of intelligence and level of expertise in many of the most complex tasks performed by humans. (shrink)

Both visual experience and conscious thought represent external objects, but in visual experience these objects seem present before the mind and available for direct access in a way that they don’t in conscious thought. In this paper, I introduce a couple of challenges that this “Scene-Immediacy” of visual experience raises for traditional versions of Representationalism. I then identify a resource to which Representationalists can appeal in addressing these challenges: the low-detail fringe of visual experience. I argue that low-detail contents within (...) visual experience provide the mind with a rich access to additional high-detail information, an access that is not found in conscious thought. This access, in turn, speaks to the challenges raised by the Scene-Immediacy of visual experience. Robert Schroer is an Assistant Professor at the University of Minnesota at Duluth. (shrink)

In this paper, I explore and defend the idea that we have epistemic responsibilities with respect to our visual searches, responsibilities that are far more fine-grained and interesting than the trivial responsibilities to keep our eyes open and “look hard”. In order to have such responsibilities, we must be able to exert fine-grained and interesting forms of control over our visual searches. I present both an intuitive case and an empirical case for thinking that we do, in fact, have such (...) forms of control over our visual searches. I then show how these forms of control can be used to aim the visual beliefs that result from our searches towards various epistemic goals. (shrink)

When considering external assistive systems for people with motor impairments, gaze has been shown to be a powerful tool as it is anticipatory to motor actions and is promising for understanding intentions of an individual even before the action. Up until now, the vast majority of studies investigating the coordinated eye and hand movement in a grasping task focused on single objects manipulation without placing them in a meaningful scene. Very little is known about the impact of the scene context (...) on how we manipulate objects in an interactive task. In the present study, it was investigated how the scene context affects human object manipulation in a pick-and-place task in a realistic scenario implemented in VR. During the experiment, participants were instructed to find the target object in a room, pick it up, and transport it to a predefined final location. Thereafter, the impact of the scene context on different stages of the task was examined using head and hand movement, as well as eye tracking. As the main result, the scene context had a significant effect on the search and transport phases, but not on the reach phase of the task. The present work provides insights into the development of potential supporting intention predicting systems, revealing the dynamics of the pick-and-place task behavior once it is realized in a realistic context-rich scenario. (shrink)

The study of eye movements is a common way to non-invasively understand and analyze human behavior. However, eye-tracking techniques are very hard to scale, and require expensive equipment and extensive expertise. In the context of web browsing, these issues could be overcome by studying the link between the eye and the computer mouse. Here, we propose new analysis methods, and a more advanced characterization of this link. To this end, we recorded the eye, mouse, and scroll movements of 151 participants (...) exploring 18 dynamic web pages while performing free viewing and visual search tasks for 20 s. The data revealed significant differences of eye, mouse, and scroll parameters over time which stabilize at the end of exploration. This suggests the existence of a task-independent relationship between eye, mouse, and scroll parameters, which are characterized by two distinct patterns: one common pattern for movement parameters and a second for dwelling/fixation parameters. Within these patterns, mouse and eye movements remained consistent with each other, while the scrolling behaved the opposite way. (shrink)

Skilled action typically requires that individuals guide their activities toward some goal. In skilled action, individuals do so excellently. We do not understand well what this capacity to guide consists in. In this paper I provide a case study of how individuals shift visual attention. Their capacity to guide visual attention toward some goal (partly) consists in an empirically discovered sub-system – the executive system. I argue that we can explain how individuals guide by appealing to the operation of this (...) sub-system. Understanding skill and skilled action thus requires appreciating the role of the executive system. (shrink)

Background: Neurotypical young adults show task-based modulation and stability of their eye movements across tasks. This study aimed to determine whether persons with aphasia (PWA) modulate their eye movements and show stability across tasks similarly to control participants. Methods: Forty-eight PWA and age-matched control participants completed four eye-tracking tasks: scene search, scene memorization, text-reading, and pseudo-reading. Results: Main effects of task emerged for mean fixation duration, saccade amplitude, and standard deviations of each, demonstrating task-based modulation of eye movements. Group by (...) task interactions indicated that PWA produced shorter fixations relative to controls. This effect was most pronounced for scene memorization and for individuals who recently suffered a stroke. PWA produced longer fixations, shorter saccades, and less variable eye movements in reading tasks compared to controls. Three-way interactions of group, aphasia subtype, and task also emerged. Text-reading and scene memorization were particularly effective at distinguishing aphasia subtype. Persons with anomic aphasia showed a reduction in reading saccade amplitudes relative to their respective control group and other PWA. Persons with conduction/Wernicke’s aphasia produced shorter scene memorization fixations relative to controls or PWA of other subtypes, suggesting a memorization specific effect. Positive correlations across most tasks emerged for fixation duration and did not significantly differ between controls and PWA. Conclusions: PWA generally produced shorter fixations and smaller saccades relative to controls particularly in scene memorization and text-reading respectively. The effect was most pronounced recently after a stroke. Selectively in reading tasks, PWA produced longer fixations and shorter saccades relative to controls, consistent with reading difficulty. PWA showed task-based modulation of eye movements, though the pattern of results was somewhat abnormal relative to controls. All subtypes of PWA also demonstrated task-based modulation of eye movements. However, persons with anomic aphasia showed reduced modulation of saccade amplitude and smaller reading saccades, possibly to improve reading comprehension. Controls and PWA generally produced stabile fixation durations across tasks and did not differ in their relationship across tasks. Overall, these results suggest there is potential to differentiate among PWA with varying subtypes and from controls using eye movement measures of task-based modulation, especially reading and scene memorization tasks. (shrink)

When producing a description of a target referent in a visual context, speakers need to choose a set of properties that distinguish it from its distractors. Computational models of language production/generation usually model this as a search process and predict that the time taken will increase both with the number of distractors in a scene and with the number of properties required to distinguish the target. These predictions are reminiscent of classic findings in visual search; however, unlike models of reference (...) production, visual search models also predict that search can become very efficient under certain conditions, something that reference production models do not consider. This paper investigates the predictions of these models empirically. In two experiments, we show that the time taken to plan a referring expression—as reflected by speech onset latencies—is influenced by distractor set size and by the number of properties required, but this crucially depends on the discriminability of the properties under consideration. We discuss the implications for current models of reference production and recent work on the role of salience in visual search. (shrink)

The human sentence processor is able to make rapid predictions about upcoming linguistic input. For example, upon hearing the verb eat, anticipatory eye-movements are launched toward edible objects in a visual scene. However, the cognitive mechanisms that underlie anticipation remain to be elucidated in ecologically valid contexts. Previous research has, in fact, mainly used clip-art scenes and object arrays, raising the possibility that anticipatory eye-movements are limited to displays containing a small number of objects in a visually impoverished context. In (...) Experiment 1, we confirm that anticipation effects occur in real-world scenes and investigate the mechanisms that underlie such anticipation. In particular, we demonstrate that real-world scenes provide contextual information that anticipation can draw on: When the target object is not present in the scene, participants infer and fixate regions that are contextually appropriate. Experiment 2 investigates whether such contextual inference requires the co-presence of the scene, or whether memory representations can be utilized instead. The same real-world scenes as in Experiment 1 are presented to participants, but the scene disappears before the sentence is heard. We find that anticipation occurs even when the screen is blank, including when contextual inference is required. We conclude that anticipatory language processing is able to draw upon global scene representations to make contextual inferences. These findings are compatible with theories assuming contextual guidance, but posit a challenge for theories assuming object-based visual indices. (shrink)

Eye movements were recorded during the display of two images of a real-world scene that were inspected to determine whether they were the same or not . In the displays where the pictures were different, one object had been changed, and this object was sometimes taken from another scene and was incongruent with the gist. The experiment established that incongruous objects attract eye fixations earlier than the congruous counterparts, but that this effect is not apparent until the picture has been (...) displayed for several seconds. By controlling the visual saliency of the objects the experiment eliminates the possibility that the incongruency effect is dependent upon the conspicuity of the changed objects. A model of scene perception is suggested whereby attention is unnecessary for the partial recognition of an object that delivers sufficient information about its visual characteristics for the viewer to know that the object is improbable in that particular scene, and in which full identification requires foveal inspection. (shrink)

Visual perception relies on stored information and environmental associations to arrive at a determinate representation of the world. This opens up the disturbing possibility that our visual experiences could themselves be subject to a kind of racial bias, simply in virtue of accurately encoding previously encountered environmental regularities. This possibility raises the following question: what, if anything, is wrong with beliefs grounded upon these prejudicial experiences? They are consistent with a range of epistemic norms, including evidentialist and reliabilist standards for (...) justification. I argue that we will struggle to locate a flaw with these sorts of perceptual beliefs so long as we focus our analysis at the level of the individual and her response to information. We should instead broaden our analysis to include the social structure within which the individual is located. Doing so lets us identify a problem with the way in which unjust social structures in particular “gerrymander” the regularities an individual is exposed to, and by extension the priors their visual system draws on. I argue that in this way, social structures can cap perceptual skill. (shrink)

Most cognitive psychology experiments evaluate models of human cognition using a relatively small, well-controlled set of stimuli. This approach stands in contrast to current work in neuroscience, perception, and computer vision, which have begun to focus on using large databases of natural images. We argue that natural images provide a powerful tool for characterizing the statistical environment in which people operate, for better evaluating psychological theories, and for bringing the insights of cognitive science closer to real applications. We discuss how (...) some of the challenges of using natural images as stimuli in experiments can be addressed through increased sample sizes, using representations from computer vision, and developing new experimental methods. Finally, we illustrate these points by summarizing recent work using large image databases to explore questions about human cognition in four different domains: modeling subjective randomness, defining a quantitative measure of representativeness, identifying prior knowledge used in word learning, and determining the structure of natural categories. (shrink)

Most everyday tasks involve multiple modalities, which raises the question of how the processing of these modalities is coordinated by the cognitive system. In this paper, we focus on the coordination of visual attention and linguistic processing during speaking. Previous research has shown that objects in a visual scene are fixated before they are mentioned, leading us to hypothesize that the scan pattern of a participant can be used to predict what he or she will say. We test this hypothesis (...) using a data set of cued scene descriptions of photo-realistic scenes. We demonstrate that similar scan patterns are correlated with similar sentences, within and between visual scenes; and that this correlation holds for three phases of the language production process (target identification, sentence planning, and speaking). We also present a simple algorithm that uses scan patterns to accurately predict associated sentences by utilizing similarity-based retrieval. (shrink)

Attention is strongly influenced by both external stimuli and internal goals. However, this useful dichotomy does not readily capture the ubiquitous and often automatic contribution of past experience stored in memory. We review recent evidence about how multiple memory systems control attention, consider how such interactions are manifested in the brain, and highlight how this framework for ‘memory-guided attention’ might help systematize previous findings and guide future research.

When referring to a target object in a visual scene, speakers are assumed to consider certain distractor objects to be more relevant than others. The current research predicts that the way in which speakers come to a set of relevant distractors depends on how they perceive the distance between the objects in the scene. It reports on the results of two language production experiments, in which participants referred to target objects in photo-realistic visual scenes. Experiment 1 manipulated three factors that (...) were expected to affect perceived distractor distance: two manipulations of perceptual grouping (region of space and type similarity), and one of presentation mode (2D vs. 3D). In line with most previous research on visually-grounded reference production, an offline measure of visual attention was taken here: the occurrence of overspecification with color. The results showed effects of region of space and type similarity on overspecification, suggesting that distractors that are perceived as being in the same group as the target are more often considered relevant distractors than distractors in a different group. Experiment 2 verified this suggestion with a direct measure of visual attention, eye tracking, and added a third manipulation of grouping: color similarity. For region of space in particular, the eye movements data indeed showed patterns in the expected direction: distractors within the same region as the target were fixated more often, and longer, than distractors in a different region. Color similarity was found to affect overspecification with color, but not gaze duration or the number of distractor fixations. Also the expected effects of presentation mode (2D vs. 3D) were not convincingly borne out by the data. Taken together, these results provide direct evidence for the close link between scene perception and language production, and indicate that perceptual grouping principles can guide speakers in determining the distractor set during reference production. (shrink)

Colour has been shown to facilitate the recognition of scene images, but only when these images contain natural scenes, for which colour is ‘diagnostic’. Here we investigate whether colour can also facilitate memory for scene images, and whether this would hold for natural scenes in particular. In the first experiment participants first studied a set of colour and greyscale natural and man-made scene images. Next, the same images were presented, randomly mixed with a different set. Participants were asked to indicate (...) whether they had seen the images during the study phase. Surprisingly, performance was better for greyscale than for coloured images, and this difference is due to the higher false alarm rate for both natural and man-made coloured scenes. We hypothesized that this increase in false alarm rate was due to a shift from scrutinizing details of the image to recognition of the gist of the image. A second experiment, utilizing images without a nameable gist, confirmed this hypothesis as participants now performed equally on greyscale and coloured images. In the final experiment we specifically targeted the more detail-based perception and recognition for greyscale images versus the more gist-based perception and recognition for coloured images with a change detection paradigm. The results show that changes to images are detected faster when image-pairs were presented in greyscale than in colour. This counterintuitive result held for both natural and man-made scenes and thus corroborates the shift from more detailed processing of images in greyscale to more gist-based processing of coloured images. (shrink)

In two experiments, we investigate to what extent various visual saliency cues in realistic visual scenes cause speakers to overspecify their definite object descriptions with a redundant color attribute. The results of the first experiment demonstrate that speakers are more likely to redundantly mention color when visual clutter is present in a scene as compared to when this is not the case. In the second experiment, we found that distractor type and distractor color affect redundant color use: Speakers are most (...) likely to overspecify if there is at least one distractor object present that has the same type, but a different color than the target referent. Reliable effects of distractor distance were not found. Taken together, our results suggest that certain visual saliency cues guide speakers in determining which objects in a visual scene are relevant distractors, and which not. We argue that this is problematic for algorithms that aim to generate human-like descriptions of objects, since these generally select properties that help to distinguish a target from all objects that are present in a scene. (shrink)