Understanding how people comprehend visual narratives (including picture stories, comics, and film) requires the combination of traditionally separate theories that span the initial sensory and perceptual processing of complex visual scenes, the perception of events over time, and comprehension of narratives. Existing piecemeal approaches fail to capture the interplay between these levels of processing. Here, we propose the Scene Perception & Event Comprehension Theory (SPECT), as applied to visual narratives, which distinguishes between front-end and back-end cognitive processes. Front-end processes occur (...) during single eye fixations and are comprised of attentional selection and information extraction. Back-end processes occur across multiple fixations and support the construction of event models, which reflect understanding of what is happening now in a narrative (stored in working memory) and over the course of the entire narrative (stored in long-term episodic memory). We describe relationships between front- and back-end processes, and medium-specific differences that likely produce variation in front-end and back-end processes across media (e.g., picture stories vs. film). We describe several novel research questions derived from SPECT that we have explored. By addressing these questions, we provide greater insight into how attention, information extraction, and event model processes are dynamically coordinated to perceive and understand complex naturalistic visual events in narratives and the real world. (shrink)

Background: Neurotypical young adults show task-based modulation and stability of their eye movements across tasks. This study aimed to determine whether persons with aphasia (PWA) modulate their eye movements and show stability across tasks similarly to control participants. Methods: Forty-eight PWA and age-matched control participants completed four eye-tracking tasks: scene search, scene memorization, text-reading, and pseudo-reading. Results: Main effects of task emerged for mean fixation duration, saccade amplitude, and standard deviations of each, demonstrating task-based modulation of eye movements. Group by (...) task interactions indicated that PWA produced shorter fixations relative to controls. This effect was most pronounced for scene memorization and for individuals who recently suffered a stroke. PWA produced longer fixations, shorter saccades, and less variable eye movements in reading tasks compared to controls. Three-way interactions of group, aphasia subtype, and task also emerged. Text-reading and scene memorization were particularly effective at distinguishing aphasia subtype. Persons with anomic aphasia showed a reduction in reading saccade amplitudes relative to their respective control group and other PWA. Persons with conduction/Wernicke’s aphasia produced shorter scene memorization fixations relative to controls or PWA of other subtypes, suggesting a memorization specific effect. Positive correlations across most tasks emerged for fixation duration and did not significantly differ between controls and PWA. Conclusions: PWA generally produced shorter fixations and smaller saccades relative to controls particularly in scene memorization and text-reading respectively. The effect was most pronounced recently after a stroke. Selectively in reading tasks, PWA produced longer fixations and shorter saccades relative to controls, consistent with reading difficulty. PWA showed task-based modulation of eye movements, though the pattern of results was somewhat abnormal relative to controls. All subtypes of PWA also demonstrated task-based modulation of eye movements. However, persons with anomic aphasia showed reduced modulation of saccade amplitude and smaller reading saccades, possibly to improve reading comprehension. Controls and PWA generally produced stabile fixation durations across tasks and did not differ in their relationship across tasks. Overall, these results suggest there is potential to differentiate among PWA with varying subtypes and from controls using eye movement measures of task-based modulation, especially reading and scene memorization tasks. (shrink)

The origin of colour categories and their relationship to colour perception have been the prime example for testing the influence of language on perception and thought and more generally for investigating the biological, ecological and cultural determination of human cognition. These themes are central to a broad range of disciplines, including vision research, neuroscience, cognitive psychology, developmental science, cultural anthropology, linguistics, computer science, and philosophy. Unfortunately, though, it has been tacitly taken for granted that the conceptual assumptions and methodological practices (...) from the dawn of empirical research on colour categorisation are the gold standard for current colour category research. Here, we show that these assumptions and practices are obsolete and flawed and have led to four fundamental and widespread misconceptions about colour categorisation: 1.) that colour perception is inherently categorical; 2.) that English Basic Colour Terms correspond to universal categories that are the end point of a fixed evolutionary sequence; 3.) that the prototypes of English basic colour terms are perceptually salient and qualify as focal colours; and 4.) that colour category research essentially revolves around the universalism-realism debate. State-of-the-art research on colour categorisation provides new, more sophisticated approaches and allows for rectifying those four statements. At the same time, some of the questions underlying those statements are not convincingly answered yet and constitute major challenges to future research. The critical considerations on colour categorisation may be transferred to research on other kinds of perceptual categorisation to inspire new, more general research questions. (shrink)

Decision making in any brain is imperfect and costly in terms of time and energy. Operating under such constraints, an organism could be in a position to improve performance if an opportunity arose to exploit informative patterns in the environment being searched. Such an improvement of performance could entail both faster and more accurate (i.e., reward-maximizing) decisions. The present study investigated the extent to which human participants could learn to take advantage of immediate patterns in the spatial arrangement of serially (...) presented foods such that a region of space would consistently be associated with greater subjective value. Eye movements leading up to choices demonstrated rapidly induced biases in the selective allocation of visual fixation and attention that were accompanied by both faster and more accurate choices of desired goods as implicit learning occurred. However, for the control condition with its spatially balanced reward environment, these subjects exhibited preexisting lateralized biases for eye and hand movements (i.e., leftward and rightward, respectively) that could act in opposition not only to each other but also to the orienting biases elicited by the experimental manipulation, producing an asymmetry between the left and right hemifields with respect to performance. Potentially owing at least in part to learned cultural conventions (e.g., reading from left to right), the findings herein particularly revealed an intrinsic leftward bias underlying initial saccades in the midst of more immediate feedback-directed processes for which spatial biases can be learned flexibly to optimize oculomotor and manual control in value-based decision making. The present study thus replicates general findings of learned attentional biases in a novel context with inherently rewarding stimuli and goes on to further elucidate the interactions between endogenous and exogenous biases. (shrink)

Forensic evidence often involves an evaluation of whether two impressions were made by the same source, such as whether a fingerprint from a crime scene has detail in agreement with an impression taken from a suspect. Human experts currently outperform computer-based comparison systems, but the strength of the evidence exemplified by the observed detail in agreement must be evaluated against the possibility that some other individual may have created the crime scene impression. Therefore, the strongest evidence comes from features in (...) agreement that are also not shared with other impressions from other individuals. We characterize the nature of human expertise by applying two extant metrics to the images used in a fingerprint recognition task and use eye gaze data from experts to both tune and validate the models. The Attention via Information Maximization model quantifies the rarity of regions in the fingerprints to determine diagnosticity for purposes of excluding alternative sources. The CoVar model captures relationships between low-level features, mimicking properties of the early visual system. Both models produced classification and generalization performance in the 75%–80% range when classifying where experts tend to look. A validation study using regions identified by the AIM model as diagnostic demonstrates that human experts perform better when given regions of high diagnosticity. The computational nature of the metrics may help guard against wrongful convictions, as well as provide a quantitative measure of the strength of evidence in casework. (shrink)

Drawing on the empirical premise that attention makes objects look more intense, Ned Block has argued for mental paint, a phenomenal residue that cannot be reduced to what is perceived or represented. If sound, Block's argument would undermine direct realism and representationism, two widely held views about the nature of conscious perception. We argue that Block's argument fails because the empirical premise it is based upon is false. Attending to an object alters its salience, but not its perceived intensity. We (...) also argue that salience should be equated with mental primer, a close cousin of mental paint that reintroduces difficulties for direct realism and representationism. The upshot is that direct realism and representationism are still in trouble, but not for the reason that Block thinks. (shrink)

To make behavioral choices that are in line with our goals and our moral beliefs, we need to gather and consider information about our current situation. Most information present in our environment is not relevant to the choices we need or would want to make and thus could interfere with our ability to behave in ways that reflect our underlying values. Certain sources of information could even lead us to make choices we later regret, and thus it would be beneficial (...) to be able to ignore that information. Our ability to exert successful self-governance depends on our ability to attend to sources of information that we deem important to our decision-making processes. We generally assume that, at any moment, we have the ability to choose what we pay attention to. However, recent research indicates that what we pay attention to is influenced by our prior experiences, including reward history and past successes and failures, even when we are not aware of this history. Even momentary distractions can cause us to miss or discount information that should have a greater influence on our decisions given our values. Such biases in attention thus raise questions about the degree to which the choices that we make may be poorly informed and not truly reflect our ability to otherwise exert self-governance. (shrink)

Following the pioneering studies of the receptive field (RF), the concept gained further significance for visual perception by the discovery of input effects from beyond the classical RF. These studies demonstrated that neuronal responses could be modulated by stimuli outside their RFs, consistent with the perception of induced brightness, color, orientation, and motion. Lesion scotomata are similarly modulated perceptually from the surround by RFs that have migrated from the interior to the outer edge of the scotoma and in this way (...) provide filling-in of the void. Large RFs are advantageous to this task. In higher visual areas, such as the middle temporal and inferotemporal lobe, RFs increase in size and lose most of their retinotopic organization while encoding increasingly complex features. Whereas lowerlevel RFs mediate perceptual filling-in, contour integration, and figure–ground segregation, RFs at higher levels serve the perception of grouping by common fate, biological motion, and other biologically relevant stimuli, such as faces. Studies in alert monkeys while freely viewing natural scenes showed that classical and nonclassical RFs cooperate in forming representations of the visual world. Today, our understanding of the mechanisms underlying the RF is undergoing a quantum leap. What had started out as a hierarchical feedforward concept for simple stimuli, such as spots, lines, and bars, now refers to mechanisms involving ascending, descending, and lateral signal flow. By extension of the bottom-up paradigm, RFs are nowadays understood as adaptive processors, enabling the predictive coding of complex scenes. Top-down effects guiding attention and tuned to task-relevant information complement the bottom-up analysis. (shrink)

Prominent models of attentional control assert a dichotomy between top-down and bottom-up control, with the former determined by current selection goals and the latter determined by physical salience. This theoretical dichotomy, however, fails to explain a growing number of cases in which neither current goals nor physical salience can account for strong selection biases. For example, equally salient stimuli associated with reward can capture attention, even when this contradicts current selection goals. Thus, although 'top-down' sources of bias are sometimes defined (...) as those that are not due to physical salience, this conception conflates distinct - and sometimes contradictory - sources of selection bias. We describe an alternative framework, in which past selection history is integrated with current goals and physical salience to shape an integrated priority map. (shrink)

This study investigates to what extent the amount of variation in a visual scene causes speakers to mention the attribute color in their definite target descriptions, focusing on scenes in which this attribute is not needed for identification of the target. The results of our three experiments show that speakers are more likely to redundantly include a color attribute when the scene variation is high as compared with when this variation is low (even if this leads to overspecified descriptions). We (...) argue that these findings are problematic for existing algorithms that aim to automatically generate psychologically realistic target descriptions, such as the Incremental Algorithm, as these algorithms make use of a fixed preference order per domain and do not take visual scene variation into account. (shrink)

The central idea behind this paper is that presuppositions of soft triggers arise from the way our attention structures the informational content of a sentence. Some aspects of the information conveyed are such that we pay attention to them by default, even in the absence of contextual information. On the other hand, contextual cues or conversational goals can divert attention to types of information that we would not pay attention to by default. Either way, whatever we do not pay attention (...) to, be it by default, or in context, is what ends up presupposed by soft triggers. This paper attempts to predict what information in the sentence is likely to end up being the main point (i.e. what we pay attention to) and what information is independent from this, and therefore likely presupposed. It is proposed that this can be calculated by making reference to event times. The notion of aboutness used to calculate independence is based on that of Demolombe and Fariñas del Cerro (In: Holdobler S (ed) Intellectics and computational logic: papers in honor of Wolfgang Bibel, 2000). (shrink)

A general definition of consciousness is: ‘consciousness is a mental aspect of a system or a process, which is a conscious experience, a conscious function, or both depending on the context’, where the term context refers to metaphysical views, constraints, specific aims, and so on. One of the aspects of visual consciousness is the visual subjective experience (SE) or the first person experience that occurs/emerges in the visual neural-network of thalamocortical system (which includes dorsal and ventral visual pathways and frontal (...) cortex) during dynamic interactions among widely distributed neuronal groups. In this article, however, consciousness and SE are interchangeably used unless noted. Consciousness can be either access (reportable) or phenomenal consciousness. For access consciousness, the interactions are between feed forward stimulus dependent signals and fronto-parietal feedback attentional signals. The necessary ingredients for access (reportable) consciousness are (i) wakefulness, (ii) reentrant interactions among neural populations, (iii) fronto-parietal and thalamic-reticular-nucleus attentional signals that modulate consciousness, (iv) working memory that retains information for consciousness, (v) stimulus at or above threshold level, and (vi) neural-network proto-experiences (PEs) that are superposed SEs embedded in a neural-network. Attention and the ability to report are not necessary for phenomenal consciousness. The neural source for the arousal system is the ascending reticular activating system in the brain stem, which brings the thalamocortical neural networks to wakeful state as a baseline for consciousness to occur. Reentrant interactions among neural populations bind stimulus attributes (such as location and features) and entail consciousness. Attention could be the results of reentry and competitive interactions, and modulates the stimulus related feed forward signal and consciousness. The ‘sources’ of attention may be thalamic reticular nucleus for bottom-up or frontal cortex for top-down direction. The ‘target’ of visual attention is ‘V4/V8/VO’ for Red-Green (R-G) channel. The neural correlates of the psychophysical entity R-G channel appear to be ‘V4/V8/VO’-neural-network (retina → LGN ↔ V1 ↔ V2 ↔ ‘V4/V8/VO’, and areas for attention, memory, and wakefulness). The psychophysical Red-Green Channel, its neurophysiological correlates V4/V8/VO-neural-network, and related experience (such as redness) are integrated. The dual-aspect-dual-mode PE-SE framework (Vimal, J Integr Neurosci 7:49–73, 2008) was used to address the explanatory gap problem of materialism. Neural-network and related SEs are the results of the co-evolution and co-development of the material aspect (mass, charge, spin, force, quanta, and space-time) and the mental aspect of fundamental particles (strings or elementary particles (fermions and bosons)). Their mental aspects are considered as the carriers of superposed multiple possible experiences (SEs/PEs) in unexpressed form. These possibilities are actualized when neural-networks are formed via neural Darwinism, and a specific SE is selected by a matching process when the necessary ingredients of consciousness/SE are satisfied. A simple experimental design is suggested to address the necessary and sufficient attributes of consciousness. (shrink)

This article sketches an idealized strategy for the identification of neural correlates of consciousness. The proposed strategy is based on a state space approach originating from the analysis of dynamical systems. The article then focuses on one constituent of consciousness, phenomenal awareness. Several rudimentary requirements for the identification of neural correlates of phenomenal awareness are suggested. These requirements are related to empirical data on selective attention, on completely intrinsic selection and on globally unconscious states. As an example, neuroscientific findings on (...) synchronized γ activity are categorized according to these requirements. (shrink)

When producing a description of a target referent in a visual context, speakers need to choose a set of properties that distinguish it from its distractors. Computational models of language production/generation usually model this as a search process and predict that the time taken will increase both with the number of distractors in a scene and with the number of properties required to distinguish the target. These predictions are reminiscent of classic findings in visual search; however, unlike models of reference (...) production, visual search models also predict that search can become very efficient under certain conditions, something that reference production models do not consider. This paper investigates the predictions of these models empirically. In two experiments, we show that the time taken to plan a referring expression—as reflected by speech onset latencies—is influenced by distractor set size and by the number of properties required, but this crucially depends on the discriminability of the properties under consideration. We discuss the implications for current models of reference production and recent work on the role of salience in visual search. (shrink)