Visual narratives like comics often are used as materials in clinical testing under a belief that they are transparent materials for individuals who may struggle with language, such as those with autism spectrum disorder (ASD), developmental language disorder (DLD), or aphasia. This review shows that this “Visual Ease Assumption” is largely unsupported, warranting reconsideration of the ways visual narratives are used with clinical populations.

Understanding how people comprehend visual narratives (including picture stories, comics, and film) requires the combination of traditionally separate theories that span the initial sensory and perceptual processing of complex visual scenes, the perception of events over time, and comprehension of narratives. Existing piecemeal approaches fail to capture the interplay between these levels of processing. Here, we propose the Scene Perception & Event Comprehension Theory (SPECT), as applied to visual narratives, which distinguishes between front-end and back-end cognitive processes. Front-end processes occur (...) during single eye fixations and are comprised of attentional selection and information extraction. Back-end processes occur across multiple fixations and support the construction of event models, which reflect understanding of what is happening now in a narrative (stored in working memory) and over the course of the entire narrative (stored in long-term episodic memory). We describe relationships between front- and back-end processes, and medium-specific differences that likely produce variation in front-end and back-end processes across media (e.g., picture stories vs. film). We describe several novel research questions derived from SPECT that we have explored. By addressing these questions, we provide greater insight into how attention, information extraction, and event model processes are dynamically coordinated to perceive and understand complex naturalistic visual events in narratives and the real world. (shrink)

In spite of their striking differences with real-life perception, films are perceived and understood without effort. Cognitive film theory attributes this to the system of continuity editing, a system of editing guidelines outlining the effect of different cuts and edits on spectators. A major principle in this framework is the 180° rule, a rule recommendation that, to avoid spectators’ attention to the editing, two edited shots of the same event or action should not be filmed from angles differing in a (...) way that expectations of spatial continuity are strongly violated. In the present study, we used high-density EEG to explore the neural underpinnings of this rule. In particular, our analysis shows that cuts and edits in general elicit early ERP component indicating the registration of syntactic violations as known from language, music, and action processing. However, continuity edits and cuts-across the line differ from each other regarding later components likely to be indicating the differences in spatial remapping as well as in the degree of conscious awareness of one's own perception. Interestingly, a time–frequency analysis of the occipital alpha rhythm did not support the hypothesis that such differences in processing routes are mainly linked to visual attention. On the contrary, our study found specific modulations of the central mu rhythm ERD as an indicator of sensorimotor activity, suggesting that sensorimotor networks might play an important role. We think that these findings shed new light on current discussions about the role of attention and embodied perception in film perception and should be considered when explaining spectators’ different experience of different kinds of cuts. (shrink)

We argue that the most ambitious version of the mental files theory of singular thought, according to which mental files are a wide-ranging psychological natural kind underlying all and only singular thinking, is unsupported by the available psychological data. Nevertheless, critical examination of the theory from a psychological perspective opens up promising avenues for research, especially concerning the relationship between our perceptual capacity to individuate and track basic individuals, and our higher level capacities for singular thought.

Narratives are an integral part of human expression. In the graphic form, they range from cave paintings to Egyptian hieroglyphics, from the Bayeux Tapestry to modern day comic books (Kunzle, 1973; McCloud, 1993). Yet not much research has addressed the structure and comprehension of narrative images, for example, how do people create meaning out of sequential images? This piece helps fill the gap by presenting a theory of Narrative Grammar. We describe the basic narrative categories and their relationship to a (...) canonical narrative arc, followed by a discussion of complex structures that extend beyond the canonical schema. This demands that the canonical arc be reconsidered as a generative schema whereby any narrative category can be expanded into a node in a tree structure. Narrative “pacing” is interpreted as a reflection of various patterns of this embedding: conjunction, left-branching trees, center-embedded constituencies, and others. Following this, diagnostic methods are proposed for testing narrative categories and constituency. Finally, we outline the applicability of this theory beyond sequential images, such as to film and verbal discourse, and compare this theory with previous approaches to narrative and discourse. (shrink)

Visual narratives like comics involve a range of complex cognitive operations in order to be understood. The Parallel Interfacing Narrative‐Semantics (PINS) Model integrates an emerging literature showing that comprehension of wordless image sequences balances two representational levels of semantic and narrative structure. The neurocognitive mechanisms that guide these processes are argued to overlap with other domains, such as language and music.

Summary This article questions certain assumptions concerning film form made by the recent psychological film research and compares them to those of precursors of film psychology like Hugo Münsterberg and Rudolf Arnheim, as well as the principles of Gestalt psychology. It is argued that principles of Gestalt psychology such as those of ‘good form’ and good continuation are still underlying the psychological research of film, becoming particularly apparent in its approach to continuity editing. Following an alternative Gestalt genealogy that links (...) Gestalt theory with more recent dynamic models of brain activity and with accounts of brain complexity and neuronal synchronisation, the article concludes that psychological research on film needs to shift the focus from form to transformation, both in conceiving the perceptual and cognitive processing of films and in approaching film aesthetics more broadly. (shrink)

Cohn's (2013) theory of “Visual Narrative Grammar” argues that sequential images take on categorical roles in a narrative structure, which organizes them into hierarchic constituents analogous to the organization of syntactic categories in sentences. This theory proposes that narrative categories, like syntactic categories, can be identified through diagnostic tests that reveal tendencies for their distribution throughout a sequence. This paper describes four experiments testing these diagnostics to provide support for the validity of these narrative categories. In Experiment 1, participants reconstructed (...) unordered panels of a comic strip into an order that makes sense. Experiment 2 measured viewing times to panels in sequences where the order of panels was reversed. In Experiment 3, participants again reconstructed strips but also deleted a panel from the sequence. Finally, in Experiment 4 participants identified where a panel had been deleted from a comic strip and rated that strip's coherence. Overall, categories had consistent distributional tendencies within experiments and complementary tendencies across experiments. These results point toward an interaction between categorical roles and a global narrative structure. (shrink)

Events can be modeled through a geometric approach, representing event structures in terms of spaces and mappings between spaces. At least two spaces are needed to describe an event, an action space and a result space. In this article, we invoke general mathematical structures in order to develop this geometric perspective. We focus on three cognitive processes that are crucially involved in events: causal thinking, control of action and learning by generalization. These cognitive processes are supported by three corresponding mathematical (...) properties: monotonicity (that we relate to qualitative causal thinking and allows extrapolation); continuity (that plays a key role in our activities of action control); and convexity (that facilitates generalization and the categorization of events, and enables interpolation). We define how such properties constrain events representations and relate them to thinking about events. We discuss the relevance of the three constraints for event segmentation and explore the implications of such constraints for semantics. We conclude by a discussion that relates our approach to other accounts of events. (shrink)

Due to the complexity of research objects, theoretical concepts, and stimuli in media research, researchers in psychology and communications presumably need sophisticated measures beyond self-report scales to answer research questions on media use processes. The present study evaluates stimulus-dependent structure in spontaneous eye-blink behavior as an objective, corroborative measure for the media use phenomenon of spatial presence. To this end, a mixed methods approach is used in an experimental setting to collect, combine, analyze, and interpret data from standardized participant self-report, (...) observation of participant behavior, and content analysis of the media stimulus. T-pattern detection is used to analyze stimulus-dependent blinking behavior, and this structural data is then contrasted with self-report data. The combined results show that behavioral indicators yield the predicted results, while self-report data shows unpredicted results that are not predicted by the underlying theory. The use of a mixed methods approach offered insights that support further theory development and theory testing beyond a traditional, mono-method experimental approach. (shrink)

Journal Name: Cognitive Linguistics Issue: Ahead of print.

This paper argues that the still-emerging paradigm of situated cognition requires a more systematic perspective on media to capture the enculturation of the human mind. By virtue of being media, cultural artifacts present central experiential models of the world for our embodied minds to latch onto. The paper identifies references to external media within embodied, extended, enactive, and predictive approaches to cognition, which remain underdeveloped in terms of the profound impact that media have on our mind. To grasp this impact, (...) I propose an enactive account of media that is based on expansive habits as media-structured, embodied ways of bringing forth meaning and new domains of values. We apply such habits, for instance, when seeing a picture or perceiving a movie. They become established through a process of reciprocal adaptation between media artifacts and organisms and define the range of viable actions within such a media ecology. Within an artifactual habit, we then become attuned to a specific media work that engages us. Both the plurality of habits and the dynamical adjustments within a habit require a more flexible neural architecture than is addressed by classical cognitive neuroscience. To detail how neural and media processes interlock, I will introduce the concept of neuromediality and discuss radical predictive processing accounts that could contribute to the externalization of the mind by treating media themselves as generative models of the world. After a short primer on general media theory, I discuss media examples in three domains: pictures and moving images; digital media; architecture and the built environment. This discussion demonstrates the need for a new cognitive media theory based on enactive artifactual habits—one that will help us gain perspective on the continuous re-mediation of our mind. (shrink)