Understanding how people comprehend visual narratives (including picture stories, comics, and film) requires the combination of traditionally separate theories that span the initial sensory and perceptual processing of complex visual scenes, the perception of events over time, and comprehension of narratives. Existing piecemeal approaches fail to capture the interplay between these levels of processing. Here, we propose the Scene Perception & Event Comprehension Theory (SPECT), as applied to visual narratives, which distinguishes between front-end and back-end cognitive processes. Front-end processes occur (...) during single eye fixations and are comprised of attentional selection and information extraction. Back-end processes occur across multiple fixations and support the construction of event models, which reflect understanding of what is happening now in a narrative (stored in working memory) and over the course of the entire narrative (stored in long-term episodic memory). We describe relationships between front- and back-end processes, and medium-specific differences that likely produce variation in front-end and back-end processes across media (e.g., picture stories vs. film). We describe several novel research questions derived from SPECT that we have explored. By addressing these questions, we provide greater insight into how attention, information extraction, and event model processes are dynamically coordinated to perceive and understand complex naturalistic visual events in narratives and the real world. (shrink)

We argue that formal linguistic theory, properly extended, can provide a unifying framework for diverse phenomena beyond traditional linguistic objects. We display applications to pictorial meanings, visual narratives, music, dance, animal communication, and, more abstractly, to logical and non-logical concepts in the ‘language of thought’ and reasoning. In many of these cases, a careful analysis reveals that classic linguistic notions are pervasive across these domains, such as for instance the constituency (or grouping) core principle of syntax, the use of logical (...) variables (for object tracking), or the variety of inference types investigated in semantics/pragmatics. The aim of this overview is to show how the application of formal linguistic concepts and methodology to non-linguistic objects yields non-trivial insights, thus opening the possibility of a general, precise theory of signs. (An appendix, found in the online supplements to this article, surveys applications of Super Linguistics to animal communication.). (shrink)

As formal theoretical linguistic methodology has matured, recent years have seen the advent of applying it to objects of study that transcend language, e.g., to the syntax and semantics of music (Lerdahl & Jackendoff 1983, Schlenker 2017a; see also Rebuschat et al. 2011). One of the aims of such extensions is to shed new light on how meaning is construed in a range of communicative systems. In this paper, we approach this goal by looking at narrative dance in the form (...) of Bharatanatyam. We argue that a semantic approach to dance can be modeled closely after the formal semantics of visual narrative proposed by Abusch (2013, 2014, 2021). A central conclusion is that dance not only shares properties of other fundamentally human means of expression, such as visual narrative and music, but that it also exhibits similarities to sign languages and the gestures of non-signers (see, e.g., Schlenker 2020) in that it uses space to track individuals in a narrative and performatively portray the actions of those individuals. From the perspective of general human cognition, these conclusions corroborate the idea that linguistic investigations beyond language (see Patel-Grosz et al. forthcoming) can yield insights into the very nature of the human mind and of the communicative devices that it avails. (shrink)

We argue that formal linguistic theory, properly extended, can provide a unifying framework for diverse phenomena beyond traditional linguistic objects. We display applications to pictorial meanings, visual narratives, music, dance, animal communication, and, more abstractly, to logical and non-logical concepts in the ‘language of thought’ and reasoning. In many of these cases, a careful analysis reveals that classic linguistic notions are pervasive across these domains, such as for instance the constituency (or grouping) core principle of syntax, the use of logical (...) variables (for object tracking), or the variety of inference types investigated in semantics/pragmatics. The aim of this overview is to show how the application of formal linguistic concepts and methodology to non-linguistic objects yields non-trivial insights, thus opening the possibility of a general, precise theory of signs. (An appendix, found in the online supplements to this article, surveys applications of Super Linguistics to animal communication.). (shrink)

While much has been written about verbal narratives, we still lack a clear account of what makes images narrative. I argue that there are narrative characteristics of images and show this with examples of single images. The argument proceeds in three steps. First, I propose that from a semantic perspective, the following two characteristics are necessary for an image to be narrative: a representation of an event and a representation of time. Second, I argue that there are paradigmatic characteristics, such (...) as at least two events, bridging connections, and unifying subjects between these events, characters with intentions, and the representation of goal-directed actions. Third, I show that it is possible to differentiate between narrative and non-narrative images while also accounting for the idea that narrativity is a matter of degree. While I do not provide a full definition of narrative images, my account constitutes a necessary first step in this direction. (shrink)

In this paper, I outline a grammar of lifting (i.e., resistance training) and compare it to that of language. I approach lifting as a system of generating complex meaning–form correspondences from regularized elements and describe the levels of mental representations and relationships between them that are involved in full command of this system. To be able to do so, I adopt a goal-based conception of meaning, which allows us to talk about mappings from complex goals to complex surface outputs in (...) systems of intentional action, signaling and non-signaling, interactive and non-interactive, in a unified way, and show how it applies in lifting. I then proceed to argue that the grammar of lifting is architecturally very similar to that of language. First, I show that both involve stable (idiomatized/lexicalized) pairings of regularized forms with regularized meanings. Second, I argue that in both lifting and language, meaning–form mapping is mediated by syntax, which, crucially, operates on non-linearized hierarchical structures of abstract objects that include both content morphemes and functional morphemes. I conclude, following and expanding on some insights from prior literature and offering further evidence for them, that neither of these architectural phenomena (idiomatized meaning–form pairings and abstract syntax) is specific to language, with both of them likely emerging in skilled action that does not necessarily involve social interaction, due to considerations of repeatability and reusability of elements in new contexts. (shrink)

The regional brain networks and the underlying neurophysiological mechanisms subserving the cognition of visual narrative in humans have largely been studied with non-invasive brain recording. In this study, we specifically investigated how regional and cross-regional cortical activities support visual narrative interpretation using intracranial stereotactic electroencephalograms recordings from thirteen human subjects (6 females, and 7 males). Widely distributed recording sites across the brain were sampled while subjects were explicitly instructed to observe images from fables presented in “sequential” order, and a set (...) of images drawn from multiple fables presented in “scrambled” order. Broadband activity mainly within the frontal and temporal lobes were found to encode if a presented image is part of a visual narrative (sequential) or random image set (scrambled). Moreover, the temporal lobe exhibits strong activation in response to visual narratives while the frontal lobe is more engaged when contextually novel stimuli are presented. We also investigated the dynamics of interregional interactions between visual narratives and contextually novel series of images. Interestingly, the interregional connectivity is also altered between sequential and scrambled sequences. Together, these results suggest that both changes in regional neuronal activity and cross-regional interactions subserve visual narrative and contextual novelty processing. (shrink)

Cognitive Science, Volume 46, Issue 7, July 2022.

Comics are complex multimodal documents that make for intriguing materials to analyze with computer vision and computational linguistics. This review summarizes the growing developments in computational modeling which have been progressing to analyze visual narratives across their various substructures.