Filmmakers use continuity editing to engender a sense of situational continuity or discontinuity at editing boundaries. The goal of this study was to assess the impact of continuity editing on how people perceive the structure of events in a narrative film and to identify brain networks that are associated with the processing of different types of continuity editing boundaries. Participants viewed a commercially produced film and segmented it into meaningful events, while brain activity was recorded with functional magnetic resonance imaging (...) (MRI). We identified three degrees of continuity that can occur at editing locations: edits that are continuous in space, time, and action; edits that are discontinuous in space or time but continuous in action; and edits that are discontinuous in action as well as space or time. Discontinuities in action had the biggest impact on behavioral event segmentation, and discontinuities in space and time had minor effects. Edits were associated with large transient increases in early visual areas. Spatial-temporal changes and action changes produced strikingly different patterns of transient change, and they provided evidence that specialized mechanisms in higher order perceptual processing regions are engaged to maintain continuity of action in the face of spatiotemporal discontinuities. These results suggest that commercial film editing is shaped to support the comprehension of meaningful events that bridge breaks in low-level visual continuity, and even breaks in continuity of spatial and temporal location. (shrink)

In addition to providing an account of the empirical facts of language, a theory that aspires to account for language as a biologically based human faculty should seek a graceful integration of linguistic phenomena with what is known about other human cognitive capacities and about the character of brain computation. The present article compares the theoretical stance of biolinguistics (Chomsky 2005, Di Sciullo and Boeckx 2011) with a constraint-based Parallel Architecture approach to the language faculty (Jackendoff 2002, Culicover and Jackendoff (...) 2005). The issues considered include the necessity of redundancy in the lexicon and the rule system, the ubiquity of recursion in cognition, derivational vs. constraint-based formalisms, the relation between lexical items and grammatical rules, the roles of phonology and semantics in the grammar, the combinatorial character of thought in humans and nonhumans, the interfaces between language, thought, and vision, and the possible course of evolution of the language faculty. In each of these areas, the Parallel Architecture offers a superior account both of linguistic facts and of the relation of language to the rest of the mind/brain. (shrink)

Understanding how people comprehend visual narratives (including picture stories, comics, and film) requires the combination of traditionally separate theories that span the initial sensory and perceptual processing of complex visual scenes, the perception of events over time, and comprehension of narratives. Existing piecemeal approaches fail to capture the interplay between these levels of processing. Here, we propose the Scene Perception & Event Comprehension Theory (SPECT), as applied to visual narratives, which distinguishes between front-end and back-end cognitive processes. Front-end processes occur (...) during single eye fixations and are comprised of attentional selection and information extraction. Back-end processes occur across multiple fixations and support the construction of event models, which reflect understanding of what is happening now in a narrative (stored in working memory) and over the course of the entire narrative (stored in long-term episodic memory). We describe relationships between front- and back-end processes, and medium-specific differences that likely produce variation in front-end and back-end processes across media (e.g., picture stories vs. film). We describe several novel research questions derived from SPECT that we have explored. By addressing these questions, we provide greater insight into how attention, information extraction, and event model processes are dynamically coordinated to perceive and understand complex naturalistic visual events in narratives and the real world. (shrink)

Narratives are an integral part of human expression. In the graphic form, they range from cave paintings to Egyptian hieroglyphics, from the Bayeux Tapestry to modern day comic books (Kunzle, 1973; McCloud, 1993). Yet not much research has addressed the structure and comprehension of narrative images, for example, how do people create meaning out of sequential images? This piece helps fill the gap by presenting a theory of Narrative Grammar. We describe the basic narrative categories and their relationship to a (...) canonical narrative arc, followed by a discussion of complex structures that extend beyond the canonical schema. This demands that the canonical arc be reconsidered as a generative schema whereby any narrative category can be expanded into a node in a tree structure. Narrative “pacing” is interpreted as a reflection of various patterns of this embedding: conjunction, left-branching trees, center-embedded constituencies, and others. Following this, diagnostic methods are proposed for testing narrative categories and constituency. Finally, we outline the applicability of this theory beyond sequential images, such as to film and verbal discourse, and compare this theory with previous approaches to narrative and discourse. (shrink)

To understand the endogenous components of the event-related brain potential (ERP), we must use data about the components' antecedent conditions to form hypotheses about the information-processing function of the underlying brain activity. These hypotheses, in turn, generate testable predictions about the consequences of the component. We review the application of this approach to the analysis of the P300 component. The amplitude of the P300 is controlled multiplicatively by the subjective probability and the task relevance of the eliciting events, whereas its (...) latency depends on the duration of stimulus evaluation. These and other factors suggest that the P300 is a manifestation of activity occurring whenever one's model of the environment must be revised. Tests of three predictions based on this “context updating” model are reviewed. Verleger's critique is based on a misconstrual of the model as well as a partial and misleading reading of the relevant literature. (shrink)

Cohn's (2013) theory of “Visual Narrative Grammar” argues that sequential images take on categorical roles in a narrative structure, which organizes them into hierarchic constituents analogous to the organization of syntactic categories in sentences. This theory proposes that narrative categories, like syntactic categories, can be identified through diagnostic tests that reveal tendencies for their distribution throughout a sequence. This paper describes four experiments testing these diagnostics to provide support for the validity of these narrative categories. In Experiment 1, participants reconstructed (...) unordered panels of a comic strip into an order that makes sense. Experiment 2 measured viewing times to panels in sequences where the order of panels was reversed. In Experiment 3, participants again reconstructed strips but also deleted a panel from the sequence. Finally, in Experiment 4 participants identified where a panel had been deleted from a comic strip and rated that strip's coherence. Overall, categories had consistent distributional tendencies within experiments and complementary tendencies across experiments. These results point toward an interaction between categorical roles and a global narrative structure. (shrink)

Ten years ago, researchers using event-related brain potentials to study language comprehension were puzzled by what looked like a Semantic Illusion: Semantically anomalous, but structurally well-formed sentences did not affect the N400 component—traditionally taken to reflect semantic integration—but instead produced a P600 effect, which is generally linked to syntactic processing. This finding led to a considerable amount of debate, and a number of complex processing models have been proposed as an explanation. What these models have in common is that they (...) postulate two or more separate processing streams, in order to reconcile the Semantic Illusion and other semantically induced P600 effects with the traditional interpretations of the N400 and the P600. Recently, however, these multi-stream models have been called into question, and a simpler single-stream model has been proposed. According to this alternative model, the N400 component reflects the retrieval of word meaning from semantic memory, and the P600 component indexes the integration of this meaning into the unfolding utterance interpretation. In the present paper, we provide support for this “Retrieval–Integration ” account by instantiating it as a neurocomputational model. This neurocomputational model is the first to successfully simulate the N400 and P600 amplitude in language comprehension, and simulations with this model provide a proof of concept of the single-stream RI account of semantically induced patterns of N400 and P600 modulations. (shrink)

We evaluate the “story grammar” approach to story understanding from three perspectives. We first examine the formal properties of the grammars and find only one to be formally adequate. We next evaluate the grammars empirically by asking whether they generate all simple stories and whether they generate only stories. We find many stories that they do not generate and one major class of nonstory that they do generate. We also evaluate the grammars' potential as comprehension models and find that they (...) would add nothing to semantic models that focus on the story content. Hence we advocate a story content oriented approach to studying story understanding instead of the structural story grammar approach. (shrink)