We argue that formal linguistic theory, properly extended, can provide a unifying framework for diverse phenomena beyond traditional linguistic objects. We display applications to pictorial meanings, visual narratives, music, dance, animal communication, and, more abstractly, to logical and non-logical concepts in the ‘language of thought’ and reasoning. In many of these cases, a careful analysis reveals that classic linguistic notions are pervasive across these domains, such as for instance the constituency (or grouping) core principle of syntax, the use of logical (...) variables (for object tracking), or the variety of inference types investigated in semantics/pragmatics. The aim of this overview is to show how the application of formal linguistic concepts and methodology to non-linguistic objects yields non-trivial insights, thus opening the possibility of a general, precise theory of signs. (An appendix, found in the online supplements to this article, surveys applications of Super Linguistics to animal communication.). (shrink)

We argue that formal linguistic theory, properly extended, can provide a unifying framework for diverse phenomena beyond traditional linguistic objects. We display applications to pictorial meanings, visual narratives, music, dance, animal communication, and, more abstractly, to logical and non-logical concepts in the ‘language of thought’ and reasoning. In many of these cases, a careful analysis reveals that classic linguistic notions are pervasive across these domains, such as for instance the constituency (or grouping) core principle of syntax, the use of logical (...) variables (for object tracking), or the variety of inference types investigated in semantics/pragmatics. The aim of this overview is to show how the application of formal linguistic concepts and methodology to non-linguistic objects yields non-trivial insights, thus opening the possibility of a general, precise theory of signs. (An appendix, found in the online supplements to this article, surveys applications of Super Linguistics to animal communication.). (shrink)

This paper develops a theory of analog representation. We first argue that the mark of the analog is to be found in the nature of a representational system’s interpretation function, rather than in its vehicles or contents alone. We then develop the rulebound structure theory of analog representation, according to which analog systems are those that use interpretive rules to map syntactic structural features onto semantic structural features. The theory involves three degree-theoretic measures that capture three independent ways in which (...) a system can be more or less analog. We explain how our theory improves upon prior accounts of analog representation, provides plausible diagnoses for novel challenge cases, extends to hybrid systems that are partially analog and partially symbolic, and accounts for some of the advantages and disadvantages of representing analogically versus symbolically. (shrink)

Semantics traditionally focuses on linguistic meaning. In recent years, the Super Linguistics movement has tried to broaden the scope of inquiry in various directions, including an extension of semantics to talk about the meaning of pictures. There are close similarities between the interpretation of language and of pictures. Most fundamentally, pictures, like utterances, can be either true or false of a given state of affairs, and hence both express propositions (Zimmermann, 2016; Greenberg, 2013; Abusch, 2015). Moreover, sequences of pictures, like (...) sequences of utterances, can be used to tell stories. Wordless picture books, comics, and film are cases in point. In this paper I pick up the project of providing a dynamic semantic account of pictorial story-telling, started by Abusch (2012) and continued by Abusch & Rooth (2017); Maier & Bimpikou (2019); Fernando (2020). More specifically, I propose here a semantics of speech and thought bubbles by adding event reference to PicDRT. To get there I first review the projection-based semantics for pictures (section 1), noting the fundamental distinction between symbolic and iconic meaning that makes speech bubbles especially interesting (section 2). I then review the dynamic PicDRT framework for pictorial narratives (section 3), add events (section 4), and propose an account of speech bubbles as quotational event modification (section 5). I end with a brief look at other conventional, symbolic enrichments in comics (section 6). (shrink)

Some have suggested that images can be arguments. Images can certainly bolster the acceptability of individual premises. We worry, though, that the static nature of images prevents them from ever playing a genuinely argumentative role. To show this, we call attention to a dilemma. The conclusion of a visual argument will either be explicit or implicit. If a visual argument includes its conclusion, then that conclusion must be demarcated from the premise or otherwise the argument will beg the question. If (...) a visual argument does not include its conclusion, then the premises on display must license that specific conclusion and not its opposite, in accordance with some demonstrable rationale. We show how major examples from the literature fail to escape this dilemma. Drawing inspiration from the graphical logic of C. S. Peirce, we suggest instead that images can be manipulated in a way that overcomes the dilemma. Diagrammatic reasoning can take one stepwise from an initial visual layout to a conclusion—thereby providing a principled rationale that bars opposite conclusions—and the visual inscription of this correct conclusion can come afterward in time—thereby distinguishing the conclusion from the premises. Even though this practical application of Peirce’s logical ideas to informal contexts requires that one make adjustments, we believe it points to a dynamic conception of visual argumentation that will prove more fertile in the long run. (shrink)

I argue that emojis are essentially little pictures, rather than words, gestures, expressives, or diagrams. ???? means that the world looks like that, from some viewpoint. I flesh out a pictorial semantics in terms of geometric projection with abstraction and stylization. Since such a semantics delivers only very minimal contents I add an account of pragmatic enrichment, driven by coherence and nonliteral interpretation. The apparent semantic distinction between emojis depicting entities (like ????) and those depicting facial expressions (like ????) I (...) analyze as a difference between truth-conditional and use-conditional pictorial content: ???? depicts what the world of evaluation looks like, while ???? depicts what the utterance context looks like. (shrink)

We propose an extension of Discourse Respresentation Theory (DRT) for analyzing pictorial narratives. We test drive our PicDRT framework by analyzing the way authors represent characters’ mental states and perception in comics. Our investigation goes beyond Abusch and Rooth (2017) in handling not just free perception sequences, but also a form of apparent perspective blending somewhat reminiscent of free indirect discourse.

Understanding how people comprehend visual narratives (including picture stories, comics, and film) requires the combination of traditionally separate theories that span the initial sensory and perceptual processing of complex visual scenes, the perception of events over time, and comprehension of narratives. Existing piecemeal approaches fail to capture the interplay between these levels of processing. Here, we propose the Scene Perception & Event Comprehension Theory (SPECT), as applied to visual narratives, which distinguishes between front-end and back-end cognitive processes. Front-end processes occur (...) during single eye fixations and are comprised of attentional selection and information extraction. Back-end processes occur across multiple fixations and support the construction of event models, which reflect understanding of what is happening now in a narrative (stored in working memory) and over the course of the entire narrative (stored in long-term episodic memory). We describe relationships between front- and back-end processes, and medium-specific differences that likely produce variation in front-end and back-end processes across media (e.g., picture stories vs. film). We describe several novel research questions derived from SPECT that we have explored. By addressing these questions, we provide greater insight into how attention, information extraction, and event model processes are dynamically coordinated to perceive and understand complex naturalistic visual events in narratives and the real world. (shrink)

In Helen Bannerman’s Little Black Sambo, the text describes its main character as witty, brave, and resourceful. The drawings of the story’s main character which accompany this text, however, present a unique kind of harm that only becomes clear when the work is read as a collection of single-panel comics rather than an illustrated book. In this chapter, I show what happens when we read drawings in books as textless comics, and, based on how things turn out from this reading, (...) motivate why we have some substantive reason to read both Little Black Sambo and other books with drawings-as-comics. (shrink)

This response takes advantage of the diverse and wide-ranging series of commentaries to clarify some aspects of the target article, and flesh out other aspects. My central point is a plea to take graphic codes seriously as codes, rather than as a kind of visual art or as a byproduct of spoken language; only in this way can the puzzle of ideography be identified and solved. In this perspective, I argue that graphic codes do not derive their expressive power from (...) iconicity alone (unlike visual arts), and I clarify the peculiar relationship that ties writing to spoken language. I then discuss three possible solutions to the puzzle of ideography. I argue that a learning account still cannot explain why ideographies fail to evolve, even if we emancipate the learning account from the version that Liberman put forward; I develop my preferred solution, the “standardization account,” and contrast it with a third solution suggested by some commentaries, which says that ideographies do not evolve because they would make communication too costly. I consider, by way of conclusion, the consequences of these views for the future evolution of ideography. (shrink)

Despite obvious advantages, no generalised ideographic codes have evolved through cultural evolution to rely on iconicity. Morin suggests that this is because of missing means of standardisation, which glottographic codes get from natural languages. Although we agree, we also point to the important role of the available media, which might support some forms of reference more effectively than others.

The paper is a study of how graphic narratives (graphic novels and the comics) represent time in external visual space as well as in inner (mental) representations. Peirce’s semiotics is the main tool of research. After a survey of various approaches to the study of time in narratives in general and in graphic narratives in particular, an outline of the various aspects of the embodiment of time in space in general is given before the forms of the embodiment of time (...) in the space of graphic narratives is examined in detail. Signs of time are signs that represent time as their object and create mental representations of time as their interpretant. On this semiotic premise, the paper proposes a reinterpretation of Genette’s theory of time in narratives. A comparison with narratives in prose reveals that narrative time in graphic narratives evinces marked discontinuities in their spatial representation. The paper distinguishes between time as continuity, time as age, and points in time. It concludes that time as continuity is typically embodied in the form of spatial diagrams whereas time as age and points in time are mainly of the nature of indexical signs. The insights into the forms of embodiment of time in space derived from these premises are applied to examples from recent graphic novels and some traditional comics. (shrink)

Formal theories of mental representation have receded from the importance they had in the early days of cognitive science. I argue that such theories are crucial in any mental domain, not just for their own sake, but to guide experimental inquiry, as well as to integrate the domain into the mind as a whole. To illustrate the criteria of adequacy for theories of mental representation, I compare two theoretical approaches to language: classical generative grammar (Chomsky, 1965, 1981, 1995) and the (...) parallel architecture (Jackendoff, 1997, 2002). The grounds for comparison include (a) the internal coherence of the theory across phonology, syntax, and semantics; (b) the relation of language to other mental faculties; (c) the relationship between grammar and lexicon; (d) relevance to theories of language processing; and (e) the possibility of languages with little or no syntax. (shrink)

There is a widely held view that visual representations (images) do not depict negation, for example, as expressed by the sentence, “the train is not coming.” The present study focuses on the real-world visual representations of photographs and comic (manga) illustrations and empirically challenges the question of whether humans and machines, that is, modern deep neural networks, can recognize visual representations as expressing negation. By collecting data on the captions humans gave to images and analyzing the occurrences of negation phrases, (...) we show some evidence that humans recognize certain images as expressing negation. Furthermore, based on this finding, we examined whether or not humans and machines can classify novel images as expressing negation. The humans were able to correctly classify images to some extent, as expected from the analysis of the image captions. On the other hand, the machine learning model of image processing was only able to perform this classification at about the chance level, not at the same level of performance as the human. Based on these results, we discuss what makes humans capable of recognizing negation in visual representations, highlighting the role of the background commonsense knowledge that humans can exploit. Comparing human and machine learning performances suggests new ways to understand human cognitive abilities and to build artificial intelligence systems with more human-like abilities to understand logical concepts. (shrink)

Visual narratives like comics involve a range of complex cognitive operations in order to be understood. The Parallel Interfacing Narrative‐Semantics (PINS) Model integrates an emerging literature showing that comprehension of wordless image sequences balances two representational levels of semantic and narrative structure. The neurocognitive mechanisms that guide these processes are argued to overlap with other domains, such as language and music.

The grammatical paradigm used to be a model for entire areas of cognitive science. Its primary tenet was that theories are axiomatic-like systems. A secondary tenet was that their predictions should be tested quickly and in great detail with introspective judgments. While the grammatical paradigm now often seems passé, we argue that in fact it continues to be as efficient as ever. Formal models are essential because they are explicit, highly predictive, and typically modular. They make numerous critical predictions, which (...) must be tested efficiently; introspective judgments do just this. We further argue that the grammatical paradigm continues to be fruitful. Within linguistics, implicature theory is a recent example, with a combination of formal explicitness, modularity, and interaction with experimental work. Beyond traditional linguistics, the grammatical paradigm has proven fruitful in the study of gestures and emojis; literature (“Free Indirect Discourse”); picture semantics and comics; music and dance cognition; and even reasoning and concepts. We argue, however, that the grammatical paradigm must be adapted to contemporary cognitive science. Computational methods are essential to derive quantitative predictions from formal models (Bayesian pragmatics is an example). And data collection techniques offer an ever richer continuum of options, from introspective judgments to large-scale experiments, which makes it possible to optimize the cost/benefit ratio of the empirical methods that are chosen to test theories. (shrink)

According to Ernst Gombrich, cartoons provide us the chance to “study the use of symbols in a circumscribed context [and] find out what role the image may play in the household of our mind” (Gombrich 1973, 190). This paper looks at some underexplored implications and outcomes of Ernst Gombrich’s conceptual schemata when such a schemata is applied to cartoons. While we might easily avoid defamatory reference when picking out a subject in writing or speech, cartoon depictions, especially those unaccompanied by (...) speech bubbles or captions, often rely on a visual symbolism typified by the warping of some features the removal of others, and the manifestation of some visual trope or other, more easily lend themselves to defamatory reference. While harms to the referent are many in cases of defamatory cartoons, this paper focuses on the harms to the viewers of such cartoons by the depicter’s message and mode of representation, unique to the cartoon form. I will focus on the harms of misinformation of the viewer by the visual schemata (Gombrich 1960) present in certain cartoons, and the way this misinformative visual schemata (Ibid) may also restrict the possibility of conceptual revisions in its viewership. Harms of this kind come out uniquely in cartoons via norms of cartooning (as a particularly stylized and symbolic mode of visual representation) and the norms of interaction employed in the act of viewing a cartoon. As we’ll see in our case study of the infamous Jyllands Posten cartoons of Muhammad the Prophet, even in cases where a cartoon representation bears no visual similarity to its referent, viewers can easily and reliably pick out the referent by calling to mind, in our focal case, the defamatory stereotypes, stock figures and icons inextricably linked with the referent. I argue that the damage to viewers of these sorts of depictions lies not in the fact that the viewer manages to pick out the intended referent of the depiction but what tools they use to pick out the referent and how such tools of reference (mis)inform their understanding of the referent they’ve picked out. (shrink)

Both mainstream and art comics often use various flourishes surrounding characters’ heads. These so-called “pictorial runes” help convey the emotional states of the characters. In this paper, using panels from Western and Indian comic albums as well as neutral emoticons and basic shapes in different colors, we focus on the following two issues: whether runes increase the awareness in comics readers about the emotional state of the character; and whether a correspondence can be found between the types of runes and (...) specific emotions. Our results show that runes help communicate emotion. Although no one-to-one correspondence was found between the tested runes and specific emotions, it was found that droplets and spikes indicate generic emotions, spirals indicate negative emotions, and twirls indicate confusion and dizziness. (shrink)

Usain Bolt’s Lightning Bolt pose, one arm highly extended to one side, suggests action. Likewise, static pictures of animals, legs extended, show animation. We tested a new cue for motion perception—extension—and in particular extension of dancer’s legs. An experiment with pictures of a dancer finds larger angles between the legs suggest greater movement, especially with in-air poses and in lateral views. Leg positions graded from simply standing to very difficult front and side splits. Liking ratings were more related to Difficulty (...) ratings than Movement ratings. (shrink)

This article argues that to disrupt legal education in a radical sense, students need to become acquainted with the art of worldmaking and the view that law is a “way of worldmaking”. First, I show that law is a cultural semiotic practice that requires decoding and, for that reason, demands a creative intervention by those that want to know, understand, and do things with law. Altogether this amounts to recognizing the different modes in which law creates, and is part of, (...) worlds. Second, I propose that due to different features of their aesthetic form, comics are a particularly effective medium to place students before the myriad ways in which law and lawyers make and reproduce worlds. Third, I illustrate the argument by exploring how the Saga comic series, through its formal multimodality and narrative and cultural complexity, can make good on that challenge. (shrink)

In several sign languages, plurals can be realized with unpunctuated or punctuated repetitions of a noun, with different semantic implications; similar repetition-based plurals have been described in some homesigns and silent gestures. Unpunctuated repetitions often get approximate ‘at least’ readings while punctuated repetitions typically correspond to ‘exactly’ readings. The prevalence of these mechanisms could be thought to be a case in which Universal Grammar does not just specify the abstract properties of grammatical elements, but also their phonological realization, at least (...) in the visual modality. We explore an alternative in which punctuated and unpunctuated repetitions arise from general properties of iconic representations. On an empirical level, we argue that in ASL, punctuated and unpunctuated repetitions are unlikely to be an exclusively grammatical mechanism, as they can be found with purely iconic representations. On a theoretical level, we argue for a modular account with three components. First, repetition-based plurals can create a simplified pictorial representation. Second, unpunctuated repetitions give rise to pictorial vagueness, resolved by way of quantification over precisifications. Third, a pragmatic process involving strategic reasoning maps these vague representations onto a set of candidate linguistic meanings, including some ‘at least’ plural readings that are best expressed by unpunctuated repetitions. (shrink)

An ideography is a general-purpose code made of pictures that do not encode language, which can be used autonomously – not just as a mnemonic prop – to encode information on a broad range of topics. Why are viable ideographies so hard to find? I contend that self-sufficient graphic codes need to be narrowly specialized. Writing systems are only an apparent exception: At their core, they are notations of a spoken language. Even if they also encode nonlinguistic information, they are (...) useless to someone who lacks linguistic competence in the encoded language or a related one. The versatility of writing is thus vicarious: Writing borrows it from spoken language. Why is it so difficult to build a fully generalist graphic code? The most widespread answer points to a learnability problem. We possess specialized cognitive resources for learning spoken language, but lack them for graphic codes. I argue in favor of a different account: What is difficult about graphic codes is not so much learning or teaching them as getting every user to learn and teach the same code. This standardization problem does not affect spoken or signed languages as much. Those are based on cheap and transient signals, allowing for easy online repairing of miscommunication, and require face-to-face interactions where the advantages of common ground are maximized. Graphic codes lack these advantages, which makes them smaller in size and more specialized. (shrink)

Drawn sequences of images, like those in comics and picture stories, are a pervasive and fundamental way that humans have communicated for millennia. Yet, the study of visual narratives has only recently gained traction in Cognitive Science. Here we explore what has held back the study of the cognition of visual narratives, and why researchers should join in scholarship of this ubiquitous aspect of expression.

Morin argues that ideographies are limited because graphic codes lack a capacity for proliferating standardization. However, natural graphic systems display rich standardization and can be placed in sequences using complex combinatorial structures. In contrast, ideographies are not natural, and their limitations lie in their attempts to artificially force a graphic system to behave like a writing system.

Film viewers make sense of films first of all at a precognitive level, triggered by their bodi­ly responses. The key notion here is movement: the movements of screen characters, the movements simulated by the viewers who perceive these characters, and the camera move­ments that mediate between the two. This review essay evaluates two monographs: Maarten Coëgnarts’ Embodied Cinema, which expands conceptual metaphor theo­ry to account for film’s unique affordances to communicate embodied meaning; and Vit­torio Gallese's and Michele Guerra’s The Empathic (...) Screen, which buttresses embod­ied simulation by film viewers experimentally by demonstrating the workings of “mirror neurons.” The review ends by discussing how these two books tie in with other develop­ments in the study of gene-culture coevolution. (shrink)

The pragmatic semiotic theory of Charles Sanders Peirce has a lot of practical benefits for enhancing visual communication in illustrations. His triadic theory of Semiosis focuses on the dynamic interrelationships between the concept to be communicated, how it is represented through a semiotic sign, and how this affects the success of how the concept is eventually interpreted. Peirce's pragmatic semiotic theory uses complex language, and although Peirce is embraced in some design disciplines, the language that defines Semiosis is problematic beyond (...) academia. This paper is an attempt to address this by providing illustrators with a basic introduction to how Semiosis can help to enhance the success of the visual communication in their illustrations. This is done by translating Peircean terminology into illustrator-centric language and providing an example of how the Semiosis is implemented in an illustration. Within the limits of a short paper, illustrators can begin to understand how the triadic nature of concept/representation/interpretation can benefit them during their ideation and sketching phase to author effective images. In doing this, this paper will mostly discuss iconic, indexical, and symbolic semiotic representation within pragmatic semiotic signs of the intended concept to be communicated in an illustration. This paper's aim is to enact a pragmatic turn in illustrators, in which Semiosis theory becomes more integrated within their practical work, by providing a more illustrator-centric dissemination of Peirce's semiotic theory. (shrink)

Intersemiotic projection is one of the most common configurations in the knowledge construction process of academic comics. Although previous studies address some general features of intersemiotic projection, further research on interdependency relations of intersemiotic projection is needed in order to map out the whole system. This study, based on the social-semiotic approach to multimodal studies, proposes a systemic framework of image-text paratactic and hypotactic projection in academic comics. This framework identifies three sub-categories of paratactic projection and hypotactic projection, respectively: Emergent, (...) Adjoined, Sequential, Compound, Subsumed, and Fused. The use of these projection configurations contributes to the realization of the communicative purposes of academic comics to evidence, exemplify, emphasize, explain, elaborate, and visualize the discussed knowledge. This research attempts not only to contribute to the theoretical development of multimodality from the social-semiotic perspective by refining and enlarging the multimodal hypotactic projection framework but also to provide an effective and applicable tool for comics analysis. (shrink)

Emoji have been ubiquitous in communication for over a decade, yet how they derive meaning remains underexplored. Here, we examine an aspect fundamental to linguistic meaning‐making: the degree to which emoji have conventional lexicalized meanings and whether that conventionalization affects processing in real‐time. Experiment 1 establishes a range of meaning agreement levels across emoji within a population; Experiment 2 measures accuracy and response times to word‐emoji pairings in a match/mismatch task. In this experiment, we found that accuracy and response time (...) both correlated significantly with the level of population‐wide meaning agreement from Experiment 1, suggesting that lexical access of single emoji may be comparable to that of words, even out of context. This is consistent with theories of a multimodal lexicon that stores links between meaning, structure, and modality in long‐term memory. Altogether, these findings suggest that emoji can allow a range of entrenched, lexicalized representations. (shrink)

Comics are complex multimodal documents that make for intriguing materials to analyze with computer vision and computational linguistics. This review summarizes the growing developments in computational modeling which have been progressing to analyze visual narratives across their various substructures.

Static depiction of motion, particularly lines trailing behind a mover, has long been of interest in the psychology literature. Empirical research has demonstrated that these “motion lines” benefited motion comprehension in static images by disambiguating the direction of movement. Yet, there is no consensus on how those lines derive their meaning. In this article, we review three accounts suggesting different interpretations of what motion lines represent. While a perceptual account considers motion lines originating from motion streaks in the primary visual (...) cortex, metaphorical and lexical accounts propose them as graphical conventions that should be learned—either through resemblance to sensory experiences (e.g., natural path marks) or directly being mapped to a conceptual category of paths. To contrast these three accounts, we integrate empirical research on motion lines and their understanding. Overall, developmental, proficiency, and cross-cultural variances indicate that the understanding of motion lines is neither innate nor universal, thus providing less support for lines having a purely perceptual origin. Rather, we argue the empirical findings suggest that motion lines are not iconic depictions of visual percepts but are graphical conventions indexing conceptual path information, which need to be learned and encoded in a visual lexicon. (shrink)