This research article examines the potential of multimodal techniques in promoting inclusive practices in the English language classroom, delving into the multidisciplinary fields of English Language Teaching (ELT) and inclusive education. This study intends to investigate how multimodal resources, such as visual aids, technology, and creative activities, may be effectively integrated to meet the various learning needs of students using a pedagogical lens. This paper examines the theoretical underpinnings and practical implications of using multimodal approaches in ELT (...) settings, shedding light on their potential to promote inclusivity, engagement, and language acquisition among learners with diverse abilities and backgrounds. It does this through a thorough literature review and analysis of pedagogical frameworks. (shrink)

The aim of this paper is to argue that our experience of artworks is normally multimodal. It is the result of perceptual processing in more than one sense modality. In other words, multimodal experience of art is not the exception; it is the rule. I use the example of music in order to demonstrate the various ways in which the visual sense modality influences the auditory processing of music and conclude that this should make us look more closely (...) at our practices of engaging with artworks. (shrink)

It is common to characterize pain with touch-related terms, like ‘cutting’, ‘pressing’, ‘sharp’, and ‘pulsing’, or temperature-related terms, like ‘hot’ or ‘burning’. This suggests that many pains are phenomenally multimodal because they are experienced as having some tactile-like or thermal-like character. The goal of this chapter is to investigate the structure of phenomenally multimodal pain experiences. It is argued that the usual accounts of multimodal structure proposed in investigations regarding exteroceptive experiences cannot be plausibly applied to (...) class='Hi'>multimodal experiences of pain. Instead, an alternative framework is proposed which characterizes the structure of tactile-like and thermal-like pains by referring to the notion of crossmodal correspondences. (shrink)

What is the correct procedure for determining the contents of perception? Philosophers tackling this question increasingly rely on empirically-oriented procedures in order to reach an answer. I argue that this constitutes an improvement over the armchair methodology constitutive of phenomenal contrast cases, but that there is a crucial respect in which current empirical procedures remain limited: they are unimodal in nature, wrongly treating the senses as isolatable faculties. I thus have two aims: first, to motivate a reorientation of the admissible (...) contents debate into a multimodal framework, charting its various significances. The second is to explore whether any experimental studies of multimodal perception support a so-called Liberal (or ‘high-level’ or ‘rich’) account of perception’s admissible contents. I conclude that the McGurk effect and the ventriloquist effect are both explicable without the postulation of high-level content, but that at least one multimodal experimental paradigm may necessitate such content: the rubber hand illusion. One upshot of this argument is that Conservatives who claim that the Liberal view intolerably broadens the scope of perceptual illusions, particularly from the perspective of perceptual psychology, should pursue other arguments against that view. (shrink)

This paper presents an analytical framework for analyzing how multimodal resources of emotion expression are semiotically materialized in discursive interactions specific to political discourse. Interested in how political personae are emotionally constructed through multimodal meaning-making practices, our analysis model assumes an interdisciplinary perspective, which integrates facial expression analysis – using FaceReader™ software –, the theory of emotional arcs and bodily actions (hand gestures) analysis that express emotions, in the analytical framework of multimodality. The results show how the (...) class='Hi'>multimodal choices that political actors make during discursive interactions allow them to build their political brand and make connections with the audience on an emotional level. (shrink)

This is an excerpt of a report that highlights and explores five questions which arose from The Unity of Consciousness and Sensory Integration conference at Brown University in November of 2011. This portion of the report explores the question: Are some of the basic units of consciousness multimodal?

This is an excerpt from a report that highlights and explores five questions which arose from the workshop on perceptual learning and perceptual recognition at the University of Toronto, Mississauga on May 10th and 11th, 2012. This excerpt explores the question: What are the origins of multimodal associations?

From the perspective of distributed cognition I will stress how abduction is essentially multimodal, in that both data and hypotheses can have a full range of verbal and sensory representations, involving words, sights, images, smells, etc., but also kinesthetic – related to the ability to sense the position and location and orientation and movement of the body and its parts – and motor experiences and other feelings such as pain, and thus all sensory modalities. The presence of kinesthetic and (...) motor aspects demonstrates that abductive reasoning can be manipulative. We can also see, in this regard, how implicit factors take part in the abductive procedure, which consequently acquires the character of a kind of “thinking through doing”. This paper further describes 1) the fact that hypotheses in science can be built through different cognitive mediators and so they can also model the same cognitive aspect in different ways; how they can be carriers/ producers of knowledge in a multimodal way; 2) the problem of the possible non-explanatory and instrumental nature of abductive reasoning and the analysis of the consequences for induction; 3) the role of manipulative abduction in building new evidence/ experiments and how they trigger smart inductive inferences. (shrink)

The aim of this paper is to propose a new approach in the form of multimodality as a semiotic method that can be used by marketers and semioticians to induce online flow, a psychological state, on a brand website. First, we refer to multimodality as a semiotic analysis that can be used for a better optimization of semiotic resource sets in meaning-making, and we distinguish it from another similar concept: multimedia. Second, after a critical literature review, we address the flow (...) construct - a state of mind sometimes experienced by people who are deeply involved and immersed in some event, object or activity. Previous quantitative studies show a correlation between online flow experience and other variables such as design, content, sound and layout that are suggested to be its antecedents. In our study, these flow antecedents are viewed as sets of signs or modes that can be orchestrated in order to create different levels of perceived challenge for brand website users. We consider that multimodality can be used for a better optimization of these various meanings as expressed by modes that are "essentially different in nature" (Bateman, 2008, p.54). Also, we refer to the online marketing outcomes of the online flow experience, in order to outline its importance and its implications in the online environment. The main conclusion of our paper is that multimodality can be used by scholars and practitioners as a premise to induce online flow on a brand website, by optimizing the sets of signs or modes. Originality – our contribution consists of an interdisciplinary approach. We use concepts from semiotics, positive psychology, and online marketing for a better understanding of how marketers and semioticians can positively influence online customer behaviour. (shrink)

The analysis of multimodal argumentation in advertising is a crucial and problematic area of research. While its importance is growing in a time characterized by images and pictorial messages, the methods used for interpreting and reconstructing the structure of arguments expressed through verbal and visual means capture only isolated dimensions of this complex phenomenon. This paper intends to propose and illustrate a methodology for the reconstruction and analysis of “double-mode” arguments in advertisements, combining the instruments developed in social semiotics, (...) pragmatics, and argumentation theory. An advertisement is processed through a five-step path. The analysis of its context, text genre, and images leads to a first representation of the messages that it encodes both pictorially and verbally (step 1). These first semantic representations are further enriched by including their polyphonic articulations and presuppositions (step 2), their explicatures (step 3), and their dialogical functions and illocutionary forces (step 4). These pragmatic steps retrieve the commitment structure of the ad, which allows a further argument analysis conducted through argumentation schemes (step 5). (shrink)

Everyday media consumption leaves no doubt that the perception of moving images from various media is characterized by experience and understanding. Corresponding research in this field has shown that the stimulus patterns flooding in on us are not only processed mentally, but also bodily. Building on this, the following study argues that incoming stimuli are processed not only visually, but multimodally, with all senses, and moreover affectively. The classical binding of a sensory organ to a medium, on whose delimitation the (...) common understanding of multimodality is based, is thus abolished. Can arguments for this assumption be found? Building on the research of cultural anthropologist Ernst Cassirer and his successors, the two American philosophers Susanne K. Langer and John M. Krois, and supported by research from the life sciences in which, in parallel, the arts also play an important role in understanding image formation, as in the work of Heinz Werner and, following on from this, in the more recent research of the child psychologist Daniel N. Stern in collaboration with mirror neuron research by the group of neuroscientists around Giacomo Rizzolatti in Parma, justifications for this will be provided. What the multimodal processing of film images is concretely based on will finally be made clear by means of an exemplary analysis of a video film by the Swiss artist Pipilotti Rist. It aims to prove, that the prerequisites for the thesis can only lie in the fact, that both the modes of perception (visual, tactile, auditory, olfactory, gustatory) as well as the medially conditioned modes of image formation are based on non-discursive, formal-abstract structures that are also processed affectively. At the same time, the analysis thus shows how the selection and composition of the vital experienced forms can provide information about the socio-culturally shaped self-image of the producer and the viewer. (shrink)

Mental imagery is early perceptual processing that is not triggered by corresponding sensory stimulation in the relevant sense modality. Multimodal mental imagery is early perceptual processing that is triggered by sensory stimulation in a different sense modality. For example, when early visual or tactile processing is triggered by auditory sensory stimulation, this amounts to multimodal mental imagery. Pulling together philosophy, psychology and neuroscience, I will argue in this paper that multimodal mental imagery plays a crucial role in (...) our engagement with musical works. Engagement with musical works is normally a multimodal phenomenon, where we get input from a number of sense modalities. But even if we screen out any input that is not auditory, multimodal mental imagery will still play an important role that musicians and composers often actively rely on. (shrink)

With a Call for Essays, the special issue Multimodality sought contributions that accept not only the material but also the body-bound dependence of media perception and understanding. To this end, contributions were included that shed light on both the structural and signifying potential of artistic works through multimodal analysis. Particular attention was paid to contributions that clarify how the structural features - the modes - of the arts, their perception, and their signifying potential in terms of content are interrelated (...) and how they are to be understood in communicative and thus socioculturally relevant terms. Thus, in addition to neuroscientific contributions, those from cultural anthropology, art history, image and art studies, and literary studies were included. -/- Articles -/- 8 Martina Sauer and Christiane Wagner - Introduction -/- 11 Giada Lombardi and Giuseppe Di Cesare - From Neuroscience to Art: The Role of ‘Vitality Forms’ in the Investigation of Multimodality -/- 25 Martina Sauer - Abstract - Affective – Multimodal: Interaction between Medium and Perception of Moving Images from the Viewpoint of Cassirer’s, Langer’s and Krois’ Embodiment Theories -/- 49 Göran Sonesson - The Secrets of Plastic Language Revealed: Multimodality, Polysemiosis, and Iconicity -/- 71 Natalia Igl - Making Sense of the Material: Multisensory Reader Involvement in Contemporary Multimodal Novels -/- 87 J. Philippe Thompson - Art is All: Proust’s Life Lessons Through Perception and Art -/- 107 Hans Dieter Huber - Philippe Parreno and the Exhibition as a Multimodal Aesthetic Experience -/- 135 Jörg U. Lensing - The Performing Human Being in a Media Interaction Space: Multi- and intermodal productions by the Düsseldorf Theater der Klänge (Theatre of Sounds) . (shrink)

Recent advances in neuroimaging technologies have rendered multimodal analysis of operators’ cognitive processes in complex task settings and environments increasingly more practical. In this exploratory study, we utilized optical brain imaging and mobile eye tracking technologies to investigate the behavioral and neurophysiological differences among expert and novice operators while they operated a human-machine interface in normal and adverse conditions. In congruence with related work, we observed that experts tended to have lower prefrontal oxygenation and exhibit gaze patterns that are (...) better aligned with the optimal task sequence with shorter fixation durations as compared to novices. These trends reached statistical significance only in the adverse condition where the operators were prompted with an unexpected error message. Comparisons between hemodynamic and gaze measures before and after the error message indicated that experts’ neurophysiological response to the error involved a systematic increase in bilateral dorsolateral prefrontal cortex (dlPFC) activity accompanied with an increase in fixation durations, which suggests a shift in their attentional state, possibly from routine process execution to problem detection and resolution. The novices’ response was not as strong as that of experts, including a slight increase only in the left dlPFC with a decreasing trend in fixation durations, which is indicative of visual search behavior for possible cues to make sense of the unanticipated situation. A linear discriminant analysis model capitalizing on the covariance structure among hemodynamic and eye movement measures could distinguish experts from novices with 91% accuracy. Despite the small sample size, the performance of the linear discriminant analysis combining eye fixation and dorsolateral oxygenation measures before and after an unexpected event suggests that multimodal approaches may be fruitful for distinguishing novice and expert performance in similar neuroergonomic applications in the field. (shrink)

Recent advances in neuroimaging technologies have rendered multimodal analysis of operators’ cognitive processes in complex task settings and environments increasingly more practical. In this exploratory study, we utilized optical brain imaging and mobile eye tracking technologies to investigate the behavioral and neurophysiological differences among expert and novice operators while they operated a human-machine interface in normal and adverse conditions. In congruence with related work, we observed that experts tended to have lower prefrontal oxygenation and exhibit gaze patterns that are (...) better aligned with the optimal task sequence with shorter fixation durations as compared to novices. These trends reached statistical significance only in the adverse condition where the operators were prompted with an unexpected error message. Comparisons between hemodynamic and gaze measures before and after the error message indicated that experts’ neurophysiological response to the error involved a systematic increase in bilateral dorsolateral prefrontal cortex (dlPFC) activity accompanied with an increase in fixation durations, which suggests a shift in their attentional state, possibly from routine process execution to problem detection and resolution. The novices’ response was not as strong as that of experts, including a slight increase only in the left dlPFC with a decreasing trend in fixation durations, which is indicative of visual search behavior for possible cues to make sense of the unanticipated situation. A linear discriminant analysis model capitalizing on the covariance structure among hemodynamic and eye movement measures could distinguish experts from novices with 91% accuracy. Despite the small sample size, the performance of the linear discriminant analysis combining eye fixation and dorsolateral oxygenation measures before and after an unexpected event suggests that multimodal approaches may be fruitful for distinguishing novice and expert performance in similar neuroergonomic applications in the field. (shrink)

It has been repeatedly suggested that synesthesia is intricately connected with unusual ways of exercising one’s mental imagery, although it is not always entirely clear what the exact connection is. My aim is to show that all forms of synesthesia are forms of (often very different kinds of) mental imagery and, further, if we consider synesthesia to be a form of mental imagery, we get significant explanatory benefits, especially concerning less central cases of synesthesia where the inducer is not sensory (...) stimulation. (shrink)

Is it true that there is an analogy between modes of creation and such of perception? Respective to the cultural anthropological research of Ernst Cassirer, Susanne K. Langer and John M. Krois and by the analysis of a tape of the Swiss video-artist Pipilotti Rist this initial thesis of Formal Aesthetics shall be supported. - I - -/- Lässt sich die Behauptung stützen, dass zwischen Gestaltungsweisen und Wahrnehmungsweisen eine Analogie besteht? Aufbauend auf den kulturanthropologischen Forschungen von Ernst Cassirer, Susanne K. (...) Langer und John M. Krois sowie der Analyse eines Tapes der Schweizer Videokünstlerin Pipilotti Rist gilt es diese ursprünglich von der formalen Ästhetik aufgeworfene These zu untermauern. (shrink)

This paper analyzes transduction as an action of transposing information from one mode to another within the communication process and its implications in terms of meaning and coherence of a multimodal message. First, I discuss the multimodal method and its conjunction with some key concepts such as: sign, meaning, mode, transduction. Secondly, I approach transduction as an essential method of translating messages across the media variety, describing my interdisciplinary approach – that brings together semiotics and communications – and (...) proposing a framework of explanation for transduction in the field of advertising. Drawing from a previous model (Culache 2015), I illustrate the way transduction takes place and identify its meaning-making issues while introducing the concept of ‘dominant mode.’. (shrink)

Argumentation is the gear of a cognitive process of reconstruction when it manifests itself in the discursive proximal space. The cognitive-semiotic perspective of multimodal argumentation suggests that, depending on the knowledge, the codes and signs employed (modes) and the context, certain multimodal structures can lead some audiences towards an accurate mental representation of argumentation, without the need forany formal standardized and verbal structure. We are able to argue with other(s) effectively without the need for a verbal translation of (...) the contents in a space that includes the semiotic interpretation of the message. (shrink)

La presente investigación tuvo el objetivo de pilotar un instrumento de recolección de información acerca de los hábitos de lectura en tutores de niños que cursan estudios preescolares. Concretamente, se realizó un pilotaje de una encuesta en la que se abordó a 40 tutores que tuviesen a cargo niños en educación preescolar. Algunos de los principales hallazgos fueron que los tutores dicen invertir más tiempo en la lectura escolar que en lectura por entretenimiento. Junto a ello, la lectura en donde (...) acompañan a los niños es poco habitual. Los tutores comentan invertir más tiempo en leer los grupos de chat de WhatsApp que en leer junto a sus niños. Estos resultados pueden evidenciar el por qué la lectura está considerada una tarea escolar y no una actividad de entretenimiento y gozo. Esta línea de investigación nos permitirá desarrollar estrategias pedagógicas que promuevan mejores hábitos de lectura, especialmente en tiempos de pandemia donde la labor de la familia en el aprendizaje de los niños queda más evidente. (shrink)

If the sight of cortically blind people were restored, could they visually recognize a cube or a sphere? This is Molyneux’s question. I argue that the answer to this question depends on the specificities of the mental setup of these cortically blind people. Some cortically blind people have (sometimes quite vivid) visual imagery. Others don’t. The answer to Molyneux’s question depends on whether the blind subjects have had visual imagery before their sight was restored. If they did, the answer to (...) Molyneux’s question is yes, if they didn’t, the answer is no. There is no generic answer to Molyneux’s question. (shrink)

Disorders of consciousness (DoC)—that is, unresponsive wakefulness syndrome/vegetative state and minimally conscious state—are debilitating conditions for which no reliable markers of consciousness recovery have yet been identified. Evidence points to the GABAergic system being altered in DoC, making it a potential target as such a marker.

This volume brings together the advanced research results obtained by the European COST Action 2102: “Cross Modal Analysis of Verbal and Nonverbal Communication”. The research published in this book was discussed at the 3rd joint EUCOGII-COST 2102 International Training School entitled “Toward Autonomous, Adaptive, and Context-Aware Multimodal Interfaces: Theoretical and Practical Issues ” and held in Caserta, Italy, on March 15-19, 2010.

Higher animals need to identify and track material objects because they depend on interactions with them for nutrition, reproduction, and social interaction. This paper investigates the perception of material objects. It argues, first, that material objects are tagged, in all five external senses, as bearers of the features detected by them. This happens through a perceptual process, here entitled Generalized Completion, which creates the appearance of objects that have properties that transcend the activation of sensory receptors. The paper shows, secondly, (...) that material objects are privileged subjects for perceived motion and interaction. That is, they are perceived as subjects for these properties while their parts seem to be subjects only derivatively. Material objects are the only perceptual subjects that are both multisensory and privileged. (shrink)

The aim of this paper is to raise some serious worries about anti-representationalism: the recently popular view according to which there are no perceptual representations. Although anti-representationalism is more and more popular, I will argue that we have strong empirical reasons for mistrusting it. More specifically, I will argue that it is inconsistent with some important empirical findings about dorsal perception and about the multimodality of perception.

In the case of ventriloquism, seeing the movement of the ventriloquist dummy’s mouth changes your experience of the auditory location of the vocals. Some have argued that cases like ventriloquism provide evidence for the view that at least some of the content of perception is fundamentally multimodal. In the ventriloquism case, this would mean your experience has constitutively audio-visual content (not just a conjunction of an audio content and visual content). In this paper, I argue that cases like ventriloquism (...) do not in fact warrant that conclusion. I then try to make sense of crossmodal cases without appealing to fundamentally multimodal content. (shrink)

This is the editors' Introduction to a special issue of the journal, Multisensory Research. European philosophers of the modern period found multisensory perception to be impossible because they thought that perceptual ideas are defined by how they are experienced. Under this conception, the individual modalities are determinables of ideas—just as colour is a determinable that embraces red and blue, so also the visual is a determinable that embraces colour and (visually experienced) shape. Since no idea is experienced as, for example, (...) both visual and auditory, there can be no such thing as audiovisual perception. This conception of modality is not directly contested, but a variety of perceptual phenomena are listed that could raise interesting questions if treated as multimodal in origin. (shrink)

Many philosophers take experience to be an essential aspect of perceptual justification. I argue against a specific variety of such an experientialist view, namely, the Looks View of perceptual justification, according to which our visual beliefs are mediately justified by beliefs about the way things look. I describe three types of cases that put pressure on the idea that perceptual justification is always related to looks-related reasons: unsophisticated cognizers, multimodal identification, and amodal completion. I then provide a tentative diagnosis (...) of what goes wrong in the Looks View: it ascribes a specific epistemic role to beliefs about looks that is actually fulfilled by subpersonal perceptual processes. (shrink)

Molyneux asked whether a newly sighted person could distinguish a sphere from a cube by sight alone, given that she was antecedently able to do so by touch. This, we contend, is a question about general ideas. To answer it, we must ask (a) whether spatial locations identified by touch can be identified also by sight, and (b) whether the integration of spatial locations into an idea of shape persists through changes of modality. Posed this way, Molyneux’s Question goes substantially (...) beyond question (a), about spatial locations, alone; for a positive answer to (a) leaves open whether a perceiver might cross-identify locations, but not be able to identify the shapes that collections of locations comprise. We further emphasize that MQ targets general ideas so as to distinguish it from corresponding questions about experiences of shape and about the property of tangible (vs. visual) shape. After proposing a generalized formulation of MQ, we extend earlier work (“Many Molyneux Questions,” Australasian Journal of Philosophy 2020) by showing that MQ does not admit a single answer across the board. Some integrative data-processes transfer across modalities; others do not. Seeing where and how such transfer succeeds and fails in individual cases has much to offer to our understanding of perception and its modalities. (shrink)

Scholarly opinions vary on what language is, how it evolved, and from where or what it evolved. Long considered uniquely human, today scholars argue for evolutionary continuity between human language and animal communication systems. But while it is generally recognized that language is an evolving communication system, scholars continue to debate from which species language evolved, and what behavioral and cognitive features are the precursors to human language. To understand the nature of protolanguage, some look for homologs in gene functionality, (...) brain areas, or anatomical structures such as the supralaryngeal vocal tract; others point toward primates, their gestural, vocal, multimodal, and in later evolving hominins also their pantomimic communication systems; and still others draw parallels between the musicality that characterizes language and the pitch found in the numerous sounds produced by animals. Accordingly, protolanguage theories today are multiple and diverse, and protolanguages might have also been diverse. This special issue on Evolving (Proto)Language/s for Lingua bundles several of the protolanguage theories that were put forward at the sixth edition of the Ways to Protolanguage conference series, held at the Calouste Gulbenkian Foundation in Lisbon, in 2019. Not aimed at surveying all the different ways there are to conceptualize, study, and model protolanguage/s, this issue provides interested readers with good overviews on the role played in current theorizing on protolanguage/s by (paleo)anthropology, genetics, physiology, developmental, evolutionary, ecological, and pragmatic research lines. (shrink)

This report highlights and explores five questions that arose from the multisensory integration workshop at the University of Toronto on May 9th and 10th, 2014: 1. What Is Multisensory Integration? 2. Do Multisensory Percepts Involve Emergent Features? 3. What Can Multisensory Processing Tell Us about Multisensory Awareness? 4. Is Language Processing a Special Kind of Multisensory Integration? 5. What Is the Purpose of Multisensory Integration?

This is an excerpt from a report on the Temporal Experience Workshop at the University of Toronto in May of 2013. This portion of the report explores the question: Do we have one central clock for time, or different clocks for each sense modality?

The predictive coding (PC) theory of attention identifies attention with the optimization of the precision weighting of prediction error. Here we provide some challenges for this identification. On the one hand, the precision weighting of prediction error is too broad a phenomenon to be identified with attention because such weighting plays a central role in multimodal integration. Cases of crossmodal illusions such as the rubber hand illusion and the McGurk effect involve the differential precision weighting of prediction error, yet (...) attention does not shift as one would predict. On the other hand, the precision weighting of prediction error is too narrow a phenomenon to be identified with attention, because it cannot accommodate the full range of attentional phenomena. We review criticisms that PC cannot account for volitional attention and affect-biased attention, and we propose that it may not be able to account for feature-based and intellectual attention. (shrink)

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: What is multisensory integration?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: Do multisensory percepts involve emergent features?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: What can multisensory processing tell us about multisensory awareness?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: What is the purpose of multisensory integration?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: Is language processing a special kind of multisensory integration?

Long-standing neglect of the chemical senses in the philosophy of perception is due, mostly, to their being regarded as ‘lower’ senses. Smell, taste, and chemically irritated touch are thought to produce mere bodily sensations. However, empirically informed theories of perception can show how these senses lead to perception of objective properties, and why they cannot be treated as special cases of perception modelled on vision. The senses of taste, touch, and smell also combine to create unified perceptions of flavour. The (...) nature of these multimodal experiences and the character of our awareness of them puts pressure on the traditional idea that each episode of perception goes one or other of the five senses. Thus, the chemical senses, far from being peripheral to the concerns of the philosophy of perception, may hold important clues to the multisensory nature of perception in general. (shrink)

We investigate an enrichment of the propositional modal language L with a "universal" modality ■ having semantics x ⊧ ■φ iff ∀y(y ⊧ φ), and a countable set of "names" - a special kind of propositional variables ranging over singleton sets of worlds. The obtained language ℒ $_{c}$ proves to have a great expressive power. It is equivalent with respect to modal definability to another enrichment ℒ(⍯) of ℒ, where ⍯ is an additional modality with the semantics x ⊧ ⍯φ (...) iff Vy(y ≠ x → y ⊧ φ). Model-theoretic characterizations of modal definability in these languages are obtained. Further we consider deductive systems in ℒ $_{c}$ . Strong completeness of the normal ℒ $_{c}$ logics is proved with respect to models in which all worlds are named. Every ℒ $_{c}$ -logic axiomatized by formulae containing only names (but not propositional variables) is proved to be strongly frame-complete. Problems concerning transfer of properties ([in]completeness, filtration, finite model property etc.) from ℒ to ℒ $_{c}$ are discussed. Finally, further perspectives for names in multimodal environment are briefly sketched. (shrink)

Human language, hominin tool production modes, and multimodal communications systems of primates and other animals are currently well-studied for how they display compositionality or combinatoriality. In all cases, the former is defined as a kind of hierarchical nesting and the latter as a lack thereof. In this article, I extend research on combinatoriality and compositionality further to investigations of everyday primate skills. Daily locomotion modes as well as behaviors associated with subsistence practices, hygiene, or body modification rely on the (...) hierarchical nesting of different behavioral and cognitive actions into complex skills. I introduce a scheme which uses hierarchical organization to differentiate combinatorial from compositional skills. Combinatorial skills are defined either as aggregational or linearly hierarchical depending on whether the skill occurs momentarily in space or unfolds sequentially over time. Compositional skills are defined either as nested or interactionally hierarchical depending on whether the skill results in new constructs or in new interactions between existing constructs. The methodology I propose combines epistemological hierarchy theory with data from primatological field research and experimental and comparative psychological research and provides a means to integrate current constructionist and extended views on cognition and action with older research on behavioral logics in psychology and operational chain thinking in anthropology. The approach furthermore synchronizes with ongoing research on teleonomy, intentionality, and creativity. -/- . (shrink)

I argue that the neural realizers of experiences of trying are not distinct from the neural realizers of actual trying . I then ask how experiences of trying might relate to the perceptual experiences one has while acting. First, I assess recent zombie action arguments regarding conscious visual experience, and I argue that contrary to what some have claimed, conscious visual experience plays a causal role for action control in some circumstances. Second, I propose a multimodal account of the (...) experience of acting. According to this account, the experience of acting is a temporally extended, co-conscious collection of agentive and perceptual experiences, functionally integrated and structured both by multimodal perceptual processing as well as by what an agent is, at the time, trying to do. (shrink)

There is converging evidence that high doses of hallucinogenic drugs can produce significant alterations of self-experience, described as the dissolution of the sense of self and the loss of boundaries between self and world. This article discusses the relevance of this phenomenon, known as “drug-induced ego dissolution (DIED)”, for cognitive neuroscience, psychology and philosophy of mind. Data from self-report questionnaires suggest that three neuropharmacological classes of drugs can induce ego dissolution: classical psychedelics, dissociative anesthetics and agonists of the kappa opioid (...) receptor (KOR). While these substances act on different neurotransmitter receptors, they all produce strong subjective effects that can be compared to the symptoms of acute psychosis, including ego dissolution. It has been suggested that neuroimaging of DIED can indirectly shed light on the neural correlates of the self. While this line of inquiry is promising, its results must be interpreted with caution. First, neural correlates of ego dissolution might reveal the necessary neurophysiological conditions for the maintenance of the sense of self, but it is more doubtful that this method can reveal its minimally sufficient conditions. Second, it is necessary to define the relevant notion of self at play in the phenomenon of DIED. This article suggests that DIED consists in the disruption of subpersonal processes underlying the “minimal” or “embodied” self, i.e., the basic experience of being a self rooted in multimodal integration of self-related stimuli. This hypothesis is consistent with Bayesian models of phenomenal selfhood, according to which the subjective structure of conscious experience ultimately results from the optimization of predictions in perception and action. Finally, it is argued that DIED is also of particular interest for philosophy of mind. On the one hand, it challenges theories according to which consciousness always involves self-awareness. On the other hand, it suggests that ordinary conscious experience might involve a minimal kind of self-awareness rooted in multisensory processing, which is what appears to fade away during DIED. (shrink)

Suppose that you are at a live jazz show. The drummer begins a solo. You see the cymbal jolt and you hear the clang. But in addition seeing the cymbal jolt and hearing the clang, you are also aware that the jolt and the clang are part of the same event. Casey O’Callaghan (forthcoming) calls this awareness “intermodal feature binding awareness.” Psychologists have long assumed that multimodal perceptions such as this one are the result of a subpersonal feature binding (...) mechanism (see Vatakis and Spence, 2007, Kubovy and Schutz, 2010, Pourtois et al., 2000, and Navarra et al., 2012). I present new evidence against this. I argue that there is no automatic feature binding mechanism that couples features like the jolt and the clang together. Instead, when you experience the jolt and the clang as part of the same event, this is the result of an associative learning process. The cymbal’s jolt and the clang are best understood as a single learned perceptual unit, rather than as automatically bound. I outline the specific learning process in perception called “unitization,” whereby we come to “chunk” the world into multimodal units. Unitization has never before been applied to multimodal cases. Yet I argue that this learning process can do the same work that intermodal binding would do, and that this issue has important philosophical implications. Specifically, whether we take multimodal cases to involve a binding mechanism or an associative process will have impact on philosophical issues from Molyneux’s question to the question of how active or passive we consider perception to be. (shrink)

It is commonly argued that values “fill the logical gap” of underdetermination of theory by evidence, namely, values affect our choice between two or more theories that fit the same evidence. The underdetermination model, however, does not exhaust the roles values play in evidential reasoning. I introduce WAVE – a novel account of the logical relations between values and evidence. WAVE states that values influence evidential reasoning by adjusting evidential weights. I argue that the weight-adjusting role of values is distinct (...) from their underdetermination gap-filling role. Values adjust weights in three ways. First, values affect our trust in the testimony of others. Second, values influence the evidential thresholds required for justified epistemic judgments. Third, values influence the relative weight of a certain type of evidence within a body of multimodal discordant evidence. WAVE explains, from an epistemic perspective, rather than psychological, how smokers, for example, can find the same evidence about the dangers of smoking less persuasive than non-smokers. WAVE allows for a wider effect of values on our accepted scientific theories and beliefs than the effect for which the underdetermination model allows alone; therefore, science studies scholars must consider WAVE in their research and analysis of evidential case studies. (shrink)

In this work we propose an encoding of Reiter’s Situation Calculus solution to the frame problem into the framework of a simple multimodal logic of actions. In particular we present the modal counterpart of the regression technique. This gives us a theorem proving method for a relevant fragment of our modal logic.

In this paper I develop a novel account of the phenomenality of language by focusing on characteristics of perceived speech. I explore the extent to which the spoken word can be said to have a horizonal structure similar to that of spatiotemporal objects: our perception of each is informed by habitual associations and expectations formed through past experiences of the object or word and other associated objects and experiences. Specifically, the horizonal structure of speech in use can fruitfully be compared (...) to that of a tool in use. The comparison suggests an account of our linguistic faculty as continuous with more foundational faculties of perception and action. I provide empirical corroboration of this account by drawing on recent neuroimaging studies of the multimodal, sensorimotor bases of speech comprehension. I then discuss how such an understanding of our linguistic ability helps advocates of embodied, non-representationalist accounts of cognition respond to a common objection. Critics grant that embodied approaches may be adequate to account for lower-level, online modes of cognition, such as perception and action, which directly engage their object. But they question whether such approaches can “scale up” to higher modes of cognition, such as imagination, memory, thought, and language, which can entertain absent, non-existent, or abstract objects. By providing a plausible account of the continuity of lower cognition and language-involving cognition, my approach responds to this objection, at least where language is concerned. (shrink)