This research article examines the potential of multimodal techniques in promoting inclusive practices in the English language classroom, delving into the multidisciplinary fields of English Language Teaching (ELT) and inclusive education. This study intends to investigate how multimodal resources, such as visual aids, technology, and creative activities, may be effectively integrated to meet the various learning needs of students using a pedagogical lens. This paper examines the theoretical underpinnings and practical implications of using multimodal approaches in ELT settings, shedding light (...) on their potential to promote inclusivity, engagement, and language acquisition among learners with diverse abilities and backgrounds. It does this through a thorough literature review and analysis of pedagogical frameworks. (shrink)

The analysis of multimodal argumentation in advertising is a crucial and problematic area of research. While its importance is growing in a time characterized by images and pictorial messages, the methods used for interpreting and reconstructing the structure of arguments expressed through verbal and visual means capture only isolated dimensions of this complex phenomenon. This paper intends to propose and illustrate a methodology for the reconstruction and analysis of “double-mode” arguments in advertisements, combining the instruments developed in social semiotics, pragmatics, (...) and argumentation theory. An advertisement is processed through a five-step path. The analysis of its context, text genre, and images leads to a first representation of the messages that it encodes both pictorially and verbally (step 1). These first semantic representations are further enriched by including their polyphonic articulations and presuppositions (step 2), their explicatures (step 3), and their dialogical functions and illocutionary forces (step 4). These pragmatic steps retrieve the commitment structure of the ad, which allows a further argument analysis conducted through argumentation schemes (step 5). (shrink)

The aim of this paper is to argue that our experience of artworks is normally multimodal. It is the result of perceptual processing in more than one sense modality. In other words, multimodal experience of art is not the exception; it is the rule. I use the example of music in order to demonstrate the various ways in which the visual sense modality influences the auditory processing of music and conclude that this should make us look more closely at our (...) practices of engaging with artworks. (shrink)

It is common to characterize pain with touch-related terms, like ‘cutting’, ‘pressing’, ‘sharp’, and ‘pulsing’, or temperature-related terms, like ‘hot’ or ‘burning’. This suggests that many pains are phenomenally multimodal because they are experienced as having some tactile-like or thermal-like character. The goal of this chapter is to investigate the structure of phenomenally multimodal pain experiences. It is argued that the usual accounts of multimodal structure proposed in investigations regarding exteroceptive experiences cannot be plausibly applied to multimodal experiences of pain. (...) Instead, an alternative framework is proposed which characterizes the structure of tactile-like and thermal-like pains by referring to the notion of crossmodal correspondences. (shrink)

What is the correct procedure for determining the contents of perception? Philosophers tackling this question increasingly rely on empirically-oriented procedures in order to reach an answer. I argue that this constitutes an improvement over the armchair methodology constitutive of phenomenal contrast cases, but that there is a crucial respect in which current empirical procedures remain limited: they are unimodal in nature, wrongly treating the senses as isolatable faculties. I thus have two aims: first, to motivate a reorientation of the admissible (...) contents debate into a multimodal framework, charting its various significances. The second is to explore whether any experimental studies of multimodal perception support a so-called Liberal (or ‘high-level’ or ‘rich’) account of perception’s admissible contents. I conclude that the McGurk effect and the ventriloquist effect are both explicable without the postulation of high-level content, but that at least one multimodal experimental paradigm may necessitate such content: the rubber hand illusion. One upshot of this argument is that Conservatives who claim that the Liberal view intolerably broadens the scope of perceptual illusions, particularly from the perspective of perceptual psychology, should pursue other arguments against that view. (shrink)

This paper presents an analytical framework for analyzing how multimodal resources of emotion expression are semiotically materialized in discursive interactions specific to political discourse. Interested in how political personae are emotionally constructed through multimodal meaning-making practices, our analysis model assumes an interdisciplinary perspective, which integrates facial expression analysis – using FaceReader™ software –, the theory of emotional arcs and bodily actions (hand gestures) analysis that express emotions, in the analytical framework of multimodality. The results show how the multimodal choices (...) that political actors make during discursive interactions allow them to build their political brand and make connections with the audience on an emotional level. (shrink)

The aim of this paper is to propose a new approach in the form of multimodality as a semiotic method that can be used by marketers and semioticians to induce online flow, a psychological state, on a brand website. First, we refer to multimodality as a semiotic analysis that can be used for a better optimization of semiotic resource sets in meaning-making, and we distinguish it from another similar concept: multimedia. Second, after a critical literature review, we address (...) the flow construct - a state of mind sometimes experienced by people who are deeply involved and immersed in some event, object or activity. Previous quantitative studies show a correlation between online flow experience and other variables such as design, content, sound and layout that are suggested to be its antecedents. In our study, these flow antecedents are viewed as sets of signs or modes that can be orchestrated in order to create different levels of perceived challenge for brand website users. We consider that multimodality can be used for a better optimization of these various meanings as expressed by modes that are "essentially different in nature" (Bateman, 2008, p.54). Also, we refer to the online marketing outcomes of the online flow experience, in order to outline its importance and its implications in the online environment. The main conclusion of our paper is that multimodality can be used by scholars and practitioners as a premise to induce online flow on a brand website, by optimizing the sets of signs or modes. Originality – our contribution consists of an interdisciplinary approach. We use concepts from semiotics, positive psychology, and online marketing for a better understanding of how marketers and semioticians can positively influence online customer behaviour. (shrink)

This is an excerpt of a report that highlights and explores five questions which arose from The Unity of Consciousness and Sensory Integration conference at Brown University in November of 2011. This portion of the report explores the question: Are some of the basic units of consciousness multimodal?

This is an excerpt from a report that highlights and explores five questions which arose from the workshop on perceptual learning and perceptual recognition at the University of Toronto, Mississauga on May 10th and 11th, 2012. This excerpt explores the question: What are the origins of multimodal associations?

Mental imagery is early perceptual processing that is not triggered by corresponding sensory stimulation in the relevant sense modality. Multimodal mental imagery is early perceptual processing that is triggered by sensory stimulation in a different sense modality. For example, when early visual or tactile processing is triggered by auditory sensory stimulation, this amounts to multimodal mental imagery. Pulling together philosophy, psychology and neuroscience, I will argue in this paper that multimodal mental imagery plays a crucial role in our engagement with (...) musical works. Engagement with musical works is normally a multimodal phenomenon, where we get input from a number of sense modalities. But even if we screen out any input that is not auditory, multimodal mental imagery will still play an important role that musicians and composers often actively rely on. (shrink)

From the perspective of distributed cognition I will stress how abduction is essentially multimodal, in that both data and hypotheses can have a full range of verbal and sensory representations, involving words, sights, images, smells, etc., but also kinesthetic – related to the ability to sense the position and location and orientation and movement of the body and its parts – and motor experiences and other feelings such as pain, and thus all sensory modalities. The presence of kinesthetic and motor (...) aspects demonstrates that abductive reasoning can be manipulative. We can also see, in this regard, how implicit factors take part in the abductive procedure, which consequently acquires the character of a kind of “thinking through doing”. This paper further describes 1) the fact that hypotheses in science can be built through different cognitive mediators and so they can also model the same cognitive aspect in different ways; how they can be carriers/ producers of knowledge in a multimodal way; 2) the problem of the possible non-explanatory and instrumental nature of abductive reasoning and the analysis of the consequences for induction; 3) the role of manipulative abduction in building new evidence/ experiments and how they trigger smart inductive inferences. (shrink)

Everyday media consumption leaves no doubt that the perception of moving images from various media is characterized by experience and understanding. Corresponding research in this field has shown that the stimulus patterns flooding in on us are not only processed mentally, but also bodily. Building on this, the following study argues that incoming stimuli are processed not only visually, but multimodally, with all senses, and moreover affectively. The classical binding of a sensory organ to a medium, on whose delimitation the (...) common understanding of multimodality is based, is thus abolished. Can arguments for this assumption be found? Building on the research of cultural anthropologist Ernst Cassirer and his successors, the two American philosophers Susanne K. Langer and John M. Krois, and supported by research from the life sciences in which, in parallel, the arts also play an important role in understanding image formation, as in the work of Heinz Werner and, following on from this, in the more recent research of the child psychologist Daniel N. Stern in collaboration with mirror neuron research by the group of neuroscientists around Giacomo Rizzolatti in Parma, justifications for this will be provided. What the multimodal processing of film images is concretely based on will finally be made clear by means of an exemplary analysis of a video film by the Swiss artist Pipilotti Rist. It aims to prove, that the prerequisites for the thesis can only lie in the fact, that both the modes of perception (visual, tactile, auditory, olfactory, gustatory) as well as the medially conditioned modes of image formation are based on non-discursive, formal-abstract structures that are also processed affectively. At the same time, the analysis thus shows how the selection and composition of the vital experienced forms can provide information about the socio-culturally shaped self-image of the producer and the viewer. (shrink)

With a Call for Essays, the special issue Multimodality sought contributions that accept not only the material but also the body-bound dependence of media perception and understanding. To this end, contributions were included that shed light on both the structural and signifying potential of artistic works through multimodal analysis. Particular attention was paid to contributions that clarify how the structural features - the modes - of the arts, their perception, and their signifying potential in terms of content are interrelated (...) and how they are to be understood in communicative and thus socioculturally relevant terms. Thus, in addition to neuroscientific contributions, those from cultural anthropology, art history, image and art studies, and literary studies were included. -/- Articles -/- 8 Martina Sauer and Christiane Wagner - Introduction -/- 11 Giada Lombardi and Giuseppe Di Cesare - From Neuroscience to Art: The Role of ‘Vitality Forms’ in the Investigation of Multimodality -/- 25 Martina Sauer - Abstract - Affective – Multimodal: Interaction between Medium and Perception of Moving Images from the Viewpoint of Cassirer’s, Langer’s and Krois’ Embodiment Theories -/- 49 Göran Sonesson - The Secrets of Plastic Language Revealed: Multimodality, Polysemiosis, and Iconicity -/- 71 Natalia Igl - Making Sense of the Material: Multisensory Reader Involvement in Contemporary Multimodal Novels -/- 87 J. Philippe Thompson - Art is All: Proust’s Life Lessons Through Perception and Art -/- 107 Hans Dieter Huber - Philippe Parreno and the Exhibition as a Multimodal Aesthetic Experience -/- 135 Jörg U. Lensing - The Performing Human Being in a Media Interaction Space: Multi- and intermodal productions by the Düsseldorf Theater der Klänge (Theatre of Sounds) . (shrink)

Is it true that there is an analogy between modes of creation and such of perception? Respective to the cultural anthropological research of Ernst Cassirer, Susanne K. Langer and John M. Krois and by the analysis of a tape of the Swiss video-artist Pipilotti Rist this initial thesis of Formal Aesthetics shall be supported. - I - -/- Lässt sich die Behauptung stützen, dass zwischen Gestaltungsweisen und Wahrnehmungsweisen eine Analogie besteht? Aufbauend auf den kulturanthropologischen Forschungen von Ernst Cassirer, Susanne K. (...) Langer und John M. Krois sowie der Analyse eines Tapes der Schweizer Videokünstlerin Pipilotti Rist gilt es diese ursprünglich von der formalen Ästhetik aufgeworfene These zu untermauern. (shrink)

Recent advances in neuroimaging technologies have rendered multimodal analysis of operators’ cognitive processes in complex task settings and environments increasingly more practical. In this exploratory study, we utilized optical brain imaging and mobile eye tracking technologies to investigate the behavioral and neurophysiological differences among expert and novice operators while they operated a human-machine interface in normal and adverse conditions. In congruence with related work, we observed that experts tended to have lower prefrontal oxygenation and exhibit gaze patterns that are better (...) aligned with the optimal task sequence with shorter fixation durations as compared to novices. These trends reached statistical significance only in the adverse condition where the operators were prompted with an unexpected error message. Comparisons between hemodynamic and gaze measures before and after the error message indicated that experts’ neurophysiological response to the error involved a systematic increase in bilateral dorsolateral prefrontal cortex (dlPFC) activity accompanied with an increase in fixation durations, which suggests a shift in their attentional state, possibly from routine process execution to problem detection and resolution. The novices’ response was not as strong as that of experts, including a slight increase only in the left dlPFC with a decreasing trend in fixation durations, which is indicative of visual search behavior for possible cues to make sense of the unanticipated situation. A linear discriminant analysis model capitalizing on the covariance structure among hemodynamic and eye movement measures could distinguish experts from novices with 91% accuracy. Despite the small sample size, the performance of the linear discriminant analysis combining eye fixation and dorsolateral oxygenation measures before and after an unexpected event suggests that multimodal approaches may be fruitful for distinguishing novice and expert performance in similar neuroergonomic applications in the field. (shrink)

Recent advances in neuroimaging technologies have rendered multimodal analysis of operators’ cognitive processes in complex task settings and environments increasingly more practical. In this exploratory study, we utilized optical brain imaging and mobile eye tracking technologies to investigate the behavioral and neurophysiological differences among expert and novice operators while they operated a human-machine interface in normal and adverse conditions. In congruence with related work, we observed that experts tended to have lower prefrontal oxygenation and exhibit gaze patterns that are better (...) aligned with the optimal task sequence with shorter fixation durations as compared to novices. These trends reached statistical significance only in the adverse condition where the operators were prompted with an unexpected error message. Comparisons between hemodynamic and gaze measures before and after the error message indicated that experts’ neurophysiological response to the error involved a systematic increase in bilateral dorsolateral prefrontal cortex (dlPFC) activity accompanied with an increase in fixation durations, which suggests a shift in their attentional state, possibly from routine process execution to problem detection and resolution. The novices’ response was not as strong as that of experts, including a slight increase only in the left dlPFC with a decreasing trend in fixation durations, which is indicative of visual search behavior for possible cues to make sense of the unanticipated situation. A linear discriminant analysis model capitalizing on the covariance structure among hemodynamic and eye movement measures could distinguish experts from novices with 91% accuracy. Despite the small sample size, the performance of the linear discriminant analysis combining eye fixation and dorsolateral oxygenation measures before and after an unexpected event suggests that multimodal approaches may be fruitful for distinguishing novice and expert performance in similar neuroergonomic applications in the field. (shrink)

It has been repeatedly suggested that synesthesia is intricately connected with unusual ways of exercising one’s mental imagery, although it is not always entirely clear what the exact connection is. My aim is to show that all forms of synesthesia are forms of (often very different kinds of) mental imagery and, further, if we consider synesthesia to be a form of mental imagery, we get significant explanatory benefits, especially concerning less central cases of synesthesia where the inducer is not sensory (...) stimulation. (shrink)

This paper analyzes transduction as an action of transposing information from one mode to another within the communication process and its implications in terms of meaning and coherence of a multimodal message. First, I discuss the multimodal method and its conjunction with some key concepts such as: sign, meaning, mode, transduction. Secondly, I approach transduction as an essential method of translating messages across the media variety, describing my interdisciplinary approach – that brings together semiotics and communications – and proposing a (...) framework of explanation for transduction in the field of advertising. Drawing from a previous model (Culache 2015), I illustrate the way transduction takes place and identify its meaning-making issues while introducing the concept of ‘dominant mode.’. (shrink)

Argumentation is the gear of a cognitive process of reconstruction when it manifests itself in the discursive proximal space. The cognitive-semiotic perspective of multimodal argumentation suggests that, depending on the knowledge, the codes and signs employed (modes) and the context, certain multimodal structures can lead some audiences towards an accurate mental representation of argumentation, without the need forany formal standardized and verbal structure. We are able to argue with other(s) effectively without the need for a verbal translation of the contents (...) in a space that includes the semiotic interpretation of the message. (shrink)

La presente investigación tuvo el objetivo de pilotar un instrumento de recolección de información acerca de los hábitos de lectura en tutores de niños que cursan estudios preescolares. Concretamente, se realizó un pilotaje de una encuesta en la que se abordó a 40 tutores que tuviesen a cargo niños en educación preescolar. Algunos de los principales hallazgos fueron que los tutores dicen invertir más tiempo en la lectura escolar que en lectura por entretenimiento. Junto a ello, la lectura en donde (...) acompañan a los niños es poco habitual. Los tutores comentan invertir más tiempo en leer los grupos de chat de WhatsApp que en leer junto a sus niños. Estos resultados pueden evidenciar el por qué la lectura está considerada una tarea escolar y no una actividad de entretenimiento y gozo. Esta línea de investigación nos permitirá desarrollar estrategias pedagógicas que promuevan mejores hábitos de lectura, especialmente en tiempos de pandemia donde la labor de la familia en el aprendizaje de los niños queda más evidente. (shrink)

If the sight of cortically blind people were restored, could they visually recognize a cube or a sphere? This is Molyneux’s question. I argue that the answer to this question depends on the specificities of the mental setup of these cortically blind people. Some cortically blind people have (sometimes quite vivid) visual imagery. Others don’t. The answer to Molyneux’s question depends on whether the blind subjects have had visual imagery before their sight was restored. If they did, the answer to (...) Molyneux’s question is yes, if they didn’t, the answer is no. There is no generic answer to Molyneux’s question. (shrink)

Disorders of consciousness (DoC)—that is, unresponsive wakefulness syndrome/vegetative state and minimally conscious state—are debilitating conditions for which no reliable markers of consciousness recovery have yet been identified. Evidence points to the GABAergic system being altered in DoC, making it a potential target as such a marker.

This volume brings together the advanced research results obtained by the European COST Action 2102: “Cross Modal Analysis of Verbal and Nonverbal Communication”. The research published in this book was discussed at the 3rd joint EUCOGII-COST 2102 International Training School entitled “Toward Autonomous, Adaptive, and Context-Aware Multimodal Interfaces: Theoretical and Practical Issues ” and held in Caserta, Italy, on March 15-19, 2010.

Higher animals need to identify and track material objects because they depend on interactions with them for nutrition, reproduction, and social interaction. This paper investigates the perception of material objects. It argues, first, that material objects are tagged, in all five external senses, as bearers of the features detected by them. This happens through a perceptual process, here entitled Generalized Completion, which creates the appearance of objects that have properties that transcend the activation of sensory receptors. The paper shows, secondly, (...) that material objects are privileged subjects for perceived motion and interaction. That is, they are perceived as subjects for these properties while their parts seem to be subjects only derivatively. Material objects are the only perceptual subjects that are both multisensory and privileged. (shrink)

This is the editors' Introduction to a special issue of the journal, Multisensory Research. European philosophers of the modern period found multisensory perception to be impossible because they thought that perceptual ideas are defined by how they are experienced. Under this conception, the individual modalities are determinables of ideas—just as colour is a determinable that embraces red and blue, so also the visual is a determinable that embraces colour and (visually experienced) shape. Since no idea is experienced as, for example, (...) both visual and auditory, there can be no such thing as audiovisual perception. This conception of modality is not directly contested, but a variety of perceptual phenomena are listed that could raise interesting questions if treated as multimodal in origin. (shrink)

The aim of this paper is to raise some serious worries about anti-representationalism: the recently popular view according to which there are no perceptual representations. Although anti-representationalism is more and more popular, I will argue that we have strong empirical reasons for mistrusting it. More specifically, I will argue that it is inconsistent with some important empirical findings about dorsal perception and about the multimodality of perception.

In the case of ventriloquism, seeing the movement of the ventriloquist dummy’s mouth changes your experience of the auditory location of the vocals. Some have argued that cases like ventriloquism provide evidence for the view that at least some of the content of perception is fundamentally multimodal. In the ventriloquism case, this would mean your experience has constitutively audio-visual content (not just a conjunction of an audio content and visual content). In this paper, I argue that cases like ventriloquism do (...) not in fact warrant that conclusion. I then try to make sense of crossmodal cases without appealing to fundamentally multimodal content. (shrink)

Many philosophers take experience to be an essential aspect of perceptual justification. I argue against a specific variety of such an experientialist view, namely, the Looks View of perceptual justification, according to which our visual beliefs are mediately justified by beliefs about the way things look. I describe three types of cases that put pressure on the idea that perceptual justification is always related to looks-related reasons: unsophisticated cognizers, multimodal identification, and amodal completion. I then provide a tentative diagnosis of (...) what goes wrong in the Looks View: it ascribes a specific epistemic role to beliefs about looks that is actually fulfilled by subpersonal perceptual processes. (shrink)

Molyneux asked whether a newly sighted person could distinguish a sphere from a cube by sight alone, given that she was antecedently able to do so by touch. This, we contend, is a question about general ideas. To answer it, we must ask (a) whether spatial locations identified by touch can be identified also by sight, and (b) whether the integration of spatial locations into an idea of shape persists through changes of modality. Posed this way, Molyneux’s Question goes substantially (...) beyond question (a), about spatial locations, alone; for a positive answer to (a) leaves open whether a perceiver might cross-identify locations, but not be able to identify the shapes that collections of locations comprise. We further emphasize that MQ targets general ideas so as to distinguish it from corresponding questions about experiences of shape and about the property of tangible (vs. visual) shape. After proposing a generalized formulation of MQ, we extend earlier work (“Many Molyneux Questions,” Australasian Journal of Philosophy 2020) by showing that MQ does not admit a single answer across the board. Some integrative data-processes transfer across modalities; others do not. Seeing where and how such transfer succeeds and fails in individual cases has much to offer to our understanding of perception and its modalities. (shrink)

The usual visual experiences possess a perspectival phenomenology as they seem to present objects from a certain perspective. Nevertheless, it is not obvious how to characterise experiential content determining such phenomenology. In particular, while there are many works investigating perspectival properties of experienced objects, a question regarding how subject is represented in visual perspectival experiences attracted less attention. In order to address this problem, I consider four popular phenomenal intuitions regarding perspectival experiences and argue that the major theories of perspectival (...) experiences do not account for all of them. Relying on these observations, I show how a theory which accommodates all these intuitions can be developed by (a) recognising that visual perspectival experiences are, in fact, multimodal visuo-bodily experiences and (b) distinguishing between egocentric and structural contents. (shrink)

Scholarly opinions vary on what language is, how it evolved, and from where or what it evolved. Long considered uniquely human, today scholars argue for evolutionary continuity between human language and animal communication systems. But while it is generally recognized that language is an evolving communication system, scholars continue to debate from which species language evolved, and what behavioral and cognitive features are the precursors to human language. To understand the nature of protolanguage, some look for homologs in gene functionality, (...) brain areas, or anatomical structures such as the supralaryngeal vocal tract; others point toward primates, their gestural, vocal, multimodal, and in later evolving hominins also their pantomimic communication systems; and still others draw parallels between the musicality that characterizes language and the pitch found in the numerous sounds produced by animals. Accordingly, protolanguage theories today are multiple and diverse, and protolanguages might have also been diverse. This special issue on Evolving (Proto)Language/s for Lingua bundles several of the protolanguage theories that were put forward at the sixth edition of the Ways to Protolanguage conference series, held at the Calouste Gulbenkian Foundation in Lisbon, in 2019. Not aimed at surveying all the different ways there are to conceptualize, study, and model protolanguage/s, this issue provides interested readers with good overviews on the role played in current theorizing on protolanguage/s by (paleo)anthropology, genetics, physiology, developmental, evolutionary, ecological, and pragmatic research lines. (shrink)

Brand Equity Planning with Structuralist Rhetorical Semiotics furnishes an innovative conceptual model and methodology for brand equity planning, with view to addressing a crucial gap in the marketing and semiotic literatures concerning how advertising multimodal textual elements may be transformed into brand associations, with an emphasis on rhetorical relata as modes of connectivity between a brand’s surface and depth grammar. The scope of this project is inter-disciplinary, spanning research areas such as brand equity, structuralist semiotics, textual semiotics, visual and film (...) semiotics, multimodal rhetoric, film theory, psychoanalysis. The proposed connectionist conceptual model of the brand trajectory of signification is operationalized through a methodological framework that encompasses a structuralist semiotic interpretative approach to the textual formation of brand equity, supported by quantitative content analysis with the aid of the software Atlas.ti and the application of multivariate mapping techniques. (shrink)

Jesse Prinz (2004) argues that emotional feelings (“state emotions”) can by themselves perceptually represent significant organism-environment relations. I object to this view mainly on the grounds that (1) it does not rule out the at least equally plausible view that emotional feelings are non-representational sensory registrations rather than perceptions, as Tyler Burge (2010) draws the distinction, and (2) perception of a relation requires perception of at least one of the relation’s relata, but an emotional feeling by itself perceives neither the (...) subject’s environment, nor in many cases the relevant subject itself. I then explore two ways in which emotional feelings as non-perceptual sensory registrations might still contribute to significant relation representation when associated with representations of the subject and/or its environment. After briefly discussing some difficulties presented by a multimodal, sensory-perceptual view of such representation, I argue in favor of a “cognitive recognition theory” that holds that significant relation instances are represented during emotion occurrences via applications of emotion-type concepts to "incoming" emotional feelings and their associated mental states. (shrink)

This report highlights and explores five questions that arose from the multisensory integration workshop at the University of Toronto on May 9th and 10th, 2014: 1. What Is Multisensory Integration? 2. Do Multisensory Percepts Involve Emergent Features? 3. What Can Multisensory Processing Tell Us about Multisensory Awareness? 4. Is Language Processing a Special Kind of Multisensory Integration? 5. What Is the Purpose of Multisensory Integration?

This is an excerpt from a report on the Temporal Experience Workshop at the University of Toronto in May of 2013. This portion of the report explores the question: Do we have one central clock for time, or different clocks for each sense modality?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: What is multisensory integration?

The predictive coding (PC) theory of attention identifies attention with the optimization of the precision weighting of prediction error. Here we provide some challenges for this identification. On the one hand, the precision weighting of prediction error is too broad a phenomenon to be identified with attention because such weighting plays a central role in multimodal integration. Cases of crossmodal illusions such as the rubber hand illusion and the McGurk effect involve the differential precision weighting of prediction error, yet attention (...) does not shift as one would predict. On the other hand, the precision weighting of prediction error is too narrow a phenomenon to be identified with attention, because it cannot accommodate the full range of attentional phenomena. We review criticisms that PC cannot account for volitional attention and affect-biased attention, and we propose that it may not be able to account for feature-based and intellectual attention. (shrink)

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: Do multisensory percepts involve emergent features?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: What can multisensory processing tell us about multisensory awareness?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: What is the purpose of multisensory integration?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: Is language processing a special kind of multisensory integration?

The concept of twofoldness plays an important role in understanding the aesthetic appreciation of pictures. My claim is that it also plays an important role in understanding the aesthetic appreciation of musical performances. I argue that when we are aesthetically appreciating the performance of a musical work, we are simultaneously attending to both the features of the performed musical work and the features of the token performance we are listening to. This twofold experience explains a number of salient aspects of (...) our experience of musical performances, from the importance of instrumentality, to the multimodal character of this experience. (shrink)

Long-standing neglect of the chemical senses in the philosophy of perception is due, mostly, to their being regarded as ‘lower’ senses. Smell, taste, and chemically irritated touch are thought to produce mere bodily sensations. However, empirically informed theories of perception can show how these senses lead to perception of objective properties, and why they cannot be treated as special cases of perception modelled on vision. The senses of taste, touch, and smell also combine to create unified perceptions of flavour. The (...) nature of these multimodal experiences and the character of our awareness of them puts pressure on the traditional idea that each episode of perception goes one or other of the five senses. Thus, the chemical senses, far from being peripheral to the concerns of the philosophy of perception, may hold important clues to the multisensory nature of perception in general. (shrink)

We investigate an enrichment of the propositional modal language L with a "universal" modality ■ having semantics x ⊧ ■φ iff ∀y(y ⊧ φ), and a countable set of "names" - a special kind of propositional variables ranging over singleton sets of worlds. The obtained language ℒ $_{c}$ proves to have a great expressive power. It is equivalent with respect to modal definability to another enrichment ℒ(⍯) of ℒ, where ⍯ is an additional modality with the semantics x ⊧ ⍯φ (...) iff Vy(y ≠ x → y ⊧ φ). Model-theoretic characterizations of modal definability in these languages are obtained. Further we consider deductive systems in ℒ $_{c}$ . Strong completeness of the normal ℒ $_{c}$ logics is proved with respect to models in which all worlds are named. Every ℒ $_{c}$ -logic axiomatized by formulae containing only names (but not propositional variables) is proved to be strongly frame-complete. Problems concerning transfer of properties ([in]completeness, filtration, finite model property etc.) from ℒ to ℒ $_{c}$ are discussed. Finally, further perspectives for names in multimodal environment are briefly sketched. (shrink)

Human language, hominin tool production modes, and multimodal communications systems of primates and other animals are currently well-studied for how they display compositionality or combinatoriality. In all cases, the former is defined as a kind of hierarchical nesting and the latter as a lack thereof. In this article, I extend research on combinatoriality and compositionality further to investigations of everyday primate skills. Daily locomotion modes as well as behaviors associated with subsistence practices, hygiene, or body modification rely on the hierarchical (...) nesting of different behavioral and cognitive actions into complex skills. I introduce a scheme which uses hierarchical organization to differentiate combinatorial from compositional skills. Combinatorial skills are defined either as aggregational or linearly hierarchical depending on whether the skill occurs momentarily in space or unfolds sequentially over time. Compositional skills are defined either as nested or interactionally hierarchical depending on whether the skill results in new constructs or in new interactions between existing constructs. The methodology I propose combines epistemological hierarchy theory with data from primatological field research and experimental and comparative psychological research and provides a means to integrate current constructionist and extended views on cognition and action with older research on behavioral logics in psychology and operational chain thinking in anthropology. The approach furthermore synchronizes with ongoing research on teleonomy, intentionality, and creativity. -/- . (shrink)

I argue that the neural realizers of experiences of trying are not distinct from the neural realizers of actual trying . I then ask how experiences of trying might relate to the perceptual experiences one has while acting. First, I assess recent zombie action arguments regarding conscious visual experience, and I argue that contrary to what some have claimed, conscious visual experience plays a causal role for action control in some circumstances. Second, I propose a multimodal account of the experience (...) of acting. According to this account, the experience of acting is a temporally extended, co-conscious collection of agentive and perceptual experiences, functionally integrated and structured both by multimodal perceptual processing as well as by what an agent is, at the time, trying to do. (shrink)

There is converging evidence that high doses of hallucinogenic drugs can produce significant alterations of self-experience, described as the dissolution of the sense of self and the loss of boundaries between self and world. This article discusses the relevance of this phenomenon, known as “drug-induced ego dissolution (DIED)”, for cognitive neuroscience, psychology and philosophy of mind. Data from self-report questionnaires suggest that three neuropharmacological classes of drugs can induce ego dissolution: classical psychedelics, dissociative anesthetics and agonists of the kappa opioid (...) receptor (KOR). While these substances act on different neurotransmitter receptors, they all produce strong subjective effects that can be compared to the symptoms of acute psychosis, including ego dissolution. It has been suggested that neuroimaging of DIED can indirectly shed light on the neural correlates of the self. While this line of inquiry is promising, its results must be interpreted with caution. First, neural correlates of ego dissolution might reveal the necessary neurophysiological conditions for the maintenance of the sense of self, but it is more doubtful that this method can reveal its minimally sufficient conditions. Second, it is necessary to define the relevant notion of self at play in the phenomenon of DIED. This article suggests that DIED consists in the disruption of subpersonal processes underlying the “minimal” or “embodied” self, i.e., the basic experience of being a self rooted in multimodal integration of self-related stimuli. This hypothesis is consistent with Bayesian models of phenomenal selfhood, according to which the subjective structure of conscious experience ultimately results from the optimization of predictions in perception and action. Finally, it is argued that DIED is also of particular interest for philosophy of mind. On the one hand, it challenges theories according to which consciousness always involves self-awareness. On the other hand, it suggests that ordinary conscious experience might involve a minimal kind of self-awareness rooted in multisensory processing, which is what appears to fade away during DIED. (shrink)