The aim of this paper is to argue that our experience of artworks is normally multimodal. It is the result of perceptual processing in more than one sense modality. In other words, multimodal experience of art is not the exception; it is the rule. I use the example of music in order to demonstrate the various ways in which the visual sense modality influences the auditory processing of music and conclude that this should make us look more closely (...) at our practices of engaging with artworks. (shrink)

Traditional medical Artificial Intelligence models, approved for clinical use, restrict themselves to single-modal data e.g. images only, limiting their applicability in the complex, multimodal environment of medical diagnosis and treatment. Multimodal Transformer Models in healthcare can effectively process and interpret diverse data forms such as text, images, and structured data. They have demonstrated impressive performance on standard benchmarks like USLME question banks and continue to improve with scale. However, the adoption of these advanced AI models is not without (...) challenges. While multimodal deep learning models like Transformers offer promising advancements in healthcare, their integration requires careful consideration of the accompanying ethical and environmental challenges. (shrink)

This research article examines the potential of multimodal techniques in promoting inclusive practices in the English language classroom, delving into the multidisciplinary fields of English Language Teaching (ELT) and inclusive education. This study intends to investigate how multimodal resources, such as visual aids, technology, and creative activities, may be effectively integrated to meet the various learning needs of students using a pedagogical lens. This paper examines the theoretical underpinnings and practical implications of using multimodal approaches in ELT (...) settings, shedding light on their potential to promote inclusivity, engagement, and language acquisition among learners with diverse abilities and backgrounds. It does this through a thorough literature review and analysis of pedagogical frameworks. (shrink)

It is common to characterize pain with touch-related terms, like ‘cutting’, ‘pressing’, ‘sharp’, and ‘pulsing’, or temperature-related terms, like ‘hot’ or ‘burning’. This suggests that many pains are phenomenally multimodal because they are experienced as having some tactile-like or thermal-like character. The goal of this chapter is to investigate the structure of phenomenally multimodal pain experiences. It is argued that the usual accounts of multimodal structure proposed in investigations regarding exteroceptive experiences cannot be plausibly applied to (...) class='Hi'>multimodal experiences of pain. Instead, an alternative framework is proposed which characterizes the structure of tactile-like and thermal-like pains by referring to the notion of crossmodal correspondences. (shrink)

From the perspective of distributed cognition I will stress how abduction is essentially multimodal, in that both data and hypotheses can have a full range of verbal and sensory representations, involving words, sights, images, smells, etc., but also kinesthetic – related to the ability to sense the position and location and orientation and movement of the body and its parts – and motor experiences and other feelings such as pain, and thus all sensory modalities. The presence of kinesthetic and (...) motor aspects demonstrates that abductive reasoning can be manipulative. We can also see, in this regard, how implicit factors take part in the abductive procedure, which consequently acquires the character of a kind of “thinking through doing”. This paper further describes 1) the fact that hypotheses in science can be built through different cognitive mediators and so they can also model the same cognitive aspect in different ways; how they can be carriers/ producers of knowledge in a multimodal way; 2) the problem of the possible non-explanatory and instrumental nature of abductive reasoning and the analysis of the consequences for induction; 3) the role of manipulative abduction in building new evidence/ experiments and how they trigger smart inductive inferences. (shrink)

This paper presents an analytical framework for analyzing how multimodal resources of emotion expression are semiotically materialized in discursive interactions specific to political discourse. Interested in how political personae are emotionally constructed through multimodal meaning-making practices, our analysis model assumes an interdisciplinary perspective, which integrates facial expression analysis – using FaceReader™ software –, the theory of emotional arcs and bodily actions (hand gestures) analysis that express emotions, in the analytical framework of multimodality. The results show how the (...) class='Hi'>multimodal choices that political actors make during discursive interactions allow them to build their political brand and make connections with the audience on an emotional level. (shrink)

This is an excerpt of a report that highlights and explores five questions which arose from The Unity of Consciousness and Sensory Integration conference at Brown University in November of 2011. This portion of the report explores the question: Are some of the basic units of consciousness multimodal?

This is an excerpt from a report that highlights and explores five questions which arose from the workshop on perceptual learning and perceptual recognition at the University of Toronto, Mississauga on May 10th and 11th, 2012. This excerpt explores the question: What are the origins of multimodal associations?

The analysis of multimodal argumentation in advertising is a crucial and problematic area of research. While its importance is growing in a time characterized by images and pictorial messages, the methods used for interpreting and reconstructing the structure of arguments expressed through verbal and visual means capture only isolated dimensions of this complex phenomenon. This paper intends to propose and illustrate a methodology for the reconstruction and analysis of “double-mode” arguments in advertisements, combining the instruments developed in social semiotics, (...) pragmatics, and argumentation theory. An advertisement is processed through a five-step path. The analysis of its context, text genre, and images leads to a first representation of the messages that it encodes both pictorially and verbally (step 1). These first semantic representations are further enriched by including their polyphonic articulations and presuppositions (step 2), their explicatures (step 3), and their dialogical functions and illocutionary forces (step 4). These pragmatic steps retrieve the commitment structure of the ad, which allows a further argument analysis conducted through argumentation schemes (step 5). (shrink)

What is the correct procedure for determining the contents of perception? Philosophers tackling this question increasingly rely on empirically-oriented procedures in order to reach an answer. I argue that this constitutes an improvement over the armchair methodology constitutive of phenomenal contrast cases, but that there is a crucial respect in which current empirical procedures remain limited: they are unimodal in nature, wrongly treating the senses as isolatable faculties. I thus have two aims: first, to motivate a reorientation of the admissible (...) contents debate into a multimodal framework, charting its various significances. The second is to explore whether any experimental studies of multimodal perception support a so-called Liberal (or ‘high-level’ or ‘rich’) account of perception’s admissible contents. I conclude that the McGurk effect and the ventriloquist effect are both explicable without the postulation of high-level content, but that at least one multimodal experimental paradigm may necessitate such content: the rubber hand illusion. One upshot of this argument is that Conservatives who claim that the Liberal view intolerably broadens the scope of perceptual illusions, particularly from the perspective of perceptual psychology, should pursue other arguments against that view. (shrink)

With a Call for Essays, the special issue Multimodality sought contributions that accept not only the material but also the body-bound dependence of media perception and understanding. To this end, contributions were included that shed light on both the structural and signifying potential of artistic works through multimodal analysis. Particular attention was paid to contributions that clarify how the structural features - the modes - of the arts, their perception, and their signifying potential in terms of content are interrelated (...) and how they are to be understood in communicative and thus socioculturally relevant terms. Thus, in addition to neuroscientific contributions, those from cultural anthropology, art history, image and art studies, and literary studies were included. -/- Articles -/- 8 Martina Sauer and Christiane Wagner - Introduction -/- 11 Giada Lombardi and Giuseppe Di Cesare - From Neuroscience to Art: The Role of ‘Vitality Forms’ in the Investigation of Multimodality -/- 25 Martina Sauer - Abstract - Affective – Multimodal: Interaction between Medium and Perception of Moving Images from the Viewpoint of Cassirer’s, Langer’s and Krois’ Embodiment Theories -/- 49 Göran Sonesson - The Secrets of Plastic Language Revealed: Multimodality, Polysemiosis, and Iconicity -/- 71 Natalia Igl - Making Sense of the Material: Multisensory Reader Involvement in Contemporary Multimodal Novels -/- 87 J. Philippe Thompson - Art is All: Proust’s Life Lessons Through Perception and Art -/- 107 Hans Dieter Huber - Philippe Parreno and the Exhibition as a Multimodal Aesthetic Experience -/- 135 Jörg U. Lensing - The Performing Human Being in a Media Interaction Space: Multi- and intermodal productions by the Düsseldorf Theater der Klänge (Theatre of Sounds) . (shrink)

Mental imagery is early perceptual processing that is not triggered by corresponding sensory stimulation in the relevant sense modality. Multimodal mental imagery is early perceptual processing that is triggered by sensory stimulation in a different sense modality. For example, when early visual or tactile processing is triggered by auditory sensory stimulation, this amounts to multimodal mental imagery. Pulling together philosophy, psychology and neuroscience, I will argue in this paper that multimodal mental imagery plays a crucial role in (...) our engagement with musical works. Engagement with musical works is normally a multimodal phenomenon, where we get input from a number of sense modalities. But even if we screen out any input that is not auditory, multimodal mental imagery will still play an important role that musicians and composers often actively rely on. (shrink)

The aim of this paper is to propose a new approach in the form of multimodality as a semiotic method that can be used by marketers and semioticians to induce online flow, a psychological state, on a brand website. First, we refer to multimodality as a semiotic analysis that can be used for a better optimization of semiotic resource sets in meaning-making, and we distinguish it from another similar concept: multimedia. Second, after a critical literature review, we address the flow (...) construct - a state of mind sometimes experienced by people who are deeply involved and immersed in some event, object or activity. Previous quantitative studies show a correlation between online flow experience and other variables such as design, content, sound and layout that are suggested to be its antecedents. In our study, these flow antecedents are viewed as sets of signs or modes that can be orchestrated in order to create different levels of perceived challenge for brand website users. We consider that multimodality can be used for a better optimization of these various meanings as expressed by modes that are "essentially different in nature" (Bateman, 2008, p.54). Also, we refer to the online marketing outcomes of the online flow experience, in order to outline its importance and its implications in the online environment. The main conclusion of our paper is that multimodality can be used by scholars and practitioners as a premise to induce online flow on a brand website, by optimizing the sets of signs or modes. Originality – our contribution consists of an interdisciplinary approach. We use concepts from semiotics, positive psychology, and online marketing for a better understanding of how marketers and semioticians can positively influence online customer behaviour. (shrink)

Everyday media consumption leaves no doubt that the perception of moving images from various media is characterized by experience and understanding. Corresponding research in this field has shown that the stimulus patterns flooding in on us are not only processed mentally, but also bodily. Building on this, the following study argues that incoming stimuli are processed not only visually, but multimodally, with all senses, and moreover affectively. The classical binding of a sensory organ to a medium, on whose delimitation the (...) common understanding of multimodality is based, is thus abolished. Can arguments for this assumption be found? Building on the research of cultural anthropologist Ernst Cassirer and his successors, the two American philosophers Susanne K. Langer and John M. Krois, and supported by research from the life sciences in which, in parallel, the arts also play an important role in understanding image formation, as in the work of Heinz Werner and, following on from this, in the more recent research of the child psychologist Daniel N. Stern in collaboration with mirror neuron research by the group of neuroscientists around Giacomo Rizzolatti in Parma, justifications for this will be provided. What the multimodal processing of film images is concretely based on will finally be made clear by means of an exemplary analysis of a video film by the Swiss artist Pipilotti Rist. It aims to prove, that the prerequisites for the thesis can only lie in the fact, that both the modes of perception (visual, tactile, auditory, olfactory, gustatory) as well as the medially conditioned modes of image formation are based on non-discursive, formal-abstract structures that are also processed affectively. At the same time, the analysis thus shows how the selection and composition of the vital experienced forms can provide information about the socio-culturally shaped self-image of the producer and the viewer. (shrink)

Recent advances in neuroimaging technologies have rendered multimodal analysis of operators’ cognitive processes in complex task settings and environments increasingly more practical. In this exploratory study, we utilized optical brain imaging and mobile eye tracking technologies to investigate the behavioral and neurophysiological differences among expert and novice operators while they operated a human-machine interface in normal and adverse conditions. In congruence with related work, we observed that experts tended to have lower prefrontal oxygenation and exhibit gaze patterns that are (...) better aligned with the optimal task sequence with shorter fixation durations as compared to novices. These trends reached statistical significance only in the adverse condition where the operators were prompted with an unexpected error message. Comparisons between hemodynamic and gaze measures before and after the error message indicated that experts’ neurophysiological response to the error involved a systematic increase in bilateral dorsolateral prefrontal cortex (dlPFC) activity accompanied with an increase in fixation durations, which suggests a shift in their attentional state, possibly from routine process execution to problem detection and resolution. The novices’ response was not as strong as that of experts, including a slight increase only in the left dlPFC with a decreasing trend in fixation durations, which is indicative of visual search behavior for possible cues to make sense of the unanticipated situation. A linear discriminant analysis model capitalizing on the covariance structure among hemodynamic and eye movement measures could distinguish experts from novices with 91% accuracy. Despite the small sample size, the performance of the linear discriminant analysis combining eye fixation and dorsolateral oxygenation measures before and after an unexpected event suggests that multimodal approaches may be fruitful for distinguishing novice and expert performance in similar neuroergonomic applications in the field. (shrink)

Recent advances in neuroimaging technologies have rendered multimodal analysis of operators’ cognitive processes in complex task settings and environments increasingly more practical. In this exploratory study, we utilized optical brain imaging and mobile eye tracking technologies to investigate the behavioral and neurophysiological differences among expert and novice operators while they operated a human-machine interface in normal and adverse conditions. In congruence with related work, we observed that experts tended to have lower prefrontal oxygenation and exhibit gaze patterns that are (...) better aligned with the optimal task sequence with shorter fixation durations as compared to novices. These trends reached statistical significance only in the adverse condition where the operators were prompted with an unexpected error message. Comparisons between hemodynamic and gaze measures before and after the error message indicated that experts’ neurophysiological response to the error involved a systematic increase in bilateral dorsolateral prefrontal cortex (dlPFC) activity accompanied with an increase in fixation durations, which suggests a shift in their attentional state, possibly from routine process execution to problem detection and resolution. The novices’ response was not as strong as that of experts, including a slight increase only in the left dlPFC with a decreasing trend in fixation durations, which is indicative of visual search behavior for possible cues to make sense of the unanticipated situation. A linear discriminant analysis model capitalizing on the covariance structure among hemodynamic and eye movement measures could distinguish experts from novices with 91% accuracy. Despite the small sample size, the performance of the linear discriminant analysis combining eye fixation and dorsolateral oxygenation measures before and after an unexpected event suggests that multimodal approaches may be fruitful for distinguishing novice and expert performance in similar neuroergonomic applications in the field. (shrink)

Is it true that there is an analogy between modes of creation and such of perception? Respective to the cultural anthropological research of Ernst Cassirer, Susanne K. Langer and John M. Krois and by the analysis of a tape of the Swiss video-artist Pipilotti Rist this initial thesis of Formal Aesthetics shall be supported. - I - -/- Lässt sich die Behauptung stützen, dass zwischen Gestaltungsweisen und Wahrnehmungsweisen eine Analogie besteht? Aufbauend auf den kulturanthropologischen Forschungen von Ernst Cassirer, Susanne K. (...) Langer und John M. Krois sowie der Analyse eines Tapes der Schweizer Videokünstlerin Pipilotti Rist gilt es diese ursprünglich von der formalen Ästhetik aufgeworfene These zu untermauern. (shrink)

It has been repeatedly suggested that synesthesia is intricately connected with unusual ways of exercising one’s mental imagery, although it is not always entirely clear what the exact connection is. My aim is to show that all forms of synesthesia are forms of (often very different kinds of) mental imagery and, further, if we consider synesthesia to be a form of mental imagery, we get significant explanatory benefits, especially concerning less central cases of synesthesia where the inducer is not sensory (...) stimulation. (shrink)

This paper analyzes transduction as an action of transposing information from one mode to another within the communication process and its implications in terms of meaning and coherence of a multimodal message. First, I discuss the multimodal method and its conjunction with some key concepts such as: sign, meaning, mode, transduction. Secondly, I approach transduction as an essential method of translating messages across the media variety, describing my interdisciplinary approach – that brings together semiotics and communications – and (...) proposing a framework of explanation for transduction in the field of advertising. Drawing from a previous model (Culache 2015), I illustrate the way transduction takes place and identify its meaning-making issues while introducing the concept of ‘dominant mode.’. (shrink)

This volume brings together the advanced research results obtained by the European COST Action 2102: “Cross Modal Analysis of Verbal and Nonverbal Communication”. The research published in this book was discussed at the 3rd joint EUCOGII-COST 2102 International Training School entitled “Toward Autonomous, Adaptive, and Context-Aware Multimodal Interfaces: Theoretical and Practical Issues ” and held in Caserta, Italy, on March 15-19, 2010.

Higher animals need to identify and track material objects because they depend on interactions with them for nutrition, reproduction, and social interaction. This paper investigates the perception of material objects. It argues, first, that material objects are tagged, in all five external senses, as bearers of the features detected by them. This happens through a perceptual process, here entitled Generalized Completion, which creates the appearance of objects that have properties that transcend the activation of sensory receptors. The paper shows, secondly, (...) that material objects are privileged subjects for perceived motion and interaction. That is, they are perceived as subjects for these properties while their parts seem to be subjects only derivatively. Material objects are the only perceptual subjects that are both multisensory and privileged. (shrink)

Purpose of this article is to describe the development of a pedagogical research approach entitled reflective documentation. Context for this inquiry is an Erasmus+ project entitled Pedagogy of Imaginative Dialogues (PIMDI). The practical examples are drawn from the PIMDI project. For fuller appreciation of the multimodal nature of reflective work and reflective documentation, we have created an exposition on Research Catalogue portal. As the outcome of this artistic pedagogical research project, we propose that reflective documentation can be seen in (...) connection with artistic research and visual ethnography as an open-ended process, without the traditional requirement to show what really happened. (shrink)

This paper examines the convergence of large language models, multimodal AI, and generative spatial technologies to enable sophisticated simulated worlds for studying agentic behavior. Recent developments in generative architectures, particularly GenEx and topology-aware mesh generation, facilitate the creation of coherent, explorable environments with artificial agents capable of complex interactions. The proposed framework for "virtual anthropology" presents novel opportunities for studying emergent behaviors and cognitive processes in controlled, generative environments, with implications for both theoretical research and practical applications in artificial (...) intelligence. (shrink)

If the sight of cortically blind people were restored, could they visually recognize a cube or a sphere? This is Molyneux’s question. I argue that the answer to this question depends on the specificities of the mental setup of these cortically blind people. Some cortically blind people have (sometimes quite vivid) visual imagery. Others don’t. The answer to Molyneux’s question depends on whether the blind subjects have had visual imagery before their sight was restored. If they did, the answer to (...) Molyneux’s question is yes, if they didn’t, the answer is no. There is no generic answer to Molyneux’s question. (shrink)

La presente investigación tuvo el objetivo de pilotar un instrumento de recolección de información acerca de los hábitos de lectura en tutores de niños que cursan estudios preescolares. Concretamente, se realizó un pilotaje de una encuesta en la que se abordó a 40 tutores que tuviesen a cargo niños en educación preescolar. Algunos de los principales hallazgos fueron que los tutores dicen invertir más tiempo en la lectura escolar que en lectura por entretenimiento. Junto a ello, la lectura en donde (...) acompañan a los niños es poco habitual. Los tutores comentan invertir más tiempo en leer los grupos de chat de WhatsApp que en leer junto a sus niños. Estos resultados pueden evidenciar el por qué la lectura está considerada una tarea escolar y no una actividad de entretenimiento y gozo. Esta línea de investigación nos permitirá desarrollar estrategias pedagógicas que promuevan mejores hábitos de lectura, especialmente en tiempos de pandemia donde la labor de la familia en el aprendizaje de los niños queda más evidente. (shrink)

Writing and genre scholarship has become increasingly attuned to how various nontextual features of written genres contribute to the kinds of social actions that the genres perform and to the activities that they mediate. Even though scholars have proposed different ways to account for nontextual features of genres, such attempts often remain undertheorized. By bringing together Writing, Activity, and Genre Research, and Multimodal Interaction Analysis, the authors propose a conceptual framework for multimodal activity-based analysis of genres, or (...) class='Hi'>Multimodal Writing, Activity, and Genre (MWAG) analysis. Furthermore, by drawing on previous studies of the laboratory notebook (lab book) genre, the article discusses the rhetorical action the genre performs and its role in mediating knowledge construction activities in science. The authors provide an illustrative example of the MWAG analysis of an emergent scientist’s lab book and discuss its contributions to his increasing participation in medical physics. The study contributes to the development of a theoretically informed analytical framework for integrative multimodal and rhetorical genre analysis, while illustrating how the proposed framework can lead to the insights into the sociorhetorical roles multimodal genres play in mediating such activities as knowledge construction and disciplinary enculturation. (shrink)

Disorders of consciousness (DoC)—that is, unresponsive wakefulness syndrome/vegetative state and minimally conscious state—are debilitating conditions for which no reliable markers of consciousness recovery have yet been identified. Evidence points to the GABAergic system being altered in DoC, making it a potential target as such a marker.

Higher animals need to identify and track material objects because they depend on interactions with them for nutrition, reproduction, and social interaction. This paper investigates the perception of material objects. It argues, first, that material objects are tagged, in all five external senses, as bearers of the features detected by them. This happens through a perceptual process, here entitled Generalized Completion, which creates the appearance of objects that have properties that transcend the activation of sensory receptors. The paper shows, secondly, (...) that material objects are privileged subjects for perceived motion and interaction. That is, they are perceived as subjects for these properties while their parts seem to be subjects only derivatively. Material objects are the only perceptual subjects that are both multisensory and privileged. (shrink)

The aim of this paper is to raise some serious worries about anti-representationalism: the recently popular view according to which there are no perceptual representations. Although anti-representationalism is more and more popular, I will argue that we have strong empirical reasons for mistrusting it. More specifically, I will argue that it is inconsistent with some important empirical findings about dorsal perception and about the multimodality of perception.

In the case of ventriloquism, seeing the movement of the ventriloquist dummy’s mouth changes your experience of the auditory location of the vocals. Some have argued that cases like ventriloquism provide evidence for the view that at least some of the content of perception is fundamentally multimodal. In the ventriloquism case, this would mean your experience has constitutively audio-visual content (not just a conjunction of an audio content and visual content). In this paper, I argue that cases like ventriloquism (...) do not in fact warrant that conclusion. I then try to make sense of crossmodal cases without appealing to fundamentally multimodal content. (shrink)

An interesting application for social robots is to act as a presenter, for example as a museum guide. In this paper, we present a fully automated system architecture for building adaptive presentations for embodied agents. The presentation is generated from a knowledge graph, which is also used to track the grounding state of information, based on multimodal feedback from the user. We introduce a novel way to use large-scale language models (GPT-3 in our case) to lexicalise arbitrary knowledge graph (...) triples, greatly simplifying the design of this aspect of the system. We also present an evaluation where 43 participants interacted with the system. The results show that users prefer the adaptive system and consider it more human-like and flexible than a static version of the same system, but only partial results are seen in their learning of the facts presented by the robot. (shrink)

This is the editors' Introduction to a special issue of the journal, Multisensory Research. European philosophers of the modern period found multisensory perception to be impossible because they thought that perceptual ideas are defined by how they are experienced. Under this conception, the individual modalities are determinables of ideas—just as colour is a determinable that embraces red and blue, so also the visual is a determinable that embraces colour and (visually experienced) shape. Since no idea is experienced as, for example, (...) both visual and auditory, there can be no such thing as audiovisual perception. This conception of modality is not directly contested, but a variety of perceptual phenomena are listed that could raise interesting questions if treated as multimodal in origin. (shrink)

Many philosophers take experience to be an essential aspect of perceptual justification. I argue against a specific variety of such an experientialist view, namely, the Looks View of perceptual justification, according to which our visual beliefs are mediately justified by beliefs about the way things look. I describe three types of cases that put pressure on the idea that perceptual justification is always related to looks-related reasons: unsophisticated cognizers, multimodal identification, and amodal completion. I then provide a tentative diagnosis (...) of what goes wrong in the Looks View: it ascribes a specific epistemic role to beliefs about looks that is actually fulfilled by subpersonal perceptual processes. (shrink)

I argue that the neural realizers of experiences of trying are not distinct from the neural realizers of actual trying . I then ask how experiences of trying might relate to the perceptual experiences one has while acting. First, I assess recent zombie action arguments regarding conscious visual experience, and I argue that contrary to what some have claimed, conscious visual experience plays a causal role for action control in some circumstances. Second, I propose a multimodal account of the (...) experience of acting. According to this account, the experience of acting is a temporally extended, co-conscious collection of agentive and perceptual experiences, functionally integrated and structured both by multimodal perceptual processing as well as by what an agent is, at the time, trying to do. (shrink)

There is converging evidence that high doses of hallucinogenic drugs can produce significant alterations of self-experience, described as the dissolution of the sense of self and the loss of boundaries between self and world. This article discusses the relevance of this phenomenon, known as “drug-induced ego dissolution (DIED)”, for cognitive neuroscience, psychology and philosophy of mind. Data from self-report questionnaires suggest that three neuropharmacological classes of drugs can induce ego dissolution: classical psychedelics, dissociative anesthetics and agonists of the kappa opioid (...) receptor (KOR). While these substances act on different neurotransmitter receptors, they all produce strong subjective effects that can be compared to the symptoms of acute psychosis, including ego dissolution. It has been suggested that neuroimaging of DIED can indirectly shed light on the neural correlates of the self. While this line of inquiry is promising, its results must be interpreted with caution. First, neural correlates of ego dissolution might reveal the necessary neurophysiological conditions for the maintenance of the sense of self, but it is more doubtful that this method can reveal its minimally sufficient conditions. Second, it is necessary to define the relevant notion of self at play in the phenomenon of DIED. This article suggests that DIED consists in the disruption of subpersonal processes underlying the “minimal” or “embodied” self, i.e., the basic experience of being a self rooted in multimodal integration of self-related stimuli. This hypothesis is consistent with Bayesian models of phenomenal selfhood, according to which the subjective structure of conscious experience ultimately results from the optimization of predictions in perception and action. Finally, it is argued that DIED is also of particular interest for philosophy of mind. On the one hand, it challenges theories according to which consciousness always involves self-awareness. On the other hand, it suggests that ordinary conscious experience might involve a minimal kind of self-awareness rooted in multisensory processing, which is what appears to fade away during DIED. (shrink)

Autism spectrum disorder is characterised by differences in unimodal and multimodal sensory and proprioceptive processing, with complex biases towards local over global processing. Many of these elements are implicated in versions of the rubber hand illusion, which were therefore studied in high-functioning individuals with ASD and a typically developing control group. Both groups experienced the illusion. A number of differences were found, related to proprioception and sensorimotor processes. The ASD group showed reduced sensitivity to visuotactile-proprioceptive discrepancy but more accurate (...) proprioception. This group also differed on acceleration in subsequent reach trials. Results are discussed in terms of weak top-down integration and precision-accuracy trade-offs. The RHI appears to be a useful tool for investigating multisensory processing in ASD. (shrink)

Long-standing neglect of the chemical senses in the philosophy of perception is due, mostly, to their being regarded as ‘lower’ senses. Smell, taste, and chemically irritated touch are thought to produce mere bodily sensations. However, empirically informed theories of perception can show how these senses lead to perception of objective properties, and why they cannot be treated as special cases of perception modelled on vision. The senses of taste, touch, and smell also combine to create unified perceptions of flavour. The (...) nature of these multimodal experiences and the character of our awareness of them puts pressure on the traditional idea that each episode of perception goes one or other of the five senses. Thus, the chemical senses, far from being peripheral to the concerns of the philosophy of perception, may hold important clues to the multisensory nature of perception in general. (shrink)

We investigate an enrichment of the propositional modal language L with a "universal" modality ■ having semantics x ⊧ ■φ iff ∀y(y ⊧ φ), and a countable set of "names" - a special kind of propositional variables ranging over singleton sets of worlds. The obtained language ℒ $_{c}$ proves to have a great expressive power. It is equivalent with respect to modal definability to another enrichment ℒ(⍯) of ℒ, where ⍯ is an additional modality with the semantics x ⊧ ⍯φ (...) iff Vy(y ≠ x → y ⊧ φ). Model-theoretic characterizations of modal definability in these languages are obtained. Further we consider deductive systems in ℒ $_{c}$ . Strong completeness of the normal ℒ $_{c}$ logics is proved with respect to models in which all worlds are named. Every ℒ $_{c}$ -logic axiomatized by formulae containing only names (but not propositional variables) is proved to be strongly frame-complete. Problems concerning transfer of properties ([in]completeness, filtration, finite model property etc.) from ℒ to ℒ $_{c}$ are discussed. Finally, further perspectives for names in multimodal environment are briefly sketched. (shrink)

Electronic text can be defined on two different, though interconnected, levels. On the one hand, electronic text can be defined by taking the notion of “text” or “printed text” as the point of departure. On the other hand, electronic text can be defined by taking the digital format as the point of departure, where everything is represented in the binary alphabet. While the notion of text in most cases lends itself to being independent of medium and embodiment, it is also (...) often tacitly assumed that it is in fact modeled on the print medium, instead of, for instance, on hand-written text or speech. In late 20th century, the notion of “text” was subjected to increasing criticism, as can be seen in the question that has been raised in literary text theory about whether “there is a text in this class.” At the same time, the notion was expanded by including extralinguistic sign modalities (images, videos). A basic question, therefore, is whether electronic text should be included in the enlarged notion that text is a new digital sign modality added to the repertoire of modalities or whether it should be included as a sign modality that is both an independent modality and a container that can hold other modalities. In the first case, the notion of electronic text would be paradigmatically formed around the e-book, which was conceived as a digital copy of a printed book but is now a deliberately closed work. Even closed works in digital form will need some sort of interface and hypertextual navigation that together constitute a particular kind of paratext needed for accessing any sort of digital material. In the second case, the electronic text is defined by the representation of content and (some parts of the) processing rules as binary sequences manifested in the binary alphabet. This wider notion would include, for instance, all sorts of scanning results, whether of the outer cosmos or the interior of our bodies and of digital traces of other processes in-between (machine readings included). Since other alphabets, such as the genetic alphabet and all sorts of images may also be represented in the binary alphabet, such materials will also belong to the textual universe within this definition. A more intriguing implication is that born-digital materials may also include scripts and interactive features as intrinsic parts of the text. The two notions define the text on different levels: one is centered on the Latin, the other on the binary alphabet, and both definitions include hypertext, interactivity, and multimodality as constituent parameters. In the first case, hypertext is included as a navigational, paratextual device; whereas in the second case, hypertext is also incorporated in the narrative within an otherwise closed work or as a constituent element on the textual universe of the web, where it serves the ongoing production of (possibly scripted) connections and disconnections between blocks of textual content. Since the early decades of early 21st century still represent only the very early stages of the globally distributed universe of web texts, this is also a history of the gradual unfolding of the dimensions of these three constituencies—hypertext, interactivity, and multimodality. The result is a still-expanding repertoire of genres, including some that are emerging via path dependency; some via remediation; and some as new genres that are unique for networked digital media, including “social media texts” and a growing variety of narrative and discursive multiple-source systems. (shrink)

Human language, hominin tool production modes, and multimodal communications systems of primates and other animals are currently well-studied for how they display compositionality or combinatoriality. In all cases, the former is defined as a kind of hierarchical nesting and the latter as a lack thereof. In this article, I extend research on combinatoriality and compositionality further to investigations of everyday primate skills. Daily locomotion modes as well as behaviors associated with subsistence practices, hygiene, or body modification rely on the (...) hierarchical nesting of different behavioral and cognitive actions into complex skills. I introduce a scheme which uses hierarchical organization to differentiate combinatorial from compositional skills. Combinatorial skills are defined either as aggregational or linearly hierarchical depending on whether the skill occurs momentarily in space or unfolds sequentially over time. Compositional skills are defined either as nested or interactionally hierarchical depending on whether the skill results in new constructs or in new interactions between existing constructs. The methodology I propose combines epistemological hierarchy theory with data from primatological field research and experimental and comparative psychological research and provides a means to integrate current constructionist and extended views on cognition and action with older research on behavioral logics in psychology and operational chain thinking in anthropology. The approach furthermore synchronizes with ongoing research on teleonomy, intentionality, and creativity. -/- . (shrink)

Perceptual processes, in particular modular processes, have long been understood as being mandatory. But exactly what mandatoriness amounts to is left to intuition. This paper identifies a crucial ambiguity in the notion of mandatoriness. Discussions of mandatory processes have run together notions of automaticity and ballisticity. Teasing apart these notions creates an important tool for the modularist's toolbox. Different putatively modular processes appear to differ in their kinds of mandatoriness. Separating out the automatic from the ballistic can help the modularist (...) diagnose and explain away some putative counterexamples to multimodal and central modules, thereby helping us to better evaluate the evidentiary status of modularity theory. (shrink)

This paper seeks to initiate a theory of “imaginative dialogues” by articulating four dialogical principles that enable such a dialogue to occur. It is part of a larger project that takes the Socratic dialogue, a widely utilized conversation technique in philosophy education, as a starting point and aims to reinterpret it by shifting emphasis to the pre-reflective, pre-linguistic, and multimodal aspects of dialogues, involving both their verbal and embodied dimensions. To integrate the verbal dimensions of a dialogue with its (...) more elusive embodied dimensions, the paper will examine the notion ‘dialogue’ from the perspective of two different strategies. The first strategy chiefly focuses on the dialogic encounter. The ‘in-between’ of this dialogic encounter enables something to emerge that transcends the individual perspective of the speakers involved. The second strategy is primarily concerned with internal differentiation. The minor differences that constitute this internal differentiation, differentiate a dialogue from within. These strategies are not mutually exclusive but indicate a variation in starting point and orientation. This paper proposes to combine these two strategies by linking accounts of the productive moments in verbal dialogues to an account of the imaginative potential of embodied dialogues. This will enable the articulation of four dialogical principles (derived from Lev Yakubinsky, Oswald Ducrot, Martin Buber, and Gottfried Wilhelm Leibniz) through which an imaginative dialogue can proceed. (shrink)

Within contemporary philosophy of perception, it is commonly claimed that flavour experiences are paradigmatic examples of multimodal perceptual experiences. In fact, virtually any sensory system, including vision and audition, is believed to influence how we experience flavours. However, there is a strong intuition, often expressed in these works, that not all of these sensory systems make an equal contribution to the phenomenology of flavour experiences. More specifically, it seems that the activities of some sensory systems are constitutive for flavour (...) perception while others merely influence how we experience flavours. This paper aims to answer the question regarding the constitutive factors of flavour perception in a twofold way. First, a theoretical framework is developed, relying on debates regarding constitutivity in analytic metaphysics and philosophy of science, which defines the stronger and weaker senses in which the activities of sensory systems may be constitutive for flavour perception. Second, relying on empirical results in flavour science, the constitutive status of activities related to distinct sensory systems in the context of flavour perception is investigated. (shrink)

It is commonly argued that values “fill the logical gap” of underdetermination of theory by evidence, namely, values affect our choice between two or more theories that fit the same evidence. The underdetermination model, however, does not exhaust the roles values play in evidential reasoning. I introduce WAVE – a novel account of the logical relations between values and evidence. WAVE states that values influence evidential reasoning by adjusting evidential weights. I argue that the weight-adjusting role of values is distinct (...) from their underdetermination gap-filling role. Values adjust weights in three ways. First, values affect our trust in the testimony of others. Second, values influence the evidential thresholds required for justified epistemic judgments. Third, values influence the relative weight of a certain type of evidence within a body of multimodal discordant evidence. WAVE explains, from an epistemic perspective, rather than psychological, how smokers, for example, can find the same evidence about the dangers of smoking less persuasive than non-smokers. WAVE allows for a wider effect of values on our accepted scientific theories and beliefs than the effect for which the underdetermination model allows alone; therefore, science studies scholars must consider WAVE in their research and analysis of evidential case studies. (shrink)

Neste trabalho, é apresentada uma investigação do que poderia ser chamado de formalização lógica do processo de mudança de teorias devido a anomalias. Por anomalia entende-se um fato observado que faz parte do escopo explanatório de uma teoria, mas que vai de encontro à previsão da mesma. Uma abordagem clássica para restaurar o poder explicativo de uma teoria ameaçada por uma anomalia é a postulação de hipóteses novas e provisórias que, em conjunto com as demais hipóteses auxiliares originais, sejam capazes (...) de resolver a anomalia. Após chegar à algumas conclusões sobre a estrutura de tal processo, propomos um framework lógico multimodal e não-monotônico capaz de representar alguns aspectos-chave dessa faceta importante da dinâmica de teorias científicas. Devido à necessidade de acomodar hipóteses provisórias incompatíveis, esse framework incorpora uma forma fraca de paraconsistência. Como um estudo de caso, analisamos o comportamento anômalo do planeta Urano que ameaçou a mecânica celeste Newtoniana por mais de meio século e ensejou a descoberta do planeta Netuno. -/- In this work, an investigation of what could be called the logical formalization of the theory change process due to anomalies is presented. An anomaly is understood as an observed fact that is part of the explanatory scope of a theory, but that goes against its prediction. A classic approach to restoring the explanatory power of a theory threatened by an anomaly is to postulate new and provisional hypotheses that, together with the other original auxiliary hypotheses, are capable of resolving the anomaly. After reaching some conclusions about the structure of such a process, we propose a multimodal and non-monotonic logical framework capable of representing some key aspects of this important facet of the dynamics of scientific theories. Due to the need to accommodate incompatible provisional hypotheses, this framework incorporates a weak form of paraconsistency. As a case study, we analyze the anomalous behavior of the planet Uranus that threatened Newtonian celestial mechanics for more than half a century and prompted the discovery of the planet Neptune. (shrink)

A typical claim in anti-representationalist approaches to cognition such as ecological psychology or radical embodied cognitive science is that ecological information is sufficient for guiding behavior. According to this view, affordances are immediately perceptually available to the agent (in the so-called “ambient energy array”), so sensory data does not require much further inner processing. As a consequence, mental representations are explanatorily idle: perception is immediate and direct. Here we offer one way to formalize this direct-perception claim and identify some important (...) limits to it. We argue that the claim should be read as saying that successful behavior just implies picking out affordance-related information from the ambient energy array. By relying on the Partial Information Decomposition framework, and more concretely on its development of the notion of synergy, we show that in multimodal perception, where various energy arrays carry affordance-related information, the “just pick out affordance-related information” approach is very inefficient, as it is bound to miss all synergistic components. Efficient multimodal information combination requires transmitting sensory-specific (and not affordance-specific) information to wherever it is that the various information streams are combined. The upshot is that some amount of computation is necessary for efficient affordance reconstruction. (shrink)

In our everyday interactions we easily and effortlessly perceive emotions in others’ facial expressions and bodily behavior. How do we do that? Philosophers and psychologists have long argued about the fundamentals of emotion perception and the debate is far from settled. While some insist on the sufficiency of the morphological information contained in facial expressions, others construe the objects of emotion perception as more complex, comprising multimodal information such as touch, tone of voice, body postures, and so on. Others, (...) in turn, have gone so far as to deny that there is a pre-given object in emotion perception in the first place, so that processes of emotion perception must be conceptualized as emergent and co-constructed phenomena in contexts of social interaction. The purpose of this paper is to argue that John Dewey’s pragmatist theory of emotions can help us make significant theoretical advancements in the understanding of emotion perception, according to which emotions are coordinated modes of behavior that we enact with others in the contexts of social interactions. In this picture, perceiving emotions is a matter of participating in a joint activity of socio- affective coordination. This theory allows us to apprehend important insights about emotion perception that appear in these contemporary debates, while at the same time avoiding their pitfalls. (shrink)