The first part of this survey article presented a cartography of some of the more extensively studied forms of multisensory processing. In this second part, I turn to examining some of the different possible ways in which the structure of conscious perceptual experience might also be characterized as multisensory. In addition, I discuss the significance of research on multisensory processing and multisensory consciousness for philosophical debates concerning the modularity of perception, cognitive penetration, and the individuation of (...) the senses. (shrink)

Multisensory processing encompasses all of the various ways in which the presence of information in one sensory modality can adaptively influence the processing of information in a different modality. In Part I of this survey article, I begin by presenting a cartography of some of the more extensively investigated forms of multisensory processing, with a special focus on two distinct types of multisensory integration. I briefly discuss the conditions under which these different forms of multisensory processing (...) occur as well as their important perceptual consequences and interrelations. In Part II, I then turn to examining of some of the different possible ways in which the structure of conscious perceptual experience might also be characterized as multisensory. In addition, I discuss the significance of research on multisensory processing and multisensory consciousness for philosophical attempts to individuate the senses. (shrink)

It is tempting to think that one’s perceptual evidence comprises just what issues from perceiving with each of the respective sensory modalities. However, empirical, rational, and phenomenological considerations show that one’s perceptual evidence can outstrip what one possesses due to perceiving with each separate sense. Some novel perceptual evidence stems from the coordinated use of multiple senses. This paper argues that some perceptual evidence in this respect is distinctively multisensory.

This chapter distinguishes between two kinds of ordinary multisensory experience that go beyond mere co-consciousness of features (e.g., the experience that results from concurrently hearing a sound in the hallway and seeing the cup on the table). In one case, a sensory experience in one modality creates a perceptual demonstrative to whose referent qualities are attributed in another sensory modality. For example, when you hear someone speak, auditory experience attributes audible qualities to a seen event, a person’s speaking motions. (...) The second kind of multisensory experience attributes features experienced in several sensory modalities to one and the same object via a process of amodal perceptual integration, i.e., integration that occurs separately from processing within the individual sensory modalities. Multisensory experiences arising from holding and seeing a tomato or from seeing the Indian curry boil and smelling it are examples of the second kind of multisensory experience. At the end of the chapter we look at synesthesia, a kind of atypical multisensory experience, and argue that one version of this phenomenon may be able to shed light on the neural mechanism underlying amodally integrated multisensory experience. (shrink)

This chapter examines how our sense modalities interact in the perception of persistence. The chapter concentrates on two questions. The first concerns perceptual processing—do perceptual computations of object persistence ever integrate and compute over representations from more than one modality? It argues that this question should be answered affirmatively. The second question concerns perceptual experience—do experiences of object persistence ever exhibit a constitutively multisensory phenomenal character, or is the phenomenology of object persistence always uniquely associated with just one modality? (...) The chapter argues that the available evidence underdetermines the answer to this question, but suggests ways it might be empirically resolved. (shrink)

This report highlights and explores five questions that arose from the multisensory integration workshop at the University of Toronto on May 9th and 10th, 2014: 1. What Is Multisensory Integration? 2. Do Multisensory Percepts Involve Emergent Features? 3. What Can Multisensory Processing Tell Us about Multisensory Awareness? 4. Is Language Processing a Special Kind of Multisensory Integration? 5. What Is the Purpose of Multisensory Integration?

According to the decomposition thesis, perceptual experiences resolve without remainder into their different modality-specific components. Contrary to this view, I argue that certain cases of multisensory integration give rise to experiences representing features of a novel type. Through the coordinated use of bodily awareness—understood here as encompassing both proprioception and kinaesthesis—and the exteroceptive sensory modalities, one becomes perceptually responsive to spatial features whose instances couldn’t be represented by any of the contributing modalities functioning in isolation. I develop an argument (...) for this conclusion focusing on two cases: 3D shape perception in haptic touch and experiencing an object’s egocentric location in crossmodally accessible, environmental space. (shrink)

This is the editors' Introduction to a special issue of the journal, Multisensory Research. European philosophers of the modern period found multisensory perception to be impossible because they thought that perceptual ideas are defined by how they are experienced. Under this conception, the individual modalities are determinables of ideas—just as colour is a determinable that embraces red and blue, so also the visual is a determinable that embraces colour and (visually experienced) shape. Since no idea is experienced as, (...) for example, both visual and auditory, there can be no such thing as audiovisual perception. This conception of modality is not directly contested, but a variety of perceptual phenomena are listed that could raise interesting questions if treated as multimodal in origin. (shrink)

Suppose that you are at a live jazz show. The drummer begins a solo. You see the cymbal jolt and you hear the clang. But in addition seeing the cymbal jolt and hearing the clang, you are also aware that the jolt and the clang are part of the same event. Casey O’Callaghan (forthcoming) calls this awareness “intermodal feature binding awareness.” Psychologists have long assumed that multimodal perceptions such as this one are the result of a subpersonal feature binding mechanism (...) (see Vatakis and Spence, 2007, Kubovy and Schutz, 2010, Pourtois et al., 2000, and Navarra et al., 2012). I present new evidence against this. I argue that there is no automatic feature binding mechanism that couples features like the jolt and the clang together. Instead, when you experience the jolt and the clang as part of the same event, this is the result of an associative learning process. The cymbal’s jolt and the clang are best understood as a single learned perceptual unit, rather than as automatically bound. I outline the specific learning process in perception called “unitization,” whereby we come to “chunk” the world into multimodal units. Unitization has never before been applied to multimodal cases. Yet I argue that this learning process can do the same work that intermodal binding would do, and that this issue has important philosophical implications. Specifically, whether we take multimodal cases to involve a binding mechanism or an associative process will have impact on philosophical issues from Molyneux’s question to the question of how active or passive we consider perception to be. (shrink)

It is common to distinguish between “holist” and “reductionist” views of brain function, where the former envisions the brain as functioning as an indivisible unit and the latter as a collection of distinct units that serve different functions. Opposing reductionism, a number of researchers have pointed out that cortical network architecture does not respect functional boundaries, and the neuroanatomist V. Braitenberg proposed to understand the cerebral cortex as a “great mixing machine” of neuronal activity from sensory inputs, motor commands, and (...) intrinsically generated processes. In this paper, we offer a contextualization of Braitenberg’s point, and we review evidence for the interactions of neuronal activity from multiple sensory inputs and intrinsic neuronal processes in the cerebral cortex. We focus on new insights from studies on audiovisual interactions and on the influence of respiration on brain functions, which do not seem to align well with “reductionist” views of areal functional boundaries. Instead, they indicate that functional boundaries are fuzzy and context dependent. In addition, we discuss the relevance of the influence of sensory, proprioceptive, and interoceptive signals on cortical activity for understanding brain-body interactions, highlight some of the consequences of these new insights for debates on embodied cognition, and offer some suggestions for future studies. (shrink)

The idea that there are five senses dates back to Aristotle, who was one of the first philosophers to examine them systematically. Though it has become conventional wisdom, many scientists and philosophers would argue that this idea is outdated and inaccurate. Indeed, they have given many different answers to this question, ranging from just three (the number of different kinds of physical energy we can detect) to 33 or more senses. Perhaps surprisingly, the issue remains controversial, partly because it is (...) not clear exactly what should count as a ‘sense’. Should proprioception, balance and kinaesthesis, amongst others, be considered additional senses? And if so, how do we figure out how many senses we really have? -/- (Published in Turkish under the title ‘Çok Duyumlu Algi’). (shrink)

Sense interaction is ubiquitous. All conscious experiences involve at least some interaction between the senses. One of the most debated questions in recent scholarship concerns the proper way of characterizing the phenomenology of multisensory experiences. According to Charles Spence and Tim Bayne (2015), the phenomenal character of multisensory integration is reducible to the co-conscious sum of modality-specific features. Following Casey O'Callaghan (2015), we can call this The Thesis of Minimal Multimodality. The main goal of the paper is to (...) refute the thesis and show that the effect of multisensory integration is reflected in experience in a way that is not exhausted by (the sum of) modality-specific phenomenal features. Whereas O'Callaghan's strategy in trying to prove this consists in providing phenomenological evidence that runs contrary to what Spence and Bayne hold, he does not, however, put into question what has become since T. Nagel's famous essay (1974) the standard conception of the phenomenal. By drawing conceptual resources in the work of Husserl and Merleau-Ponty, I will elaborate another approach and explain why the phenomenal character of experience cannot be reduced to its "what it's like" character. Phenomenal experience is thicker, and also includes various forms of bodily self-experiences and felt possibilities of action and behaviours. Building on these and like insights, the main objective of the paper is to outline a phenomenological account of multimodal perception and sensory interaction. I will draw three main conclusions. First, I will argue that both Husserl and Merleau-Ponty would, too, reject The Thesis of Minimal Multimodality and endorse the strong, Constitutive Thesis that O'Callaghan defends. More interestingly still, we will see, secondly, that the widening of phenomenal consciousness that they argue for allows to generalize the constitutive thesis, thereby showing that all perceptual experience constitutively depend on the interplay of two or more senses. Spelling out the details of this claim will bring me, thirdly, to specify the intrinsic relation between normativity and perceptual experience. In short, I will argue that the mechanisms responsible for multimodality make a phenomenological difference evaluable in normative terms. (shrink)

In this article, I consider the reception of images that are present in a city space. I focus on the juxtaposition of computer‑generated images covering fences surrounding construction sites and the real spaces which they screen from view. I postulate that a visual experience is dependent on input from the other human senses. While looking at objects, we are not only standing in front of them but are being influenced by them. Seeing does not leave a physical trace on the (...) object; instead the interference is more subtle — it influences the way in which we perceive space. Following in the footsteps of Sarah Pink, Michael Taussig and William J. T. Mitchell, I show that seeing (to paraphrase the title of an article by the last of the above mentioned scholars) is a cultural practice. The last part of the article presents a visual essay as a method that can contribute to cultural urban studies. I give as an example of such a method a photo‑essay about chosen construction sites in Poznań, which I photographed between December 2014 and June 2015. (shrink)

In crossmodal identification, a subject token identifies an item perceived in one sensory modality with an item perceived in another sensory modality. Does crossmodal identification always occur in cognition, or does crossmodal identification sometimes take place in perception? This paper argues that crossmodal identification occurs in cognition, and not in perception. Nevertheless, multisensory perception is not unalive to crossmodal identity. Experimental evidence demonstrates that perception is differentially sensitive to the identity of individuals presented to distinct senses. Such sensitivity enhances (...) recognition and improves action. This approach relies on distinguishing crossmodal identification from perceiving crossmodal identity. Perception registers crossmodal identity, but crossmodal identification as such belongs to thought. (shrink)

Long-standing neglect of the chemical senses in the philosophy of perception is due, mostly, to their being regarded as ‘lower’ senses. Smell, taste, and chemically irritated touch are thought to produce mere bodily sensations. However, empirically informed theories of perception can show how these senses lead to perception of objective properties, and why they cannot be treated as special cases of perception modelled on vision. The senses of taste, touch, and smell also combine to create unified perceptions of flavour. The (...) nature of these multimodal experiences and the character of our awareness of them puts pressure on the traditional idea that each episode of perception goes one or other of the five senses. Thus, the chemical senses, far from being peripheral to the concerns of the philosophy of perception, may hold important clues to the multisensory nature of perception in general. (shrink)

From playing basketball to ordering at a food counter, we frequently and effortlessly coordinate our attention with others towards a common focus: we look at the ball, or point at a piece of cake. This non-verbal coordination of attention plays a fundamental role in our social lives: it ensures that we refer to the same object, develop a shared language, understand each other’s mental states, and coordinate our actions. Models of joint attention generally attribute this accomplishment to gaze coordination. But (...) are visual attentional mechanisms sufficient to achieve joint attention, in all cases? Besides cases where visual information is missing, we show how combining it with other senses can be helpful, and even necessary to certain uses of joint attention. We explain the two ways in which non-visual cues contribute to joint attention: either as enhancers, when they complement gaze and pointing gestures in order to coordinate joint attention on visible objects, or as modality pointers, when joint attention needs to be shifted away from the whole object to one of its properties, say weight or texture. This multisensory approach to joint attention has important implications for social robotics, clinical diagnostics, pedagogy and theoretical debates on the construction of a shared world. (shrink)

In this paper we argue that object perception may be affected by what we call “perceptual frames.” Perceptual frames are adaptations of the perceptual system that guide how perceptual objects are singled out from a sensory environment. These adaptations are caused by perceptual learning and realized through bottom-up functional processes such that sensory information is organized in a subject-dependent way leading to idiosyncratic perceptual object representations. Through domain-specific training, perceptual learning, and the acquisition of object-knowledge, it is possible to modulate (...) the adaptive perceptual system such that its ability to represent becomes bespoke. Different perceivers with different perceptual frames may, therefore, receive the same sensory information and perceive different perceptual objects due to the effects of framing. Consequently, we demonstrate the plausibility of this account by surveying empirical data concerning the functions of (1) multisensory integration, (2) amodal completion, and (3) predictive anticipation. Regarding (1), we argue that the perceptual system’s optimization processes employ perceptual frames to facilitate multisensory feature binding. Regarding (2), we argue that amodal completion can occur with or without the help of mental imagery, yet either instance of amodal completion requires perceptual frames. Regarding (3), we demonstrate that perceptually anticipating an object’s motion involves the implementation of perceptual frames. We conclude that framing effects are a matter of perceptual diversity and highlight the need to accommodate unique perspectives in the philosophy and science of perception. (shrink)

The dual role of olfaction in both smelling and tasting, i.e. flavour perception, makes it an important test case for philosophical theories of sensory individuation. Indeed, the psychologist Paul Rozin claimed that olfaction is a “dual sense”, leading some scientists and philosophers to propose that we have not one, but two senses of smell: orthonasal and retronasal olfaction. In this paper I consider how best to understand Rozin’s claim, and upon what grounds one might judge there to be one or (...) two distinct olfactory modalities. I conclude that while Rozin may be right that humans have dual occurrences of an olfactory ‘sense’, the concept of a sense-modality, and hence the ‘sense’ of smell, is ambiguous between two different notions: a physiological sensory channel and an experiential modality, along the lines suggested by J. J. Gibson. Furthermore, recognising that these are complementary rather than competing conceptions of a sense-modality enables the formulation of a powerful ‘dual-concept’ framework for posing and addressing questions concerning the complex architecture of human multisensory experience. (shrink)

Higher animals need to identify and track material objects because they depend on interactions with them for nutrition, reproduction, and social interaction. This paper investigates the perception of material objects. It argues, first, that material objects are tagged, in all five external senses, as bearers of the features detected by them. This happens through a perceptual process, here entitled Generalized Completion, which creates the appearance of objects that have properties that transcend the activation of sensory receptors. The paper shows, secondly, (...) that material objects are privileged subjects for perceived motion and interaction. That is, they are perceived as subjects for these properties while their parts seem to be subjects only derivatively. Material objects are the only perceptual subjects that are both multisensory and privileged. (shrink)

Humans coordinate their focus of attention with others, either by gaze following or prior agreement. Though the effects of joint attention on perceptual and cognitive processing tend to be examined in purely visual environments, they should also show in multisensory settings. According to a prevalent hypothesis, joint attention enhances visual information encoding and processing, over and above individual attention. If two individuals jointly attend to the visual components of an audiovisual event, this should affect the weighing of visual information (...) during multisensory integration. We tested this prediction in this preregistered study, using the well-documented sound-induced flash illusions, where the integration of an incongruent number of visual flashes and auditory beeps results in a single flash being seen as two (fission illusion) and two flashes as one (fusion illusion). Participants were asked to count flashes either alone or together, and expected to be less prone to both fission and fusion illusions when they jointly attended to the visual targets. However, illusions were as frequent when people attended to the flashes alone or with someone else, even though they responded faster during joint attention. Our results reveal the limitations of the theory that joint attention enhances visual processing as it does not affect temporal audiovisual integration. (shrink)

Each of our sensory modalities—vision, touch, taste, etc.—works on a slightly different timescale, with differing temporal resolutions and processing lag. This raises the question of how, or indeed whether, these sensory streams are co-ordinated or ‘bound’ into a coherent multisensory experience of the perceptual ‘now’. In this paper I evaluate one account of how temporal binding is achieved: the temporal windows hypothesis, concluding that, in its simplest form, this hypothesis is inadequate to capture a variety of multisensory phenomena. (...) Rather, the evidence suggests the existence of a more complex temporal structure in which multiple overlapping windows support distinct functional mechanisms. To aid in the precise formulation of such views, I propose a taxonomy of temporal window types and their characteristics that in turn suggests promising avenues for future empirical and philosophical research. I conclude by examining some philosophical implications of multi-window models for the metaphysics of perception and perceptual experience more generally. (shrink)

This paper presents an account of the senses and what differentiates them that is compatible with richly multisensory perception and consciousness. According to this proposal, senses are ways of perceiving. Each sense is a subfaculty that comprises a collection of perceptual capacities. What each sense shares and what differentiates one sense from another is the manner in which those capacities are exercised. Each way of perceiving involves a distinct type of information gathering, individuated by the information it functions to (...) extract and the medium from which it does so. This approach distinguishes the project of characterizing and differentiating senses from that of attributing experiences to sensory modalities. Perceptual experiences are episodes in which perceptual capacities are exer- cised. Conscious perceptual episodes may be ascribed to distinct sensory modalities, according to the manners in which perceptual capacities are deployed on an occasion. According to this account, senses are not exclusive. First, their capacities may overlap. Second, perceptual episodes, including conscious experiences, may belong to multiple senses. Indeed, some episodes require the joint use of several senses. In this account, subjects have only limited first-person knowledge of the senses they employ. (shrink)

We can discern two opposing viewpoints regarding synesthesia. According to the first, it is an oddity, an outlier, or a disordered condition. According to the second, synesthesia is pervasive, driving creativity, metaphor, or language itself. Which is it? Ultimately, I favor the first perspective, according to which cross-sensory synesthesia is an outlying condition. But the second perspective is not wholly misguided. My discussion has three lessons. First, synesthesia is just one of a variety of effects in which one sense modality (...) causally impacts and reshapes experience associated with another. These effects are utterly common. However, due to their unfamiliarity, and to their conflict with a widespread conception of the role of the senses in perception and perceptual experience, until recently they have been surprising. Second, synesthesia nevertheless must be distinguished from other inter-modal effects that lead to misperception, such as crossmodal illusions. Third, synesthesia also may be distinguished from the potentially much broader class of synesthetic effects, which could be common across the population and within individuals. (shrink)

There is converging evidence that high doses of hallucinogenic drugs can produce significant alterations of self-experience, described as the dissolution of the sense of self and the loss of boundaries between self and world. This article discusses the relevance of this phenomenon, known as “drug-induced ego dissolution (DIED)”, for cognitive neuroscience, psychology and philosophy of mind. Data from self-report questionnaires suggest that three neuropharmacological classes of drugs can induce ego dissolution: classical psychedelics, dissociative anesthetics and agonists of the kappa opioid (...) receptor (KOR). While these substances act on different neurotransmitter receptors, they all produce strong subjective effects that can be compared to the symptoms of acute psychosis, including ego dissolution. It has been suggested that neuroimaging of DIED can indirectly shed light on the neural correlates of the self. While this line of inquiry is promising, its results must be interpreted with caution. First, neural correlates of ego dissolution might reveal the necessary neurophysiological conditions for the maintenance of the sense of self, but it is more doubtful that this method can reveal its minimally sufficient conditions. Second, it is necessary to define the relevant notion of self at play in the phenomenon of DIED. This article suggests that DIED consists in the disruption of subpersonal processes underlying the “minimal” or “embodied” self, i.e., the basic experience of being a self rooted in multimodal integration of self-related stimuli. This hypothesis is consistent with Bayesian models of phenomenal selfhood, according to which the subjective structure of conscious experience ultimately results from the optimization of predictions in perception and action. Finally, it is argued that DIED is also of particular interest for philosophy of mind. On the one hand, it challenges theories according to which consciousness always involves self-awareness. On the other hand, it suggests that ordinary conscious experience might involve a minimal kind of self-awareness rooted in multisensory processing, which is what appears to fade away during DIED. (shrink)

We argue that animals are not cognitively stuck in time. Evidence pertaining to multisensory temporal order perception strongly suggests that animals can represent at least some temporal relations of perceived events.

I am aware of the red and orange autumn leaves. Am I aware of my awareness of the leaves? Not so according to many philosophers. By contrast, many meditative traditions report an experience of awareness itself. I argue that such a pure awareness experience must have a non-sensory phenomenal character. I use Douglas Harding’s first-person experiments for assisting in recognising pure awareness. In particular, I investigate the gap where one cannot see one’s head. This is not a mere gap because (...) I seem to be looking from here. Critically, I claim, the experience of looking from here has a non-sensory phenomenal character. I argue that this sense of being aware cannot be reduced to egocentric visual-spatial relations nor the viewpoint because it continues when I close my eyes. Neither is a multisensory origin sufficient to explain why I seem to be at this central point rather than elsewhere. Traditionally, claims of a pure awareness experience have been restricted to highly trained individuals in very restricted circumstances. The innovation of Harding’s approach is that it reliably isolates a candidate for pure awareness using methods which can be replicated at any time. (shrink)

In recent years, the scientific study of meditation and psychedelic drugs has seen remarkable developments. The increased focus on meditation in cognitive neuroscience has led to a cross-cultural classification of standard meditation styles validated by functional and structural neuroanatomical data. Meanwhile, the renaissance of psychedelic research has shed light on the neurophysiology of altered states of consciousness induced by classical psychedelics, such as psilocybin and LSD, whose effects are mainly mediated by agonism of serotonin receptors. Few attempts have been made (...) at bridging these two domains of inquiry, despite intriguing evidence of overlap between the phenomenology and neurophysiology of meditation practice and psychedelic states. In particular, many contemplative traditions explicitly aim at dissolving the sense of self by eliciting altered states of consciousness through meditation, while classical psychedelics are known to produce significant disruptions of self-consciousness, a phenomenon known as drug-induced ego dissolution. In this article, we discuss available evidence regarding convergences and differences between phenomenological and neurophysiological data on meditation practice and psychedelic drug-induced states, with a particular emphasis on alterations of self-experience. While both meditation and psychedelics may disrupt self-consciousness and underlying neural processes, we emphasize that neither meditation nor psychedelic states can be conceived as simple, uniform categories. Moreover, we suggest that there are important phenomenological differences even between conscious states described as experiences of self-loss. As a result, we propose that self-consciousness may be best construed as a multidimensional construct, and that “self-loss,” far from being an unequivocal phenomenon, can take several forms. Indeed, various aspects of self-consciousness, including narrative aspects linked to autobiographical memory, self-related thoughts and mental time travel, and embodied aspects rooted in multisensory processes, may be differently affected by psychedelics and meditation practices. Finally, we consider long-term outcomes of experiences of self-loss induced by meditation and psychedelics on individual traits and prosocial behavior. We call for caution regarding the problematic conflation of temporary states of self-loss with “selflessness” as a behavioral or social trait, although there is preliminary evidence that correlations between short-term experiences of self-loss and long-term trait alterations may exist. (shrink)

Higher animals need to identify and track material objects because they depend on interactions with them for nutrition, reproduction, and social interaction. This paper investigates the perception of material objects. It argues, first, that material objects are tagged, in all five external senses, as bearers of the features detected by them. This happens through a perceptual process, here entitled Generalized Completion, which creates the appearance of objects that have properties that transcend the activation of sensory receptors. The paper shows, secondly, (...) that material objects are privileged subjects for perceived motion and interaction. That is, they are perceived as subjects for these properties while their parts seem to be subjects only derivatively. Material objects are the only perceptual subjects that are both multisensory and privileged. (shrink)

Autism spectrum disorder is characterised by differences in unimodal and multimodal sensory and proprioceptive processing, with complex biases towards local over global processing. Many of these elements are implicated in versions of the rubber hand illusion, which were therefore studied in high-functioning individuals with ASD and a typically developing control group. Both groups experienced the illusion. A number of differences were found, related to proprioception and sensorimotor processes. The ASD group showed reduced sensitivity to visuotactile-proprioceptive discrepancy but more accurate proprioception. (...) This group also differed on acceleration in subsequent reach trials. Results are discussed in terms of weak top-down integration and precision-accuracy trade-offs. The RHI appears to be a useful tool for investigating multisensory processing in ASD. (shrink)

This article proposes a sensory studies methodology for the interpretation of museum objects. The proposed method unfolds in two phases: virtual encounter via an on-line catalog and actual exposure in the context of a handling workshop. In addition to exploring the écart between image and object, the “Sensing Art and Artifacts” exercise articulates a framework for arriving at a multisensory, cross-cultural, interactive understanding of aesthetic value. The case studies presented here involve four objects from the collection of the Hunterian (...) Museum as sensed and interpreted by scholars of psychology, philosophy, sociology, and anthropology. It is proposed that aesthetic judgment in the expanded (cross-cultural) sense contemplated here involves apprehending the museum object through multiple sensory modalities in place of the conventional Western fixation on visible form. (shrink)

Some philosophers have argued that we do not hear sounds as located in the environment. Others have objected that this straightforwardly contradicts the phenomenology of auditory experience. And from this they draw metaphysical conclusions about the nature of sounds—that they are events or properties of vibrating surfaces rather than waves or sensations. I argue that there is a minimal, but recognizable, notion of audition to which this phenomenal objection does not apply. While this notion doesn’t correspond to our ordinary notion (...) of auditory experience, it does—in conjunction with our lack of an uncontroversial individuation of the senses and recent interest in distinctively multisensory features of perceptual experiences—raise the possibility of more expansive notions of audition, including some that do plausibly count as corresponding to our everyday notion of audition, that lack the spatial phenomenology cited in the objection. Until this possibility is ruled out, the phenomenal objection and metaphysical conclusions drawn from it remain inconclusive. (shrink)

Inducing the rubber hand illusion (RHI) requires that participants look at an imitation hand while it is stroked in synchrony with their occluded biological hand. Previous explanations of the RHI have emphasized multisensory integration, and excluded higher cognitive functions. We investigated the relationship between the RHI and higher cognitive functions by experimentally testing task switch (as measured by switch cost) and mind wandering (as measured by SART score); we also included a questionnaire for attentional control that comprises two subscales, (...) attention-shift and attention-focus. To assess experience of RHI, the Botvinick and Cohen (1998) questionnaire was used and illusion onset time was recorded. Our results indicate that rapidity of onset reliably indicates illusion strength. Regression analysis revealed that participants evincing less switch cost and higher attention-shift scores had faster RHI onset times, and that those with higher attention-shift scores experienced the RHI more vividly. These results suggest that the multi-sensory hypothesis is not sufficient to explain the illusion: higher cognitive functions should be taken into account when explaining variation in the experience of ownership for the rubber hand. (shrink)

Mirror neuron research has come a long way since the early 1990s, and many theorists are now stressing the heterogeneity and complexity of the sensorimotor properties of fronto-parietal circuits. However, core aspects of the initial ‘ mirror mechanism ’ theory, i.e. the idea of a symmetric encapsulated mirroring function translating sensory action perceptions into motor formats, still appears to be shaping much of the debate. This article challenges the empirical plausibility of the sensorimotor segregation implicit in the original mirror metaphor. (...) It is proposed instead that the teleological organization found in the broader fronto-parietal circuits might be inherently sensorimotor. Thus the idea of an independent ‘purely perceptual’ goal understanding process is questioned. Further, it is hypothesized that the often asymmetric, heterogeneous and contextually modulated mirror and canonical neurons support a function of multisensory mapping and tracking of the perceiving agents affordance space. Such a shift in the interpretative framework offers a different theoretical handle on how sensorimotor processes might ground various aspects of intentional action choice and social cognition. Mirror neurons would under the proposed “social affordance model” be seen as dynamic parts of larger circuits, which support tracking of currently shared and competing action possibilities. These circuits support action selection processes—but also our understanding of the options and action potentials that we and perhaps others have in the affordance space. In terms of social cognition ‘ mirror ’ circuits might thus help us understand not only the intentional actions others are actually performing—but also what they could have done, did not do and might do shortly. (shrink)

Technology-enhanced learning settings are changing quickly and complexly in the contemporary digital era, making it possible for students with disabilities to learn more effectively than before. The words "multisensory" and "media" together, however, suggest that this strategy entails incorporating several sensory modalities in educational media to improve learning experiences for children with disabilities. It can entail integrating visual, aural, tactile, and kinesthetic elements to meet various learning requirements and styles. This article examines how Mulsemedia, one of these cutting-edge technologies, (...) enhances learning methodologies, improving the teaching and learning of unique pedagogies and emphasizing teaching and learning-related modules for students with special needs. The researchers used a qualitative research design to understand and review the ten papers with secondary data based connected to mulsemedia in special education: A novel teaching approach for the next generation. The article also describes the extremely encouraging outcomes of case studies conducted with engineering students with disabilities in several schools in India. This critical article finds that multimedia-enhanced instruction significantly improves special students' learning experiences and ability to learn new information for future development. (shrink)

Two universal functional principles of Grossberg’s Adaptive Resonance Theory [19] decipher the brain code of all biological learning and adaptive intelligence. Low-level representations of multisensory stimuli in their immediate environmental context are formed on the basis of bottom-up activation and under the control of top-down matching rules that integrate high-level long-term traces of contextual configuration. These universal coding principles lead to the establishment of lasting brain signatures of perceptual experience in all living species, from aplysiae to primates. They are (...) re-visited in this concept paper here on the basis of examples drawn from the original code and from some of the most recent related empirical findings on contextual modulation in the brain, highlighting the potential of Grossberg’s pioneering insights and groundbreaking theoretical work for intelligent solutions in the domain of developmental and cognitive robotics. (shrink)

A myriad of sensations inform and direct us when we engage with the environment. To understand their influence on the development of our habitus it is important to focus on unifying processes in sensing. This approach allows us to include phenomena that elude a rather narrow view that focuses on each of the five discrete senses in isolation. One of the central questions addressed in this volume is whether there is something like a sensual habitus, and if there is, how (...) it can be defined. This is especially done by exploring the formation and habituation of the senses in and by a culturally shaped habitat. Two key concepts, Synaesthesia and Kinaesthetics, are addressed as essential components for an understanding of the interface of habitat and the rich and multisensory experience of a perceiving subject. At a Berlin-based conference Synaesthesia and Kinaesthetics, scholars from various disciplines gathered to discuss these issues. In bringing together the outcome of these discussions, this book gives new insights into the key phenomena of sensory integration and synaesthetic experiences, it enriches the perspectives on sensually embedded interaction and its habituation, and it expands this interdisciplinary inquiry to questions about the cultures of sensory habitus. (shrink)

We can see ruins as objects that have a foot in three different times: the past, the present, and the future. This is the nature of the ruin: they help us imagine the past, affords us interesting aesthetic opportunities in the present, and asks us to project ourselves (and it) into the future. We think about those who once lived, our own current experience, and what will be. There are two kinds of aesthetic experiences of ruins that are often conflated: (...) that of recording the ruin be it in poetry, prose, or picture and that of engaging with a ruin qua ruin (the experience of the ruin in situ). In this paper I discuss ruin appreciation and the major and minor appreciations discussed by Rose Macaulay. Minor forms of appreciation include graffiti while contemplating the sublime and engaging in multisensory ruinlust are considered major forms of appreciation. (shrink)

To create an immersive experience, it is not important to have a plethora of stimuli for different sensors, but to have their coherence, or in other words, multisensory integration. This coherence becomes the necessary basis for immersing oneself in a story, a game, a movie, or any other experience. Sound plays a key role in creating such an experience, both by accompanying the action and emphasizing the narrative flow, and as an independent actor that takes a central role and (...) directs attention. The aim of the paper and poster is to explore the means by which sound can influence immersion in computer-generated virtual environments and the psychological mechanisms involved in the musical induction of emotions. The main question to be answered is how these techniques and models can be used to create a sense of immersion. The first part of the article examines immersiveness itself. The second part is devoted to the practical aspects of sound design for immersive technologies, describing the tools and techniques used to create realistic and compelling soundscapes. The third part touches briefly on the psychological mechanisms behind the induction of musical emotions. Taking a holistic view of these components and using them to reveal each other is how sound can create a sense of presence and place. (shrink)

Studies have shown that both serotonin and glutamate receptor systems play a crucial role in the mechanisms underlying drug-induced synesthesia. The specific nature of these mechanisms, however, continues to remain elusive. Here we propose two distinct hypotheses for how synesthesia triggered by hallucinogens in the serotonin-agonist family may occur. One hypothesis is that the drug-induced destabilization of thalamic projections via GABAergic neuronal circuits from sensory areas leads to a disruption of low-level, spontaneous integration of multisensory stimuli. This sort of (...) integration regularly occurs when spatial and temporal attributes match. Destabilization of feedback loops, however, can result in incongruent experiences or binding of random thalamus activation with sensory input in a particular sensory modality. The second hypothesis builds on embodied cognition, cases in which visual images of external stimuli activate task-related neural regions. On this proposal, binding processes that do not normally generate awareness become accessible to consciousness as a result of decreased attentional discrimination among incoming stimuli. (shrink)

Our emotional states affect how we perceive the world. If I am stressed, annoyed, or irritated, I might experience the sound of children laughing and screaming as they play around the house in a negative manner — it is unpleasant, loud, piercing, and so on. Yet, if I’m in a relaxed, happy, loving mood, the very same sounds might be experienced as pleasant, playful, warm, and so on. The sounds being made by the children are the same in both cases, (...) but how they are experienced differs. The auditory experience is affected by the emotional state that I am in. I perceive the sounds differently depending on how I’m feeling. Although this might at first seem to be a trivial observation, it certainly is not. We take for granted that our perceptual experiences give rise to justified beliefs and knowledge. Thus, if my perceptual experience of an object can differ based on my emotional state, and I take my perceptual experience to justify my beliefs and even lead to knowledge, then we might have a problem. If emotions — which are not fixed and in large part uncontrolled — affect our ability to accurately perceive the world, then they may undermine the justified beliefs and knowledge gained on the basis of our perceptual experience of the world. My goal here is to explore this potential problem. -/- I consider how we might understand the effect that emotions have on the justification of our perceptual beliefs about the world, beliefs that we acquire from a variety of sensory modalities — audition, gustation olfaction, and so on. I take the problem to be associated with one of two forms of perceptual influence: penetration or multisensory integration. In any given perceptual moment there are multiple sensory modalities and mental states at play, each affecting the overall experience. Whether we understand the influence of emotion on perception as a form of non-perceptual penetration or a form of non-visual sensory perception of the inner body — interoception — the potential epistemological difficulties remain: how can we be said to acquire justified beliefs and knowledge on the basis of such influenced perceptual experience. As has been the norm, only the five exteroceptive senses of vision, audition, olfaction, taste and touch are typically discussed in the context of sensory perception. However, as I argue, there is strong reason to accept the claim that emotional experience is a form of interoception, and that interoception ought to be considered when discussing sensory perception. In this way, then, I propose that clarifying the role played by interoception in sense perception across modalities is necessary if we are to make progress on the epistemological problems at hand. (shrink)

Susceptibility to the rubber hand illusion (RHI) varies. To date, however, there is no consensus explanation of this variability. Previous studies, focused on the role of multisensory integration, have searched for neural correlates of the illusion. But those studies have failed to identify a sufficient set of functionally specific neural correlates. Because some evidence suggests that frontal α power is one means of tracking neural instantiations of self, we hypothesized that the higher the frontal α power during the eyes-closed (...) resting state, the more stable the self. As a corollary, we infer that the more stable the self, the less susceptible are participants to a blurring of boundaries—to feeling that the rubber hand belongs to them. Indeed, we found that frontal α amplitude oscillations negatively correlate with susceptibility. Moreover, since lower frequencies often modulate higher frequencies, we explored the possibility that this might be the case for the RHI. Indeed, some evidence suggests that high frontal α power observed in low-RHI participants is modulated by δ frequency oscillations. We conclude that while neural correlates of multisensory integration might be necessary for the RHI, sufficient explanation involves variable intrinsic neural activity that modulates how the brain responds to incompatible sensory stimuli. (shrink)

This contribution examines two radically different explanations of our phenomenal intuitions, one reductive and one strongly non-reductive, and identifies two germane ideas that could benefit many other theories of consciousness. Firstly, the ability of sophisticated agent architectures with a purely physical implementation to support certain functional forms of qualia or proto-qualia appears to entail the possibility of machine consciousness with qualia, not only for reductive theories but also for the nonreductive ones that regard consciousness as ubiquitous in Nature. Secondly, analysis (...) of introspective psychological material seems to hint that, under the threshold of our ordinary waking awareness, there exist further ‘submerged’ or ‘subliminal’ layers of consciousness which constitute a hidden foundation and support and another source of our phenomenal intuitions. These ‘submerged’ layers might help explain certain puzzling phenomena concerning subliminal perception, such as the apparently ‘unconscious’ multisensory integration and learning of subliminal stimuli. (shrink)

In the age of ubiquitous technology, humans are reshaped through each transaction they are involved in. AI-driven networks, online games, and multisensory interactive environments make up alternate realities. Within such alternate worlds, users are reshaped as deterministic agents. Technology’s focus on reducing complexity leads to a human being dependent on prediction-driven machines and behaving like them. Meaning and information are disconnected. Existence is reduced to energy processes. The immense gain in efficiency translates as prosperity. Citizens of advanced economies, hurrying (...) in the rhythm of machine-driven interactions, feel entitled to it. Successful at the price of self-awareness, they no longer know what this means. Happiness and prosperity are not consubstantial. Lack of happiness leads to aggression. This is the image of the world as we see it, no longer looking at each other, eye to eye, but screen to screen. The questions eliminated in the process of transferring responsibility from the individual to machines will inevitably become society’s new focus. When the goal is to get everyone to behave like a machine, the Singularity hypothesis becomes self-fulfilling prophecy. If, in addition to having exhausted natural resources, society does not want to end up making the human mind superfluous, it has to seek a better understanding of what makes anticipation possible. As a definitory characteristic of the living, corresponding to its complexity, anticipation can no longer be taken for granted, while every effort is made to reduce complexity for efficiency’s sake. Awareness of the processes conducive to its expression in successful human action will position human beings as masters of their destiny, not slaves of their own making. Antecapere ergo sum might be the counterclaim to Descartes’ Dubito ergo sum . To resist being perfected into oblivion, that is, unsustainable prosperity, means to reclaim the knowledge corresponding to higher levels of complexity. (shrink)

This chapter offers an indirect defence of the Evansian conception of egocentric space, by showing how it resolves a puzzle concerning the unity of egocentric spatial perception. The chapter outlines several common assumptions about egocentric perspectival structure and argues that a subject’s experience, both within and across her sensory modalities, may involve multiple structures of this kind. This raises the question of how perspectival unity is achieved, such that these perspectival structures form a complex whole, rather than merely disunified set (...) of individually, distinctively structured experiences. The shortcomings of variety of accounts are considered: switch accounts ; sensory accounts; transformation accounts; and ultimate accounts. These shortcomings are addressed by a further kind of account provided by the Evansian conception – an agentive account – according to which egocentrically structured experiences present the world in relation to parts of a single thing, the body as a dynamic unity. (shrink)

With a Call for Essays, the special issue Multimodality sought contributions that accept not only the material but also the body-bound dependence of media perception and understanding. To this end, contributions were included that shed light on both the structural and signifying potential of artistic works through multimodal analysis. Particular attention was paid to contributions that clarify how the structural features - the modes - of the arts, their perception, and their signifying potential in terms of content are interrelated and (...) how they are to be understood in communicative and thus socioculturally relevant terms. Thus, in addition to neuroscientific contributions, those from cultural anthropology, art history, image and art studies, and literary studies were included. -/- Articles -/- 8 Martina Sauer and Christiane Wagner - Introduction -/- 11 Giada Lombardi and Giuseppe Di Cesare - From Neuroscience to Art: The Role of ‘Vitality Forms’ in the Investigation of Multimodality -/- 25 Martina Sauer - Abstract - Affective – Multimodal: Interaction between Medium and Perception of Moving Images from the Viewpoint of Cassirer’s, Langer’s and Krois’ Embodiment Theories -/- 49 Göran Sonesson - The Secrets of Plastic Language Revealed: Multimodality, Polysemiosis, and Iconicity -/- 71 Natalia Igl - Making Sense of the Material: Multisensory Reader Involvement in Contemporary Multimodal Novels -/- 87 J. Philippe Thompson - Art is All: Proust’s Life Lessons Through Perception and Art -/- 107 Hans Dieter Huber - Philippe Parreno and the Exhibition as a Multimodal Aesthetic Experience -/- 135 Jörg U. Lensing - The Performing Human Being in a Media Interaction Space: Multi- and intermodal productions by the Düsseldorf Theater der Klänge (Theatre of Sounds) . (shrink)

When a user integrates a sensory substitution device into her life, the process involves perceptual learning, that is, ‘relatively long-lasting changes to an organism’s perceptual system that improve its ability to respond to its environment’ (Goldstone 1998: 585). In this paper, I explore ways in which the extensive literature on perceptual learning can be applied to help improve sensory substitution devices. I then use these findings to answer a philosophical question. Much of the philosophical debate surrounding sensory substitution devices concerns (...) what happens after perceptual learning occurs. In particular, should the resultant perceptual experience be classified in the substituted modality (as vision), in the substituting modality (as auditory or tactile), or in a new sense modality? I propose a novel empirical test to help resolve this philosophical debate. (shrink)

We perceive in many ways. But several dubious presuppositions about the senses mask this diversity of perception. Philosophers, scientists, and engineers alike too often presuppose that the senses (vision, audition, etc.) are independent sources of information, perception being a sum of these independent contributions. We too often presuppose that we can generalize from vision to other senses. We too often presuppose that vision itself is best understood as a passive receptacle for an image thrown by a lens. In this essay (...) we illustrate how reflection on results from the cognitive sciences undermine these (and many other) traditional presuppositions. Our illustration situates the remaining essays in the volume, Perception and its Modalities, which further extend the case against various traditional presuppositions, and offer alternatives ways to understand the senses and their role in perception. While many of the essays in this volume represent competing views and draw on different fields, from conceptual analysis to engineering, most contributors believe that to understand perception properly, we need to treat as foundational the idea that the senses work together. From that agreement, new paradigms for theorizing about perception begin to emerge. (shrink)

Some investigators of the rubber hand illusion (RHI) have suggested that when standard RHI induction procedures are employed, if the rubber hand is experienced by participants as owned, their corresponding biological hands are experienced as disowned. Others have demurred: drawing upon a variety of experimental data and conceptual considerations, they infer that experience of the RHI might include the experience of a supernumerary limb, but that experienced disownership of biological hands does not occur. Indeed, some investigators even categorically deny that (...) any experimental paradigm has been employed or any evidence can be adduced to support the claim that disownership experiences occur during the RHI. It goes without saying that RHI experiences can be elusive, and that there is some evidence to support claims that supernumerary limb experiences can sometimes occur. Here, however, we test the claim that the conscious experience of disownership can occur during the RHI. In order to test this claim, we developed two new online proxies—onset time for the illusion and illusion duration—and combined these with established questionnaires that concern the conscious contents of the RHI, in particular ownership/disownership experiences. Both online proxy data and post hoc questionnaire data converge in supporting the claim that disownership experiences do occur, at least when the left hand is the object of investigation. Our findings that onset time and illusion duration are reliable measures suggest that investigations of the RHI stand to benefit by devoting more attention to data collected while the RHI is being experienced, in particular data concerning temporal dynamics. (shrink)

The First Sense: A Philosophical Study of Human Touch is one of the rare contributions in theoretical and philosophical psychology exclusively on human's sense of touch in the past decades. Although the study is conducted from a philosophical point of view, it is highly empirically informed. The author seeks to base his distinctions and arguments on empirical findings, but also offers his own original ideas and theses. In Section The Structure and Contents of the Book I discuss the structure and (...) contents of the book with an emphasis on the central thesis “the unity of touch,” and in Section A Different Perspective I contrast that central thesis with a different perspective. (shrink)

It has been repeatedly suggested that synesthesia is intricately connected with unusual ways of exercising one’s mental imagery, although it is not always entirely clear what the exact connection is. My aim is to show that all forms of synesthesia are forms of (often very different kinds of) mental imagery and, further, if we consider synesthesia to be a form of mental imagery, we get significant explanatory benefits, especially concerning less central cases of synesthesia where the inducer is not sensory (...) stimulation. (shrink)