Crossmodal Perception

We can discern two opposing viewpoints regarding synesthesia. According to the first, it is an oddity, an outlier, or a disordered condition. According to the second, synesthesia is pervasive, driving creativity, metaphor, or language itself. Which is it? Ultimately, I favor the first perspective, according to which cross-sensory synesthesia is an outlying condition. But the second perspective is not wholly misguided. My discussion has three lessons. First, synesthesia is just one of a variety of effects in which one sense modality (...) causally impacts and reshapes experience associated with another. These effects are utterly common. However, due to their unfamiliarity, and to their conflict with a widespread conception of the role of the senses in perception and perceptual experience, until recently they have been surprising. Second, synesthesia nevertheless must be distinguished from other inter-modal effects that lead to misperception, such as crossmodal illusions. Third, synesthesia also may be distinguished from the potentially much broader class of synesthetic effects, which could be common across the population and within individuals. (shrink)

Suppose that you are at a live jazz show. The drummer begins a solo. You see the cymbal jolt and you hear the clang. But in addition seeing the cymbal jolt and hearing the clang, you are also aware that the jolt and the clang are part of the same event. Casey O’Callaghan (forthcoming) calls this awareness “intermodal feature binding awareness.” Psychologists have long assumed that multimodal perceptions such as this one are the result of a subpersonal feature binding mechanism (...) (see Vatakis and Spence, 2007, Kubovy and Schutz, 2010, Pourtois et al., 2000, and Navarra et al., 2012). I present new evidence against this. I argue that there is no automatic feature binding mechanism that couples features like the jolt and the clang together. Instead, when you experience the jolt and the clang as part of the same event, this is the result of an associative learning process. The cymbal’s jolt and the clang are best understood as a single learned perceptual unit, rather than as automatically bound. I outline the specific learning process in perception called “unitization,” whereby we come to “chunk” the world into multimodal units. Unitization has never before been applied to multimodal cases. Yet I argue that this learning process can do the same work that intermodal binding would do, and that this issue has important philosophical implications. Specifically, whether we take multimodal cases to involve a binding mechanism or an associative process will have impact on philosophical issues from Molyneux’s question to the question of how active or passive we consider perception to be. (shrink)

According to the decomposition thesis, a subject’s total perceptual experience at a time is an aggregate of discrete, modality‐specific experiences. Contrary to this view, I argue that certain cases of multisensory integration give rise to experiences that represent features of a novel type. Through the coordinated use of bodily awareness – understood here as encompassing both proprioception and kinaesthesis – and the exteroceptive sensory modalities, one becomes perceptually responsive to spatial features whose instances couldn’t be represented by any of the (...) contributing modalities functioning in isolation. I develop an argument for this conclusion focusing on two cases: 3D shape perception in haptic touch and experiencing an object’s egocentric location in crossmodally accessible space. (shrink)

Inducing the rubber hand illusion (RHI) requires that participants look at an imitation hand while it is stroked in synchrony with their occluded biological hand. Previous explanations of the RHI have emphasized multisensory integration, and excluded higher cognitive functions. We investigated the relationship between the RHI and higher cognitive functions by experimentally testing task switch (as measured by switch cost) and mind wandering (as measured by SART score); we also included a questionnaire for attentional control that comprises two subscales, attention-shift (...) and attention-focus. To assess experience of RHI, the Botvinick and Cohen (1998) questionnaire was used and illusion onset time was recorded. Our results indicate that rapidity of onset reliably indicates illusion strength. Regression analysis revealed that participants evincing less switch cost and higher attention-shift scores had faster RHI onset times, and that those with higher attention-shift scores experienced the RHI more vividly. These results suggest that the multi-sensory hypothesis is not sufficient to explain the illusion: higher cognitive functions should be taken into account when explaining variation in the experience of ownership for the rubber hand. (shrink)

Intermodal representationalists hold that the phenomenal characters of experiences are fully determined by their contents. In contrast, intramodal representationalists hold that the phenomenal characters of experiences are determined by their contents together with their intentional modes or manners of representation, which are nonrepresentational features corresponding roughly to the sensory modalities. This paper discusses a kind of experience that provides evidence for an intermodal representationalist view: intermodal experiences, experiences that unify experiences in different modalities. I argue that such experiences are much (...) easier to explain on the intermodal view. (shrink)

(This paper will appear in the Australasian Journal of Philosophy. The Accepted Version will be posted when it is published.) -/- Molyneux's Question (MQ) concerns whether a newly sighted man would recognize/distinguish a sphere and a cube by vision, assuming he could previously do this by touch. We argue that (MQ) splits into questions about (a) shared representations of space in different perceptual systems, and about (b) shared ways of constructing higher dimensional spatiotemporal features from information about lower dimensional ones, (...) most of the technical difficulty centring on (b). So understood, MQ resists any monolithic answer: everything depends on the constraints faced by particular perceptual systems in extracting features of higher dimensionality from those of lower. Each individual question of this type is empirical and must be investigated separately. We present several variations on MQ based on different levels of dimensional integration—some of these are familiar, some novel adaptations of problems known elsewhere, and some completely novel. Organizing these cases in this way is useful because it unifies a set of disparate questions about intermodal transfer that have held philosophical and psychological interest, suggests a new range of questions of the same type, sheds light on similarities and differences between members of the family, and allows us to formulate a much-augmented set of principles and questions concerning the intermodal transfer of spatiotemporal organization. (shrink)

This paper is a defense of an internalist view of the perception of shapes. A basic assumption of the paper is that perceptual experiences have certain parts which account both for the phenomenal character associated with perceiving shapes—phenomenal shapes—and for the intentional content presenting shapes—intentional shapes. Internalism about perceptions of shapes is defined as the claim that phenomenal shapes determine the intentional shapes. Externalism is defined as the claim that perceptual experiences represent whatever shape the phenomenal shape reliably tracks. The (...) argument against externalism proceeds in three steps. First, it is argued that phenomenal shapes are modality specific, such that a phenomenal shape that features in a visual perceptual experience cannot feature in a haptic perceptual experience, and vice versa. Second, it is argued that intentional shapes are amodal. Third, it is argued that externalism is incompatible with the fact that phenomenal shapes are modality specific and intentional shapes amodal. (shrink)

The first part of this survey article presented a cartography of some of the more extensively studied forms of multisensory processing. In this second part, I turn to examining some of the different possible ways in which the structure of conscious perceptual experience might also be characterized as multisensory. In addition, I discuss the significance of research on multisensory processing and multisensory consciousness for philosophical debates concerning the modularity of perception, cognitive penetration, and the individuation of the senses.

A recent empirical study claims to show that the answer to Molyneux’s question is negative, but, as John Schwenkler points out, its findings are inconclusive: Subjects tested in this study probably lacked the visual acuity required for a fair assessment of the question. Schwenkler is undeterred. He argues that the study could be improved by lowering the visual demands placed on subjects, a suggestion later endorsed and developed by Kevin Connolly. I suggest that Connolly and Schwenkler both underestimate the difficulties (...) involved in rectifying the study they seek to fix. The problem is that the experimental paradigm under consideration fails to account for the role that rational inference plays in newly sighted subjects’ ability or inability to recognize spatial properties across modalities. Since answering Molyneux’s question requires establishing whether spatial properties can be recognized, across modalities, by newly sighted subjects without recourse to rational inference, this is a problem. Indeed, it is a problem that may be worsened by Schwenkler and Connolly’s suggestions regarding the lowering of visual demands on subjects in cross-modal matching tasks. (shrink)

Multisensory processing encompasses all of the various ways in which the presence of information in one sensory modality can adaptively influence the processing of information in a different modality. In Part I of this survey article, I begin by presenting a cartography of some of the more extensively investigated forms of multisensory processing, with a special focus on two distinct types of multisensory integration. I briefly discuss the conditions under which these different forms of multisensory processing occur as well as (...) their important perceptual consequences and interrelations. In Part II, I then turn to examining of some of the different possible ways in which the structure of conscious perceptual experience might also be characterized as multisensory. In addition, I discuss the significance of research on multisensory processing and multisensory consciousness for philosophical attempts to individuate the senses. (shrink)

Synesthesia is the “union of the senses” whereby two or more of the five senses that are normally experienced separately are involuntarily and automatically joined together in experience. For example, some synesthetes experience a color when they hear a sound or see a letter. In this paper, I examine two cases of synesthesia in light of the notions of “experiential parts” and “conscious unity.” I first provide some background on the unity of consciousness and the question of experiential parts. I (...) then describe two very different cases of synesthesia. Finally, I critically examine the cases in light of two central notions of “unity.” I argue that there is good reason to think that the neural “vehicles” of conscious states are distributed widely and can include multiple modalities. I also argue that some synesthetic experiences do not really enjoy the same “object unity” associated with normal vision. (shrink)

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: Do multisensory percepts involve emergent features?

This report highlights and explores five questions that arose from the multisensory integration workshop at the University of Toronto on May 9th and 10th, 2014: 1. What Is Multisensory Integration? 2. Do Multisensory Percepts Involve Emergent Features? 3. What Can Multisensory Processing Tell Us about Multisensory Awareness? 4. Is Language Processing a Special Kind of Multisensory Integration? 5. What Is the Purpose of Multisensory Integration?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: What is the purpose of multisensory integration?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: What can multisensory processing tell us about multisensory awareness?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: What is multisensory integration?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: Is language processing a special kind of multisensory integration?

This paper provides a categorization of cross-modal experiences. There are myriad forms. Doing so allows us to think clearly about the nature of different cross-modal experiences and allows us to clearly formulate competing hypotheses about the kind of experiences involved in different cross-modal phenomena.

We perceive in many ways. But several dubious presuppositions about the senses mask this diversity of perception. Philosophers, scientists, and engineers alike too often presuppose that the senses (vision, audition, etc.) are independent sources of information, perception being a sum of these independent contributions. We too often presuppose that we can generalize from vision to other senses. We too often presuppose that vision itself is best understood as a passive receptacle for an image thrown by a lens. In this essay (...) we illustrate how reflection on results from the cognitive sciences undermine these (and many other) traditional presuppositions. Our illustration situates the remaining essays in the volume, Perception and its Modalities, which further extend the case against various traditional presuppositions, and offer alternatives ways to understand the senses and their role in perception. While many of the essays in this volume represent competing views and draw on different fields, from conceptual analysis to engineering, most contributors believe that to understand perception properly, we need to treat as foundational the idea that the senses work together. From that agreement, new paradigms for theorizing about perception begin to emerge. (shrink)

This is an excerpt from a report on the Sensory Substitution and Augmentation Conference at the British Academy in March of 2013. This portion of the report explores the question: What are the limitations of sensory substitution.

This is an excerpt from a report on the Sensory Substitution and Augmentation Conference at the British Academy in March of 2013. This portion of the report explores the question: Does sensory substitution generate perceptual or cognitive states?

This is an excerpt from a report on the Sensory Substitution and Augmentation Conference at the British Academy in March of 2013. This portion of the report explores the question: How does sensory substitution interact with the brain’s architecture?

This is an excerpt from a report on the Sensory Substitution and Augmentation Conference at the British Academy in March of 2013. This portion of the report explores the question: What can sensory substitution tell us about perceptual learning?

This report highlights and explores five questions that arose from the workshop on sensory substitution and augmentation at the British Academy, March 26th through 28th, 2013.

In the case of ventriloquism, seeing the movement of the ventriloquist dummy’s mouth changes your experience of the auditory location of the vocals. Some have argued that cases like ventriloquism provide evidence for the view that at least some of the content of perception is fundamentally multimodal. In the ventriloquism case, this would mean your experience has constitutively audio-visual content (not just a conjunction of an audio content and visual content). In this paper, I argue that cases like ventriloquism do (...) not in fact warrant that conclusion. I then try to make sense of crossmodal cases without appealing to fundamentally multimodal content. (shrink)

Kant argued that the perceptual representations of space and time were templates for the perceived spatiotemporal ordering of objects, and common to all modalities. His idea is that these perceptual representations were specific to no modality, but prior to all—they are pre-modal, so to speak. In this paper, it is argued that active perception—purposeful interactive exploration of the environment by the senses—demands premodal representations of time and space.

Ordinary perceiving relies heavily on our sensing the spatial properties of objects, e.g., their shapes, sizes, and locations. Such spatial perception is central in everyday life. We safely cross a street by seeing and hearing the locations of oncoming vehicles. And we often identify objects by seeing and feeling their distinctive shapes. -/- To understand how we perceive spatial properties, we must explain the nature of the mental states figuring in spatial perception. The experience one has when seeing a cube, (...) e.g., differs from the experiences one has when seeing other shapes, e.g., spheres and pyramids. We must explain how such experiences differ to fully understand how we perceive differences in the spatial properties of objects. This presents a challenge often overlooked in philosophy and cognitive science. Whereas we can differentiate physical objects by their spatial properties, we cannot differentiate the experiences involved in perception in respect of their own spatial properties. Experiences are mental states, not physical objects, so they do not themselves have spatial properties; a visual experience of a 50 ft. tall cube, e.g., isn’t itself 50 ft. tall or cubical. So we must differentiate our perceptual experiences of those objects some other way, in terms of their own properties. -/- I argue the experiences figuring in spatial perception have mental properties distinct from, but analogous to, the spatial properties we perceive. The experience one has when seeing a square, e.g., has a property that resembles and differs from other such mental properties in ways parallel to the ways physical squares resemble and differ from other shapes. Just as squares are more similar to rectangles than triangles, the mental property of an experience of a square is more similar to that of an experience of a rectangle than that of an experience of a triangle. -/- I show how this theory helps solve several problems in philosophy and cognitive science; explaining change blindness, accounting for our ability to perceive combinations of distinct properties, e.g., color and shape, and determining whether the properties of experiences pertaining to the same spatial properties in different sensory modalities are themselves the same. (shrink)