From playing basketball to ordering at a food counter, we frequently and effortlessly coordinate our attention with others towards a common focus: we look at the ball, or point at a piece of cake. This non-verbal coordination of attention plays a fundamental role in our social lives: it ensures that we refer to the same object, develop a shared language, understand each other’s mental states, and coordinate our actions. Models of joint attention generally attribute this accomplishment to gaze coordination. But (...) are visual attentional mechanisms sufficient to achieve joint attention, in all cases? Besides cases where visual information is missing, we show how combining it with other senses can be helpful, and even necessary to certain uses of joint attention. We explain the two ways in which non-visual cues contribute to joint attention: either as enhancers, when they complement gaze and pointing gestures in order to coordinate joint attention on visible objects, or as modality pointers, when joint attention needs to be shifted away from the whole object to one of its properties, say weight or texture. This multisensory approach to joint attention has important implications for social robotics, clinical diagnostics, pedagogy and theoretical debates on the construction of a shared world. (shrink)

It is commonly believed that human perceptual experiences can be, and usually are, multimodal. What is more, a stronger thesis is often proposed that some perceptual multimodal characters cannot be described simply as a conjunction of unimodal phenomenal elements. If it is the case, then a question arises: what is the additional mode of combination that is required to adequately describe the phenomenal structure of multimodal experiences? The paper investigates what types of audio–visual experiences have phenomenal character that cannot be (...) analysed as a mere conjunction of visual and auditory elements; and how can we properly characterise the required, additional mode of perceptual combination. Three main modes of combination are considered: instantiation, parthood, and grouping. It is argued that some phenomena involving intermodal relations, like spatial and temporal ventriloquism, can be analysed in terms of audio–visual, perceptual grouping. On the other hand, cases of intermodal binding need a different treatment. Experiences involving audio–visual binding should be analysed as experiences presenting objects or events which instantiate, or which have a proper part instantiating, both visually and auditorily determined properties. (shrink)

Experimental research has shown that pairs of stimuli which are congruent and assumed to 'go together' are recalled more effectively than an item presented in isolation. Will this multisensory memory benefit occur when stimuli are richer and longer, in an ecological setting? In the present study, we focused on an everyday situation of audio-visual learning and manipulated the relationship between audio guide tracks and viewed portraits in the galleries of the Tate Britain. By varying the gender and narrative style of (...) the voice-over, we examined how the perceived congruency and assumed unity of the audio guide track with painted portraits affected subsequent recall. We show that tracks perceived as best matching the viewed portraits led to greater recall of both sensory and linguistic content. We provide the first evidence that manipulating crossmodal congruence and unity assumptions can effectively impact memory in a multisensory ecological setting, even in the absence of precise temporal alignment between sensory cues. (shrink)

An audiovisual correspondence (AVC) refers to an observer’s seemingly arbitrary yet consistent matching of sensory features across the two modalities; for example, between an auditory pitch and visual size. Research on AVCs has frequently used a speeded classification procedure in which participants are asked to rapidly classify an image when it is either accompanied by a congruent or an incongruent sound (or vice versa). When, as is typically the case, classification is faster in the presence of a congruent stimulus, researchers (...) have inferred that the AVC is automatic and bottom-up. Such an inference is incomplete because the procedure does not show that the AVC is not subject to top-down influences. To remedy this problem, we devised a procedure that allows us to assess the degree of “bottom-up-ness” and “top-down-ness” in the processing of an AVC. We did this in studies of AVCs between pitch and five visual features: size, height, spatial frequency, brightness, and angularity. We find that all the AVCs we studied involve both bottom-up and top-down processing, thus undermining the prevalent generalization that AVCs are automatic. (shrink)

It is common to characterize pain with touch-related terms, like ‘cutting’, ‘pressing’, ‘sharp’, and ‘pulsing’, or temperature-related terms, like ‘hot’ or ‘burning’. This suggests that many pains are phenomenally multimodal because they are experienced as having some tactile-like or thermal-like character. The goal of this chapter is to investigate the structure of phenomenally multimodal pain experiences. It is argued that the usual accounts of multimodal structure proposed in investigations regarding exteroceptive experiences cannot be plausibly applied to multimodal experiences of pain. (...) Instead, an alternative framework is proposed which characterizes the structure of tactile-like and thermal-like pains by referring to the notion of crossmodal correspondences. (shrink)

According to cognitive psychology, virtually every sensory system influences the way in which flavours are experienced. However, it is less clear which systems are actually constitutive of flavour perception and which have merely causal influence. The paper focuses on the status of vision and audition, which are usually not treated as constitutive in the context of flavour perception. First, it is proposed that the mechanistic explanation debate provides conceptual resources which allow the constitutivity of sensory systems to be assessed. Second, (...) it is argued that the contemporary state of the art provides reasons for accepting the constitutive role of audition, but analogous evidence is not available in the case of vision. (shrink)

Space—an essential dimension of our life—is analyzable from different viewpoints, which often gives rise to contrasting conceptualizations. Perceptual analyses shed light on the intrinsic anisotropy and deformations of perceived space, raising the issue of which geometry may be able to represent perceptual space. Pictorial drawings and painting have been relevant sources of information about the nature of living and perceived space. Although the geometry of perceptual space is still in its infancy, contributions are beginning to appear. This special issue contributes (...) to the development of the geometry of perceptual space. (shrink)

Previous studies have shown that Chinese speakers and non-Chinese speakers exhibit different patterns of cross-modal congruence for the lexical tones of Mandarin Chinese, depending on which features of the pitch they attend to. But is this pattern of language-specific listening a conscious cultural strategy or an automatic processing effect? If automatic, does it also apply when the same pitch contours no longer sound like speech? Implicit Association Tests (IATs) provide an indirect measure of cross-modal association. In a series of IAT (...) studies, conducted with participants with three kinds of language backgrounds (Chinese-dominant bilinguals, Chinese balanced bilinguals, and English speakers with no Chinese experience) we find language-specific congruence effects for Mandarin lexical tones but not for matched sine-wave stimuli. That is, for linguistic stimuli, non-Chinese speakers show advantages for pitch-height congruence (high-pointy, low-curvy); no congruence effects were found for Chinese speakers. For non-linguistic stimuli, all participant groups showed advantages for pitch-height congruence. The present findings suggest that non-lexical tone congruence (high-pointy, low-curvy) is a basic congruence pattern, and the acquisition of a language with lexical tone can alter this perception. (shrink)