Putting the principles of multisensory teaching into practice, this study investigated the effect of audio-visual vocabulary repetition on L2 sentence comprehension. Forty participants were randomly assigned to experimental and control groups. A sensory-based model of instruction was used to teach a list of unfamiliar vocabularies to the two groups. Following the instruction, the experimental group repeated the instructed words twice, while the control group received no vocabulary repetition. Afterward, their electrophysiological neural activities were recorded through electroencephalography while doing a sentence (...) acceptability judgment task with 216 sentences under acceptable and unacceptable conditions. A one-way analysis of variance, a multivariate analysis of variance, and a Bayesian repeated-measures ANOVA were used to compare the behavioral and neurocognitive responses [N400 as the main language-related event-related brain potential effect] of the two groups. The results showed no significant N400 amplitude difference in favor of any of the groups. The findings corroborated the ineffectiveness of two repetitions preceded by multisensory instruction on L2 sentence comprehension. (shrink)

From playing basketball to ordering at a food counter, we frequently and effortlessly coordinate our attention with others towards a common focus: we look at the ball, or point at a piece of cake. This non-verbal coordination of attention plays a fundamental role in our social lives: it ensures that we refer to the same object, develop a shared language, understand each other’s mental states, and coordinate our actions. Models of joint attention generally attribute this accomplishment to gaze coordination. But (...) are visual attentional mechanisms sufficient to achieve joint attention, in all cases? Besides cases where visual information is missing, we show how combining it with other senses can be helpful, and even necessary to certain uses of joint attention. We explain the two ways in which non-visual cues contribute to joint attention: either as enhancers, when they complement gaze and pointing gestures in order to coordinate joint attention on visible objects, or as modality pointers, when joint attention needs to be shifted away from the whole object to one of its properties, say weight or texture. This multisensory approach to joint attention has important implications for social robotics, clinical diagnostics, pedagogy and theoretical debates on the construction of a shared world. (shrink)

How do people learn multisensory, or amodal, representations, and what consequences do these representations have for perceptual performance? We address this question by performing a rational analysis of the problem of learning multisensory representations. This analysis makes use of a Bayesian nonparametric model that acquires latent multisensory features that optimally explain the unisensory features arising in individual sensory modalities. The model qualitatively accounts for several important aspects of multisensory perception: (a) it integrates information from multiple sensory sources in such a (...) way that it leads to superior performances in, for example, categorization tasks; (b) its performances suggest that multisensory training leads to better learning than unisensory training, even when testing is conducted in unisensory conditions; (c) its multisensory representations are modality invariant; and (d) it predicts ‘‘missing” sensory representations in modalities when the input to those modalities is absent. Our rational analysis indicates that all of these aspects emerge as part of the optimal solution to the problem of learning to represent complex multisensory environments. (shrink)

Different styles of social interaction are one of the core characteristics of autism spectrum disorder. Social differences among individuals with ASD often include difficulty in discerning the emotions of neurotypical people based on their facial expressions. This review first covers the rich body of literature studying differences in facial emotion recognition in those with ASD, including behavioral studies and neurological findings. In particular, we highlight subtle emotion recognition and various factors related to inconsistent findings in behavioral studies of FER in (...) ASD. Then, we discuss the dual problem of FER – namely facial emotion expression or the production of facial expressions of emotion. Despite being less studied, social interaction involves both the ability to recognize emotions and to produce appropriate facial expressions. How others perceive facial expressions of emotion in those with ASD has remained an under-researched area. Finally, we propose a method for teaching FER [FER teaching hierarchy ] based on recent research investigating FER in ASD, considering the use of posed vs. genuine emotions and static vs. dynamic stimuli. We also propose two possible teaching approaches: a standard method of teaching progressively from simple drawings and cartoon characters to more complex audio-visual video clips of genuine human expressions of emotion with context clues or teaching in a field of images that includes posed and genuine emotions to improve generalizability before progressing to more complex audio-visual stimuli. Lastly, we advocate for autism interventionists to use FER stimuli developed primarily for research purposes to facilitate the incorporation of well-controlled stimuli to teach FER and bridge the gap between intervention and research in this area. (shrink)

Previous research has demonstrated that ratings of the perceived pleasantness and quality of odors can be modulated by auditory stimuli presented at around the same time. Here, we extend these results by assessing whether the hedonic congruence between odor and sound stimuli can modulate the perception of odor intensity, pleasantness, and quality in untrained participants. Unexpectedly, our results reveal that broadband white noise, which was rated as unpleasant in a follow-up experiment, actually had a more pronounced effect on participants’ odor (...) ratings than either the consonant or dissonant musical selections. In particular, participants rated the six smells used as being less pleasant and less sweet when they happened to be listening to white noise, as compared to any one of the other music conditions. What is more, these results also add evidence to support the existence of a close relationship between an odor’s hedonic character and the perception of odor quality. So, for example, independent of the sound condition, pleasant odors were rated as sweeter, less dry, and brighter than the unpleasant odors. These results are discussed in terms of their implications for the understanding of crossmodal correspondences between olfactory and auditory stimuli. (shrink)

Linking arbitrary shapes to personal labels or reward values results in immediate processing benefits for those stimuli that happen to be associated with the self or high rewards in perceptual matching tasks. Here we further explored how social and reward associations interact with multisensory stimuli by pairing labels and objects with tones. We also investigated whether self and reward biases persist for multisensory stimuli with the label removed after an association had been made. Both high reward stimuli and those associated (...) with the self, resulted in faster responses and improved discriminability, which persisted for multisensory stimuli even when the labels were removed. However, these self- and reward-biases partly depended on the specific alignment between the physical tones and the conceptual order. Performance for reward associations improved when the endpoints of low or high rewards were paired with low or high tones; meanwhile, for personal associations, there was a benefit when the self was paired with either low or high tones, but there was no effect when the stranger was associated with either endpoint. These results indicate that, unlike reward, social personal associations are not represented along a continuum with two marked endpoints but rather with a single reference point. (shrink)

Recent advances in the field of statistical learning have established that learners are able to track regularities of multimodal stimuli, yet it is unknown whether the statistical computations are performed on integrated representations or on separate, unimodal representations. In the present study, we investigated the ability of adults to integrate audio and visual input during statistical learning. We presented learners with a speech stream synchronized with a video of a speaker’s face. In the critical condition, the visual (e.g. /gi/) and (...) auditory (e.g. /mi/) signals were occasionally incongruent, which we predicted would produce the McGurk illusion, resulting in the perception of an audiovisual syllable (e.g. /ni/). In this way, we used the McGurk illusion to manipulate the underlying statistical structure of the speech streams, such that perception of these illusory syllables facilitated participants’ ability to segment the speech stream. Our results therefore demonstrate that participants can integrate audio and visual input to perceive the McGurk illusion during statistical learning. We interpret our findings as support for modality-interactive accounts of statistical learning. (shrink)

Word learning is basic to foreign language acquisition, however time consuming and not always successful. Empirical studies have shown that traditional (visual) word learning can be enhanced by gestures. The gesture benefit has been attributed to depth of encoding. Gestures can lead to depth of encoding because they trigger semantic processing and sensorimotor enrichment of the novel word. However, the neural underpinning of depth of encoding is still unclear. Here, we combined an fMRI and a behavioral study to investigate word (...) encoding online. In the scanner, participants encoded 30 novel words of an artificial language created for experimental purposes and their translation into the subjects’ native language. Participants encoded the words three times visually, audiovisually, and by additionally observing semantically related gestures performed by an actress. Hemodynamic activity during word encoding revealed the recruitment of cortical areas involved in stimulus processing. In this study, depth of encoding can be spelt out in terms of sensorimotor brain networks that grow larger the more sensory modalities are linked to the novel word. Word retention outside the scanner documented a positive effect of gestures in a free recall test in the short-term. (shrink)

This article explores the concept of hybrid epistemology in relation with the author's theological teaching of his neighbours from the northern townships of Pretoria and the students of the University of Pretoria. It is written from the perspective of a black African mission practitioner who values with equal footing the diverse ways human beings can acquire knowledge. He longs to see a symbiotic relationship between different epistemologies and be prioritised in the theological training of Africans. He stresses that the value (...) in authenticity would allow the diversity of epistemologies to weave together in a symbiotic way. This article is a case study that reflects on the symbiotic relationship between different epistemologies using the five human senses as a multi-sensory approach to knowing. It discusses the experiences with students from InnerCHANGE and the University of Pretoria. (shrink)

Experimental research has shown that pairs of stimuli which are congruent and assumed to 'go together' are recalled more effectively than an item presented in isolation. Will this multisensory memory benefit occur when stimuli are richer and longer, in an ecological setting? In the present study, we focused on an everyday situation of audio-visual learning and manipulated the relationship between audio guide tracks and viewed portraits in the galleries of the Tate Britain. By varying the gender and narrative style of (...) the voice-over, we examined how the perceived congruency and assumed unity of the audio guide track with painted portraits affected subsequent recall. We show that tracks perceived as best matching the viewed portraits led to greater recall of both sensory and linguistic content. We provide the first evidence that manipulating crossmodal congruence and unity assumptions can effectively impact memory in a multisensory ecological setting, even in the absence of precise temporal alignment between sensory cues. (shrink)

We introduce a new and time-efficient memory-encoding paradigm for functional magnetic resonance imaging. This paradigm is optimized for mapping multiple contrasts using a mixed design, using auditory and visual stimuli. We demonstrate that the paradigm evokes robust neuronal activity in typical sensory and memory networks. We were able to detect auditory and visual sensory-specific encoding activities in auditory and visual cortices. Also, we detected stimulus-selective activation in environmental-, voice-, scene-, and face-selective brain regions. A subsequent recognition task allowed the detection (...) of sensory-specific encoding success activity in both auditory and visual cortices, as well as sensory-unspecific positive ESA in the hippocampus. Further, sensory-unspecific negative ESA was observed in the precuneus. Among others, the parallel mixed design enabled sustained and transient activity comparison in contrast to rest blocks. Sustained and transient activations showed great overlap in most sensory brain regions, whereas several regions, typically associated with the default-mode network, showed transient rather than sustained deactivation. We also show that the use of a parallel mixed model had relatively little influence on positive or negative ESA. Together, these results demonstrate a feasible, versatile, and brief memory-encoding task, which includes multiple sensory stimuli to guarantee a comprehensive measurement. This task is especially suitable for large-scale clinical or population studies, which aim to test task-evoked sensory-specific and sensory-unspecific memory-encoding performance as well as broad sensory activity across the life span within a very limited time frame. (shrink)

Multisensory integration can alter information processing, and previous research has shown that such processes are modulated by sensory switch costs and prior experience. Here we report an incidental finding demonstrating, for the first time, the interplay between these processes and experimental factors, specifically the presence of the experimenter in the testing room. Experiment 1 demonstrates that multisensory motor facilitation in response to audiovisual stimuli is higher in those trials in which the sensory modality switches than when it repeats. Those participants (...) who completed the study while alone exhibited increased RT variability. Experiment 2 replicated these findings using the letters “b” and “d” presented as unisensory stimuli or congruent and incongruent multisensory stimuli. Multisensory enhancements were inflated following a sensory switch; that is, congruent and incongruent multisensory stimuli resulted in significant gains following a sensory switch in the monitored condition. However, when the participants were left alone, multisensory enhancements were only observed for repeating incongruent multisensory stimuli. These incidental findings therefore suggest that the effects of letter congruence and sensory switching on multisensory integration are partly modulated by the presence of an experimenter. (shrink)