Drawing from ethnographic, empirical, and historical / cultural perspectives, we examine the extent to which visual aspects of music contribute to the communication that takes place between performers and their listeners. First, we introduce a framework for understanding how media and genres shape aural and visual experiences of music. Second, we present case studies of two performances, and describe the relation between visual and aural aspects of performance. Third, we report empirical evidence that visual aspects of performance reliably influence perceptions (...) of musical structure and affective interpretations of music. Finally, we trace new and old media trajectories of aural and visual dimensions of music, and highlight how our conceptions, perceptions and appreciation of music are intertwined with technological innovation and media deployment strategies. (shrink)

We investigate the dynamics of sensory integration for perceiving musical performance, a complex natural behavior. Thirty musically trained participants saw, heard, or both saw and heard, performances by two clarinetists. All participants used a sliding potentiometer to make continuous judgments of tension (a measure correlated with emotional response) and continuous judgments of phrasing (a measure correlated with perceived musical structure) as performances were presented. The data analysis sought to reveal relations between the sensory modalities (vision and audition) and to quantify (...) the effect of seeing the performances on participants' overall subjective experience of the music. In addition to traditional statistics, functional data analysis techniques were employed to analyze time-varying aspects of the data. The auditory and visual channels were found to convey similar experiences of phrasing but different experiences of tension through much of the performances. We found that visual information served both to augment and to reduce the experience of tension at different points in the musical piece (as revealed by functional linear modeling and functional significance testing). In addition, the musicians' movements served to extend the sense of phrasing, to cue the beginning of new phrases, to indicate musical interpretation, and to anticipate changes in emotional content. Evidence for an interaction effect suggests that there may exist an emergent quality when musical performances are both seen and heard. The investigation augments knowledge of human communicative processes spanning language and music, and involving multiple modalities of emotion and information transfer. (shrink)