When young children learn to use language, they start to use their hands in co‐verbal gesturing. There are, however, considerable differences between children, and it is not completely understood what these individual differences are due to. We studied how children at 4 years of age employ speech and iconic gestures to convey meaning in different kinds of spatial event descriptions, and how this relates to their cognitive abilities. Focusing on spontaneous illustrations of actions, we applied a semantic feature (SF) analysis (...) to characterize combinations of speech and gesture meaning and related them to the child's visual‐spatial abilities or abstract/concrete reasoning abilities (measured using the standardized SON‐R test). Results show that children with higher cognitive abilities convey significantly more meaning via gesture and less solely via speech. These findings suggest that young children's use of cospeech representational gesturing is positively related to their mental representation and reasoning abilities. (shrink)

The economic principle of communication, according to which successful communication can be reached by least effort, has been studied for verbal communication. With respect to nonverbal behavior, it implies that forms of iconic gestures change over the course of communication and become reduced in the sense of less pronounced. These changes and their effects on learning are currently unexplored in relevant literature. Addressing this research gap, we conducted a word learning study to test the effects of changing gestures on children’s (...) slow mapping. We applied a within-subject design and tested 51 children, aged 6.7 years, who learned unknown words from a story. The storyteller acted on the basis of two conditions: In one condition, in which half of the target words were presented, the story presentation was enhanced with progressively reduced iconic gestures ; in the other condition, half of the target words were accompanied by fully executed iconic gestures. To ensure a reliable gesture presentation, children were exposed to a recorded person telling a story in both conditions. We tested the slow mapping effects on children’s productive and receptive word knowledge three minutes as well as two to three days after being presented the story. The results suggest that children’s production of the target words, but not their understanding thereof, was enhanced by PRG. (shrink)

Co-speech hand gestures offer a rich avenue for research into studying emotion communication because they serve as both prominent expressive bodily cues and an integral part of language. Despite such a strategic relevance, gesture-speech integration and interaction have received less research focus on its emotional function compared to its cognitive function. This review aims to shed light on the current state of the field regarding the interplay between co-speech hand gestures and emotions, focusing specifically on the role of gestures in (...) expressing and understanding both others’ and one's own emotions. The article concludes by addressing current limitations in the field and proposing future directions for researchers investigating gesture-emotion interaction. Our goal is to provide a roadmap to researchers in their exploration of the role of gestures in emotions, ultimately contributing to a more comprehensive understanding of how gestures and emotions intersect. (shrink)

Gestures—hand movements that accompany speech and express ideas—can help children learn how to solve problems, flexibly generalize learning to novel problem‐solving contexts, and retain what they have learned. But does it matter who is doing the gesturing? We know that producing gesture leads to better comprehension of a message than watching someone else produce gesture. But we do not know how producing versus observing gesture impacts deeper learning outcomes such as generalization and retention across time. Moreover, not all children benefit (...) equally from gesture instruction, suggesting that there are individual differences that may play a role in who learns from gesture. Here, we consider two factors that might impact whether gesture leads to learning, generalization, and retention after mathematical instruction: (1) whether children see gesture or do gesture and (2) whether a child spontaneously gestures before instruction when explaining their problem‐solving reasoning. For children who spontaneously gestured before instruction, both doing and seeing gesture led to better generalization and retention of the knowledge gained than a comparison manipulative action. For children who did not spontaneously gesture before instruction, doing gesture was less effective than the comparison action for learning, generalization, and retention. Importantly, this learning deficit was specific to gesture, as these children did benefit from doing the comparison manipulative action. Our findings are the first evidence that a child's use of a particular representational format for communication (gesture) directly predicts that child's propensity to learn from using the same representational format. (shrink)

When people talk, they gesture. Gesture is a fundamental component of language that contributes meaningful and unique information to a spoken message and reflects the speaker’s underlying knowledge and experiences. Theoretical perspectives of speech and gesture propose that they share a common conceptual origin and have a tightly integrated relationship, overlapping in time, meaning, and function to enrich the communicative context. We review a robust literature from the field of psychology documenting the benefits of gesture for communication for both speakers (...) and listeners, as well as its important cognitive functions for organizing spoken language, and facilitating problem-solving, learning, and memory. Despite this evidence, gesture has been relatively understudied in populations with neurogenic communication disorders. While few studies have examined the rehabilitative potential of gesture in these populations, others have ignored gesture entirely or even discouraged its use. We review the literature characterizing gesture production and its role in intervention for people with aphasia, as well as describe the much sparser literature on gesture in cognitive communication disorders including right hemisphere damage, traumatic brain injury, and Alzheimer’s disease. The neuroanatomical and behavioral profiles of these patient populations provide a unique opportunity to test theories of the relationship of speech and gesture and advance our understanding of their neural correlates. This review highlights several gaps in the field of communication disorders which may serve as a bridge for applying the psychological literature of gesture to the study of language disorders. Such future work would benefit from considering theoretical perspectives of gesture and using more rigorous and quantitative empirical methods in its approaches. We discuss implications for leveraging gesture to explore its untapped potential in understanding and rehabilitating neurogenic communication disorders. (shrink)

There is plenty of evidence that speech and gesture form a tightly integrated system, as reflected in parallelisms in language production, comprehension, and development ( McNeill, 1992 ; Kendon, 2004 ). Yet, it is a common assumption that speakers use gestures to compensate for their expressive difficulties, a notion found in developmental studies of both first and second language acquisition, and in theoretical proposals concerning the gesture-speech relationship. If gestures are compensatory, they should mainly occur in disfluent stretches of speech. (...) However, the evidence is sparse and conflicting. This study extends previous studies and tests the putative compensatory role of gestures by comparing the gestural behavior in fluent vs. disfluent stretches of narratives by competent speakers in two languages (Dutch and Italian), and by language learners (children and adult L2 learners). The results reveal that (1) in all groups speakers overwhelmingly produce gestures during fluent speech and only rarely during disfluencies. However, L2 learners are significantly more likely to gesture in disfluency than the other groups; (2) in all groups gestures during disfluencies tend to be holds; (3) in all groups the rare gestures completed in disfluencies have both referential and pragmatic functions. Overall, the data strongly suggest that when speech stops, so does gesture. The findings constitute an important challenge to both gesture and language acquisition theories assuming a mainly (lexical) compensatory role for (referential) gestures. Instead, the results provide strong support for the notion that speech and gestures form an integrated system. (shrink)

Theories of grounded and embodied cognition offer a range of accounts of how reasoning and body‐based processes are related to each other. To advance theories of grounded and embodied cognition, we explore the cognitive relevance of particular body states to associated math concepts. We test competing models of action‐cognition transduction to investigate the cognitive relevance of directed actions to students’ mathematical reasoning in the area of geometry. The hypotheses we test include (1) that cognitively relevant directed actions have a direct (...) effect on performance (direct cognitive relevance hypothesis), (2) that cognitively relevant directed actions lead to more frequent production of gestures during explanations, which leads to improved performance (mediated cognitive relevance hypothesis), and (3) that performance effects of directed actions are influenced by the presence or absence of gesture production during mathematical explanations (moderated cognitive relevance hypothesis). We explore these hypotheses in an experiment where high school students (N = 85) evaluated the truth of geometry conjectures after performing cognitively relevant or cognitively irrelevant directed actions while playing a movement‐based video game. Contrary to the direct and mediated cognitive relevance hypotheses, we found no overall differences in performance or gesture production between relevant and irrelevant conditions. Consistent with the moderated cognitive relevance hypothesis, cognitive relevance influenced mathematical performance, as measured by the accuracy of students’ intuitions, insights, and the validity of their proofs, provided that students produced certain kinds of gestures during mathematical explanations (i.e., with explanatory gestures as the moderator). Implications for theories of grounded and embodied cognition and the design of embodied forms of educational interventions are discussed. (shrink)

This article proposes an analysis of the gestural production of people suffering from fluent or non-fluent aphasia in comparison with the gestures performed by speakers not suffering from any language disorder. The interest of such a study is twofold: firstly, it allows a better understanding of what happens to gestures during the numerous speech dysfluencies caused by the speech impairment. Among other things, it sheds light on the links between speech and gestures in multimodal face-to-face interaction. Despite the profusion of (...) work on aphasia, very little is still known about the characteristics of the gestures performed spontaneously by aphasic people. The study presented here is based on video examples from the AphasiaBank corpus to illustrate the specific features of the speech and gestures of atypical speakers and gives a more detailed description of the gestural production of people with aphasia. This knowledge can inform post-stroke rehabilitation procedures that rely on some form of multimodal speech therapy in order to adapt to the various communication needs of the patients. (shrink)

Pantomime has long been considered distinct from co-speech gesture. It has therefore been argued that pantomime cannot be part of gesture-speech integration. We examine pantomime as distinct from silent gesture, focusing on non-co-speech gestures that occur in the midst of children’s spoken narratives. We propose that gestures with features of pantomime are an infrequent but meaningful component of a multimodal communicative strategy. We examined spontaneous non-co-speech representational gesture production in the narratives of 30 monolingual English-speaking children between the ages of (...) 8- and 11-years. We compared the use of co-speech and non-co-speech gestures in both autobiographical and fictional narratives and examined viewpoint and the use of non-manual articulators, as well as the length of responses and narrative quality. The use of non-co-speech gestures was associated with longer narratives of equal or higher quality than those using only co-speech gestures. Non-co-speech gestures were most likely to adopt character-viewpoint and use non-manual articulators. The present study supports a deeper understanding of the term pantomime and its multimodal use by children in the integration of speech and gesture. (shrink)

The role of nonverbal communication in patients with post-stroke language impairment (aphasia) is not yet fully understood. This study investigated how aphasic patients perceive and produce co-speech gestures during face-to-face interaction, and whether distinct brain lesions would predict the frequency of spontaneous co-speech gesturing. For this purpose, we recorded samples of conversations in patients with aphasia and healthy participants. Gesture perception was assessed by means of a head-mounted eye-tracking system, and the produced co-speech gestures were coded according to a linguistic (...) classification system. The main results are that meaning-laden gestures (e.g., iconic gestures representing object shapes) are more likely to attract visual attention than meaningless hand movements, and that patients with aphasia are more likely to fixate co-speech gestures overall than healthy participants. This implies that patients with aphasia may benefit from the multimodal information provided by co-speech gestures. On the level of co-speech gesture production, we found that patients with damage to the anterior part of the arcuate fasciculus showed a higher frequency of meaning-laden gestures. This area lies in close vicinity to the premotor cortex and is considered to be important for speech production. This may suggest that the use of meaning-laden gestures depends on the integrity of patients’ speech production abilities. (shrink)

BackgroundResearch on interpersonal synchrony has mostly focused on a single modality, and hence little is known about the connections between different types of social attunement. In this study, the relationship between sympathetic nervous system synchrony, movement synchrony, and the amount of speech were studied in couple therapy.MethodsData comprised 12 couple therapy cases. Synchrony in electrodermal activity, head and body movement, and the amount of speech and simultaneous speech during the sessions were analyzed in 12 sessions at the start of couple (...) therapy and eight sessions at the end of therapy. Synchrony was calculated from cross-correlations using time lags and compared to segment-shuffled pseudo synchrony. The associations between the synchrony modalities and speech were analyzed using complex modeling.FindingsCouple therapy participants’ synchrony mostly occurred in-phase. Anti-phase synchrony was more common in movement than in sympathetic nervous system activity. Synchrony in sympathetic nervous system activity only correlated with movement synchrony between the client-therapist dyads. Movement synchrony and the amount of speech correlated negatively between spouses and co-therapists, meaning that the more time the dyad members talked during the session, the less bodily synchrony they exhibited.ConclusionThe different roles and relationships in couple therapy were associated with the extent to which synchrony modalities were linked with each other. In the relationship between clients and therapists, synchrony in arousal levels and movement “walked hand in hand”, whereas in the other relationships they were not linked. Generally, more talk time by the therapy participants was associated with anti-phase movement synchrony. If, as suggested, emotions prepare us for motor action, an important finding of this study is that sympathetic nervous system activity can also synchronize with that of others independently of motor action. (shrink)

Although children learn more when teachers gesture, it is not clear how gesture supports learning. Here, we sought to investigate the nature of the memory processes that underlie the observed benefits of gesture on lasting learning. We hypothesized that instruction with gesture might create memory representations that are particularly resistant to interference. We investigated this possibility in a classroom study with 402 second‐ and third‐grade children. Participants received classroom‐level instruction in mathematical equivalence using videos with or without accompanying gesture. After (...) instruction, children solved problems that were either visually similar to the problems that were taught, and consistent with an operational interpretation of the equal sign (interference), or visually distinct from equivalence problems and without an equal sign (control) in order to assess the role of gesture in resisting interference after learning. Gesture facilitated learning, but the effects of gesture and interference varied depending on type of problem being solved and the strategies that children used to solve problems prior to instruction. Some children benefitted from gesture, while others did not. These findings have implications for understanding the mechanisms underlying the beneficial effect of gesture on mathematical learning, revealing that gesture does not work via a general mechanism like enhancing attention or engagement that would apply to children with all forms of prior knowledge. (shrink)

Cognitive Science, Volume 46, Issue 2, February 2022.

Producing content-related gestures has been found to impact students’ learning, whether such gestures are spontaneously generated by the learner in the course of problem-solving, or participants are instructed to pose based on experimenter instructions during problem-solving and word learning. Few studies, however, have investigated the effect of (a) performing instructed gestures while learning concepts or (b) producing gestures without there being an implied connection between the gestures and the concepts being learned. The two studies reported here investigate the impact of (...) instructed hand movements on students’ subsequent understanding of a concept. Students were asked to watch an instructional video—focused on the concept of statistical model—three times. Two experimental groups were given a secondary task to perform while watching the video, which involved moving their hands to mimic the placement and orientation of red rectangular bars overlaid on the video. Students were told that the focus of the study was multitasking, and that the instructed hand movements were unrelated to the material being learned. In the content-match group the placement of the hands reinforced the concept being explained, and in the content-mismatch group it did not. A control group was not asked to perform a secondary task. In both studies, findings indicate that students in the content-match group performed better on the posttest, and showed less variation in performance, than did students in the content-mismatch group, with control students falling in between. Instructed hand movement—even when presented as an unrelated, secondary task—can affect students’ learning of a complex concept. (shrink)

When faced with an ambiguous pronoun, comprehenders use both multimodal cues and linguistic cues to identify the antecedent. While research has shown that gestures facilitate language comprehension, improve reference tracking, and influence the interpretation of ambiguous pronouns, literature on reference resolution suggests that a wide set of linguistic constraints influences the successful resolution of ambiguous pronouns and that linguistic cues are more powerful than some multimodal cues. To address the outstanding question of the importance of gesture as a cue in (...) reference resolution relative to cues in the speech signal, we have previously investigated the comprehension of contrastive gestures that indexed abstract referents – in this case expressions of personal preference – and found that such gestures did facilitate the resolution of ambiguous statements of preference. In this study, we extend this work to investigate whether the effect of gesture on resolution is diminished when the gesture indexes a statement that is less likely to be interpreted as the correct referent. Participants watched videos in which a speaker contrasted two ideas that were either neutral or moral acceptable). A gesture to the left or right side co-occurred with speech expressing each position. In gesture-disambiguating trials, an ambiguous phrase was accompanied by a gesture to one side or the other. In gesture non-disambiguating trials, no third gesture occurred with the ambiguous phrase. Participants were more likely to choose the idea accompanied by gesture as the stimulus speaker’s preference. We found no effect of scenario type. Regardless of whether the linguistic cue expressed a view that was morally charged or neutral, observers used gesture to understand the speaker’s opinion. This finding contributes to our understanding of the strength and range of cues, both linguistic and multimodal, that listeners use to resolve ambiguous references. (shrink)

Research has shown that gesture production supports learning across a number of tasks. It is unclear, however, whether gesture production during encoding can support narrative recall, who gesture production benefits most, and whether certain types of gestures are more beneficial than others. This study, therefore, investigated the effect of gesture production during the encoding of a narrative on subsequent narrative recall, and whether individuals’ levels of verbal and nonverbal memory moderated this effect. Additionally, this study investigated whether producing certain types (...) of gestures during encoding was more beneficial than others. Participants (N = 90, Mage = 20.43) read aloud a narrative while under instruction to produce gestures, under no specific instruction to produce gestures, or were required to keep their hands behind their back to prevent them from gesturing. Participants completed measures assessing verbal and nonverbal memory. While gesture production during encoding benefitted narrative recall (as measured through specific questions), verbal memory moderated the effect, such that gesture production was more beneficial for individuals with higher than lower verbal memory. Furthermore, producing representational gestures during encoding benefitted recall of points in the narrative at which those gestures were produced, while beat gestures had no effect. Findings have implications for understanding the mechanisms underlying the links between gesture and learning, as well as practical implications in instructional settings. (shrink)

Representational transformation is a technique for deepening our understanding of mathematical concepts and facilitating the process of solving mathematical problems. This article suggests that representational transformation can contribute to thought processes in four ways. Firstly, representational transformation helps people employ a wider range of cognitive resources. When one representation of a concept or a problem is transformed into another representation, the sensory-motor resources that are involved in the processing of the base representation are employed to process the target representation. Secondly, (...) representational transformation helps in the process of organizing scattered information into suitable units. This gives structure to various parts of information and makes them much easier to handle. Thirdly, representational transformation helps in the process of suppressing irrelevant information and unnecessary elements of problems. This is a key step in organizing and structuring information. Fourthly, representational transformation helps in the process of discovering general rules and turning them into explicit knowledge. (shrink)

This study examines the form and function of gestural depictions that develop over extended stretches of concept explanation by a philosopher. Building onStreeck’s (2009)explorations of depiction by gesture, we examine how this speaker’s process of exposition involves sequences of multimodal, analogical depiction by which the philosophical concepts are not only expressed through gesture forms, but also dynamically analyzed and construed through gestural activity. Drawing on perspectives of gesture as active meaning making (Müller 2014,2016,Streeck 2009), we argue that the build-up of (...) gestures in depiction sequences, activated through a multimodal metaphor (Müller & Cienki 2009), engages the wider philosophical standpoint of the speaker. Using video analysis supported by interview data, we demonstrate how examination of gestures within and across discourse can lead to understanding of how dynamic, embodied, and subjective processes of conceptualization contribute to philosophical theorizing. (shrink)

Speakers often use gesture to demonstrate how to perform actions—for example, they might show how to open the top of a jar by making a twisting motion above the jar. Yet it is unclear whether listeners learn as much from seeing such gestures as they learn from seeing actions that physically change the position of objects (i.e., actually opening the jar). Here, we examined participants' implicit and explicit understanding about a series of movements that demonstrated how to move a set (...) of objects. The movements were either shown with actions that physically relocated each object or with gestures that represented the relocation without touching the objects. Further, the end location that was indicated for each object covaried with whether the object was grasped with one or two hands. We found that memory for the end location of each object was better after seeing the physical relocation of the objects, that is, after seeing action, than after seeing gesture, regardless of whether speech was absent (Experiment 1) or present (Experiment 2). However, gesture and action built similar implicit understanding of how a particular handgrasp corresponded with a particular end location. Although gestures miss the benefit of showing the end state of objects that have been acted upon, the data show that gestures are as good as action in building knowledge of how to perform an action. (shrink)

Co-speech gestures are closely connected to speech, but little attention has been paid to the associations between gesture and L2 speech performance. This study explored the associations between four types of co-speech gestures and the meaning, form, and discourse dimensions of L2 speech performance. Gesture and speech data were collected by asking 61 lower-intermediate English learners whose first language is Chinese to retell a cartoon clip. Results showed that all the four types of co-speech gestures had positive associations with meaning (...) and discourse L2 speech measures but no association with form-related speech measures, except the positive association between metaphorics and the percentage of error-free clauses. The findings suggest that co-speech gestures may have a tighter connection with meaning construction in producing L2 speech. (shrink)

When learners self-explain, they try to make sense of new information. Although research has shown that bodily actions and written notes are an important part of learning, previous analyses of self-explanations rarely take into account written and non-verbal data produced spontaneously. In this paper, the extent to which interpretations of self-explanations are influenced by the systematic consideration of such data is investigated. The video recordings of 33 undergraduate students, who learned with worked-out examples dealing with complex numbers, were categorized successively (...) including three different data bases: verbal data, verbal and written data, and verbal, written and non-verbal data. Results reveal that including written data and non-verbal data leads to a more accurate analysis of self-explanations than an analysis solely based on verbal data. This influence is even stronger for the categorization of self-explanations as adequate or inadequate. (shrink)

The framework of depicting put forward byClark (2016)offers a schematic vantage point from which to examine iconic language use. Confronting the framework with empirical data, we consider some of its key theoretical notions. Crucially, by reconceptualizing the typology of depictions, we identify an overlooked domain in the literature: “speech-embedded nonverbal depictions,” namely cases where meaning is communicated iconically, nonverbally, and without simultaneously co-occurring speech. In addition to contextualizing the phenomenon in relation to existing research, we demonstrate, with examples from American (...) TV talk shows, how such depictions function in real-life language use, offering a brief sketch of their complexities and arguing also for their theoretical significance. (shrink)

Gestures and speech are clearly synchronized in many ways. However, previous studies have shown that the semantic similarity between gestures and speech breaks down as people approach transitions in understanding. Explanations for these gesture–speech mismatches, which focus on gestures and speech expressing different cognitive strategies, have been criticized for disregarding gestures’ and speech's integration and synchronization. In the current study, we applied three different perspectives to investigate gesture–speech synchronization in an easy and a difficult task: temporal alignment, semantic similarity, and (...) complexity matching. Participants engaged in a simple cognitive task and were assigned to either an easy or a difficult condition. We automatically measured pointing gestures, and we coded participant's speech, to determine the temporal alignment and semantic similarity between gestures and speech. Multifractal detrended fluctuation analysis was used to determine the extent of complexity matching between gestures and speech. We found that task difficulty indeed influenced gesture–speech synchronization in all three domains. We thereby extended the phenomenon of gesture–speech mismatches to difficult tasks in general. Furthermore, we investigated how temporal alignment, semantic similarity, and complexity matching were related in each condition, and how they predicted participants’ task performance. Our study illustrates how combining multiple perspectives, originating from different research areas (i.e., coordination dynamics, complexity science, cognitive psychology), provides novel understanding about cognitive concepts in general and about gesture–speech synchronization and task difficulty in particular. (shrink)