The tradeoff hypothesis in the speech–gesture relationship claims that (a) when gesturing gets harder, speakers will rely relatively more on speech, and (b) when speaking gets harder, speakers will rely relatively more on gestures. We tested the second part of this hypothesis in an experimental collaborative referring paradigm where pairs of participants (directors and matchers) identified targets to each other from an array visible to both of them. We manipulated two factors known to affect the difficulty of speaking to assess (...) their effects on the gesture rate per 100 words. The first factor, codability, is the ease with which targets can be described. The second factor, repetition, is whether the targets are old or new (having been already described once or twice). We also manipulated a third factor, mutual visibility, because it is known to affect the rate and type of gesture produced. None of the manipulations systematically affected the gesture rate. Our data are thus mostly inconsistent with the tradeoff hypothesis. However, the gesture rate was sensitive to concurrent features of referring expressions, suggesting that gesture parallels aspects of speech. We argue that the redundancy between speech and gesture is communicatively motivated. (shrink)

Gestures and speech are clearly synchronized in many ways. However, previous studies have shown that the semantic similarity between gestures and speech breaks down as people approach transitions in understanding. Explanations for these gesture–speech mismatches, which focus on gestures and speech expressing different cognitive strategies, have been criticized for disregarding gestures’ and speech's integration and synchronization. In the current study, we applied three different perspectives to investigate gesture–speech synchronization in an easy and a difficult task: temporal alignment, semantic similarity, and (...) complexity matching. Participants engaged in a simple cognitive task and were assigned to either an easy or a difficult condition. We automatically measured pointing gestures, and we coded participant's speech, to determine the temporal alignment and semantic similarity between gestures and speech. Multifractal detrended fluctuation analysis was used to determine the extent of complexity matching between gestures and speech. We found that task difficulty indeed influenced gesture–speech synchronization in all three domains. We thereby extended the phenomenon of gesture–speech mismatches to difficult tasks in general. Furthermore, we investigated how temporal alignment, semantic similarity, and complexity matching were related in each condition, and how they predicted participants’ task performance. Our study illustrates how combining multiple perspectives, originating from different research areas (i.e., coordination dynamics, complexity science, cognitive psychology), provides novel understanding about cognitive concepts in general and about gesture–speech synchronization and task difficulty in particular. (shrink)

People with aphasia use gestures not only to communicate relevant content but also to compensate for their verbal limitations. The Sketch Model (De Ruiter, 2000) assumes a flexible relationship between gesture and speech with the possibility of a compensatory use of the two modalities. In the successor of the Sketch Model, the AR‐Sketch Model (De Ruiter, 2017), the relationship between iconic gestures and speech is no longer assumed to be flexible and compensatory, but instead iconic gestures are assumed to express (...) information that is redundant to speech. In this study, we evaluated the contradictory predictions of the Sketch Model and the AR‐Sketch Model using data collected from people with aphasia as well as a group of people without language impairment. We only found compensatory use of gesture in the people with aphasia, whereas the people without language impairments made very little compensatory use of gestures. Hence, the people with aphasia gestured according to the prediction of the Sketch Model, whereas the people without language impairment did not. We conclude that aphasia fundamentally changes the relationship of gesture and speech. (shrink)

The study of signed languages has inspired scientific' speculation regarding foundations of human language. Relationships between the acquisition of sign language in apes and man are discounted on logical grounds. Evidence from the differential hreakdown of sign language and manual pantomime places limits on the degree of overlap between language and nonlanguage motor systems. Evidence from functional magnetic resonance imaging reveals neural areas of convergence and divergence underlying signed and spoken languages.

I enlarge on the theme that the brain mechanisms required for languageand other aspects of the human mind evolved through selective changes in the regulatory genes governing growth. Extension of the period of postnatal growth increases the role of the environment in structuring the brain, and spatiotemporal programming (heterochrony) ofgrowth might explain hierarchical representation, hemispheric specialization, and perhaps sex differences.

The strong predominance of right-handedness appears to be a uniquely human characteristic, whereas the left-cerebral dominance for vocalization occurs in many species, including frogs, birds, and mammals. Right-handedness may have arisen because of an association between manual gestures and vocalization in the evolution of language. I argue that language evolved from manual gestures, gradually incorporating vocal elements. The transition may be traced through changes in the function of Broca's area. Its homologue in monkeys has nothing to do with vocal control, (...) but contains the so-called “mirror neurons,” the code for both the production of manual reaching movements and the perception of the same movements performed by others. This system is bilateral in monkeys, but predominantly left-hemispheric in humans, and in humans is involved with vocalization as well as manual actions. There is evidence that Broca's area is enlarged on the left side in Homo habilis, suggesting that a link between gesture and vocalization may go back at least two million years, although other evidence suggests that speech may not have become fully autonomous until Homo sapiens appeared some 170,000 years ago, or perhaps even later. The removal of manual gesture as a necessary component of language may explain the rapid advance of technology, allowing late migrations of Homo sapiens from Africa to replace all other hominids in other parts of the world, including the Neanderthals in Europe and Homo erectus in Asia. Nevertheless, the long association of vocalization with manual gesture left us a legacy of right-handedness. Key Words: cerebral dominance; gestures; handedness; hominids; language evolution; primates; speech; vocalization. (shrink)

In an analysis of three college lectures, Corts and Pollio found that figurative language and gesture often appeared together in ‘bursts’. These bursts were initially characterized as novel figurative expression that centered on the primary topic of the lecture. The current study is an attempt to provide clearer description of how and why figurative language and gesture so often appear together in academic discourse. In addition, this study extends earlier findings to additional speakers and academic disciplines to improve generalizability. In (...) accordance with Corts and Pollio, the bursts of figurative activity are predominately novel rather than cliched, coherent with a root metaphor, and centered around the main topics of the lecture. The present study adds that figures within bursts largely include analogies and metaphor that represent a concept whereas topical figures outside of bursts are descriptive phrases that often include other types of figures such as hyperbole, litote, etc. The regularity of such bursts suggests that they serve an important cognitive function independent of personal style of the lecturer or any demands of a particular academic field. (shrink)

Speakers frequently produce elaborate hand movements during talk that have been shown to serve a communicative function. Nevertheless, two-thirds of the semantic content of these hand movements is encoded linguistically elsewhere in the discourse . The present experiment demonstrated that while 62.9% of semantic information in gesture was elsewhere, most gestures retained at least one semantic feature that was never represented linguistically. Semantic features were more explicit when they occurred in gesture than when represented linguistically. Even in cases where the (...) imagistic gesture appeared somewhat redundant, gesture at the narrative level preserves a discernable communicative function. (shrink)

Name der Zeitschrift: Cognitive Linguistics Jahrgang: 27 Heft: 4 Seiten: 603-618.

Müller argues that double dissociations do not imply underlying modularity of the cognitive system, citing neural networks as examples of fully distributed systems that can give rise to double dissociations. We challenge this claim, noting that suchdouble dissociations typically do not “scale-up,” and that even some singledissociations can be difficult to account for in a distributed system.

Müller misconstrues autonomy to mean strict locality of brain function, something quite different from the functional autonomy that linguists claim. Similarly, he misperceives the interaction of learned and innate components hypothesized in current generative models. Evidence from sign languages, Creole languages, and neurological studies of rare forms of aphasia also argues against his conclusions.

People with aphasia use gestures not only to communicate relevant content but also to compensate for their verbal limitations. The Sketch Model (De Ruiter, 2000) assumes a flexible relationship between gesture and speech with the possibility of a compensatory use of the two modalities. In the successor of the Sketch Model, the AR‐Sketch Model (De Ruiter, 2017), the relationship between iconic gestures and speech is no longer assumed to be flexible and compensatory, but instead iconic gestures are assumed to express (...) information that is redundant to speech. In this study, we evaluated the contradictory predictions of the Sketch Model and the AR‐Sketch Model using data collected from people with aphasia as well as a group of people without language impairment. We only found compensatory use of gesture in the people with aphasia, whereas the people without language impairments made very little compensatory use of gestures. Hence, the people with aphasia gestured according to the prediction of the Sketch Model, whereas the people without language impairment did not. We conclude that aphasia fundamentally changes the relationship of gesture and speech. (shrink)

This book proposes a theory of human cognitive evolution, drawing from paleontology, linguistics, anthropology, cognitive science, and especially neuropsychology. The properties of humankind's brain, culture, and cognition have coevolved in a tight iterative loop; the main event in human evolution has occurred at the cognitive level, however, mediating change at the anatomical and cultural levels. During the past two million years humans have passed through three major cognitive transitions, each of which has left the human mind with a new way (...) of representing reality and a new form of culture. Modern humans consequently have three systems of memory representation that were not available to our closest primate relatives: mimetic skill, language, and external symbols. These three systems are supported by new types of “hard” storage devices, two of which (mimetic and linguistic) are biological, one technological. Full symbolic literacy consists of a complex of skills for interacting with the external memory system. The independence of these three uniquely human ways of representing knowledge is suggested in the way the mind breaks down after brain injury and confirmed by various other lines of evidence. Each of the three systems is based on aninventivecapacity, and the products of those capacities – such as languages, symbols, gestures, social rituals, and images – continue to be invented and vetted in the social arena. Cognitive evolution is not yet complete: the externalization of memory has altered the actual memory architecture within which humans think. This is changing the role of biological memory and the way in which the human brain deploys its resources; it is also changing the form of modern culture. (shrink)

Chomskyian claims of a genetically hard-wired and cognitively autonomous “universal grammar” are being promoted by generative linguistics as facts about language to the present day. The related doctrine of an evolutionary discontinuity in language emergence, however, is based on misconceptions about the notions of homology and preadaptation. The obvious lack of equivalence between symbolic communicative capacities in existing nonhuman primates and human language does not preclude common roots. Normal and disordered language development is strongly influenced by the genome, but there (...) is no evidence for the existence of specific genes underlying “universal grammar.” In the mature brain, stages of language processing can be distinguished and “first-pass” syntactic analyses appear to precede semantic decoding. However, this partial seriality – as well as behavioral and clinical dissociations between lexical and functional categories – can be best described, not in terms of serially activated and discrete modules, but in terms of classes of cell assemblies that differ in their distributional properties. Whereas cell assemblies involved in semantic interpretation (“content word” assemblies) are widely distributed and are generally less vulnerable to focal lesion, those involved in structural decoding (functor assemblies) are primarily distributed within the left perisylvian cortices and are selectively vulnerable to left perisylvian lesion. These distributional differences are explained in terms of the perceptuomotor components involved in the acquisition of relevant representations. The emerging “motivated” or “toposemantic” brain regional specificity can only be accommodated with “soft” and maturational versions of modularity. The failure to reproduce double dissociations in current connectionist models is due to overly simple neuroscientific assumptions, notably that of overall equipotentiality. Linguistic models should not be expected to be “implemented” in the brain, but need to be constrained by neuroscientific evidence on how biological brains function. (shrink)

Contrary to Müller's claims, and in support of modular theories, genetic factors play a substantial and significant role in language. The finding that some children with specific language impairment (SLI) have nonlinguistic impairments may reflect improper diagnosis of SLI or impairments that are secondary to linguistic impairments. Thus, such findings do not argue against the modularity thesis. The lexical/functional distinction appears to be innate and specifically linguistic and could be instantiated in either symbolic or connectionist systems.

The belief that syntax is an innate, autonomous, species-specific module is highly questionable. Syntax demonstrates the mosaic nature of evolutionary change, in that it made use of (and led to the enhancement of) numerous preexisting neurocognitive features. It is best understood as an emergent characteristic of the explosion of semantic complexity that occurred during hominid evolution.

Müller's target article aims to summarize approaches to the question of how language elements (phonemes, morphemes, etc.) and rules are laid down in the brain. However, it suffers from being too vague about basic assumptions and empirical predictions of neurobiological models, and the empirical evidence available to test the models is not appropriately evaluated. (1) In a neuroscientific model of language, different cortical localizations of words can only be based on biological principles. These need to be made explicit. (2) Evidence (...) for and against word class differences could be evaluated more rigorously. (3) All (and only) humans are able to learn languages with complex syntactic structures; it is, therefore, not appropriate to deny innateness and universality of syntactic principles. The real question appears to be the following: Which neurobiological principles are the linguistic principles based on? (shrink)

The concepts of the innateness, universality, species-specificity, and autonomy of the human language capacity have had an extreme impact on the psycholinguistic debate for over thirty years. These concepts are evaluated from several neurobiological perspectives, with an emphasis on the emergence of language and its decay due to brain lesion and progressive brain disease.Evidence of perceptuomotor homologies and preadaptations for human language in nonhuman primates suggests a gradual emergence of language during hominid evolution. Regarding ontogeny, the innate component of language (...) capacity is likely to be polygenic and shared with other developmental domains. Dissociations between verbal and nonverbal development are probably rooted in the perceptuomotor specializations of neural substrates rather than the autonomy of a grammar module. Aphasiologicaldata often assumed to suggest modular linguistic subsystems can be accounted for in terms of a neurofunctional model incorporating perceptuomotor-based regional specializationsand distributivity of representations. Thus, dissociations between grammatical functors and content words are due to different conditions of acquisition and resulting differences in neural representation. Human brains are characterized by multifactorial interindividual variability, and strict universality of functional organization is biologically unrealistic.A theoretical alternative is proposed according to which (1) linguistic specialization of brain areas is due to epigenetic and probabilistic maturational events, not to genetic ”hard-wiring,” and (2) linguistic knowledge is neurally represented in distributed cell assemblies whose topography reflects the perceptuomotor modalities involved in the acquisition and use of a given item of knowledge. (shrink)

ABSTRACT This article aims to analyze the process of mutual understanding from the perspective of Discursive Neurolinguistics. As a methodology, a case study was carried out on a child with a linguistic profile of verbal restriction, which concerns the dimension of utterances involving words and their morphology. Data interpretation was based on the Bakhtinian perspective, which addresses ‘understanding’ as an active-responsive process and meaning as emerging from the relationship between utterances. The results showed that the interlocutors’ common ground allowed the (...) joint construction of meaning even in the face of verbal restrictions, and aspects such as nonverbal elements and the recognition of the child’s active-responsive role also favored mutual understanding. As a conclusion, the attention to different multimodal resources as legitimate modes of expression allows children with verbal restrictions to fulfill the role of “speakers” regardless of their modes of expression. (shrink)

A persistent controversy in language evolution research has been whether language emerged in the gestural-visual or in the vocal-auditory modality. A “dialectic” solution to this age-old debate has now been gaining ground: language was fully multimodal from the start and remains so to this day. In this paper, we show this solution to be too simplistic and outline a more specific theoretical proposal, which we designate as pantomime-first. To decide between the multimodal-first and pantomime-first alternatives, we review several lines of (...) interdisciplinary evidence and complement it with a cognitive-semiotic experiment. In the study, the participants saw – and then matched to hand-drawn images – recordings of short transitive events enacted by 4 actors in two conditions: visual and multimodal. Significantly, the matching accuracy was greater in the visual than the multimodal condition, though a follow-up experiment revealed that the emotional profiles of the events enacted in the multimodal condition could be reliably detected from the sound alone. We see these results as supporting the proposed pantomime-first scenario. (shrink)

Using a cross-modal semantic priming paradigm, both experiments of the present study investigated the link between the mental representations of iconic gestures and words. Two groups of the participants performed a primed lexical decision task where they had to discriminate between visually presented words and nonwords (e.g., flirp). Word targets (e.g., bird) were preceded by video clips depicting either semantically related (e.g., pair of hands flapping) or semantically unrelated (e.g., drawing a square with both hands) gestures. The duration of gestures (...) was on average 3,500 ms in Experiment 1 but only 1,000 ms in Experiment 2. Significant priming effects were observed in both experiments, with faster response latencies for related gesture–word pairs than unrelated pairs. These results are consistent with the idea of interactions between the gestural and lexical representational systems, such that mere exposure to iconic gestures facilitates the recognition of semantically related words. (shrink)

Using a cross‐modal semantic priming paradigm, both experiments of the present study investigated the link between the mental representations of iconic gestures and words. Two groups of the participants performed a primed lexical decision task where they had to discriminate between visually presented words and nonwords (e.g., flirp). Word targets (e.g., bird) were preceded by video clips depicting either semantically related (e.g., pair of hands flapping) or semantically unrelated (e.g., drawing a square with both hands) gestures. The duration of gestures (...) was on average 3,500 ms in Experiment 1 but only 1,000 ms in Experiment 2. Significant priming effects were observed in both experiments, with faster response latencies for related gesture–word pairs than unrelated pairs. These results are consistent with the idea of interactions between the gestural and lexical representational systems, such that mere exposure to iconic gestures facilitates the recognition of semantically related words. (shrink)

Toolmaking requires motor skills that in turn require handedness, so that there is no competition between the two sides of the brain. Thus, handedness is not necessarily linked to vocalization but to the origin of causal beliefs required for making complex tools. Language may have evolved from these processes.

One can disagree with Müller that it is neurobiologically questionable to suppose that human language is innate, specialized, and species-specific, yet agree that the precise brain mechanisms controlling language in any individual will be influenced by epigenesis and genetic variability, and that the interplay between inherited and acquired aspects of linguistic capacity deserves to be investigated.

This study investigates whether there is a relation between the semantics of linguistic expressions that indicate temporal distance and the spatial properties of their co-speech gestures. To this date, research on time gestures has focused on features such as gesture axis, direction, and shape. Here we focus on a gesture property that has been overlooked so far: the distance of the gesture in relation to the body. To achieve this, we investigate two types of temporal linguistic expressions are addressed: proximal (...) (e.g., near future, near past) and distal (e.g., distant past, distant future). Data was obtained through the NewsScape library, a multimodal corpus of television news. A total of 121 co-speech gestures were collected and divided into the two categories. The gestures were later annotated in terms of gesture space and classified in three categories: (i) center, (ii) periphery, and (iii) extreme periphery. Our results suggest that gesture and language are coherent in the expression of temporal distance: when speakers locate an event far from them, they tend to gesture further from their body; similarly, when locating an event close to them, they gesture closer to their body. These results thus reveal how co-speech gestures also reflect a space-time mapping in the dimension of distance. (shrink)

Researchers in the fields of sign language and gesture studies frequently present their participants with video stimuli showing actors performing linguistic signs or co-speech gestures. Up to now, such video stimuli have been mostly controlled only for some of the technical aspects of the video material, leaving open the possibility that systematic differences in video stimulus materials may be concealed in the actual motion properties of the actor’s movements. Computer vision methods such as OpenPose enable the fitting of body-pose models (...) to the consecutive frames of a video clip and thereby make it possible to recover the movements performed by the actor in a particular video clip without the use of a point-based or markerless motion-tracking system during recording. The OpenPoseR package provides a straightforward and reproducible way of working with these body-pose model data extracted from video clips using OpenPose, allowing researchers in the fields of sign language and gesture studies to quantify the amount of motion pertaining only to the movements performed by the actor in a video clip. These quantitative measures can be used for controlling differences in the movements of an actor in stimulus video clips or, for example, between different conditions of an experiment. In addition, the package also provides a set of functions for generating plots for data visualization, as well as an easy-to-use way of automatically extracting metadata from large sets of video files. (shrink)

Evidence is presented from a polyglot savant to suggest that double dissociations between linguistic and nonverbal abilities are more important than Müller's target article implies. It is also argued that the special nature of syntax makes its assimilation to other aspects of language or to nonhuman communication systems radically implausible.